Q: Why do my logs show my LISTSERV server sending mail to other LISTSERV sites?
New LISTSERV customers are often confused to look in their log files and find that their LISTSERV server is sending and receiving mail to and from other LISTSERV sites. For example:
7 Jun 2011 00:00:01 Processing file 9430674 from LISTSERV@XXX.EASE.LSOFT.COM
7 Jun 2011 00:00:01 From LISTSERV@LISTSERV.NET: DIST2 I=Y FROM=LISTSERV@XXX.MONTGOMERYCOUNTYTN.ORG POOL=N FORW(CRC) HOST(726 6 (...)
7 Jun 2011 00:00:01 Distributing file "X-SUPD JOB" from LISTSERV@XXX.MONTGOMERYCOUNTYTN.ORG...
7 Jun 2011 00:00:01 File forwarded to LISTSERV@SEARN for 1 recipient.
7 Jun 2011 00:00:01 File forwarded to LISTSERV.ARIZONA.EDU for 16 recipients.
7 Jun 2011 00:00:01 File forwarded to LISTSERV.CUNY.EDU for 7 recipients.
7 Jun 2011 00:00:01 File forwarded to LISTSERV.NODAK.EDU for 3 recipients.
7 Jun 2011 00:00:01 File forwarded to LISTSERV.UC.EDU for 2 recipients.
7 Jun 2011 00:00:01 File forwarded to LISTSERV.VT.EDU for 3 recipients.
7 Jun 2011 00:00:01 File forwarded to LISTSERV.WVNET.EDU for 1 recipient.
7 Jun 2011 00:00:01 File forwarded to LISTSERV.UTPA.EDU for 1 recipient.
7 Jun 2011 00:00:01 File forwarded to LISTSERV.ECU.EDU for 1 recipient.
7 Jun 2011 00:00:01 File forwarded to HERMES.GWU.EDU for 1 recipient.
7 Jun 2011 00:00:01 File forwarded to LISTS.QUEENSU.CA for 1 recipient.
7 Jun 2011 00:00:01 File forwarded to LISTSERV.BUFFALO.EDU for 1 recipient.
7 Jun 2011 00:00:01 File "X-SUPD JOB" distributed to LISTSERV@XXX.EASE.LSOFT.COM
7 Jun 2011 00:00:01 Done - 12 jobs (38 rcpts), 1 outbound file (1 rcpt).
The short answer is that by default, LISTSERV servers communicate with one another to share things like spam alerts, network routing updates and list mail distribution. This feature can be enabled or disabled by way of the RUNMODE site configuration variable. For the longer answer, we need to go back in time to the days when bandwidth was scarce, server time was expensive and network connectivity wasn't necessarily an "always-on" resource.
The Good/Bad Old Days
To understand the nature of LISTSERV's inter-server updates, it helps to understand a bit about the world into which LISTSERV was born. For most users today, the word "Internet" is nearly synonymous with the World Wide Web. Even for non-web services like email, a great many users access those services by way of a web browser. But in 1986, when Eric Thomas's "Revised LISTSERV" was released, the World Wide Web didn't yet exist, and the Internet itself was still in its infancy. It consisted of a number of disparate networks – ARPANET, BITNET, NSFNET, USENET, etc. – the nodes of which may or may not have had gateways to one another. Messaging between those nodes was neither direct nor instantaneous, and messages were usually sent or retrieved in batches as nodes connected to the network temporarily to communicate. For something like a mailing list, things were even more complicated because the sending server had to connect not just to one remote system to send a message to it, but to multiple recipient systems. Doing so was slow and costly – in terms of both money and network resources. Such was the electronic world into which Revised LISTSERV was set loose in 1986.
In Search of a Better Way
On July 25, 1986, Eric Thomas sent the following message to the LSTSRV-L list:
Let's start dreaming and consider the following situation. A revised LISTSERV
machine has been installed in the major hub nodes of all countries. It has been
granted enough disk space to hold the massive BITEARN NODES file, and enough
CPU time to process it at initialization to set various GLOBALV variables. One
of the network coordinators has volunteered to maintain a list of all the sites
who have installed revised LISTSERV, and to generate a kind of "routing table"
for LISTSERV. All of this is wishful thinking, but what's wrong about dreaming?
:-) Now a user at our node (a dead-end node indeed) wants to send a 400-lines
file to a list of 500 persons all over the network (let's say it's an electro-
nic magazine about computer networks, or something like that). Instead of doing
a SENDFILE, which would hog the network during days (128 link.Mbytes load, assu
ming an average distance of 8 links between two randomly selected BITNET nodes
-- see Hank's statistics), he issues a LSEND command (same syntax). Instead of
going directly to RSCS, the file goes to the nearest LISTSERV, with a list of
recipients appended to it. The server will distribute the file to all the reci-
pients in its 'service area' (ideally only the local node). It will then scan
the list of remaining recipients, search its "routing table" for the correspon-
ding LISTSERV's userid@node, and build a list of them. It will eventually ship
the file to all the selected LISTSERVs, after having edited the list of recipi-
ents so that only the nodes which were routed through a given server appear on
the list of recipients sent to that server. I can't find a formula to compute
the resulting network load, because it depends on the configuration of the
servers, but it would probably be 10 times smaller....
In short: what if each LISTSERV server knew about each of the others, and they used each other for shared mail distribution, passing messages only to and from their nearest neighbors, something like a fireman's bucket brigade? A crazy dream, indeed, but one that was to become a core feature of the LISTSERV service: the LISTSERV backbone.
The LISTSERV Backbone
As LISTSERV sites came online, they had (and still have) the opportunity to participate in the LISTSERV network, as well as the opportunity to join the LISTSERV backbone. LISTSERV nodes registered as backbone sites are sites that are assumed to be online 24 hours a day, 7 days a week and running the latest version of the LISTSERV software. As the name implies, LISTSERV backbone sites are the "spine" of the LISTSERV network – they pass mail along from one node to another along the backbone, and on to the local network nodes that are terminal points on the network but do not themselves relay mail to each other the way that the backbone sites do. The backbone model of LISTSERV mail distribution allowed for much more efficient use of network resources and greatly reduced the overhead involved in distributing list mail – a great benefit, indeed, in the early days of the Internet.
As the network grew and new challenges emerged, the LISTSERV network was put to purposes other than just mailing list distribution. For the network to operate effectively, it was important that all sites keep up-to-date copies of the network tables. Those tables themselves (contained in files like INTPEERS.NAMES and BITEARN.NODES) are pushed along the LISTSERV network via automatic updates. Networked LISTSERV sites also send along a list of their public email lists, all of which are added to CataList, the official catalog of LISTSERV lists. As spam became more of a problem and LISTSERV came to include spam-detection algorithms, the network was again leveraged. If one LISTSERV server detected a piece of spam, it could push a spam update including a message checksum across the network to the other LISTSERV nodes, so that if they received the same piece of spam, it could be immediately recognized and quarantined.
The most common types of LISTSERV backbone mail that you may see in your logs are as follows:
- DISTRIBUTE mail – Mail distribution for your local users for messages that originate from subscriptions to lists at remote LISTSERV hosts.
- Subscription Summary (X-SUPD) and List Update (X-LUPD) messages – Mail notifying the LISTSERV network of your currently available public lists, and summary statistics for those lists.
- Spam Notification (X-SPAM) messages – Updates about spam detected by networked LISTSERV sites.
Controlling the RUNMODE
LISTSERV's participation in the LISTSERV network is voluntary and controlled by the RUNMODE site configuration variable. The possible settings are as follows:
- Networked: Full participation in the LISTSERV network.
- Tableless: Limited participation in the LISTSERV network. The server still receives updates from the LISTSERV backbone but does not keep its own local copy of the networking tables. Instead, it "piggybacks" on one of the networked servers. This is the setting always used for the LISTSERV Lite/Free Edition.
- Standalone No participation in the LISTSERV network.
LISTSERV Classic/HPO sites that wish to participate in the LISTSERV network should complete the registration form at: http://www.lsoft.com/regform.html. Sites that do not wish to participate do not need to register and should set their RUNMODE to "Standalone".
1. LISTSERV Site Manager's Manual, section 5.6 ("Server Registration")
2. LISTSERV Site Manager's Manual, section 19.3 ("Interpreting the LISTSERV Log")
Subscribe to LISTSERV at Work.