Introduction

This section describes how to create new Mailman lists equivalent to the existing Listserv lists.

When we did this, there were about two weeks when we made subscriber changes in parallel, to both the old Listserv lists and the new Mailman lists, while we converted the archives and got ready for the final cutover.

We were able to do this because all our subscriber changes (subs, unsubs, and address changes) were done through CGI scripts on our website which generated email commands to LISTSERV—so we just modified the scripts to also send equivalent commands to the Mailman administrative command handler we described in the last section.

In other words, if someone subscribed to ListX on our website, the CGI script sent commands to subscribe them to both the Listserv ListX (soon-to-be-old server) and the Mailman ListX (soon-to-be-new server), even though the latter weren’t get getting any posts because ListX@lists.ourhost.org still pointed to the Listserv server.

When we were ready for the actual cutover we just changed lists.ourhost.org to point from the Listserv host to the Mailman host, and Mailman started handling the new posts right away. Then once everything was verified as working properly, we went back and updated the CGI scripts to stop sending subscriber changes to the old Listserv host.

Identifying And Grouping Current Lists

This may be easy for you, but we had 70+ lists to migrate, so not only did we have to identify them, we had to group them in terms of their settings so we’d be able to use a script to create multiple lists with similar settings in a batch instead of having to create them one at a time.

To do this you obviously need to understand all of your old Listserv list settings AND the equivalent new Mailman settings.

So if you  have a large number of lists you’ll need to examine all of their settings and see if they need to be categorized into settings groups. For example, after analyzing our 70+ lists, I identified the following settings groups:

• announcement lists – moderated lists allowing only a few people to post directly (no approval needed), but forcing all other posters’ submissions to go into an approval queue, with replies going to the poster by default, not the list, and with anyone able to subscribe to these lists

• member lists – discussion lists that any of our organization’s members (not the general public) could subscribe to and then post to, with replies going to the list by default

• public lists – discussion lists anyone in the general public could subscribe to and then post to, with replies going to the list by default

• public review lists – discussion lists anyone in the general public could subscribe to, but with messages requiring moderator approval, and replies going to the list by default

• open post lists – special purpose lists that anyone in the general public could post to, but which required special list owner approval for subscriptions (the subscriber lists for these were very small)

• restricted lists – discussion lists whose subscribers were limited to special organizational subgroups, requiring each subscriber to be added or removed by a list owner, with replies going to the list by default

To do this for your lists, ask basic questions like the following:

1. Who can subscribe to the lists? (anyone in the public, only people in your organization, only a few special people, etc.)
2. Who can post to the list? (you might have an announcement list that anyone can subscribe to, but only a few people can post to)
3. Do posts require moderator approval? (and if so, is the moderator just the list owner, or do you need to have additional non-owner moderators?)
4. Where do replies to posts go by default? (to the list or to the poster?)

Examining your current list settings should answer these questions and maybe prompt additional ones you’ll need to answer for all your lists. Then you can determine if all your lists have the same settings or they need to be categorized into settings groups as ours did.

Exploring Mailman List Config Options

To see all the possible Mailman settings for a list, I used the following command to view the settings of the previously-created test list:

/usr/lib/mailman/bin/config_list -o orig.listconfig test-list

(your config_list location might differ, but it will be with all the other “mailman bin” commands)

The goal of this was, for each possible Mailman list setting, to determine if we wanted the same setting to apply to all our lists or if the setting’s value would differ depending on which settings group the list was in.

If you do this and look at orig.listconfig you’ll see that it’s about 80% comments, about 10% blank lines and 10% Python code; list config files like this are (and must be) valid Python code since they are read back in by the config_list script with the -i option to make list changes.

(Note that if you’re just changing one or two things, you don’t need to feed the entire config file back into config_list; it’s smart enough that if you feed it a file with a small number of settings, it will just change those settings on the given list and keep the other settings at their previous values.)

It’s also important to note that these “config files” have no direct ties to Mailman. Once you dump a list’s configuration settings into a text file using config_list -o (o=output) as above, you can do whatever you want with that text file and Mailman won’t care.

The only time the text file matters again is if you feed it back into Mailman with config_list -i (i=input), at which point Mailman will take each setting in that file and use it to update the real list settings it keeps internally. Once you’ve done that, you’re free to delete the file if you want—you can always recreate it if necessary with config_list -o.

I wanted to keep orig.listconfig as a set of Mailman’s default values, so I copied it to announcement.listconfig so that the latter could represent the configuration of our announcement lists (see our settings groups in the bulleted list above).

Then I edited announcement.listconfig, first replacing every reference to “test-list” with “abc-listname” (no quotes; all our mailing lists begin with abc-). This allowed us to later batch-create multiple lists with almost identical configurations by replacing abc-listname.

Second, I added a comment of “ALWAYS” or “DIFFERS” next to every setting in announcement.listconfig, for “always use this setting value for all our lists” or “this will differ from one list settings group to another”. This allowed me to copy announcement.listconfig to other settings group listconfig files, search for each DIFFERS comment, and make changes to just those options.

I also made notes next to each ALWAYS or DIFFERS comment to explain the choice of setting value and make notes about the configuration option itself if necessary. For example:

# Should any existing Reply-To: header found in the original message be
# stripped?  If so, this will be done regardless of whether an explict
# Reply-To: header is added by Mailman or not.
#
# legal values are:
#    0 = "No"
#    1 = "Yes"
# ALWAYS - force replies to list or, if possible, list AND poster
#   either way, need to strip original reply to, so always true
first_strip_reply_to = True

The bold text is what I added, a note that I wanted this option to be true for all our lists, applying the “reply to list” or “reply to poster” policy equally for all posts and never honoring an original Reply-To header included by a poster.

Here’s an example of a DIFFERS comment explaining the choice and which list settings groups should have which values for this setting:

# When this option is enabled, all list traffic is emergency moderated,
# i.e. held for moderation.  Turn this option on when your list is
# experiencing a flamewar and you want a cooling off period.
#
# legal values are:
#    0 = "No"
#    1 = "Yes"
# DIFFERS - set to no for almost all lists; yes for moderated lists
emergency = False

Finally, since a file that’s intended for feeding back into config_list -i must be valid Python code, I wanted to make sure I hadn’t made any typos so I ran python announcement.listconfig to make sure the script diddn’t bomb; a successful run just gives you back the command line with no output since config files just set a lot of variables.

I’d love to explain all of our option choices, but you’ll need to examine them all for yourself anyway and the in-file comment documentation is usually pretty good. One option I would like to explain though is the “new_member_options” setting, an opaque numerical value with a nebulous comment:

# When a new member is subscribed to this list, their initial set of
# options is taken from the this variable's setting.
new_member_options = 256

The new_member_options name implied it was pretty important and determined the default options for all new subscribers, and I needed to know if it should be the same across all our lists or if it needed to differ per settings group.

However, as you can see there was very little comment documentation for this option, and I couldn’t find anything online either. Ultimately I had to delve into the source code, where I found the following in the source tree’s Mailman/Defaults.py.in file:

# Digests             = 0 # handled by other mechanism, doesn't need a flag.
# DisableDelivery     = 1 # Obsolete; use set/getDeliveryStatus()
# DontReceiveOwnPosts = 2 # Non-digesters only
# AcknowledgePosts    = 4
# DisableMime         = 8 # Digesters only
# ConcealSubscription = 16
# SuppressPasswordReminder = 32
# ReceiveNonmatchingTopics = 64
# Moderate = 128
# DontReceiveDuplicates = 256

I believe the way this works is that you decide which options (“flags”) you want to be “on” for new subscribers and then add up all the values for those options; the total is what new_member_options should be.

In other words, since the default (in a new Mailman list) new_member_options value is 256, that means the only option turned on for new subscribers is DontReceiveDuplicates.

I examined the above options and determined that the only setting we wanted was SuppressPasswordReminder (if someone subscribed to 20 of our lists it would be awful to get 20 monthly password reminders!), so I made our new_member_options value 32 instead of the default 256, and made an ALWAYS note that this should apply to all our lists.

(Side note: I got the sense that new_member_options is older/obsolete since some of its flags seemed covered by other configuration options and this style of determining default subscriber options is a lot more difficult for most people to use.)

Anyway, after replacing test-list with abc-listname in the announcement config file, and then adding ALWAYS or DIFFERS comments for each option, I copied announcement.listconfig to members.listconfig, then went through all the DIFFERS comments and made sure that the settings values were correct for “members” type lists.

When done with members.listconfig I copied it to openpost.listconfig, edited all the DIFFERS settings, and then repeated the copy/edit process for the remaining list settings groups.

I ended up with six .listconfig files, matching our list settings groups in the bulleted list above: announcement.listconfig, members.listconfig, openpost.listconfig, public.listconfig, pubreview.listconfig, and restricted.listconfig. These files were 1) identical to each other except where lists of that settings group should differ, and 2) had a generic list name of “abc-listname” instead of the real list name.

Listserv And Mailman List Configuration Option Equivalents

In order to make good choices about the Mailman configuration settings for our list settings groups, I needed to understand which old Listserv configuration settings translated to which new Mailman configuration settings.

I wasn’t able to find much online comparing Listserv and Mailman list settings; in fact the only page I found was the MIT Mailman User Guide: ListServ Keywords in Current Use and its accompanying “cheat sheet” PDF.

I can only discuss the Listserv options we used and give you their Mailman equivalents, but hopefully between the options here, the MIT page, the Listserv list header documentation, and the Mailman config file comments, you can figure everything out for your site.

Listserv Header Keyword	Mailman Config File Option
Review= (who can get subscriber lists)	private_roster (found in Privacy options… → Membership exposure in the Mailman web admin interface)
Subscription= (who can subscribe to the list)	subscribe_policy (Privacy options… → Subscribing)
Send= (who can post to the list)	emergency (causes all posts to be moderated; General Options → Additional settings), accept_these_nonmembers (set to ‘^.*@*.^‘ to allow anyone to post; Privacy options… → Sender filters → Non-member filters), include_list_post_header (General Options → Additional settings)
Notify= (if list owner is notified of new subscriptions, deletions, etc.)	admin_immed_notify, admin_notify_mchanges (both are in General Options → Notifications)
Reply-to= (if replies go to list/sender, and if reply-to header is honored)	reply_goes_to_list, first_strip_reply_to, reply_to_address (these three are in General Options → Reply-To header munging), anonymous_list (General Options → General list personality)
Default-Options= (initial settings for new subscribers)	new_member_options (General Options → Additional settings), digest_is_default (Digest options), default_member_moderation (Privacy options… → Sender filters → Member filters)
Files= (obsolete option, see Listserv doc)	no equivalent
Validate= (how to validate commands as authentic; with a password, confirmation and reply, etc.)	no equivalent, all Mailman web changes require password login
Filter= (whether to auto-identify mailing loops and suspicious/spammy From addresses)	header_filter_rules (loosely; Privacy options… → Header filters), mostly handled automatically
Confidential= (whether list’s existence is public knowledge)	advertised (Privacy options… → Subscription rules → Subscribing)
X-Tags= (whether X-To and X-Cc headers in posts are passed to subscribers)	no equivalent, presumably they are passed on but I have not confirmed
Stats= (obsolete option, see Listserv doc)	no equivalent
Ack= (whether to respond to postings to let sender know post got through)	autorespond_postings, autoresponse_postings_text, other autoresponse_ options (all in Auto-responder section of web admin)
Notebook= (whether to keep archives and how often to rotate)	archive, archive_private, archive_volume_frequency (all in Archiving Options section of web admin)
Auto-Delete= (whether and how aggressively to auto-remove bad subscriber addresses)	bounce_processing and other bounce_ options (all in Bounce processing section of web admin)
SizeLim= (in lines unless otherwise specfied)	max_message_size (Kb; in General Options → Additional settings)
Daily-threshold= (max msgs per day before list is automatically “held”)	no equivalent, though you can manually set “emergency” if there is a mail loop or flame war (in General Options → Additional settings)
List-Address= (domain for the list, so it is listname@list-address)	host_name (in General Options → Additional settings)
Attachments= (whether to allow attachment, reject with an error, or silently strip them)	all filter_ options, pass_mime_types, pass_filename_extensions, collapse_alternatives (these four options are in Content filtering), scrub_nondigest (in Non-digest options)
Digest= (whether to support digests for this list and if so, how many messages/lines in each)	nondigestable (in Non-digest options), digestable, mime_is_default_digest, digest_is_default and other digest_ options (all in Digest options)
Language= (whether to strip/convert HTML to text)	convert_html_to_plaintext (in Content filtering section of web admin)
Editor= (who can review/approve other people’s posts, and bypass similar moderation	moderator (General Options → General list personality)
Owner= (the list owner or owners)	owner (General Options → General list personality)
Errors-To= (address that errors should go to)	no equivalent, errors go to list owners

Creating Mailman Lists In Bulk

In the “Identifying And Grouping Current Lists” section we created several different settingsgroup.listconfig files, where settingsgroup was a name for a group of lists with all the same configuration settings except for the listname itself. We’d put abc-listname as the list name in the group settings file so we could search and replace it in the batch creation process, which is described in this section.

While we came up with .listconfig files for each settings group, we still had to have a way to say which settings group each list belongs to. So I created settingsgroup.lists files such as announcement.lists, member.lists, public.lists, pubreview.lists, etc. Each file simply had, the lists belonging to that settings group, one list per line.

For example, we only had one announcement type list to create so our announce.lists file had just one line:

ABC-Announce

If we’d had more announcement type lists, they would have each been on a separate line in the announce.lists file.

So the idea was that announcement.lists had a roster of lists (one per line) to create according to the announcement.listconfig file settings, with abc-listname in that settings file to be replaced with each list name from announcement.lists. And the same thing with public.lists, pubreview.lists, etc., with each .lists file containing the lists to be created with the corresponding .listconfig file settings.

There were probably many ways to do the batch creation, taking each line in the whatever.lists file and replacing abc-listname in the whatever.listconfig, then using the “newlist” Mailman bin command to create the new list with that name.

I chose to just use a series of single-line commands, one for each settings group. The command I used was the following, pasted into the shell and run as a single line:

perl -ne ‘chomp; $mclist = $_; $lclist = lc($_); print `echo cp -f announcement.listconfig /tmp/$lclist.listconfig`; print `echo perl -pi -e s#ABC-Listname#$mclist#g /tmp/$lclist.listconfig`; print `echo perl -pi -e s#abc-listname#$lclist#g /tmp/$lclist.listconfig`; print `echo newlist -q $lclist abc-listmaster\@abc.org listpassword >> aliases-to-make.txt`; print `echo config_list -i /tmp/$lclist.listconfig $lclist`;’ announcement.lists

(This assumes that you’re already in the Mailman bin directory, such as /usr/lib/mailman/bin, or that you’ve put that directory in your $PATH. If not, you will want to write out the full paths to newlist, config_list, and change_pw above.)

Note:  The command as written above won’t actually do anything, it will just print what it would do if you removed all the “echo” commands. Whenever I do anything complex like this, repeating commands over a lot of items, I first use echo statements to print out what’s going to happen and then look it over to check for errors before running the real commands.

The overall idea is that Perl is used to iterate over each line of the announcement.lists file, where each line is the mixed-case name of a list we want to create. The names are mixed case because in some settings (like the list description) Mailman lets you use mixed case, while in others it just takes lower case.

The command above sets $mclist for the mixed-case list name, and $lclist for lowercase, then it copies announcement.listconfig to a temp location, replaces ABC-Listname in the temp file with the mixed case value using the $mclist variable, and does the same with abc-listname and $lclist.

Then it uses the Mailman newlist bin command to create the list using abc-listmaster@abc.org as the initial list owner and “listpassword” as the initial list password, though both can be overridden in the group settings file which is applied next.

Important Note: Since we had 70 different email lists, we did want them all to have the same password. Your situation may be different and if you need each list password to be different you might use the above commands to set them all the same and then use the Mailman change_pw command afterward to customize the password on a per-list basis.

The version of the newlist command above also appends the output of the command to a file called aliases-to-make.txt in case you need to create mail aliases for your new lists. We didn’t need to do that though because our new list host had set up postfix-to-mailman.py so that postfix automatically recognized the Mailman aliases for new lists. (There’s a qmail-to-mailman.py file too if you’re running qmail, very handy.)

(Note: Actually the “>> aliases-to-make.txt” redirection in the “echo newlist” portion causes that command to get redirected into aliases-to-make.txt rather than printing on screen, so for testing the echo you’ll have to remove “>> aliases-to-make.txt”, just be sure to add it back in before running the commands for real!)

After the new list is created, config_list is used with the temp config file to configure it.

If you’re comfortable with all of this, try it out and examine the output from the echo commands.

You can even try the commands for real, one at a time in the shell with one list, to make sure they’ll work as you expect them to (e.g., that there are no path or permission errors). In that case you can check the contents of aliases-to-make.txt to make sure the output is what you expected, and even run config_list -o – newlistname to double-check the configuration.

When you’re comfortable with what will happen from the looping commands above, remove the echo statements and run the commands for real with a small number of lists. If that works, you can then run the commands with each successive settings group (which for us meant changing “announcement” to “member”, then “public”, “pubreview”, etc.), ultimately creating all new Mailman lists in batches according to their old Listserv settings groups.

Next: Migrating Subscribers And Keeping Them In Sync or Up: Table of Contents