Commands

• To see all the settings for a mailman list: /usr/lib/mailman/bin/config_list -o – ‘listname’  (outputs to screen; use -o filename to output to a file)
• To see a list’s settings minus comments and whitespace: config_list -o – ‘listname’ | perl -ne ‘print unless /(^#|^\s*$)/’
• To change the password for a list: /usr/lib/mailman/bin/change_pw -l ‘listname’ -p ‘newpass’ -q (the -q prevents a notice going to the list owners)
• To show the subscribers of a list: /usr/lib/mailman/bin/list_members -f ‘listname’ (the -f shows full names for subscribers)
• To show the owners of a list: /usr/lib/mailman/bin/list_owners ‘listname’ (if you use list_admins instead, it seems to do the same thing)
• To rebuild all list archives (in bash): for list in `/usr/lib/mailman/bin/list_lists -b | egrep -v ‘^mailman$’`; do /usr/lib/mailman/bin/arch –wipe $list; done
• To restart Mailman (as root or mailman user): /etc/init.d/mailman restart
• To one or more list settings for a list: config_list -o – ‘listname’ > tmp.txt; then edit tmp.txt to leave only the setting you want to change; config_list -i tmp.txt ‘listname’
• To apply a setting change to multiple lists, use config_list -o – ‘listname’ > tmp.txt and edit tmp.txt as above, but then use shell looping: or list in adf-foo adf-bar etc; do config_list -i setting.txt $list ; done

Notes

I’d intended to list here each code file I’m making available along with a description, but instead I put the code descriptions in the Apache directory pages so just look there.

The Mailman logs on our server are in /var/log/mailman and /var/log/mailman/error contains the most recent errors.

Important: Our list host informed me that the default “mailman” list must be kept around, you can’t just delete it or important Mailman things will (apparently) break. So you’ve been warned

To increase the max # of recipients for a message to be able to get posted (in case you have lists where legitimate posts get rejected due to having too many recipients), see the max_num_recipients setting, which is in the Privacy Options -> Recipient Filters part of the web interface.

I created the arch_wipe_all script because I’d tried running the commands in it from the shell but it took so many hours (70+ lists and 10+ years of archives) that my shell would get disconnected and I’d have to start over completely. By putting the commands in a script I could preface it with nohup so it would keep running even if I got disconnected.

I noticed in /etc/mailman/postfix-to-mailman.py that if DEB_LISTMASTER isn’t set in /etc/mailman/mm_cfg.py or /usr/lib/mailman/Mailman/Defaults.py that error mail will go to postmaster@localhost, but I wanted it sent to another address in our organization so I added DEB_LISTMASTER = ‘addr@ourhost.org’ to the bottom of /etc/mailman/mm_cfg.py.

Pre-migration messages in people’s mailboxes had links to the old Listserv archives  in the footer, so even though we changed the footer for Mailman, we needed to provide redirects in case someone used an old link. We did this with an Apache URL Rewrite rule to redirect http://lists.ourhost.org/archives/listname.html to http://lists.ourhost.org/cgi-bin/mailman/private/listname/:

RewriteEngine on
RewriteRule ^archives/(.+)\.html$ http://lists.us.org/cgi-bin/mailman/private/$1/ [R]

(the second line, the RewriteRule, needs to be all on one line)

During one part of our disk issues converting our archives, our server admin said we’d maxed out our number of inodes (files and directories), probably because of the large number of archive related files (70+ lists for 10+ years). If you see errors like “unable to write file”, you can check the inodes used on the disk itself with df -i, and check inodes for your user with quota -sv username.

Bookmarks

The following pages weren’t listed in other parts of the guide and may be useful:

• Summary of the Mailman bin command line programs
• Mailman List Management Guide (2001 though, but has good settings descriptions)
• Changing List Owners On A Mailman Mailing List
• How do I integrate Mailman with PHP, or any other web technology?
• Mailman Installation Manual
• Mailman List Member Manual
• Regenerating Large Mailman Archives (I did not have to do the steps in this page even though our archives are 10+ years)
• How to edit the archives of a Mailman List
• How to remove a post from a Mailman list archive, or an entire archive (Note: to remove a post completely, I prefer to use Thunderbird to edit the mbox file)
• Mailman bounce processing explanations
• Mailman Frequently Asked Questions

Up: Table of Contents

Introduction

If you’re converting Listserv lists to Mailman, you probably want to keep your list archives too. The good news is that the Listserv archive format is plain text and is actually pretty well documented. The bad news is that it’s a custom proprietary format, not something common like the Unix “mbox” format, so conversion will be necessary.

Mailman actually wanted the archives to be in mbox format, so that’s what we converted the Listserv archives to. In fact, one of our lists had a lot of spam in its archives so after we did the conversion, but before we imported into Mailman, we opened the mbox file in Thunderbird, deleted the spam messages, and then saved the file back to mbox for importing into Mailman.

After converting the Listserv archives to mbox format, we had to move the mbox files to the right locations for Mailman to generate HTML archive pages, and then use Swish to index the HTML pages and do a little Mailman fiddling to keep private archives private (including search results). But first, more about the archive conversion process.

Converting the Archive Files

My archive conversion journey began with a Perl script called ls2mail.pl that was posted in 1999 and claimed to convert from Listserv’s archive format to mbox for Mailman.

The admin of our old Listserv modified it a bit to fix an unspecified “potentially nasty bug”, and I modified it further to 1) skip messages with dates earlier than the earliest legitimate post and later than the current year (to weed out spam messages with invalid dates), and 2) better match the mboxrd format (by quoting body lines beginning with “From”).

First, our Listserv archives were broken up into weekly archive files named listname.logYYMMw where YY is the year of the archive file, MM is the month, and w is the week (a = first week, b = second, up to e for a month with five weeks). For example, listname.log0901a was the Listserv archive file for the first week of January, 2009.

I decided it would be better to rename the files to use four digit years instead of two (i.e., listname.logYYYYMMw), so I used a small Perl utility called perlren to rename the files according to a Perl regular expression. So for each list I did:

perlren 's#log1#log201#' *.log*
perlren 's#log9#log199#' *.log*
perlren 's#log0#log200#' *.log*

Then I wanted a master file for each list which would have a sorted listing of all archive files (and because of the rename above, an alpha sorting was also a date sorting):

ls -1 *.log* > archive-files.txt

(I used a general name like archive-files.txt since all the log files for each list were grouped into a separate directory for each list.)

Then I used that file as a basis of making a file containing all the Listserv log files combined into one big file, in order:

perl -ne 'print "Processing $_"; chomp; print `cat $_ >> abc-listname.ls`;' \
  archive-files.txt

Finally, I ran the ls2mail.pl conversion script on the master Listserv log file to convert it to mbox format:

perl ~/mailman/ls2mail.pl < abc-listname.ls > abc-listname.mbox

Generating Mailman’s HTML Archive Pages

To generate the viewable HTML pages for Mailman’s web archive, I just had to move this new mbox file to the appropriate Mailman list archive directory (e.g., /var/lib/mailman/archives/private/abc-listname.mbox/) and run the Mailman bin “arch” command to generate the HTML archive pages from the mbox file:

mv abc-listname.mbox /var/lib/mailman/archives/private/abc-listname.mbox/
/usr/lib/mailman/bin/arch --wipe abc-listname \
  /var/lib/mailman/archives/private/abc-listname.mbox/abc-listname.mbox

(the –wipe option tells arch to overwrite any existing HTML pages with newly-generated ones from the mbox file, but since this was a new list that wasn’t a problem)

Once I’d tested this with one list, I repeated the process automatically with the other lists by doing something like the following (using command looping in the bash shell again):

cd /var/lib/mailman/archives/private
for list in abc-list1 abc-list2 abc-list3 etc;
do cd $list;
perlren 's#log1#log201#' *.log*;
perlren 's#log9#log199#' *.log*;
perlren 's#log0#log200#' *.log*;
ls -1 *.log* > archive-files.txt;
perl -ne 'print "Processing $_"; chomp; print `cat $_ >> archive.ls`;' \
  archive-files.txt;
perl ~/mailman/ls2mail.pl < archive.ls > $list.mbox;
mv $list.mbox ../$list.mbox/;
/usr/lib/mailman/bin/arch --wipe $list ../$list.mbox/$list.mbox;
cd ..;
done

By doing this, all the existing list archives were converted from Listserv archives (combined into .ls files) to Mailman .mbox archive files, and HTML web-viewable archives pages were generated by Mailman’s arch command.

A Word About Disk Space

I ran into disk space limits several times during this conversion process.

First, our old list host gave us the Listserv archives as one large compressed .tar.gz file—which expanded to triple the size, requiring 4x the size to accommodate both the gzip file and its uncompressed files.

Furthermore, I found that after concatenating all the individual Listserv .log files into one giant Listserv notebook archive .ls file (so, disk space times two for that step), the conversion to mbox format caused the resulting mbox file to take up about 60% more space than the .ls files.

Then running the Mailman arch command on the mbox files generated HTML pages that took up about twice as much space as the corresponding mbox files.

And then the Swish search index files took up about 55% of the size of the HTML archive pages, so that was another bunch of disk space.

The following table shows, step by step, how a 350MB compressed Listserv archive file ballooned up to almost 10GB:

You can mitigate some of those increases by deleting interim files (e.g., deleting the individual .log files after creating the concatenated .ls Listserv archive files), but the overall disk usage still ends up a lot more than you might think because the mbox files take more space than the Listserv notebooks, the HTML archive files are twice the size of the mbox files, and the Swish search index files are even more space.

We filled our disk up several times during the conversion, and one of the times this happened caused all mail delivery for all our just-converted lists to stop completely until some disk space was freed up and Mailman was restarted (actually it required a server restart, not just Mailman, which they’d tried with “/etc/init.d/mailman restart”).

By the way, after filling the disk a few times I created a script called disk_space_check and installed a daily cron job to send an email if disk usage was too high:

/bin/df -h | $HOME/bin/disk_space_check

In the next section we’ll add searching to Mailman’s archives, and make it so search on private lists (and search results) are limited to just subscribers.

Next: Setting Up Archive Search or Up: Table of Contents

This guide was written in short bursts over several months, sometimes at the same time as performing our conversion from Listserv (pricey commercial mailing list manager) to Mailman (free software mailing list manager) and sometimes after the fact. (If you stumbled across this page and don’t know the difference, see Mailman is not Listserv.)

So it is not by any means perfect or even necessarily free from mistakes, and it certainly needs editing and cleaning up. In particular you may find the narrative/explanatory style vexing, along with occasional changes in person/tense.

For these flaws I can only apologize in advance. I will try to clean them up and improve these pages over time, but even with its current shortcomings, it is still the guide (and code!) I wish had been available when I started my own conversion of 70+ Listserv lists to Mailman.

Note: This guide and the code I’m making available were both written against Mailman version 2.1.12. I’d hope that future versions of Mailman wouldn’t change the API functions I called, but obviously there is no guarantee that won’t happen, so if you use my code with a later version of Mailman it might not work. In particular, if you’re going to use the administrative command handler, I’d recommend running its test program first to see if anything breaks.

The Problem

The problem: A small non-profit organization which had more than 70 Listserv mailing lists, most with 10+ years of archives, needed to convert all of the lists and archives to GNU Mailman because its Listserv host was going to cease all operations within a year’s time (i.e., we got 9 months’ warning).

The old Listserv list host had had a special, grandfathered non-profit site license for Listserv, and LSoft was unwilling to allow that license to transfer with our organization to another host, so sticking with Listserv simply wasn’t an option.

(If you hunt around the LSoft website and finally dig up their pricing, you’ll see that for most small organizations even the cheapest Listserv license is prohibitively expensive; it certainly was for us, a non-profit org with 70 lists and a total annual IT budget of about $1000/year.)

Searching for other hosts that already had Listserv lists proved fruitless as well. I could hardly find any that had Listserv, and the few that did have it were way too pricey.

Also, it seems that in the modern Internet age, “mailing list” means “email marketing newsletter”, not “discussion list”. This distinction was important, and came up repeatedly as a problem because almost all of our our 70 lists were discussion lists.

I bring this up because early on someone in our organization pointed out that our website host already provided “mailing list” software called DadaMail… which is actually just meant for one-way marketing newsletter emails.

In fact, one of DadaMail’s main features is that it has email throttling. Why? Because most ISPs now limit customers’ outgoing email to a maximum number of messages per hour. Why? Probably because spammers crack people’s websites through common application vulnerabilities and then use the victims’ sites to send spam.

Why did the hourly email limit on our web host matter? Because if you have 1000 customers and want to send them all a single newsletter message, that’s fine—if your host limits you to 500 messages/hour then you set DadaMail or your mailing script to send one message every 2-3 seconds.

The problem arises when you have an actual discussion list (and we had 70 of them!). If you have a discussion list with 600 subscribers and someone sends a post to the list, the ISP tries to send that post to 600 subscribers and hits the hourly limit.

You could spread delivery of the post over two hours but what happens when someone who gets it replies? That’s another 600 messages going out, when some people haven’t even gotten the first one yet! Clearly, if there is actual discussion (i.e., replies), email thottling (as with DadaMail and other announcement-oriented programs) just won’t work.

So what I learned from my research into mailing list software and hosts was that there are really two problems: 1) The cost of the mailing list software itself, and 2) the difficulty finding a list host which won’t cap/throttle your email message output at a certain number per hour.

The Solution

The answer to the problem of the exorbitant cost of Listserv was fairly simple. I had noticed GNU Mailman years before, and it seemed to be an open source replacement for Listserv which was reasonably mature and somewhat Listserv-like. I was right about this, but it required a lot of customization which is why this documentation exists.

For the second problem, there just didn’t seem to be any affordable hosts offering mailing lists, especially for 70+ lists.

One possible solution would have been to get a dedicated server and install Mailman on it. That solution wouldn’t have worked for my non-profit because they just don’t have the resources to manage a dedicated server—they barely have enough manpower to manage their website and membership database.

So we basically had to find a host that would already have Mailman, or allow us to install it, would not have a per-hour message throttling/limit, would let us have command line (shell) access to run the Mailman “bin” commands and extend Mailman’s functionality (such as adding an administrative command handler), and would otherwise manage the server (backups, security updates, performance tweaking, etc.). And all for a price within our small budget, of course.

After a lot of searching we eventually found one, Binhost, which we’ve been pretty happy with so far. If you set up an account with them, please mention me—I won’t personally get anything for it, but my non-profit org will get a small account credit for the referral. Or if you have any questions about our experiences with them, feel free to contact me.

Development and Production Servers

Due to the long time it took to find a list host, I’d already started playing with Mailman on our main web server just to learn it, even knowing that we couldn’t actually have our lists on that server.

So in effect, our web host became my development server for playing with Mailman and determining what kinds of customizations we’d need for a smooth transition from Listserv when we did find a list host.

I only mention this because there were a few differences between the dev box setup (a shared web host) and the list host we ultimately settled on (really a special case managed server), and this guide will occasionally mention differences between these two servers so I wanted to describe their setup a bit here.

Files for Downloading

I’ve tried to link to relevant code files in the text, but you can also browse all the code at once. The utils/ subdirectory has scripts I found helpful, and used indirectly in the project, but which weren’t directly related to the conversion. For example, tgrep searches through files like grep, but ignores non-text (binary) files.

Next: Installing Mailman or Up: Table of Contents