Preface and Disclaimer

This guide was written in short bursts over several months, sometimes at the same time as performing our conversion from Listserv (pricey commercial mailing list manager) to Mailman (free software mailing list manager) and sometimes after the fact. (If you stumbled across this page and don’t know the difference, see Mailman is not Listserv.)

So it is not by any means perfect or even necessarily free from mistakes, and it certainly needs editing and cleaning up. In particular you may find the narrative/explanatory style vexing, along with occasional changes in person/tense.

For these flaws I can only apologize in advance. I will try to clean them up and improve these pages over time, but even with its current shortcomings, it is still the guide (and code!) I wish had been available when I started my own conversion of 70+ Listserv lists to Mailman.

Note: This guide and the code I’m making available were both written against Mailman version 2.1.12. I’d hope that future versions of Mailman wouldn’t change the API functions I called, but obviously there is no guarantee that won’t happen, so if you use my code with a later version of Mailman it might not work. In particular, if you’re going to use the administrative command handler, I’d recommend running its test program first to see if anything breaks.

The Problem

The problem: A small non-profit organization which had more than 70 Listserv mailing lists, most with 10+ years of archives, needed to convert all of the lists and archives to GNU Mailman because its Listserv host was going to cease all operations within a year’s time (i.e., we got 9 months’ warning).

The old Listserv list host had had a special, grandfathered non-profit site license for Listserv, and LSoft was unwilling to allow that license to transfer with our organization to another host, so sticking with Listserv simply wasn’t an option.

(If you hunt around the LSoft website and finally dig up their pricing, you’ll see that for most small organizations even the cheapest Listserv license is prohibitively expensive; it certainly was for us, a non-profit org with 70 lists and a total annual IT budget of about $1000/year.)

Searching for other hosts that already had Listserv lists proved fruitless as well. I could hardly find any that had Listserv, and the few that did have it were way too pricey.

Also, it seems that in the modern Internet age, “mailing list” means “email marketing newsletter”, not “discussion list”. This distinction was important, and came up repeatedly as a problem because almost all of our our 70 lists were discussion lists.

I bring this up because early on someone in our organization pointed out that our website host already provided “mailing list” software called DadaMail… which is actually just meant for one-way marketing newsletter emails.

In fact, one of DadaMail’s main features is that it has email throttling. Why? Because most ISPs now limit customers’ outgoing email to a maximum number of messages per hour. Why? Probably because spammers crack people’s websites through common application vulnerabilities and then use the victims’ sites to send spam.

Why did the hourly email limit on our web host matter? Because if you have 1000 customers and want to send them all a single newsletter message, that’s fine—if your host limits you to 500 messages/hour then you set DadaMail or your mailing script to send one message every 2-3 seconds.

The problem arises when you have an actual discussion list (and we had 70 of them!). If you have a discussion list with 600 subscribers and someone sends a post to the list, the ISP tries to send that post to 600 subscribers and hits the hourly limit.

You could spread delivery of the post over two hours but what happens when someone who gets it replies? That’s another 600 messages going out, when some people haven’t even gotten the first one yet! Clearly, if there is actual discussion (i.e., replies), email thottling (as with DadaMail and other announcement-oriented programs) just won’t work.

So what I learned from my research into mailing list software and hosts was that there are really two problems: 1) The cost of the mailing list software itself, and 2) the difficulty finding a list host which won’t cap/throttle your email message output at a certain number per hour.

The Solution

The answer to the problem of the exorbitant cost of Listserv was fairly simple. I had noticed GNU Mailman years before, and it seemed to be an open source replacement for Listserv which was reasonably mature and somewhat Listserv-like. I was right about this, but it required a lot of customization which is why this documentation exists.

For the second problem, there just didn’t seem to be any affordable hosts offering mailing lists, especially for 70+ lists.

One possible solution would have been to get a dedicated server and install Mailman on it. That solution wouldn’t have worked for my non-profit because they just don’t have the resources to manage a dedicated server—they barely have enough manpower to manage their website and membership database.

So we basically had to find a host that would already have Mailman, or allow us to install it, would not have a per-hour message throttling/limit, would let us have command line (shell) access to run the Mailman “bin” commands and extend Mailman’s functionality (such as adding an administrative command handler), and would otherwise manage the server (backups, security updates, performance tweaking, etc.). And all for a price within our small budget, of course.

After a lot of searching we eventually found one, Binhost, which we’ve been pretty happy with so far. If you set up an account with them, please mention me—I won’t personally get anything for it, but my non-profit org will get a small account credit for the referral. Or if you have any questions about our experiences with them, feel free to contact me.

Development and Production Servers

Due to the long time it took to find a list host, I’d already started playing with Mailman on our main web server just to learn it, even knowing that we couldn’t actually have our lists on that server.

So in effect, our web host became my development server for playing with Mailman and determining what kinds of customizations we’d need for a smooth transition from Listserv when we did find a list host.

I only mention this because there were a few differences between the dev box setup (a shared web host) and the list host we ultimately settled on (really a special case managed server), and this guide will occasionally mention differences between these two servers so I wanted to describe their setup a bit here.

Files for Downloading

I’ve tried to link to relevant code files in the text, but you can also browse all the code at once. The utils/ subdirectory has scripts I found helpful, and used indirectly in the project, but which weren’t directly related to the conversion. For example, tgrep searches through files like grep, but ignores non-text (binary) files.

Next: Installing Mailman or Up: Table of Contents