Listserv to Mailman Part 1.1: Installing Mailman

Installing from Source

This page describes how I installed Mailman from source on our development server (a shared web hosting environment). However, it’s also possible to install through a binary package manager such as RPM or Apt, which is actually how our list host admins installed it onto our production server. So things might end up in different locations depending on how Mailman is installed.

Choosing a Version and Downloading

My first question for installing Mailman on our dev box was which version. This mattered because we had not yet chosen a list host but I assumed that whatever host we ultimately chose would probably have the latest stable and mature version.

Figuring out the latest stable and mature version of Mailman turned out to be a bit challenging. By examining the “timeline” graph at https://launchpad.net/mailman/+series (specifically, by hovering over the graph and hitting the little magnify-minus sign several times so I could eventually see all the way to the right), I found that at the time of checking (early Dec 2009), version 2.1.12 was the latest stable version since it was the most recent on the 2.1 “latest stable release” line.

I tried to download mailman-2.1.12.tgz from launchpad.net several times, but it never worked, so I finally downloaded it from http://ftp.gnu.org/gnu/mailman/?C=M;O=D

(Both of the above download sites were found on the main Mailman site at http://www.gnu.org/software/mailman/download.html)

Installing Mailman to a Custom Directory

I’ll skip past the usual technical details of downloading and unpacking files (I did it from the shell command line with: wget http://ftp.gnu.org/gnu/mailman/mailman-2.1.12.tgz and then: tar xvfz mailman-2.1.12.tgz).

In fact I wouldn’t even mention the installation process except that my first attempt failed because the web host I was using (which is not the one I am happily using now, Dreamhost) had pre-mapped the /mailman URL for me, and it resisted any attempts to override/fix it.

This was a problem because when I first installed Mailman on our dev box, created a test list (with the newlist command), and then tried to go to something like http://ourdomain.org/mailman/listinfo/test-list it told me there was no such list—even though I had just created it. I banged my head against the wall for a while until I figured out that my web host had pre-mapped /mailman and that there was no way around it.

(If you want to see if your web host has done this, I recommend going to http://www.yourhost.com/mailman/ before installing Mailman, and seeing if you get the same error page as when going to http://www.yourhost.com/badpagelink/ or if you get some unexpectedly-different page instead.)

So eventually I had to delete that first installation and install again into a different directory with the following options:

./configure --prefix=/home/ouruser/public_html/mm
--with-username=ouruser --with-groupname=ouruser
--with-mail-gid=ouruser --with-cgi-gid=ouruser
--with-mailhost=mail.ourhost.org --with-cgi-ext=.cgi

(I had to use the –prefix option, rather than accepting the default installation location, because my dev box was a shared hosting environment where I was just a regular user, rather than having root/admin permissions and being able to install into system directories like /var/mailman.)

After running configure and make install I then created a test list by running the mailman “bin” program newlist test-list (in ~/public_html/mm/bin). Then I tried going to http://ourdomain.org/mailman/listinfo/test-list and things seemed to work fine until I tried to log in, at which point the fact that Mailman insists on setting cookies with a Path of /mailman caused my logins to not “stick”.

I thought this would be easy to fix by editing the cookie set by Mailman from /mailman to /mm, but it turned out that most of the cookie-related add-ons for Firefox do not allow you to edit cookie values.

The Add N Edit Cookies Firefox add-on seemed promising, but when I edited the Path value of the cookies set after logging in to Mailman, to change from /mailman to /mm, it didn’t actually save the changes. But I figured out a workaround: copying the cookie values for the old login cookie (Name, Content, Host, Path, Expires) to a temp text file, deleting the cookie, then creating a new cookie with the same values except a different Path (and plus 20 years for the Expires value too).

All these machinations were just to deal with having to install Mailman to /mm instead of /mailman on our dev box because the web host had pre-mapped /mailman for us and we couldn’t override it. Hopefully you won’t have that problem at all; I highly recommend just installing using the defaults if you can!

For more information about installing Mailman see the official GNU Mailman Installation Manual. There’s also a helpful page covering more technical aspects of installing on various flavors of Linux (including the easy way of just installing from an RPM or YUM package) at the YoLinux Tutorial: GNU Mailman Email List Installation and Configuration.

Testing the Installation

If you’ve installed Mailman into the default /mailman and used newlist to create a “test-list”, you should be able to use the following URLs to check if your installation is correct:

  • http://yourhost.com/mailman/listinfo/test-list for info about the test list
  • http://yourhost.com/mailman/admin/test-list for admin pages about test list
  • http://yourhost.com/mailman/listinfo for info on all lists on the server

If you had to install Mailman to a custom directory like /mm and/or use a custom CGI file extension like .cgi, then you’d want to use something like the following:

  • http://yourhost.com/mm/cgi-bin/listinfo.cgi/test-list for info about the test list
  • http://yourhost.com/mm/cgi-bin/admin.cgi/test-list for admin pages for test list
  • http://yourhost.com/mm/cgi-bin/listinfo.cgi for info on all lists on the server

I haven’t covered setting up the automatic Mailman daemons and cron jobs because I couldn’t do that on our dev box’s shared hosting environment, and they were already set up on our production server. If you’re installing on your own box as root, the default options to configure and make install should take care of all that for you.

Next: Installing Swish for Archive Searching or Up: Table of Contents

Listserv to Mailman Part 1.0: Overview

Preface and Disclaimer

This guide was written in short bursts over several months, sometimes at the same time as performing our conversion from Listserv (pricey commercial mailing list manager) to Mailman (free software mailing list manager) and sometimes after the fact. (If you stumbled across this page and don’t know the difference, see Mailman is not Listserv.)

So it is not by any means perfect or even necessarily free from mistakes, and it certainly needs editing and cleaning up. In particular you may find the narrative/explanatory style vexing, along with occasional changes in person/tense.

For these flaws I can only apologize in advance. I will try to clean them up and improve these pages over time, but even with its current shortcomings, it is still the guide (and code!) I wish had been available when I started my own conversion of 70+ Listserv lists to Mailman.

Note: This guide and the code I’m making available were both written against Mailman version 2.1.12. I’d hope that future versions of Mailman wouldn’t change the API functions I called, but obviously there is no guarantee that won’t happen, so if you use my code with a later version of Mailman it might not work. In particular, if you’re going to use the administrative command handler, I’d recommend running its test program first to see if anything breaks.

The Problem

The problem: A small non-profit organization which had more than 70 Listserv mailing lists, most with 10+ years of archives, needed to convert all of the lists and archives to GNU Mailman because its Listserv host was going to cease all operations within a year’s time (i.e., we got 9 months’ warning).

The old Listserv list host had had a special, grandfathered non-profit site license for Listserv, and LSoft was unwilling to allow that license to transfer with our organization to another host, so sticking with Listserv simply wasn’t an option.

(If you hunt around the LSoft website and finally dig up their pricing, you’ll see that for most small organizations even the cheapest Listserv license is prohibitively expensive; it certainly was for us, a non-profit org with 70 lists and a total annual IT budget of about $1000/year.)

Searching for other hosts that already had Listserv lists proved fruitless as well. I could hardly find any that had Listserv, and the few that did have it were way too pricey.

Also, it seems that in the modern Internet age, “mailing list” means “email marketing newsletter”, not “discussion list”. This distinction was important, and came up repeatedly as a problem because almost all of our our 70 lists were discussion lists.

I bring this up because early on someone in our organization pointed out that our website host already provided “mailing list” software called DadaMail… which is actually just meant for one-way marketing newsletter emails.

In fact, one of DadaMail’s main features is that it has email throttling. Why? Because most ISPs now limit customers’ outgoing email to a maximum number of messages per hour. Why? Probably because spammers crack people’s websites through common application vulnerabilities and then use the victims’ sites to send spam.

Why did the hourly email limit on our web host matter? Because if you have 1000 customers and want to send them all a single newsletter message, that’s fine—if your host limits you to 500 messages/hour then you set DadaMail or your mailing script to send one message every 2-3 seconds.

The problem arises when you have an actual discussion list (and we had 70 of them!). If you have a discussion list with 600 subscribers and someone sends a post to the list, the ISP tries to send that post to 600 subscribers and hits the hourly limit.

You could spread delivery of the post over two hours but what happens when someone who gets it replies? That’s another 600 messages going out, when some people haven’t even gotten the first one yet! Clearly, if there is actual discussion (i.e., replies), email thottling (as with DadaMail and other announcement-oriented programs) just won’t work.

So what I learned from my research into mailing list software and hosts was that there are really two problems: 1) The cost of the mailing list software itself, and 2) the difficulty finding a list host which won’t cap/throttle your email message output at a certain number per hour.

The Solution

The answer to the problem of the exorbitant cost of Listserv was fairly simple. I had noticed GNU Mailman years before, and it seemed to be an open source replacement for Listserv which was reasonably mature and somewhat Listserv-like. I was right about this, but it required a lot of customization which is why this documentation exists.

For the second problem, there just didn’t seem to be any affordable hosts offering mailing lists, especially for 70+ lists.

One possible solution would have been to get a dedicated server and install Mailman on it. That solution wouldn’t have worked for my non-profit because they just don’t have the resources to manage a dedicated server—they barely have enough manpower to manage their website and membership database.

So we basically had to find a host that would already have Mailman, or allow us to install it, would not have a per-hour message throttling/limit, would let us have command line (shell) access to run the Mailman “bin” commands and extend Mailman’s functionality (such as adding an administrative command handler), and would otherwise manage the server (backups, security updates, performance tweaking, etc.). And all for a price within our small budget, of course.

After a lot of searching we eventually found one, Binhost, which we’ve been pretty happy with so far. If you set up an account with them, please mention me—I won’t personally get anything for it, but my non-profit org will get a small account credit for the referral. Or if you have any questions about our experiences with them, feel free to contact me.

Development and Production Servers

Due to the long time it took to find a list host, I’d already started playing with Mailman on our main web server just to learn it, even knowing that we couldn’t actually have our lists on that server.

So in effect, our web host became my development server for playing with Mailman and determining what kinds of customizations we’d need for a smooth transition from Listserv when we did find a list host.

I only mention this because there were a few differences between the dev box setup (a shared web host) and the list host we ultimately settled on (really a special case managed server), and this guide will occasionally mention differences between these two servers so I wanted to describe their setup a bit here.

Files for Downloading

I’ve tried to link to relevant code files in the text, but you can also browse all the code at once. The utils/ subdirectory has scripts I found helpful, and used indirectly in the project, but which weren’t directly related to the conversion. For example, tgrep searches through files like grep, but ignores non-text (binary) files.

Next: Installing Mailman or Up: Table of Contents

Objectives and Key Results

I was intrigued enough by the title of a recent article (Every Worker Should Be C.E.O. of Something) to do a quick read-through. As soon as I saw it was an interview with a CEO I was suspicious it would just be self- and company-promoting, and it is a little but, but I found something of value on the second page I wanted to share:

John Doerr [the venture capitalist] sold me on this idea of O.K.R.’s, which stands for objectives and key results. The idea is that the whole company and every group has one objective and three measurable key results – if you achieve two of the three, you achieve your overall objective, and if you achieve all three, you’ve really killed it.

We put the whole company on that, so everyone knows their O.K.R.’s. And that is a good, simple organizing principle that keeps people focused on the three things that matter — not the 10.

Then I ask everybody to write down on Sunday night or Monday morning what are your three priorities for the week, and then on Friday see how you did against them. It’s the only way people can stay focused and not burn out. And if I look at your road map and you have 10 priorities for you and your team, you probably don’t know which of the three matter, and probably none of the 10 are right.

I can look at everyone’s piece of paper, and their road map shows every item you were going to do and your predicted results and actual results, and then the results are in red if you missed them, yellow if they’re close and green if you passed them. I think road maps are a great principle just for managing your life. It keeps everybody focused, and it lets me know what trains are on or off the tracks.

(Actually I won’t lie, I originally only intended to bookmark it with del.icio.us but they wouldn’t let me put all of the excerpt in the Notes field, so I figured I’d make it a blog post instead :))

Hello, world

I’m leaving the title of this default post more or less unaltered since the “hello world” program is a tradition in the field of programming / computer science / software engineering / whatever we’re calling it these days 🙂

(Though I have changed the title from “Hello world!” to “Hello, world” on the theory that someone is sitting out there with an automated scanner looking for brandy new blogs to hack :))

This blog will mostly be miscellaneous technical things I run across – a blog-ification of the “tips file” concept: Since 1995 or so I’ve kept a plain text file which various tech tips I’ve run across and found useful enough that I wanted them handy for easy lookup.

But don’t worry, I’m not going to go back through 15 years worth of tips file entries here.

(Especially since some of them would be quite dated at this point, like the one where I described how to telnet to port 80 on the localhost of a web server to manually try out HTTP commands… OK maybe that one would actually be cool to write a blog entry on, but maybe the one about starting a Perl script with #!/usr/bin/perl isn’t necessary – and yes I do own that T-shirt, though I haven’t worn it in probably 12+ years or so :))

My main reason for starting this blog in earnest, after wanting “to do” it for years, is that I’m in the middle of a project to convert about 70 Listserv mailing lists to Mailman. (Since Listserv is a commercial product and the current host is ceasing operations, and the organization can’t afford a Listserv license, I’m doing the conversion to Mailman which is a free / open source alternative.)

All of my web searches for resources on how to do this turned up very cryptic stuff, usually arcane bits from mailing lists the Mailman developers were on. So I’d really like to write the guide I wish had existed when I started this process.

Eventually I’m going to gather up all the blog posts on the topic and collect them into a more coherent bona fide “guide”, but for right now I really need to do the conversion. So blogging about the process as I do it is a middle ground of actually capturing all the niggling little details so I don’t forget them, and writing a full fledged (more or less) “guide”.

Oh, and the blog will also occasionally have astrology bits/posts since I find that an interesting intellectual consideration. It (astrology) may, for all I know, be psychological wankery which is nothing more than some kind of self-serving bias, but I did take classes on it for years when I lived in NYC and still do astrological readings for people on a semi-regular basis, so I actually do know what I’m doing with it. And so far it seems much more accurate than not, especially when it comes to timing of major life events/processes.

Some time soon I’ll be resurrecting the astrology portion of my main site, so you can read all about it there, but in the meantime if you are interested in an astrological reading feel free to contact me.

It’s $75 for an hour and includes discussion of basic personality and life issues along with predictive information for the next year or two, with a focus on anything special you request. And if you’re doing the math and think wow, $75 an hour is a lot, bear in mind that it takes me several hours to prepare for a reading.

Plus… what is the value of having good advice about what’s going on in your life, when it will change, what the purpose of it all is, etc.?