Commands

• To see all the settings for a mailman list: /usr/lib/mailman/bin/config_list -o – ‘listname’  (outputs to screen; use -o filename to output to a file)
• To see a list’s settings minus comments and whitespace: config_list -o – ‘listname’ | perl -ne ‘print unless /(^#|^\s*$)/’
• To change the password for a list: /usr/lib/mailman/bin/change_pw -l ‘listname’ -p ‘newpass’ -q (the -q prevents a notice going to the list owners)
• To show the subscribers of a list: /usr/lib/mailman/bin/list_members -f ‘listname’ (the -f shows full names for subscribers)
• To show the owners of a list: /usr/lib/mailman/bin/list_owners ‘listname’ (if you use list_admins instead, it seems to do the same thing)
• To rebuild all list archives (in bash): for list in `/usr/lib/mailman/bin/list_lists -b | egrep -v ‘^mailman$’`; do /usr/lib/mailman/bin/arch –wipe $list; done
• To restart Mailman (as root or mailman user): /etc/init.d/mailman restart
• To one or more list settings for a list: config_list -o – ‘listname’ > tmp.txt; then edit tmp.txt to leave only the setting you want to change; config_list -i tmp.txt ‘listname’
• To apply a setting change to multiple lists, use config_list -o – ‘listname’ > tmp.txt and edit tmp.txt as above, but then use shell looping: or list in adf-foo adf-bar etc; do config_list -i setting.txt $list ; done

Notes

I’d intended to list here each code file I’m making available along with a description, but instead I put the code descriptions in the Apache directory pages so just look there.

The Mailman logs on our server are in /var/log/mailman and /var/log/mailman/error contains the most recent errors.

Important: Our list host informed me that the default “mailman” list must be kept around, you can’t just delete it or important Mailman things will (apparently) break. So you’ve been warned

To increase the max # of recipients for a message to be able to get posted (in case you have lists where legitimate posts get rejected due to having too many recipients), see the max_num_recipients setting, which is in the Privacy Options -> Recipient Filters part of the web interface.

I created the arch_wipe_all script because I’d tried running the commands in it from the shell but it took so many hours (70+ lists and 10+ years of archives) that my shell would get disconnected and I’d have to start over completely. By putting the commands in a script I could preface it with nohup so it would keep running even if I got disconnected.

I noticed in /etc/mailman/postfix-to-mailman.py that if DEB_LISTMASTER isn’t set in /etc/mailman/mm_cfg.py or /usr/lib/mailman/Mailman/Defaults.py that error mail will go to postmaster@localhost, but I wanted it sent to another address in our organization so I added DEB_LISTMASTER = ‘addr@ourhost.org’ to the bottom of /etc/mailman/mm_cfg.py.

Pre-migration messages in people’s mailboxes had links to the old Listserv archives  in the footer, so even though we changed the footer for Mailman, we needed to provide redirects in case someone used an old link. We did this with an Apache URL Rewrite rule to redirect http://lists.ourhost.org/archives/listname.html to http://lists.ourhost.org/cgi-bin/mailman/private/listname/:

RewriteEngine on
RewriteRule ^archives/(.+)\.html$ http://lists.us.org/cgi-bin/mailman/private/$1/ [R]

(the second line, the RewriteRule, needs to be all on one line)

During one part of our disk issues converting our archives, our server admin said we’d maxed out our number of inodes (files and directories), probably because of the large number of archive related files (70+ lists for 10+ years). If you see errors like “unable to write file”, you can check the inodes used on the disk itself with df -i, and check inodes for your user with quota -sv username.

Bookmarks

The following pages weren’t listed in other parts of the guide and may be useful:

• Summary of the Mailman bin command line programs
• Mailman List Management Guide (2001 though, but has good settings descriptions)
• Changing List Owners On A Mailman Mailing List
• How do I integrate Mailman with PHP, or any other web technology?
• Mailman Installation Manual
• Mailman List Member Manual
• Regenerating Large Mailman Archives (I did not have to do the steps in this page even though our archives are 10+ years)
• How to edit the archives of a Mailman List
• How to remove a post from a Mailman list archive, or an entire archive (Note: to remove a post completely, I prefer to use Thunderbird to edit the mbox file)
• Mailman bounce processing explanations
• Mailman Frequently Asked Questions

Up: Table of Contents

Mailman doesn’t come with built-in archive searching. Instead, maybe in the Unix tradition of “do one thing and do it well, leave the rest to other tools”, you must find, install, and integrate your own search package. In some places it’s mentioned that people have used the Swish search package successfully, but usually no further explanation of how to do this is given.

(Note that there are actually two “swish” index/search packages out there, Swish-E and Swish++. For this guide, and on our server, I used Swish-E because we already used it for search on our main website.)

Mailman seems to add each message to the list archive (mbox file and HTML archive pages) as it’s processed for delivery/handling. We needed to somehow periodically have Swish generate a search index from all the HTML archive pages.

We also had to tweak the Mailman installation to add a search box to the list template pages, and add other tweaks to limit searching and search results to subscribers only (since all our lists and their archives were private).

Adding a Search Box to List Templates

To enable list archive searching, we needed a place where people could enter their search parameters, which required editing a few Mailman list template files.

As written at http://wpkg.org/Integrating_Mailman_with_a_Swish-e_search_engine, we didn’t want to edit the installed Mailman templates, we wanted to copy the default templates to a special area and then edit the copies to override the default ones. This made sure that our customized templates wouldn’t be overwritten by a Mailman upgrade.

On our server, the default templates were in /etc/mailman/en/ so we created /etc/mailman/site/en/ and copied the archidxhead.html, archtoc.html, and archtocnombox.html files from the former directory to the latter.

Then I edited archidxhead.html to add the following line:

<li><b><a href="/cgi-bin/mailman/search/%(listname)s"
>Search the archives of this list</a></b></li>

archtoc.html was edited to have:

<p>You can get <a href="%(listinfo)s">more information about this list</a>,
<a href="%(fullarch)s">download the full raw archive</a> (%(size)s),
or <a href="/cgi-bin/mailman/search/%(listname)s"
>search all the archives of the list</a>.</p>

Finally, archtocnombox.html was edited to have:

<p>You can get <a href="%(listinfo)s">more information about this list</a>
or <a href="/cgi-bin/mailman/search/%(listname)s"
>search the list archives</a>.</p>

Mailman seems to load templates into memory when starting up, so to get it to recognize the custom template overrides, we had to restart Mailman (which I think was done with /etc/init.d/mailman restart but I’m not 100% sure because our server admins actually did the restart).

Note that if your mailman CGI scripts end with cgi (e.g., if you built from source and used the –with-cgi-ext=.cgi flag to the configure script), it would be search.cgi in the above snippets.

After putting links to the search script on the Mailman templates, we had to put the search script in place to receive search queries. This was a bit complicated due to our requirement to keep search (and search results) limited to subscribers only, and it’s where we had to deviate from the Integrating Mailman with a Swish-e search engine page because that didn’t cover keeping searches private.

The script I wrote to create the Swish search index for each list (arch_index.py) also does as much setup for Swish searching as possible but there are some one-time setup things we had to do manually first.

Before I describe those one-time setup steps, I want to describe the general process of using arch_index.py to create the Swish index files for Mailman lists.

Generating a Search Index with Swish

Swish doesn’t search the HTML pages in the archive directly, probably because it would be too slow. Instead, it searches against its own optimized index file. That’s faster, but periodically the index file itself needs to be updated (regenerated completely, actually, as Swish doesn’t seem to support incrementally adding to index files).

Since I’d have to do the same steps to generate the Swish index file for each list archive, I automated the process with a script called arch_index.py, named somewhat in honor of the Mailman bin program “arch”.

In addition to generating the Swish index file for a list(s), I also had arch_index.py set up the CGI script which uses Swish to search a given list (based on the swish.cgi provided with the Swish package) , and set up a configuration file for the CGI script to work correctly.

If arch_index.py is just given the Mailman archive directory as a parameter, and no other options, it creates Swish search indexes for all lists (except the built-in “mailman” list, though there is a flag to force indexing of that too):

arch_index.py /var/lib/mailman/archives/

To create the index (and search config files, etc.) for just a particular list you can do:

arch_index.py -l some-listname /var/lib/mailman/archives/

arch_index.py assumes the Swish indexer program is located at /usr/bin/swish-e and the default Swish CGI search script was installed at /usr/lib/swish-e/swish.cgi—though you can override both on the command line or by editing arch_index.py itself. To see all options, just type arch_index.py with no arguments.

arch_index.py also enables searching list archives by date. Swish’s search by date feature looks at the modification times of indexed files, but for a list converted from Listserv all the HTML archive pages would be generated at once and have the same modification time. So to support date searching, arch_index.py looks at each posting’s date and then sets the file modification time to that date.

We should probably describe exactly what arch_index.py does though…

What arch_index.py Does

You can look at the source code yourself of course, but here’s a quick summary of what arch_index.py does:

1. Checks command line options and figures out what, if any, lists to index, and whether the swish-e and swish.cgi files are available
2. Copies the default swish.cgi file (if it hasn’t been copied already) and customizes it
3. Creates a config file for the customized swish.cgi script (if it hasn’t been created before)
4. Updates the modification times of any new HTML archive message files for each list, to match their message posting dates
5. Creates a swish indexing config file for each list, if necessary
6. Actually runs swish for each list to create the swish index files

Creating a Custom Swish Search Template for All Lists

Part of the customization in item #2 above is changing the default swish.cgi file to refer to a custom template file, similar to the process described in Integrating Mailman with a Swish-e search engine.

So we set up the custom template file by copying TemplateDefault.pm in /usr/lib/swish-e/perl/SWISH/ to TemplateDefault_MM.pm in the same directory and making four changes:

1. package SWISH::TemplateDefault was changed to package SWISH::TemplateDefault_MM
2. my $advanced_link = qq[<small><a href="$form">advanced form</a></small>] was changed to my $advanced_link = qq[<small><a href="$form$ENV{'PATH_INFO'}">advanced form</a></small>]
3. <form method=”get” action=”$form” enctype=”application/x-www-form-urlencoded” class=”form”> was changed to <form method=”get” action=”$form$ENV{‘PATH_INFO’}” enctype=”application/x-www-form-urlencoded” class=”form”>
4. The following was added after $query_href and $pages:
   $query_href =~ s#search\?#search$ENV{‘PATH_INFO’}\?#g;
   $pages =~ s#search\?#search$ENV{‘PATH_INFO’}\?#g;

(If you prefer patch files you can download TemplateDefault_MM.patch, change into /usr/lib/swish-e/perl/SWISH/ and then run “patch < TemplateDefault_MM.patch”; see The Ten Minute Guide to diff and patch for more info.)

$ENV{‘PATH_INFO’} had to be added in all those places to let one master Swish search template work for all lists, because the listname is passed to the search script as an extra path info parameter (i.e., /cgi-bin/mailman/search/listname).

arch_index.py copies the swish.cgi file in /usr/lib/swish-e/ into /var/lib/mailman/archives/private/ and then 1) changes SWISH::TemplateDefault to SWISH::TemplateDefault_MM and 2) changes $DEFAULT_CONFIG_FILE to point to /var/lib/mailman/archives/private/swish.cgi.conf which arch_index.py also creates.

The /var/lib/mailman/archives/private/swish.cgi.conf file created by arch_index.py uses an $ENV{‘LISTNAME’} environment variable that /cgi-bin/mailman/search (which we haven’t covered yet) sets from its extra path info parameter. In other words, using an environment variable in swish.cgi.conf file allows there to be one master configuration file which can dynamically refer to a different search index file for each list.

Setting Up the Search CGI Script – Background/Explanations

At this point we’d edited the search templates to provide links to the search pages, created Swish search indexes for all the HTML message files, and created a customized swish.cgi script and config file (in /var/lib/mailman/archives/private/) to do the actual searching with Swish and return results.

We could have then tried to hook up the web pages’ search links with our version of swish.cgi to do the search and return the results, as described on the Integrating Mailman with a Swish-e search engine page (though it uses a somewhat different integration method with Server Side Includes).

The problem with that approach is that while the full list messages themselves are protected by the Mailman “private” access control mechanism (for private lists), the search results themselves contain message excerpts so if a mailing list had confidential information it would be exposed to non-subscribers. For our purposes, that wasn’t acceptable.

So we also had to restrict running the search and displaying the results to list subscribers, which proved to be fairly involved.

My initial thought was to try to figure out Mailman’s authentication mechanism and then wrap it around the Swish search CGI script.

I looked at the files in the /cgi-bin/mailman directory (actually /usr/lib/cgi-bin/mailman/ in our installation) and the “file” command said that they were all “setgid ELF 32-bit LSB executable” files, i.e., compiled executables.

This confused me since most of Mailman seems written in Python, but I noticed that the Python files in /usr/lib/mailman/Mailman/Cgi/ had the same names as the compiled programs in /usr/lib/cgi-bin/mailman/. Further, changing one of the interpreted .py files in /usr/lib/mailman/Mailman/Cgi/ confirmed that the compiled files with the same name in /usr/lib/cgi-bin/mailman/ were calling them.

I tried copying the interpreted Python file /usr/lib/mailman/Mailman/Cgi/private.py to /usr/lib/cgi-bin/mailman/search and then editing it to call Swish’s CGI search program (/var/lib/mailman/archives/private/swish.cgi) instead of displaying private archive pages as private.py normally does.

While this worked on our development box, unfortunately on our production host it resulted in permission errors reading list configuration files because the web server wouldn’t use the setgid mechanism to run an interpreted file as the same “mailman” group as the other Mailman programs (which it wouldn’t do for good security reasons).

I’d thought the CGI files in /usr/lib/cgi-bin/mailman/ were compiled for performance reasons, but it turns out they were compiled to allow the web server to run the CGI scripts with the correct permissions via the setgid mechanism.

At this point I downloaded the source files for our version of Mailman because I wanted to confirm this and because I suspected I’d need to compile my own version of a “search” script (a modified “private” script). (I needed to download the source files because our production server admins had installed Mailman with a precompiled binary package.)

After downloading the mailman-2.x.yy.tar.gz file for our version of Mailman and unpacking it, I went into the src/ subdirectory and found a cgi-wrapper.c program that, along with common.c, confirmed my theory about the compiled binary wrapper programs existing just for security reasons. In particular the following comments in common.c were helpful:

/* We want to tightly control how the CGI scripts get executed.
 * For portability and security, the path to the Python executable
 * is hard-coded into this C wrapper, rather than encoded in the #!
 * line of the script that gets executed.  So we invoke those
 * scripts by passing the script name on the command line to the
 * Python executable.
 *
 * We also need to hack on the PYTHONPATH environment variable so
 * that the path to the installed Mailman modules will show up
 * first on sys.path.
 */

While this whole compilation thing was a pain, at least the C wrappers appeared to be as thin as possible and seemed to exist merely to call the corresponding Python scripts in /usr/lib/mailman/Mailman/Cgi/

(Incidentally, the part above about “hard-coding the path to the Python executable” finally explained why the Python scripts in /usr/lib/mailman/Mailman/Cgi/ didn’t have #!/usr/bin/python at the top!)

What I ultimately had to do was: 1) Compile my own “search” binary CGI wrapper which had the setgid bit like the other Mailman CGI programs in /usr/lib/cgi-bin/mailman/, and 2) Create a corresponding search.py file in /usr/lib/mailman/Mailman/Cgi/ as a version of the private.py script which authenticated the user and then called swish.cgi to do the actual search.

Compiling a CGI Wrapper for the Search Script

Even though our list server admins had installed Mailman from a binary package, to compile a custom binary CGI wrapper for the search script I needed to download the source for the same version of Mailman as installed on the server (which I determined by running the Mailman bin program “version”).

I created a directory at ~/src and then ran wget http://ftp.gnu.org/gnu/mailman/mailman-2.x.yy.tgz to download the tarfile for our Mailman version (e.g., 2.1.12) and tar xvfz mailman-2.x.yy.tgz to unpack it, which created a mailman-2.x.yy/ subdirectory.

I also realized I might need a place to “install” the compiled binary wrapper file, so I did mkdir ~/src/mailman and then chmod g+s ~/src/mailman because that’s required by the configure script below.

Then I changed into the newly-created mailman-2.x.yy directory and ran ./configure –prefix=$HOME/src/mailman to generate Makefiles from the corresponding Makefile.in files (if you’re curious how the Makefile system works, see make: Automating Your Recipes).

(Actually, I had to run ./configure –prefix=$HOME/src/mailman –with-username=list –with-groupname=list because our list server is set up to use “list” instead of “mailman” for the Mailman user and group, but chances are your server will use the default mailman account/group which configure looks for, so that won’t be necessary.)

Then I changed into the src/ subdirectory (making the current directory ~/src/mailman-2.x.yy/src/) which had the following files:

• cgi-wrapper.c
• common.c
• common.h
• mail-wrapper.c
• Makefile
• Makefile.in
• vsnprintf.c

All of those files had come with the source archive except for Makefile, which was created by running the “configure” command above. Then I needed to edit that file (~/src/mailman-2.x.yy/src/Makefile) to make the following changes:

• Changed prefix= /home/ouruser/src/mailman to prefix= /usr/lib/mailman (because our /home/ouruser/src/mailman was just an empty place for our newly-compiled program to be saved into)
• Copied and pasted the two $(CGI_PROGS) target lines and then in the duplicated lines changed $(CGI_PROGS) to search; this resulted in adding the following two lines:
  search: $(srcdir)/cgi-wrapper.c $(COMMONOBJS)
  $(CC) -DSCRIPT=”\”$@\”" -I. $(CGI_FLAGS) $(CFLAGS) $(COMMONOBJS) -o $@ $(srcdir)/cgi-wrapper.c

(You can use Makefile.patch to make the changes, with “patch < Makefile.patch”, but you’ll need to edit the patch file first to replace “ouruser” with your own username. If you make the changes by hand instead, keep in mind that the leading space before $(CC) is a TAB and not just spaces, which has been called one of the worst design botches in the history of Unix.)

Having added a new target for our search wrapper program in ~/src/mailman-2.x.yy/src/Makefile, I then ran make search to do the actual compilation.

I’d expected it to get compiled into the ~/src/mailman directory, but in fact it got compiled into ~/src/mailman-2.x.yy/src/ (the same directory as the just-edited Makefile). I typed ./search to run the program and it produced the following output:

Content-type: text/html

<head>
<title>Mailman CGI error!!!</title>
</head><body>
<h1>Mailman CGI error!!!</h1>
The Mailman CGI wrapper encountered a fatal error. This entry
is being stored in your syslog:
<pre>
Group mismatch error... (etc. etc. etc.)

This was a hell of a lot better than a segfault, and confirmed that the custom CGI wrapper for search had compiled successfully.

I copied the new “search” program to the same place as the other Mailman CGI scripts (with cp ./search /usr/lib/cgi-bin/mailman/) and had our server admins change the user and group to match the other CGI programs in that directory.

I also—importantly—had them do chmod g+s /usr/lib/cgi-bin/mailman/search to make it a setgid script, because the whole point of this process was to have a a compiled setgid script so that the web server would run the search program as the same group as the other Mailman programs and be able to read all the Mailman files without permission problems.

The last step was then creating a Python search script that would get called by the compiled search CGI wrapper program we just compiled and installed.

Creating search.py in Mailman/Cgi to Run the Swish Search

Compared to compiling a “search” binary CGI wrapper program, creating the corresponding Python search.py script was relatively simple.

As written above, the CGI wrapper programs in /usr/lib/cgi-bin/mailman/ only seem to exist to let Apache run the real Python scripts with the necessary security permissions as the main Mailman user (via the setgid mechanism).

These Python scripts that actually do the work are in the Mailman/Cgi/ directory, which on our server is /usr/lib/mailman/Mailman/Cgi/, and have the same names as the CGI wrapper programs in /usr/lib/cgi-bin/mailman/, so it’s pretty easy to see the one-to-one correspondence between them.

Basically, I just needed to do cp private.py search.py from within the /usr/lib/mailman/Mailman/Cgi/ directory and then edit search.py to do the following:

1. Removed everything between
   doc.set_language(lang)
   and
   if __name__ == ‘__main__’:
2. Added the following two lines in place of what was just removed:
   os.environ['LISTNAME'] = Utils.websafe(listname)
   os.execv(‘/var/lib/mailman/archives/private/swish.cgi’, [])

(or you can copy private.py to search.py and then use search.patch to make the changes with “patch < search.patch”)

This ended up replacing the part of private.py that displays HTML archive files after authentication, with a call to the swish.cgi search program to do the search and display the results.

To test this, I went to http://lists.ourhost.org/cgi-bin/mailman/private/abc-listname/ and clicked on the “search the list archives” link. This ran http://lists.ourhost.org/cgi-bin/mailman/search/abc-listname which ended up calling swish.cgi to display the default search form for that list. Putting in a test search term and clicking the Search button searched that list’s index file (which swish.cgi figured out from the listname in the URL) and displayed the results—after doing the same authentication that private.py uses before displaying private list messages.

Next: Appendix – Tips, Tricks, and Notes or Up: Table of Contents

Introduction

If you’re converting Listserv lists to Mailman, you probably want to keep your list archives too. The good news is that the Listserv archive format is plain text and is actually pretty well documented. The bad news is that it’s a custom proprietary format, not something common like the Unix “mbox” format, so conversion will be necessary.

Mailman actually wanted the archives to be in mbox format, so that’s what we converted the Listserv archives to. In fact, one of our lists had a lot of spam in its archives so after we did the conversion, but before we imported into Mailman, we opened the mbox file in Thunderbird, deleted the spam messages, and then saved the file back to mbox for importing into Mailman.

After converting the Listserv archives to mbox format, we had to move the mbox files to the right locations for Mailman to generate HTML archive pages, and then use Swish to index the HTML pages and do a little Mailman fiddling to keep private archives private (including search results). But first, more about the archive conversion process.

Converting the Archive Files

My archive conversion journey began with a Perl script called ls2mail.pl that was posted in 1999 and claimed to convert from Listserv’s archive format to mbox for Mailman.

The admin of our old Listserv modified it a bit to fix an unspecified “potentially nasty bug”, and I modified it further to 1) skip messages with dates earlier than the earliest legitimate post and later than the current year (to weed out spam messages with invalid dates), and 2) better match the mboxrd format (by quoting body lines beginning with “From”).

First, our Listserv archives were broken up into weekly archive files named listname.logYYMMw where YY is the year of the archive file, MM is the month, and w is the week (a = first week, b = second, up to e for a month with five weeks). For example, listname.log0901a was the Listserv archive file for the first week of January, 2009.

I decided it would be better to rename the files to use four digit years instead of two (i.e., listname.logYYYYMMw), so I used a small Perl utility called perlren to rename the files according to a Perl regular expression. So for each list I did:

perlren 's#log1#log201#' *.log*
perlren 's#log9#log199#' *.log*
perlren 's#log0#log200#' *.log*

Then I wanted a master file for each list which would have a sorted listing of all archive files (and because of the rename above, an alpha sorting was also a date sorting):

ls -1 *.log* > archive-files.txt

(I used a general name like archive-files.txt since all the log files for each list were grouped into a separate directory for each list.)

Then I used that file as a basis of making a file containing all the Listserv log files combined into one big file, in order:

perl -ne 'print "Processing $_"; chomp; print `cat $_ >> abc-listname.ls`;' \
  archive-files.txt

Finally, I ran the ls2mail.pl conversion script on the master Listserv log file to convert it to mbox format:

perl ~/mailman/ls2mail.pl < abc-listname.ls > abc-listname.mbox

Generating Mailman’s HTML Archive Pages

To generate the viewable HTML pages for Mailman’s web archive, I just had to move this new mbox file to the appropriate Mailman list archive directory (e.g., /var/lib/mailman/archives/private/abc-listname.mbox/) and run the Mailman bin “arch” command to generate the HTML archive pages from the mbox file:

mv abc-listname.mbox /var/lib/mailman/archives/private/abc-listname.mbox/
/usr/lib/mailman/bin/arch --wipe abc-listname \
  /var/lib/mailman/archives/private/abc-listname.mbox/abc-listname.mbox

(the –wipe option tells arch to overwrite any existing HTML pages with newly-generated ones from the mbox file, but since this was a new list that wasn’t a problem)

Once I’d tested this with one list, I repeated the process automatically with the other lists by doing something like the following (using command looping in the bash shell again):

cd /var/lib/mailman/archives/private
for list in abc-list1 abc-list2 abc-list3 etc;
do cd $list;
perlren 's#log1#log201#' *.log*;
perlren 's#log9#log199#' *.log*;
perlren 's#log0#log200#' *.log*;
ls -1 *.log* > archive-files.txt;
perl -ne 'print "Processing $_"; chomp; print `cat $_ >> archive.ls`;' \
  archive-files.txt;
perl ~/mailman/ls2mail.pl < archive.ls > $list.mbox;
mv $list.mbox ../$list.mbox/;
/usr/lib/mailman/bin/arch --wipe $list ../$list.mbox/$list.mbox;
cd ..;
done

By doing this, all the existing list archives were converted from Listserv archives (combined into .ls files) to Mailman .mbox archive files, and HTML web-viewable archives pages were generated by Mailman’s arch command.

A Word About Disk Space

I ran into disk space limits several times during this conversion process.

First, our old list host gave us the Listserv archives as one large compressed .tar.gz file—which expanded to triple the size, requiring 4x the size to accommodate both the gzip file and its uncompressed files.

Furthermore, I found that after concatenating all the individual Listserv .log files into one giant Listserv notebook archive .ls file (so, disk space times two for that step), the conversion to mbox format caused the resulting mbox file to take up about 60% more space than the .ls files.

Then running the Mailman arch command on the mbox files generated HTML pages that took up about twice as much space as the corresponding mbox files.

And then the Swish search index files took up about 55% of the size of the HTML archive pages, so that was another bunch of disk space.

The following table shows, step by step, how a 350MB compressed Listserv archive file ballooned up to almost 10GB:

You can mitigate some of those increases by deleting interim files (e.g., deleting the individual .log files after creating the concatenated .ls Listserv archive files), but the overall disk usage still ends up a lot more than you might think because the mbox files take more space than the Listserv notebooks, the HTML archive files are twice the size of the mbox files, and the Swish search index files are even more space.

We filled our disk up several times during the conversion, and one of the times this happened caused all mail delivery for all our just-converted lists to stop completely until some disk space was freed up and Mailman was restarted (actually it required a server restart, not just Mailman, which they’d tried with “/etc/init.d/mailman restart”).

By the way, after filling the disk a few times I created a script called disk_space_check and installed a daily cron job to send an email if disk usage was too high:

/bin/df -h | $HOME/bin/disk_space_check

In the next section we’ll add searching to Mailman’s archives, and make it so search on private lists (and search results) are limited to just subscribers.

Next: Setting Up Archive Search or Up: Table of Contents

To migrate subscribers, ideally we’d just get the addresses on each Listserv list with a REVIEW command and then subscribe those addresses to each new Mailman list of the same name.

The problem is that we need to copy not only the subscribers but their list option settings, such as NOMAIL, DIGEST, etc. This makes it much more complicated.

The first thing I did was identify the few “concealed” addresses on our Listserv lists so I could change them to NOCONCEAL. I did this by sending QUERY LISTNAME WITH CONCEAL FOR *@* to LISTSERV (replacing LISTNAME with one of the Listserv list names), for each list.

Since there weren’t a lot of concealed addresses, I set them “noconceal” by sending QUIET SET LISTNAME NOCONCEAL FOR ADDRESS PW=LISTPASSWORD for each subscriber to LISTSERV (replacing LISTNAME with the list name, ADDRESS with the subscriber address, and LISTPASSWORD with the Listserv list password).

I wanted to eliminate concealed statuses because I needed to make sure all subscribers would show up when I did a REVIEW command to generate one subscriber file for each list (more on this later).

Listserv Subscriber Settings And Mailman Equivalents

You may find it helpful to see a list of possible Listserv subscriber options and what settings/actions are available in Mailman to match them:

Listserv Subscriber Options	Mailman Equivalent
MAIL/NOMAIL (whether to get posts; MAIL is assumed unless NOMAIL is set)	each subscriber can have the delivery flag set “on” or “off”
DIGEST/NODIGEST (whether posts are batched up into periodic digests; NODIGEST is assumed unless DIGEST is set)	add_members bin command has a -d option to add people with the digest setting; afterwards, a subscriber can have the digest flag set “plain”, “mime”, or “off”; see also the digest_is_default and other digest_ list config setting
MIME/NOMIME (whether digests are sent to a user in MIME format; NOMIME is assumed unless MIME is set)	each subscriber can have the digest flag set “plain”, “mime”, or “off”; plain is equivalent to NOMIME and mime is equivalent to MIME; see also the mime_is_default_digest list config setting
INDEX/NOINDEX (a variant of digest mode; instead of digest batches of messages, just the subject lines of posts are sent as a summary)	no equivalent that I am aware of, and that’s fine because I don’t really see how useful a summary is with just subjects
ACK/NOACK/MSGACK (sends a “your post was received” confirmation message, or not; NOACK is assumed unless ACK is set; MSGACK is obsolete)	each subscriber can have the ack flag set “on” or “off”, though I don’t see the point—it seems to make more sense to just use getting your own posts back (with the myposts flag) as the acknowledgement message
SUBJECTHDR (whether [list name] is put in the Subject field of every post, for filtering purposes)	I believe this is only settable on a list-wide (not subscriber-specific) basis via the subject_prefix list config setting
CONCEAL/NOCONCEAL (whether someone shows up on a normal REVIEW of the list)	each subscriber can have the hide flag set “on” or “off”; also the private_roster list config setting lets you restrict member addresses to other members or list admins; my mailman_admin_command_handler.py doesn’t support the hide option though since I think it’s a PITA for admins
REPRO/NOREPRO (whether someone gets copies of their own posts; REPRO is assumed unless NOREPRO is set)	each subscriber can have the myposts flag set “on” or “off”
TOPICS (a way to let people only see certain categories of posts; I never understood or used this though, seemed easier just to create more lists)	Mailman definitely seems to support this but we did not have to deal with it; there are a number of topics_ list config settings and supposedly users can choose to just get certain topics; see Mailing list topics in the docs
POST/NOPOST (whether someone is allowed to post to the list; POST is assumed unless NOPOST is set)	no equivalent that I am aware of, though you can use the mod subscriber flag to cause someone’s posts to require approval, and then just never approve them
EDITOR/NOEDITOR (whether someone can bypass normal approval mechanisms required for posting to certain lists)	I don’t think there is a per-user subscriber flag for this, but the moderator list configuration setting allows you to specify all moderator/editors at once; the only downside is that then they also get notices of any posts held for review and have the ability to approve/deny them too; can also use the accept_these_nonmembers list config setting to allow people to bypass approval, but I think it only works for non-subscribers
REVIEW/NOREVIEW (whether someone’s posts are specially held for approval on an otherwise unmoderated list—usually for bad behavior)	each subscriber can have the mod flag set “on” or “off” (on = they are “moderated” and need their posts approve)
RENEW/NORENEW (whether someone should get periodic stay-subscribed confirmation emails, overriding the list setting)	no equivalent that I am aware of, since I don’t believe Mailman sends out stay-subscribed (i.e., renew your subscription) confirmation messages

Note that some of these settings (like the mod flag for users) were difficult to discover, and sometimes required delving into the source code (especially the Mailman/Commands/cmd_set.py module).

More Preparation Work

At this point we’d changed anyone who was set CONCEALed to NOCONCEAL so we wouldn’t miss them in the conversion process. Then we needed to decide which of the subscriber settings were important to copy in the conversion and which were not. (The table above should help you think about the various subscriber settings to consider.)

For example, I decided that we’d support (i.e., carry over the subscriber settings on our old Listserv lists to equivalent Mailman settings on the new lists) the NOMAIL, DIGEST, MIME, NOPOST, NOREPRO, and REVIEW options.

I’d already decided to include list names in subject lines for all subscribers on all our lists, regardless of what their old Listserv SUBJECTHDR settings were, due to the fact that Mailman only has this setting at the list level.

(For people who had relied on list names in subject lines, to not include this setting on the Mailman lists would destroy their mail filters, while for those who didn’t rely on it, at worst it would be a mild annoyance to suddenly see list names in subject lines—though before our cutover I warned subscribers that list names in subject lines would soon be the default.)

The Listserv EDITOR flag was also something Mailman only supported on a per-list basis, but that wasn’t a problem for us because we’d always used the Editor= line in the old Listserv list headers, rather than the EDITOR flag on individual subscribers, so we dealt with it at the list setting level instead of here at the subscriber setting level.

I’d decided against supporting the CONCEAL flag, even though there is an equivalent setting on Mailman lists (“hide”), because I don’t think it really protects subscriber addresses (this is really best done with the “private_roster” Mailman setting), and over the years CONCEAL caused me a lot of headaches (e.g., I’d waste a lot of time trying to figure out a subscriber problem only to finally realize that they were concealed).

Finally, I decided we wouldn’t support the INDEX, ACK, or RENEW/NORENEW options, as Mailman doesn’t really have equivalents for those and I doubt anyone had set them anyway. I also didn’t bother with TOPICS since we’d never used it in Listserv.

So, for all of the “supported” options above, I had to send a message to LISTSERV seeing who, for each list, had these supported options set and saving the responses to each message in a separate text file. The message to LISTSERV was QUERY LISTNAME WITH OPTION FOR *@* (LISTNAME was the list name and OPTION was something like NOMAIL or MIME).

The reason we did it this way was that based on the way Mailman worked, it seemed best to do this in two steps: 1) Subscribe all subscribers of each Listserv list to the corresponding Mailman list, regardless of their settings, and then 2) go back and apply the special settings for certain people (such as REVIEW) by hand or by script.

Copying Subscribers And Settings

With our old Listserv setup we used cron on our web server to do an hourly REVIEW of each list and then save the results (which included all subscribers) to files, one file per list with the listname as the filename.

I’d written a quick and dirty script called queryopt.pl to email a query to Listserv to find out all subscribers with a given option on a given list, such as DIGEST, NOMAIL, EDITOR, etc.

Since all our list names began with abc- (where abc is our organization’s acronym), this allowed me to go into the directory containing all those review output files and do something like perl queryopt.pl nomail abc-* and have it send a bunch of Listserv queries, one for each list (because abc-* is expanded by the shell to all the abc- files, or all our list names).

I used the following procmail filter to save the results of these query emails to files named option-listname.txt (e.g., nomail-abc-foo.txt would have everyone with the NOMAIL option on the abc-foo list):

PMDIR=$HOME/.procmail
# Save results of certain queries; pipe to prog for massage and save
:0 Hc
* ^From:.*LISTSERV@
* ^Subject:.*Re:.* query
| cat - | $PMDIR/save_opt_query.py

This matched the sender and subject line and then sent the email with LISTSERV’s query response into a program called save_opt_query.py.

One minor issue I had, with no easy fix, was that save_opt_query.py expected the Listserv response message to be like:

Subscription options for Mike Smily <mikednahelix@HOTMAIL.COM>, list
ABC-ANNOUNCE:

DIGEST         You receive list digests, rather than individual postings
SUBJECTHDR     Full (normal) mail headers with list name in message
               subject
REPRO          You receive a copy of your own postings
MSGACK         Short "TELL" acknowledgement of successfully processed
               postings

Subscription date: 30 Jul 2004

Subscription options for "Alexandria ." <alextheconquerer@YAHOO.COM>,
list ABC-ANNOUNCE:

DIGEST         You receive list digests, rather than individual postings
SUBJECTHDR     Full (normal) mail headers with list name in message
               subject
REPRO          You receive a copy of your own postings
MSGACK         Short "TELL" acknowledgement of successfully processed
               postings

That worked most of the time, but sometimes the name/email was too long so the name was on one line and the email was on the next. So after importing our subscribers I had to go back and manually fix those few cases (by adding those addresses to lists and setting their options), but it was only about 30 people for 70 lists so, while annoying, it was an acceptable trade-off to keep save_opt_query.py as simple as possible.

(As to how I identified the problem records, I looked for lines in the saved option-listname.txt files that did not have the <foo@bar.com> brackets, for example with: egrep -v ‘<’ *.txt)

As mentioned before, the Listserv subscriber options I decided to support (i.e., copy over to the new Mailman lists) were: DIGEST, MIME (a special case of digest), NOMAIL, NOPOST, NOREPRO, and REVIEW. I also wanted to run queryopt.pl for NODIGEST to get the “regular” (non-digest) subscribers, because the Mailman add_members command has flags to specify a file with non-digest subscribers and a file with digest subscribers to add.

So after testing the procmail filter to make sure it would catch and save the option query output into files correctly, I did the following to run it for all the options:

for option in digest nodigest mime nomail nopost norepro review; \
do perl queryopt.pl $option abc-*; done

(Note that this used looping syntax specific to the bash shell; your shell’s loop and variable substitution might vary, such as requiring parentheses—if you don’t know what I’m talking about read your shell’s man page, search the web, or ask another developer/admin.)

To check if all responses had been saved properly I just did a file count based on filenames:

for option in digest nodigest mime nomail nopost norepro review; \
do echo $option; ls -1 ${option}-* | wc -l; done

(checking that the number of each option file equaled the number of lists we have, for example with 71 lists we should have had 71 nodigest-* files)

Since I’d been doing all this on our web server where the hourly Listserv REVIEW files resided (which allowed me to do abc-* in the shell to expand to all our listnames), I had to copy these files (which were the results of queries to LISTSERV to find out the options people had on our soon-to-be-old Listserv lists) over to our new list host for automated subscription and option setting on the new Mailman lists.

Importing Subscribers and Setting Options

Now we had a bunch of option-related text files (option-listname.txt) on our new Mailman list server. I used shell looping again to do a batch import of subscribers followed by option setting for subscribers with special options like NOMAIL.

Since I’d already created new Mailman lists on our new list host equivalent to the old Listserv lists I could do the following:

for lst in `/usr/lib/mailman/bin/list_lists -b | egrep -v '^mailman$'`; do
echo add_members -r nodigest-${lst}.txt -d digest-${lst}.txt -w n -a n $lst
echo set_option.py mod on ${lst} 'tree#bard' review-${lst}.txt
echo set_option.py mod on ${lst} 'tree#bard' nopost-${lst}.txt
echo set_option.py myposts off ${lst} 'tree#bard' norepro-${lst}.txt
echo set_option.py delivery off ${lst} 'tree#bard' nomail-${lst}.txt
echo set_option.py digest mime ${lst} 'tree#bard' mime-${lst}.txt
done

As before, when using shell looping I like to use “echo” statements to check the commands before running them. If all looks good, I remove the echos to really run the commands.

(In fact, I first ran the add_members and set_option.py commands manually on one of our lists and verified their success by sending “who” and “show” commands to the admin command handler, before using the shell loop to run them on ALL lists.)

The commands above first subscribe people in bulk using the Mailman add_members bin command with options to specify files with non-digest and digest subscribers, and to not send any notifications of the subscriptions. Then they use another script I wrote (set_option.py) to apply the Mailman option settings to subscribers and thus copy the old Listserv settings to the new subscribers.

Keeping Subscribers and Options In Sync

At this point I had the Mailman lists set up as a snapshot in time of the Listserv lists, but I needed to do additional work to make sure they’d stay in sync going forward (i.e., to run them in parallel with the Listserv lists, in terms of subscribers and subscriber options).

Fortunately, all of our subscribers seemed to use our CGI-based web subscription management pages rather than emailing LISTSERV directly or using the Listserv web interface and, as described before, those CGI scripts just generated admin email commands to LISTSERV.

This worked out well for our migration because I was able to add Mailman code right after Listserv related code, to do the same things for the Mailman lists (such as subscribe, change options, unsubscribe, etc.).

To get a little more detailed, our web pages called CGI scripts which in turn called a MailingLists.py module to do the actual work for our Listserv lists. For the migration I copied MailingLists.py to MailmanLists.py and kept the function names the same but changed the actual commands to use the custom Mailman admin command handler. Then I could just make sure that every time a CGI script used a function in MailingLists.py it called the same function in MailmanLists.py to keep the old Listserv and new Mailman lists in sync.

This was still a little time-consuming, searching all our CGI scripts to see where MailingLists.py was called and then adding calls to MailmanLists.py, but it was worth it.

But it wasn’t just CGI scripts—I also had to check for cron jobs or shell scripts I’d written that used MailingLists.py functions too, and add equivalent calls to MailmanLists.py like in the CGI scripts (and keep track of all these changes so that I could go back and remove the MailingLists.py calls after the migration was done).

Fortunately, there were relatively few cron jobs or shell scripts which used MailingLists.py (really just one weekly cron job that checked whether the email addresses on “members only” lists actually corresponded to current members).

But this is definitely a case where utility scripts like tgrep came in handy, such as running a command like the following from the home directory on our web server, which also contained the web tree as a subdirectory:

find . -type f -size -512k -exec tgrep MailingLists {} \; > /var/tmp/mls.txt

This started in the current directory (.) and looked through all files (-type f) less than 512Kb (-size -512k) and ran tgrep (which only searches text files) for “MailingLists” and printed out the filename and matching lines. The output was redirected someplace outside the current directory tree because otherwise find/tgrep would have kept matching lines in the output file itself, resulting in an ever-growing output file (ask me how I know this!).

(The unix find command is very powerful and the above only scratches the surface of the many awesome things you can do with it; for more information on its many useful options, Google something like using find linux.)

Keeping Subscriber Files Updated With Mailman

This may not apply to you at all, but I want to mention that we still use the loosely coupled web subscription pages whose input goes to CGI scripts which use objects to generate admin email commands to the list server; the only real difference after our migration is that the list server has changed from Listserv to Mailman. (See the lib directory, particularly MailmanLists.py—the other files in there are just supporting objects for it.)

You may remember that I said our old Listserv setup had an hourly cron job which sent REVIEW commands to LISTSERV and saved the replies into text files, one per list. That provided subscriber files our MailmanLists.py object could use, allowing the CGI scripts to have (relatively) recent information about which subscribers were on which lists.

We could have used the same setup with Mailman, sending email messages to the admin command handler every hour to get an updated subscriber roster for each list, but since we were allowed to submit cron jobs on our new list host, it seemed better to just run a job hourly on the list host that would directly extract the subscriber rosters, save them to files, and then copy them over to our web host using secure copy (scp).

By exchanging SSH key files ahead of time we were able to allow the scp copy to happen automatically without having to enter passwords for the file transfer. If you don’t know what I’m talking about, Google something like ssh passwordless login (since scp uses ssh underneath).

If you want to do something like this automated saving of list subscriber rosters to files, and then copying them somewhere else, you might find my short review_and_copy utility script useful.

Conversely, if you want to do something like we used to do with Listserv, where you regularly email Mailman’s admin command handler a “who listname” command and then save the results to a file, see the revmmlists script.

Similarly, if you’re paranoid about backing up your Mailman list configurations to dump files on a regular basis (with content equal to running config_list -o dumpfile listname at the command line), you might want to look at my backup_list_configs file.

And if you do a lot from the command line you might be interested in the sub, unsub, group_sub, and group_unsub scripts in the utils area.

That’s not necessary by any means, but I spent so much time configuring our lists that I want to know I can always recreate those configurations instantly if needed (with config_list -i configfile listname).

A Note About Passwords

I need to mention that since our website also had a private, login-based “members only” area, we wanted list subscribers who were also members of our organization to have their Mailman list passwords be the same as their website login passwords.

On the other hand, if someone was a subscriber but not a member, we wanted all their passwords set as the same default password.

This didn’t matter so much for subscription management because we had our own custom pages for subscribing/unsubscribing/etc. In fact, we didn’t actually want people going to the Mailman list pages for those things, we wanted them to go to our pages so we could also do things like keeping mailing list and website passwords in sync.

However, subscriber passwords did matter for accessing list archives since all our list archives are password-protected to subscribers only, so we wanted the password to be a no-brainer, just-use-the-same-password-as-your-website-login thing.

Shortly copying the subscribers from Listserv to Mailman and setting their options, then adding code to keep Listserv and Mailman subscribers/options in sync, I realized that I needed to also keep subscriber passwords in sync, and not just between Listserv and Mailman, but in the case of members, in sync with each member’s corresponding website login password too.

I dealt with this in a two-stage manner, just as I had for the options. The first stage was doing a massive sync to get everything correct as of a snapshot point in time. Then the second stage was to make sure it would be correct going forward.

For the first item, the one-time-fix, I coded a utility script called sync-list-passwords that used objects representing our membership database and website login system to check whether someone was a member and if so, whether they’d set up a login on our site. The script checked these and then set the Mailman list passwords appropriately depending on which case each subscriber fit into.

For the second item, I simply made sure in the MailmanLists.py code that whenever someone got added to a list it would see if they were a member and had a website login, and then send a password setting command to the admin command handler. If someone were removed from a list there was nothing that needed to be done, and if someone changed their address the clone_member Mailman code took care of copying over their password.

I also had to add code to our website login system so that if someone changed their login password there, it sent a message to the Mailman admin command handler to change their password on all their list subscriptions as well.

I’d be very surprised if your setup matched ours or was even as complicated, but the point is that Mailman does assign passwords to subscribers for each list (randomly I think, if you don’t specify), and you will need to think about this if you intend to have private list archives or intend to allow subscribers direct access to the Mailman web interface pages (since subscribers need to log in to change anything).

Further, if you have a lot of lists you’ll need to think about how you’re going to handle the fact that if you don’t do something to actively keep subscriber passwords the same across all those lists, they could end up with different random passwords on different lists, which could be a big problem.

And it’s a lot better to come up with a password-handling strategy up front, before you start your migration, than in a panic after the migration has already started (ask me how I know this!).

Next: Converting Listserv Archives to Mailman or Up: Table of Contents

This section describes how to create new Mailman lists equivalent to the existing Listserv lists.

When we did this, there were about two weeks when we made subscriber changes in parallel, to both the old Listserv lists and the new Mailman lists, while we converted the archives and got ready for the final cutover.

We were able to do this because all our subscriber changes (subs, unsubs, and address changes) were done through CGI scripts on our website which generated email commands to LISTSERV—so we just modified the scripts to also send equivalent commands to the Mailman administrative command handler we described in the last section.

In other words, if someone subscribed to ListX on our website, the CGI script sent commands to subscribe them to both the Listserv ListX (soon-to-be-old server) and the Mailman ListX (soon-to-be-new server), even though the latter weren’t get getting any posts because ListX@lists.ourhost.org still pointed to the Listserv server.

When we were ready for the actual cutover we just changed lists.ourhost.org to point from the Listserv host to the Mailman host, and Mailman started handling the new posts right away. Then once everything was verified as working properly, we went back and updated the CGI scripts to stop sending subscriber changes to the old Listserv host.

Identifying And Grouping Current Lists

This may be easy for you, but we had 70+ lists to migrate, so not only did we have to identify them, we had to group them in terms of their settings so we’d be able to use a script to create multiple lists with similar settings in a batch instead of having to create them one at a time.

To do this you obviously need to understand all of your old Listserv list settings AND the equivalent new Mailman settings.

So if you  have a large number of lists you’ll need to examine all of their settings and see if they need to be categorized into settings groups. For example, after analyzing our 70+ lists, I identified the following settings groups:

• announcement lists – moderated lists allowing only a few people to post directly (no approval needed), but forcing all other posters’ submissions to go into an approval queue, with replies going to the poster by default, not the list, and with anyone able to subscribe to these lists

• member lists – discussion lists that any of our organization’s members (not the general public) could subscribe to and then post to, with replies going to the list by default

• public lists – discussion lists anyone in the general public could subscribe to and then post to, with replies going to the list by default

• public review lists – discussion lists anyone in the general public could subscribe to, but with messages requiring moderator approval, and replies going to the list by default

• open post lists – special purpose lists that anyone in the general public could post to, but which required special list owner approval for subscriptions (the subscriber lists for these were very small)

• restricted lists – discussion lists whose subscribers were limited to special organizational subgroups, requiring each subscriber to be added or removed by a list owner, with replies going to the list by default

To do this for your lists, ask basic questions like the following:

1. Who can subscribe to the lists? (anyone in the public, only people in your organization, only a few special people, etc.)
2. Who can post to the list? (you might have an announcement list that anyone can subscribe to, but only a few people can post to)
3. Do posts require moderator approval? (and if so, is the moderator just the list owner, or do you need to have additional non-owner moderators?)
4. Where do replies to posts go by default? (to the list or to the poster?)

Examining your current list settings should answer these questions and maybe prompt additional ones you’ll need to answer for all your lists. Then you can determine if all your lists have the same settings or they need to be categorized into settings groups as ours did.

Exploring Mailman List Config Options

To see all the possible Mailman settings for a list, I used the following command to view the settings of the previously-created test list:

/usr/lib/mailman/bin/config_list -o orig.listconfig test-list

(your config_list location might differ, but it will be with all the other “mailman bin” commands)

The goal of this was, for each possible Mailman list setting, to determine if we wanted the same setting to apply to all our lists or if the setting’s value would differ depending on which settings group the list was in.

If you do this and look at orig.listconfig you’ll see that it’s about 80% comments, about 10% blank lines and 10% Python code; list config files like this are (and must be) valid Python code since they are read back in by the config_list script with the -i option to make list changes.

(Note that if you’re just changing one or two things, you don’t need to feed the entire config file back into config_list; it’s smart enough that if you feed it a file with a small number of settings, it will just change those settings on the given list and keep the other settings at their previous values.)

It’s also important to note that these “config files” have no direct ties to Mailman. Once you dump a list’s configuration settings into a text file using config_list -o (o=output) as above, you can do whatever you want with that text file and Mailman won’t care.

The only time the text file matters again is if you feed it back into Mailman with config_list -i (i=input), at which point Mailman will take each setting in that file and use it to update the real list settings it keeps internally. Once you’ve done that, you’re free to delete the file if you want—you can always recreate it if necessary with config_list -o.

I wanted to keep orig.listconfig as a set of Mailman’s default values, so I copied it to announcement.listconfig so that the latter could represent the configuration of our announcement lists (see our settings groups in the bulleted list above).

Then I edited announcement.listconfig, first replacing every reference to “test-list” with “abc-listname” (no quotes; all our mailing lists begin with abc-). This allowed us to later batch-create multiple lists with almost identical configurations by replacing abc-listname.

Second, I added a comment of “ALWAYS” or “DIFFERS” next to every setting in announcement.listconfig, for “always use this setting value for all our lists” or “this will differ from one list settings group to another”. This allowed me to copy announcement.listconfig to other settings group listconfig files, search for each DIFFERS comment, and make changes to just those options.

I also made notes next to each ALWAYS or DIFFERS comment to explain the choice of setting value and make notes about the configuration option itself if necessary. For example:

# Should any existing Reply-To: header found in the original message be
# stripped?  If so, this will be done regardless of whether an explict
# Reply-To: header is added by Mailman or not.
#
# legal values are:
#    0 = "No"
#    1 = "Yes"
# ALWAYS - force replies to list or, if possible, list AND poster
#   either way, need to strip original reply to, so always true
first_strip_reply_to = True

The bold text is what I added, a note that I wanted this option to be true for all our lists, applying the “reply to list” or “reply to poster” policy equally for all posts and never honoring an original Reply-To header included by a poster.

Here’s an example of a DIFFERS comment explaining the choice and which list settings groups should have which values for this setting:

# When this option is enabled, all list traffic is emergency moderated,
# i.e. held for moderation.  Turn this option on when your list is
# experiencing a flamewar and you want a cooling off period.
#
# legal values are:
#    0 = "No"
#    1 = "Yes"
# DIFFERS - set to no for almost all lists; yes for moderated lists
emergency = False

Finally, since a file that’s intended for feeding back into config_list -i must be valid Python code, I wanted to make sure I hadn’t made any typos so I ran python announcement.listconfig to make sure the script diddn’t bomb; a successful run just gives you back the command line with no output since config files just set a lot of variables.

I’d love to explain all of our option choices, but you’ll need to examine them all for yourself anyway and the in-file comment documentation is usually pretty good. One option I would like to explain though is the “new_member_options” setting, an opaque numerical value with a nebulous comment:

# When a new member is subscribed to this list, their initial set of
# options is taken from the this variable's setting.
new_member_options = 256

The new_member_options name implied it was pretty important and determined the default options for all new subscribers, and I needed to know if it should be the same across all our lists or if it needed to differ per settings group.

However, as you can see there was very little comment documentation for this option, and I couldn’t find anything online either. Ultimately I had to delve into the source code, where I found the following in the source tree’s Mailman/Defaults.py.in file:

# Digests             = 0 # handled by other mechanism, doesn't need a flag.
# DisableDelivery     = 1 # Obsolete; use set/getDeliveryStatus()
# DontReceiveOwnPosts = 2 # Non-digesters only
# AcknowledgePosts    = 4
# DisableMime         = 8 # Digesters only
# ConcealSubscription = 16
# SuppressPasswordReminder = 32
# ReceiveNonmatchingTopics = 64
# Moderate = 128
# DontReceiveDuplicates = 256

I believe the way this works is that you decide which options (“flags”) you want to be “on” for new subscribers and then add up all the values for those options; the total is what new_member_options should be.

In other words, since the default (in a new Mailman list) new_member_options value is 256, that means the only option turned on for new subscribers is DontReceiveDuplicates.

I examined the above options and determined that the only setting we wanted was SuppressPasswordReminder (if someone subscribed to 20 of our lists it would be awful to get 20 monthly password reminders!), so I made our new_member_options value 32 instead of the default 256, and made an ALWAYS note that this should apply to all our lists.

(Side note: I got the sense that new_member_options is older/obsolete since some of its flags seemed covered by other configuration options and this style of determining default subscriber options is a lot more difficult for most people to use.)

Anyway, after replacing test-list with abc-listname in the announcement config file, and then adding ALWAYS or DIFFERS comments for each option, I copied announcement.listconfig to members.listconfig, then went through all the DIFFERS comments and made sure that the settings values were correct for “members” type lists.

When done with members.listconfig I copied it to openpost.listconfig, edited all the DIFFERS settings, and then repeated the copy/edit process for the remaining list settings groups.

I ended up with six .listconfig files, matching our list settings groups in the bulleted list above: announcement.listconfig, members.listconfig, openpost.listconfig, public.listconfig, pubreview.listconfig, and restricted.listconfig. These files were 1) identical to each other except where lists of that settings group should differ, and 2) had a generic list name of “abc-listname” instead of the real list name.

Listserv And Mailman List Configuration Option Equivalents

In order to make good choices about the Mailman configuration settings for our list settings groups, I needed to understand which old Listserv configuration settings translated to which new Mailman configuration settings.

I wasn’t able to find much online comparing Listserv and Mailman list settings; in fact the only page I found was the MIT Mailman User Guide: ListServ Keywords in Current Use and its accompanying “cheat sheet” PDF.

I can only discuss the Listserv options we used and give you their Mailman equivalents, but hopefully between the options here, the MIT page, the Listserv list header documentation, and the Mailman config file comments, you can figure everything out for your site.

Creating Mailman Lists In Bulk

In the “Identifying And Grouping Current Lists” section we created several different settingsgroup.listconfig files, where settingsgroup was a name for a group of lists with all the same configuration settings except for the listname itself. We’d put abc-listname as the list name in the group settings file so we could search and replace it in the batch creation process, which is described in this section.

While we came up with .listconfig files for each settings group, we still had to have a way to say which settings group each list belongs to. So I created settingsgroup.lists files such as announcement.lists, member.lists, public.lists, pubreview.lists, etc. Each file simply had, the lists belonging to that settings group, one list per line.

For example, we only had one announcement type list to create so our announce.lists file had just one line:

ABC-Announce

If we’d had more announcement type lists, they would have each been on a separate line in the announce.lists file.

So the idea was that announcement.lists had a roster of lists (one per line) to create according to the announcement.listconfig file settings, with abc-listname in that settings file to be replaced with each list name from announcement.lists. And the same thing with public.lists, pubreview.lists, etc., with each .lists file containing the lists to be created with the corresponding .listconfig file settings.

There were probably many ways to do the batch creation, taking each line in the whatever.lists file and replacing abc-listname in the whatever.listconfig, then using the “newlist” Mailman bin command to create the new list with that name.

I chose to just use a series of single-line commands, one for each settings group. The command I used was the following, pasted into the shell and run as a single line:

perl -ne ‘chomp; $mclist = $_; $lclist = lc($_); print `echo cp -f announcement.listconfig /tmp/$lclist.listconfig`; print `echo perl -pi -e s#ABC-Listname#$mclist#g /tmp/$lclist.listconfig`; print `echo perl -pi -e s#abc-listname#$lclist#g /tmp/$lclist.listconfig`; print `echo newlist -q $lclist abc-listmaster\@abc.org listpassword >> aliases-to-make.txt`; print `echo config_list -i /tmp/$lclist.listconfig $lclist`;’ announcement.lists

(This assumes that you’re already in the Mailman bin directory, such as /usr/lib/mailman/bin, or that you’ve put that directory in your $PATH. If not, you will want to write out the full paths to newlist, config_list, and change_pw above.)

Note:  The command as written above won’t actually do anything, it will just print what it would do if you removed all the “echo” commands. Whenever I do anything complex like this, repeating commands over a lot of items, I first use echo statements to print out what’s going to happen and then look it over to check for errors before running the real commands.

The overall idea is that Perl is used to iterate over each line of the announcement.lists file, where each line is the mixed-case name of a list we want to create. The names are mixed case because in some settings (like the list description) Mailman lets you use mixed case, while in others it just takes lower case.

The command above sets $mclist for the mixed-case list name, and $lclist for lowercase, then it copies announcement.listconfig to a temp location, replaces ABC-Listname in the temp file with the mixed case value using the $mclist variable, and does the same with abc-listname and $lclist.

Then it uses the Mailman newlist bin command to create the list using abc-listmaster@abc.org as the initial list owner and “listpassword” as the initial list password, though both can be overridden in the group settings file which is applied next.

Important Note: Since we had 70 different email lists, we did want them all to have the same password. Your situation may be different and if you need each list password to be different you might use the above commands to set them all the same and then use the Mailman change_pw command afterward to customize the password on a per-list basis.

The version of the newlist command above also appends the output of the command to a file called aliases-to-make.txt in case you need to create mail aliases for your new lists. We didn’t need to do that though because our new list host had set up postfix-to-mailman.py so that postfix automatically recognized the Mailman aliases for new lists. (There’s a qmail-to-mailman.py file too if you’re running qmail, very handy.)

(Note: Actually the “>> aliases-to-make.txt” redirection in the “echo newlist” portion causes that command to get redirected into aliases-to-make.txt rather than printing on screen, so for testing the echo you’ll have to remove “>> aliases-to-make.txt”, just be sure to add it back in before running the commands for real!)

After the new list is created, config_list is used with the temp config file to configure it.

If you’re comfortable with all of this, try it out and examine the output from the echo commands.

You can even try the commands for real, one at a time in the shell with one list, to make sure they’ll work as you expect them to (e.g., that there are no path or permission errors). In that case you can check the contents of aliases-to-make.txt to make sure the output is what you expected, and even run config_list -o – newlistname to double-check the configuration.

When you’re comfortable with what will happen from the looping commands above, remove the echo statements and run the commands for real with a small number of lists. If that works, you can then run the commands with each successive settings group (which for us meant changing “announcement” to “member”, then “public”, “pubreview”, etc.), ultimately creating all new Mailman lists in batches according to their old Listserv settings groups.

Next: Migrating Subscribers And Keeping Them In Sync or Up: Table of Contents

This section describes how to install an “administrative command handler”, something you can email list management commands to, just like emailing commands to LISTSERV. If you never used the Listserv email interface then this entire section may be unnecessary for you.

Background

One of the first things I did when investigating Mailman was try to find out how to send mailing list management commands by email.

That was important because in administering our 70 email lists over the years, we’d developed a set of helper scripts and modules to make list management actions like subscribing/unsubscribing easier.

If someone wanted to subscribe to our lists they’d put their name and email into a form, check off the lists they wanted, and submit. Then behind the scenes our scripts would send an email to LISTSERV with the appropriate commands to make it happen.

This was all aimed at 1) allowing subscribers to manage their own subscriptions on the web, and 2) allowing list admins to easily manage subscribers over a lot of lists.

Unfortunately, it soon became apparent that Mailman has no administrative email interface. Instead, Mailman provides a simple but clean web-based interface for list admins to manage users on each mailing list.

Going to each list’s web interface to manage a given subscriber is fine for one or a few mailing lists, but it’s completely unmanageable at 10, 20, or 70 lists—you need a simple one-stop place to set options, change addresses, etc. for a subscriber on all lists at once.

This meant I either had to abandon Mailman as the mailing list software in favor of something which did have an administrative email interface, or stick with Mailman and program it myself. Since Mailman seemed like the only viable replacement for Listserv, I decided to scratch my own itch.

Fortunately, I already knew Python, which most of Mailman is written in, so programming an administrative email interface which could call built-in Mailman code to do its work was actually possible.

Issues Setting Up Admin Command Handler

I’d noticed when I created the “test-list” on my development box with the newlist command that it told me I still needed to create some email aliases for various list-related addresses to pipe messages into mailman for handling:

To finish creating your mailing list, you must edit your
/etc/aliases (or equivalent) file by adding the following
lines, and possibly running the `newaliases' program:

## test-list mailing list
test-list:              "|/path/to/mailman post test-list"
test-list-admin:        "|/path/to/mailman admin test-list"
test-list-bounces:      "|/path/to/mailman bounces test-list"
...

This got me to thinking that I’d need an email address to receive the Mailman admin commands. My first thought, based on the above, was something like test-list-admin-cmd, but I didn’t want to have to make a new admin alias for every new list.

I really wanted something like our old listserv@lists.ourhost.com, which all our web and shell scripts were sending admin command messages to.

So I created a command handler called mailman_admin_cmd_handler.py and in the control panel for the development box set it so that mail sent to mailman-admin-cmd@lists.ourhost.com would get piped into mailman_admin_cmd_handler.py

(Our host’s control panel was cPanel, and I used the Forwarders option in the Mail section to “Add a Forwarder”, then “Address to Forward” of mailman-admin-cmd, then “Advanced Options”, then “Pipe to a Program”, then /path/to/mailman_admin_cmd_handler.py)

If you use /etc/aliases for Sendmail or Exim, as the newlist output suggests, then you’d probably do something like:

## admin interface for all mailing lists
mailman-admin-cmd:   "|/path/to/mailman_admin_cmd_handler.py"

Actually that’s a simplification of what I really did. I really used procmail because that’s what handled all our other mail forwarding needs. So in “Pipe to a Program” I piped it to |/usr/bin/procmail -m /home/ouruser/.procmail/rc.mailman-admin-cmd and then in that procmail recipe file I put:

:0 H
* ^TOmailman-admin-cmd@lists.ourhost.com
| $HOME/.procmail/mailman_admin_cmd_handler.py

(actually I had a few other things in that file, but a procmail tutorial is beyond the scope of this guide—type “man procmailrc” or “man procmailex” at a shell for more info)

This was all on our dev box, and part of why I stuck mailman_admin_cmd_handler.py in ~/.procmail was that my goal was not to make any modifications to the base Mailman installation in case our production list host didn’t let us muck with those files.

This placement had a few minor consequences. One is that I had to create symbolic links for two mailman “bin” programs, list_members and clone_member, into the same directory as the handler script, adding .py to the symbolic links so the handler script could import them. For example:

cd /home/ouruser/.procmail
ln -s /path/to/mailman/bin/clone_member ./clone_member.py
ln -s /path/to/mailman/bin/list_members ./list_members.py

I also had to install a helper module (ListStream.py) and configure mailman_admin_cmd_handler.py with custom settings like the Mailman installation directory, the SMTP server, an optional string/password for the subject line of admin command messages (for added security), etc.

Even if you can put mailman_admin_cmd_handler.py in the mailman bin directory itself, you’ll still need to symlink clone_member to clone_member.py (and the same for list_members) within that directory so mailman_admin_cmd_handler.py can import and use those scripts, you’ll still need to save ListStream.py there, etc.

(But I’d still recommend keeping these custom scripts outside the installed Mailman tree, even if you have permissions to modify those files, simply because if you ever upgrade your Mailman installation, you don’t want to inadvertently delete/move the custom files and break a lot of stuff. Plus it’s just nice to have standard-install Mailman stuff in one area, and custom add-on stuff in another.)

I should also mention that when we finally chose a list host, they didn’t use alias forwarding/piping or procmail at all, they used a specialized postfix-to-mailman.py script so that no aliases needed to be made when new lists were created.

(If you’re using Postfix with Mailman see the GNU Mailman Installation Manual, particularly the Integrating Postfix and Mailman section; see also a related discussion on FreeBSDDiary.)

Downloading the Files

Here are the basic files for setting up the administrative email interface; click to view, right-click and Save to download:

• mailman_admin_cmd_handler.py – the main handler program
• ListStream.py – a helper module used by mailman_admin_cmd_handler
• clone_member.py.txt – a reminder that you need to symlink clone_member
• list_members.py.txt – a reminder that you need to symlink list_members
• test_mailman_admin_cmd_handler.py – optional unit tests of the handler

test_mailman_admin_cmd_handler.py is completely optional, but I’d recommend downloading and running it just to check that you made all the necessary customization changes in mailman_admin_cmd_handler.py. Just be sure to look through it thoroughly before running it, because you’ll need to choose your own test values.

IMPORTANT NOTE: I’ve tried to make sure that any code I’m providing has the word INSTALL in comments to mark any places you’ll need to modify the code for your own installation (i.e., places where your installation will probably differ from mine). So you need to go through every file you download and search/grep for INSTALL and update the corresponding settings.

Installing

The following are the steps I used to install the administrative email command handler on our production host. In some cases your environment may differ, such as the difference between using Sendmail/procmail filtering (on our dev box) and using Postfix (on our production box) as described above.

First, I created a directory to hold all custom Mailman related files, ~/mailman.

I also noted the locations of the system (installed) Mailman files, which seemed to be /etc/mailman, /var/lib/mailman, and /usr/lib/mailman. In particular, I needed to know the path to the directory tree holding the Mailman bin scripts and helper objects, which the admin command handler uses. (For us, those were /usr/lib/mailman/bin/ and /usr/lib/mailman/Mailman/ respectively.)

After downloading the files in the “Downloading the Files” section above into the ~/mailman directory I needed to modify the /etc/mailman/postfix-to-mailman.py file in order to allow commands sent to mailman-admin-cmd@lists.ourhost.org to get sent to the handler script:

                  ("/usr/sbin/sendmail", MailmanOwner))
         sys.exit(0)

+    # Local customization
+    elif local == 'mailman-admin-cmd':
+        mm_admin_pgm = '/home/adf/mailman/mailman_admin_cmd_handler.py'
+        os.execv(mm_admin_pgm, [])
+
     # Assume normal posting to a mailing list
     mlist, func = local, 'post'

(If you’re not familiar with diff output, that means I added the four lines with the pluses to the area between the sys.exit(0) line and the # Assume lines.)

At this point I really wanted to send a command to see what would happen, but I thought it would be better to run the handler’s unit tests first to see if I’d missed something basic.

To do this, I needed to search the downloaded files for server-specific things to edit that had been flagged with INSTALL comments (such as the location of our Mailman files in /usr/lib/mailman).

I also had to, as described above, create the list_members.py and clone_member.py symlink files in ~/mailman pointing to list_members and clone_member in /usr/lib/mailman/bin.

The unit tests assume a test list called test-list, so I had to use newlist test-list in /usr/lib/mailman/bin/ to make it (you can delete it later with rmlist if you wish).

The unit tests in test_mailman_admin_cmd_handler.py also assume there is a test user already on the test list, so I did this with:

echo "test@ourhost.org" | /usr/lib/mailman/bin/add_members -r - test-list

Once I’d done that, and made sure all the other settings in test_mailman_admin_cmd_handler.py were correct (especially in the setUp function), I ran test_mailman_admin_cmd_handler.py at the command line and verified that all the unit tests passed.

(If you do this and the tests don’t pass, you’ll need to look at the tests that failed in test_mailman_admin_cmd_handler.py and figure out what went wrong and why.)

Testing

Now, having verified that the code was set up correctly via the unit tests in test_mailman_admin_cmd_handler.py, I wanted to send a test command by email.

I should mention that I hit a snag at this point that you might not.

I had decided, as part of the overall migration strategy, to have our new Mailman list host be lists2.ourhost.org while continuing to run our lists on the soon-to-be-old Listserv lists.ourhost.org.

When Mailman was set up properly, including creating all lists and copying all users and their options, so that the two hosts were in sync, the plan for the cutover was to just change where lists.ourhost.org pointed to. This worked out pretty smoothly in the end, by the way, but I wanted to explain the lists2.ourhost.org thing now.

So I sent a command to mailman-admin-cmd@lists2.ourhost.org, putting help in the subject line and body.

That should have generated an error because the default setup requires any emailed admin commands to have a password in the subject line for security, and of course most list-related commands require their own passwords too (the list password, the site admin password, or the list owner’s password if the list owner is on the list and the email is sent from that account).

Unfortunately that simple “help” test produced the following error:

550-Verification failed for <mailman-admin-cmd@lists2.ourhost.org>
550-The mail server could not deliver mail to
mailman-admin-cmd@lists2.ourhost.org. The account or domain may not
exist, they may be blacklisted, or missing the proper dns entries.
550 Sender verify failed (in reply to RCPT TO command))

Also, the error only happened when I sent from myaddress@ourhost.org; when I sent the help test message from anotheracct@gmail.com (or any domain which was not ourhost.org) it worked fine and produced the expected “Required password not present in subject line.”

The admin for our new list provider said, “Because this test domain, lists2.ourhost.org, does not have Sender Verify set up on it, it’s failing the check and your ourhost.org mailserver is rejecting it.” I researched this and found that what he called “sender verify” refers to the Sender Policy Framework, aka SPF record, an anti-spam related DNS record.

Eventually our old host (which was still running the DNS for ourhost.org) solved the problem and when I asked them how, they wrote:

I’m not sure I remember all the steps along the way. I originally
created an A record for lists2.ourhost.org, and an MX record that
referred to the same IP address. When all was said and done I had
an A record, an MX record that referred to lists2.ourhost.org (instead
of to its IP address) and a TXT field with the following contents:

spf1 a mx a:208.85.173.148 -all

Here are the tinydns source records that I used:
+lists2.ourhost.org:208.85.173.148:3600
@lists2.ourhost.org::lists2.ourhost.org:20
‘lists2.ourhost.org:v=spf1 a mx a\072208.85.173.148 -all:3600

(I’ve altered the IP addresses for the privacy of the org I did this for, btw.)

I wish I could give more information about this Sender Verify problem and what really solved it, but at least if you encounter the same thing this info might help.

Anyway, having solved the Sender Verify problem, I re-sent the test message to mailman-admin-cmd@lists2.ourhost.org with help in the subject and body and got the expected “Required password not present in subject line” response.

Then I sent another test message, this time with help PASSWORD in the subject (no quotes, where PASSWORD was the value of SUBJECT_PASSWD in mailman_admin_cmd_handler.py) and help in the body, and got the expected response, “Valid commands:” followed by a list of valid commands for the admin command handler.

I didn’t do an exhaustive test of every command, since the unit tests in test_mailman_admin_cmd_handler.py sort of do that, but I did try one more command besides “help” just to make sure the basic mechanism was working.

I tried sending the “who” command to see who was subscribed to the test list I’d set up. Specifically, I sent a message with a subject of who test-list PASSWORD (where PASSWORD was the SUBJECT_PASSWD value in mailman_admin_cmd_handler.py) and a body of who test-list LISTPASSWORD (where LISTPASSWORD was the password I set for the list when creating it).

(Note: You can actually use anything in the subject line as long as it has the admin command handler password, but I like to keep it as descriptive as possible, and for a single command that usually means using the command itself minus the list password.)

After a few issues involving list ownership (I’d mistakenly sent the command from an account that wasn’t one of the list owners for the test list), I finally got a roster of subscribers for test-list so I felt a little more confident that the basic system was working correctly.

Next: Creating Mailman Lists Based On Listserv Lists or Up: Table of Contents

It occurred to me recently that by now, this limit must surely have been lifted, especially as Moore’s Law has improved hardware speeds and thus the ability to crack passwords of 8 characters easily via brute force.

(See Password Exhaustion: Predicting the End of Password Usefulness – PDF for more good info on this.)

So I asked about this over on Stack Overflow (SO), and got a lot of good responses. But in thinking about the answers more, and in particular trying to code something to find the maximum length on a given system, I had some more thoughts.

In particular, while the limit of 8 characters probably went away on most Linux systems in the late 1990s or early 2000s, there’s always a lingering question (in my mind anyway) of, “Yeah, but does this system I’m currently on support characters > 8 characters?”

Here’s what I coded up as a quick Perl script to try finding the maximum length of passwords for the system it’s run on:

#!/usr/bin/perl -w
$| = 1; # turn off output buffering so each . prints out
my($newpasswd) = chr(rand(95) + 32); # ASCII char 32-126
my($lasthash) = crypt($newpasswd, '$1$12345678'); # MD5=$1$
while (1) {
    print '.'; # for debugging/progress
    $newpasswd = join('', $newpasswd, chr(rand(95) + 32));
    $newhash = crypt($newpasswd, '$1$12345678');
    if ($newhash eq $lasthash) {
        print "Maximum password length: ";
        print length($newpasswd) - 1, "\n";
        last;
    }
}

(loosely based on some code posted in response to my question on SO)

After looking at this blog post on Password Length, I figured the program would stop at 79 or 127 or something, but to my surprise it didn’t, it just kept going on and on and on until eventually it was killed by my web host as a runaway program (which, I guess, it sort of was )

This puzzled me, and vexed me, so in my obsessiveness I wanted to find out why this would happen. I mean, that blog post said there was a limit, so why wasn’t I finding one?

It turns out that as long as you’re making the Unix crypt() function use a hashing method other than the classic/default DES-based algorithm, by using a salt prefix like $1$ for MD5, there really is no limit to the password because it’s like producing an MD5 hash for a file—you can get a different MD5 hash for files of any length.

I believe where the real limit—if there is one on a given system—comes in, is not in the Unix crypt() function, but by the password programs such as “passwd” that use it. Those programs themselves may have input limits, which is the only way I can explain the limits shown on that password length blog post.

As someone on SO commented though, you’re probably always safe using a password longer than 8 characters—if you used “mybigbigapple” as a password and the system only paid attention to the first 8 characters, it would just ignore the “apple” part. It’s just a question of whether it wall also accept “mybigbigorange”.

And if it’s easier for you to remember a longer password, then that’s a consideration too.

I guess for me it’s just a question of knowing whether the system I’m on is stronger (supporting passwords > 8 characters) or weaker with passwords, and for some reason I just want to know whether the system is ignoring everything I type past 8.

I did find it a little ironic, though, that if your password gets into the very long category, there’s actually a greater chance (at least with MD5 I believe) of a hash collision, meaning that another password (or one entered by a password cracker) might produce the same hash and thus let someone use that alternate password instead of your real password.

(If you have no idea what I’m talking about re: password hashes, see the Wikipedia article on Passwords, particularly the Background section.)

So where does that leave us? Well, I think it’s relatively easily to check whether crypt() supports passwords longer than 8 characters:

perl -e 'if (crypt(q^12345678^, q^$1$@@$^) eq
  crypt(q^123456789^, q^$1$@@$^)) { print "crypt() appears
  limited to 8 characters\n" } else { print "crypt() appears
  to support more than 8 characters\n"; }'

(typed on the command line, all in one line of course)

Exactly how much longer than 8 characters seems like it would depend on the particular program and not crypt() itself. The password length blog post seems to say that 72 or 79 might be a common limit, and since the average English word length appears to be 5.1 letters, that’s about 14 words.

So as a rule of thumb, we can probably figure that maybe a 10 word maximum for a password (really a “pass phrase”) is about right. That’s what I’ll go with anyway, at least after I use the command above to check that such a longer password/passphrase will actually be worth the trouble on a given system

At least until passwords go away completely of course!

(And while I’m on the topic of security, let me recommend the excellent book Schneier on Security. I picked it up in the bookstore recently with a stack of other books to breeze through, and long after putting the other books down I was still reading it! Very good thinking on a wide range of computer security topics.)

Mailman, unlike Listserv, doesn’t come with built-in list archive searching. (Which is funny since I’d always thought Listserv’s archive search was clunky and dated—but at least it had archive search! )

I think Mailman may not have archive searching because the developers wanted to keep their focus on the mailing list software itself rather than trying to write and maintain search software too. They probably figured it would be better to leave search to search software developers, and also give people the freedom to install whatever search package they want.

Nonetheless, it would have been a great relief if Mailman had just come with a default search package for list archives that could be uninstalled/overridden if necessary, rather than making everyone who wants to provide list archive searching (which must be a pretty common requirement) re-invent the search wheel by hunting down a package and figuring out how to integrate it into Mailman.

The Swish-E Package

After a good deal of searching (ha) I eventually found that many others who also wanted searchable list archives seemed to lean toward using Swish-E (swish-e.org, Wikipedia).

(Note: Since writing this guide, I’ve found that others have used htdig successfully for archive searching too; see the _README file at msapiro.net/mm/ and this documentation.)

Fortunately, I already had experience with Swish because our organization already used Swish for the search feature on its main website.

I wish I could provide step-by-step instructions for setting up Swish , but I forget the steps I used to install it from source on our development box (web host) back in 2003, and the server admins on our production list host installed it for me, probably using a binary package manager like RPM or Apt.

Even though I don’t have step-by-step instructions for installing Swish (though if you are root, downloading/unpacking the source, running “configure” and “make install” will probably do the trick), I wanted to include a section about it for a few reasons.

Aside from pointing you to the main swish-e.org site, in particular the Download and Documentation sections (the latter including a nice INSTALL page) , I also wanted to point you to the extremely helpful Integrating Mailman with a Swish-e Search Engine page.

I ended up only following some of that page’s advice, but it was invaluable for getting started on the Listserv to Mailman archive conversion as far as what issues to consider. So that page is definitely worth checking out just to see what’s involved. (Though note that I ended up writing code to automate some of the things on that page, which I’ll cover later.)

First though, just focus on installing Swish-e itself, which indexes and searches 1) the Mailman HTML archive pages (one page per message) and, optionally, 2) any message attachment files such as PDFs or Word docs by using extra Swish add-on/filter programs.

Using Swish-E for Searching Message Attachments

While the basic setup of Swish for HTML archive messages is fairly easy, setting it up to search non-HTML/text files such as Word, Excel, and PDF attachments is a little more tricky. I struggled with this when I set up Swish to search binary files on our website, so I wanted to give some info on it in case you want to support searching message attachments too.

(Mailman does allow attachments to be included in list archives, it just separates each from its associated message and puts a link to the attachment file at the bottom of the archived message.)

Note though: This entire section (the rest of this page) is optional. To be blunt, if you don’t have to support searching archive message attachments, make your life easier and don’t do it; you can always go back and add it later if you need to. We didn’t even do it for our list archives, I’m just including this info about using Swish to index binary files based on my experience doing that on our main website.

For a general article about indexing HTML pages and other file types with Swish-E, see How to Index Anything from the Linux Journal in 2003. At first glance that page is very good, but I haven’t examined it in detail and it’s possible that some details have changed since 2003.

Basically, for non-text/HTML files Swish relies on external helper programs to extract text from each file. For example, it uses the pdftotext program in the xpdf package for extracting text from PDF files, the catdoc program to get text from Word .doc files, etc. See the “Optional But Recommended Packages” section of the Swish-E INSTALL doc for more info on what’s available.

(I haven’t yet tackled the issue of extracting text from Word’s newer .docx file format for Swish; if I do, I’ll update this page, but if you’ve already done so please let me know; some possibilities are 1) docx2txt, 2) unoconv though that might unusable by Swish because it requires a running OpenOffice instance, or 3) maybe even a quick-and-dirty unzip/sed/grep combination.)

For more information about supporting searching of PDF, Word, etc. files, see “How do I index my PDF, Word, and compressed documents?” and the sections after it on the Swish-E FAQ page as well as the example filters in the “Document Filter Directives” section of the SWISH-CONFIG man page.

Note: I had to tweak the example given on that last page for PDF and DOC files, which were the only two binary file types I included in our website search. Specifically, the SWISH-CONFIG page gave the example of:

FileFilter .pdf       pdftotext   "%p -"

and that produced an error during indexing; every time the Swish indexer encountered a PDF file and tried to run pdftotext, it printed pdftotext’s usage info:

pdftotext version 3.02
Copyright 1996-2007 Glyph & Cog, LLC
Usage: pdftotext [options] <PDF-file> [<text-file>]
  -f <int>          : first page to convert
  -l <int>          : last page to convert
  ... etc.

To fix this, I had to tweak it to:

FileFilter .pdf pdftotext "'%p' -"

Actually I chose to spell out the full /path/to/pdftotext instead of just pdftotext there, but you get the idea—the main difference is in the quoting at the end, to put %p within its own single quotes.

I had to do the same thing with the catdoc example; the SWISH-CONFIG page suggested:

FileFilter .doc     /usr/local/bin/catdoc "-s8859-1 -d8859-1 %p"

… but that failed too so I enclosed the %p in single quotes there as well:

FileFilter .doc /path/to/our/catdoc "-s8859-1 -d8859-1 '%p'"

Character Set Problems While Indexing

Another issue was that at some point I started getting character set error messages when running the Swish indexer. I wish I’d documented the problem and my solution when it happened, but I’ve done my best to reconstruct the issue in case it’s useful to someone.

I believe the error may have been something about catdoc not being able to find the ascii.replchars and/or ascii.specchars character set files. I think I hunted around to find these files, but since catdoc seems to be a fairly old (and seemingly unmaintained) package, my search was fruitless.

Ultimately I think my solution was that I noticed I did have ascii.rpl and ascii.spc files, so I copied those to ascii.replchars and ascii.specchars and that fixed the problem.

I’m also still getting the following error (from pdftotext I believe) when running the Swish indexer on our web site:

Error: Unknown character collection 'Adobe-Korea1'

I searched online and couldn’t find anything for that specific character set, but when I searched for “swish unknown character collection” I found this post which recommended upgrading the xpdf package as a possible solution. I haven’t tried it yet because it’s only a few errors and I hate to upgrade things unless I absolutely have to, but I wanted to mention it here in case someone else gets similar errors using Swish to index PDFs.

Next: Installing an Administrative Command Handler or Up: Table of Contents

This page describes how I installed Mailman from source on our development server (a shared web hosting environment). However, it’s also possible to install through a binary package manager such as RPM or Apt, which is actually how our list host admins installed it onto our production server. So things might end up in different locations depending on how Mailman is installed.

Choosing a Version and Downloading

My first question for installing Mailman on our dev box was which version. This mattered because we had not yet chosen a list host but I assumed that whatever host we ultimately chose would probably have the latest stable and mature version.

Figuring out the latest stable and mature version of Mailman turned out to be a bit challenging. By examining the “timeline” graph at https://launchpad.net/mailman/+series (specifically, by hovering over the graph and hitting the little magnify-minus sign several times so I could eventually see all the way to the right), I found that at the time of checking (early Dec 2009), version 2.1.12 was the latest stable version since it was the most recent on the 2.1 “latest stable release” line.

I tried to download mailman-2.1.12.tgz from launchpad.net several times, but it never worked, so I finally downloaded it from http://ftp.gnu.org/gnu/mailman/?C=M;O=D

(Both of the above download sites were found on the main Mailman site at http://www.gnu.org/software/mailman/download.html)

Installing Mailman to a Custom Directory

I’ll skip past the usual technical details of downloading and unpacking files (I did it from the shell command line with: wget http://ftp.gnu.org/gnu/mailman/mailman-2.1.12.tgz and then: tar xvfz mailman-2.1.12.tgz).

In fact I wouldn’t even mention the installation process except that my first attempt failed because the web host I was using (which is not the one I am happily using now, Dreamhost) had pre-mapped the /mailman URL for me, and it resisted any attempts to override/fix it.

This was a problem because when I first installed Mailman on our dev box, created a test list (with the newlist command), and then tried to go to something like http://ourdomain.org/mailman/listinfo/test-list it told me there was no such list—even though I had just created it. I banged my head against the wall for a while until I figured out that my web host had pre-mapped /mailman and that there was no way around it.

(If you want to see if your web host has done this, I recommend going to http://www.yourhost.com/mailman/ before installing Mailman, and seeing if you get the same error page as when going to http://www.yourhost.com/badpagelink/ or if you get some unexpectedly-different page instead.)

So eventually I had to delete that first installation and install again into a different directory with the following options:

./configure --prefix=/home/ouruser/public_html/mm
--with-username=ouruser --with-groupname=ouruser
--with-mail-gid=ouruser --with-cgi-gid=ouruser
--with-mailhost=mail.ourhost.org --with-cgi-ext=.cgi

(I had to use the –prefix option, rather than accepting the default installation location, because my dev box was a shared hosting environment where I was just a regular user, rather than having root/admin permissions and being able to install into system directories like /var/mailman.)

After running configure and make install I then created a test list by running the mailman “bin” program newlist test-list (in ~/public_html/mm/bin). Then I tried going to http://ourdomain.org/mailman/listinfo/test-list and things seemed to work fine until I tried to log in, at which point the fact that Mailman insists on setting cookies with a Path of /mailman caused my logins to not “stick”.

I thought this would be easy to fix by editing the cookie set by Mailman from /mailman to /mm, but it turned out that most of the cookie-related add-ons for Firefox do not allow you to edit cookie values.

The Add N Edit Cookies Firefox add-on seemed promising, but when I edited the Path value of the cookies set after logging in to Mailman, to change from /mailman to /mm, it didn’t actually save the changes. But I figured out a workaround: copying the cookie values for the old login cookie (Name, Content, Host, Path, Expires) to a temp text file, deleting the cookie, then creating a new cookie with the same values except a different Path (and plus 20 years for the Expires value too).

All these machinations were just to deal with having to install Mailman to /mm instead of /mailman on our dev box because the web host had pre-mapped /mailman for us and we couldn’t override it. Hopefully you won’t have that problem at all; I highly recommend just installing using the defaults if you can!

For more information about installing Mailman see the official GNU Mailman Installation Manual. There’s also a helpful page covering more technical aspects of installing on various flavors of Linux (including the easy way of just installing from an RPM or YUM package) at the YoLinux Tutorial: GNU Mailman Email List Installation and Configuration.

Testing the Installation

If you’ve installed Mailman into the default /mailman and used newlist to create a “test-list”, you should be able to use the following URLs to check if your installation is correct:

• http://yourhost.com/mailman/listinfo/test-list for info about the test list
• http://yourhost.com/mailman/admin/test-list for admin pages about test list
• http://yourhost.com/mailman/listinfo for info on all lists on the server

If you had to install Mailman to a custom directory like /mm and/or use a custom CGI file extension like .cgi, then you’d want to use something like the following:

• http://yourhost.com/mm/cgi-bin/listinfo.cgi/test-list for info about the test list
• http://yourhost.com/mm/cgi-bin/admin.cgi/test-list for admin pages for test list
• http://yourhost.com/mm/cgi-bin/listinfo.cgi for info on all lists on the server

I haven’t covered setting up the automatic Mailman daemons and cron jobs because I couldn’t do that on our dev box’s shared hosting environment, and they were already set up on our production server. If you’re installing on your own box as root, the default options to configure and make install should take care of all that for you.

Next: Installing Swish for Archive Searching or Up: Table of Contents

This guide was written in short bursts over several months, sometimes at the same time as performing our conversion from Listserv (pricey commercial mailing list manager) to Mailman (free software mailing list manager) and sometimes after the fact. (If you stumbled across this page and don’t know the difference, see Mailman is not Listserv.)

So it is not by any means perfect or even necessarily free from mistakes, and it certainly needs editing and cleaning up. In particular you may find the narrative/explanatory style vexing, along with occasional changes in person/tense.

For these flaws I can only apologize in advance. I will try to clean them up and improve these pages over time, but even with its current shortcomings, it is still the guide (and code!) I wish had been available when I started my own conversion of 70+ Listserv lists to Mailman.

Note: This guide and the code I’m making available were both written against Mailman version 2.1.12. I’d hope that future versions of Mailman wouldn’t change the API functions I called, but obviously there is no guarantee that won’t happen, so if you use my code with a later version of Mailman it might not work. In particular, if you’re going to use the administrative command handler, I’d recommend running its test program first to see if anything breaks.

The Problem

The problem: A small non-profit organization which had more than 70 Listserv mailing lists, most with 10+ years of archives, needed to convert all of the lists and archives to GNU Mailman because its Listserv host was going to cease all operations within a year’s time (i.e., we got 9 months’ warning).

The old Listserv list host had had a special, grandfathered non-profit site license for Listserv, and LSoft was unwilling to allow that license to transfer with our organization to another host, so sticking with Listserv simply wasn’t an option.

(If you hunt around the LSoft website and finally dig up their pricing, you’ll see that for most small organizations even the cheapest Listserv license is prohibitively expensive; it certainly was for us, a non-profit org with 70 lists and a total annual IT budget of about $1000/year.)

Searching for other hosts that already had Listserv lists proved fruitless as well. I could hardly find any that had Listserv, and the few that did have it were way too pricey.

Also, it seems that in the modern Internet age, “mailing list” means “email marketing newsletter”, not “discussion list”. This distinction was important, and came up repeatedly as a problem because almost all of our our 70 lists were discussion lists.

I bring this up because early on someone in our organization pointed out that our website host already provided “mailing list” software called DadaMail… which is actually just meant for one-way marketing newsletter emails.

In fact, one of DadaMail’s main features is that it has email throttling. Why? Because most ISPs now limit customers’ outgoing email to a maximum number of messages per hour. Why? Probably because spammers crack people’s websites through common application vulnerabilities and then use the victims’ sites to send spam.

Why did the hourly email limit on our web host matter? Because if you have 1000 customers and want to send them all a single newsletter message, that’s fine—if your host limits you to 500 messages/hour then you set DadaMail or your mailing script to send one message every 2-3 seconds.

The problem arises when you have an actual discussion list (and we had 70 of them!). If you have a discussion list with 600 subscribers and someone sends a post to the list, the ISP tries to send that post to 600 subscribers and hits the hourly limit.

You could spread delivery of the post over two hours but what happens when someone who gets it replies? That’s another 600 messages going out, when some people haven’t even gotten the first one yet! Clearly, if there is actual discussion (i.e., replies), email thottling (as with DadaMail and other announcement-oriented programs) just won’t work.

So what I learned from my research into mailing list software and hosts was that there are really two problems: 1) The cost of the mailing list software itself, and 2) the difficulty finding a list host which won’t cap/throttle your email message output at a certain number per hour.

The Solution

The answer to the problem of the exorbitant cost of Listserv was fairly simple. I had noticed GNU Mailman years before, and it seemed to be an open source replacement for Listserv which was reasonably mature and somewhat Listserv-like. I was right about this, but it required a lot of customization which is why this documentation exists.

For the second problem, there just didn’t seem to be any affordable hosts offering mailing lists, especially for 70+ lists.

One possible solution would have been to get a dedicated server and install Mailman on it. That solution wouldn’t have worked for my non-profit because they just don’t have the resources to manage a dedicated server—they barely have enough manpower to manage their website and membership database.

So we basically had to find a host that would already have Mailman, or allow us to install it, would not have a per-hour message throttling/limit, would let us have command line (shell) access to run the Mailman “bin” commands and extend Mailman’s functionality (such as adding an administrative command handler), and would otherwise manage the server (backups, security updates, performance tweaking, etc.). And all for a price within our small budget, of course.

After a lot of searching we eventually found one, Binhost, which we’ve been pretty happy with so far. If you set up an account with them, please mention me—I won’t personally get anything for it, but my non-profit org will get a small account credit for the referral. Or if you have any questions about our experiences with them, feel free to contact me.

Development and Production Servers

Due to the long time it took to find a list host, I’d already started playing with Mailman on our main web server just to learn it, even knowing that we couldn’t actually have our lists on that server.

So in effect, our web host became my development server for playing with Mailman and determining what kinds of customizations we’d need for a smooth transition from Listserv when we did find a list host.

I only mention this because there were a few differences between the dev box setup (a shared web host) and the list host we ultimately settled on (really a special case managed server), and this guide will occasionally mention differences between these two servers so I wanted to describe their setup a bit here.

Files for Downloading

I’ve tried to link to relevant code files in the text, but you can also browse all the code at once. The utils/ subdirectory has scripts I found helpful, and used indirectly in the project, but which weren’t directly related to the conversion. For example, tgrep searches through files like grep, but ignores non-text (binary) files.

Next: Installing Mailman or Up: Table of Contents