Basic facts

At 6PM Wed Feb 1, 2006, I am going to turn off our existing mail server. You will not be able to read your mail, nor send out any mail through Nevis. By 7PM, I hope a new mail server will be up, and you'll able to read and write your e-mail again. No e-mail should be lost during this change.

More details

Our existing mail server is about four years old. Although that's not old compared to some of our other servers (you don't want to know how old our web server is!), it's beginning to slow down at times; there's more about this issue in the FAQ.

The new mail server has been configured, tested, and appears to work. If you're feeling experimental, you can even try it out: in your mail reader, replace mail.nevis.columbia.edu (or franklin or whatever) with dhcp194.nevis.columbia.edu. Note that our firewall blocks off all the DHCP addresses, so you can only run this test from a machine at Nevis. (Don't forget to change this name back when you're done with your tests!)

The basic transition plan is this:

• At 6PM Wed 01-Feb-06, I will shut off all the mail services (SMTP, POP3, and IMAP) on our existing mail server.

• The new server is already synchronizing files with the old server in a nightly job. I will run this file synchronization one last time. This is the part that will take most of the hour between 6 and 7 PM.

• The old server will be shut down; the new one put in its place. The IP address and network parameters of the new box will be changed to those of franklin. At that point, the new server will be running and processing our mail.

If this schedule is inconvenient for you, or there are any issues that have not been addressed, please contact me. However, please read the next section first, since your question may be answered there.

Questions and answers

• Will we lose any mail?

  No. When our mail server at Nevis goes down, mail is re-directed to our backup mail server in the Nevis Annex. The mail will be stored there until our mail server comes back on-line.

• Why are we upgrading the mail server now?

  As I mentioned above, we have older server boxes at Nevis, so why upgrade this particular one? The answer is that the existing mail server appears to be slowing down.

  In theory, this should not happen. We receive roughly 3000 e-mail messages per day at Nevis, at least 10% of which are spam. (We probably receive much more spam than that, but I can only track those messages flagged by users who use SpamAssassin.) In and of itself, that's not a lot.

  But we've all experienced long mail server delays at one time or another. The problem appears to be three-fold:

  • We scan each message that comes in for viruses. The database of virus signatures is updated regularly. While the number of messages we get each day is roughly constant, the number of viruses for which the server must check is monotonically increasing.

  • The modern graphical mail readers (like Thunderbird) tend to make multiple IMAP connections at the same time, typically one for each folder you click on.

  • Again, the number of legitimate mail connections we receive each day is roughly constant. However, the number of invalid mail connections we get is increasing. This includes connection attempts from non-existent e-mail addresses, non-existent domains, forged sender information, hacking attempts, and so on. Our mail server has no problem rejecting these connections, but each individual connection absorbs some processing time.

    (I don't believe this is a bandwidth issue; our traffic monitor does not show any significant load no matter how many connection errors I see.)

  My best guess: As more users shift to newer mail readers, and with everyone trying to read their mail between 9AM-10AM every day, and with the mail server receiving (and virus-scanning) more messages at that time, the current mail server can't handle the load.

• How do the old and new mail servers compare?

  	"old" mail server	"new" mail server
  Processors	550 MhZ Pentium III	2.8 GHz Intel Penium D (dual-core)
  RAM	384 MB	2 GB
  disk space	56 GB	200 GB
  operating system	Redhat 9	Fedora Core 4

• So once the new mail server is in place, all the speed problems will go away?

  I hope so!

  We perform quite a few anti-virus and anti-spam checks on each message, and some of those checks involve connecting to servers outside of Nevis. I've worked to optimize and buffer these checks as best I know how; if those checks are what's slowing down the server, the hardware upgrades may have little effect. If that happens, we'll have to make some tough choices between the signal-to-noise ratio in the mail we receive versus mail server efficiency.

• Will everything work exactly the same way, or will I have to change my mail reader configuration?

  I've tried to make everything as transparent as possible. However, the new versions of the mail software are a little pickier about security issues than our current server. I've updated our standard mail reader configuration guide with any changes you might expect. Also, the procedure to set up Pine without passwords has changed, but if you've already set that up, there should be no difference.