Sunday, April 17, 2011

Mail server rebuilt

As the previous post notes, our primary mail server (war.OCF aka mail.OCF) was down on Friday and parts of Thursday and Saturday.

At around 3:30a Thursday the server began to slow down on processing mail, and by 3:15p Friday the server had crashed. We could not turn the server back on, suspecting file corruption on the SPARC hypervisor the Solaris LDOM runs on.

We began rebuilding the mail server with Debian 6 on physical x86 hardware Friday night, and after 9am Saturday services were coming up. IMAP, POP, SMTP, and SMTPAUTH should all be up as of 12:30p Saturday.

We're using a new SSL certificate on the new mail server.

Since we're still running parts of mail on a mix of other machines running Solaris and Debian 5, this is a temporary solution. Maintenance work should be expected soon. We apologize for the inconvenience.

As always, you can email any suggestions or comments to staff@OCF.Berkeley.EDU.

UPDATE Sat 5:15p: Our incoming mail server (sandstorm.OCF) has a backlog of emails to process. There may be delays in receiving emails.

UPDATE Sat 11:45p: special local (slocal) mail delivery mail processing is no longer supported, please update your .forward files accordingly.

UPDATE Tues 11:55p: We've had difficulties with the incoming mail server, so mail may have been temporarily (or possibly in some rare cases, permanently rejected) when sent to an OCF address. We've replaced the Solaris machine with the newly-rebuilt mail server. We apologize for the inconvenience, and hope that there will be no bugs.