Thursday, August 16, 2012

Unexpected downtime

NFS home directories were unavailable between 6:30pm yesterday and 9:30am this morning. Most OCF services were affected. The outage appears to be related to networking changes for the move to Hearst Gym.

Edit 4:00pm: We discovered that the mail server ran out of space during the outage and was mishandling mail between 6:25am and now. Domino effect.

Edit Fri 4:00am: Unfortunately parts of the mail server became corrupted. Mail was being temporarily rejected until we resolved numerous software issues by 3:00am. Delayed mail should now be coming in.