Email virtual hosting downtime

Email virtual hosting (e.g., email forwarding for person@group.berkeley.edu) was unavailable today, April 27, between 2am and 11:30pm. Email senders during this time would have received a bounce message that said "Relay access denied". We apologize for the inconvenience.

Campus-wide outage

The entire campus appears to have been inaccessible from outside between 12:19am and 2:11am, so OCF was inaccessible from off-campus as well. We do not yet know the cause of the widespread outage.

Printing daily quota

Starting today, only 30 pages may be printed per day at the OCF lab by each member. Each member is still also limited to printing 250 pages per semester.

Why are we doing this?

• We print 500,000 pages per semester (and increasing), but as a student group we don't have the financial resources or volunteer manpower to sustain this.
• We print 2.5 times our printers' maximum recommended monthly print volumes on average (often more). Our printers require constant cleaning due to frequent toner leaks, which reduce reliability and print quality.
• We have had quota complaints (members forgetting to log off, botched jobs), which would be minimized if each member's semesterly quota could only decreased by a certain amount each day.
• The lengths of lines waiting for printing continues to increase.

See the February 12 OCF Board of Directors meeting minutes.

File storage system (NFS) restructure

We will be restructuring our file storage system sometime over the next semester. Please be aware that the u1-u8 file structure will be gone. If you have any files that depend on that file structure, they will not work anymore. They can be fixed by changing them to use your home directory (e.g. /home/f/fo/foo instead of /mnt/u3/home/f/fo/foo).

Update 1/25: We will be taking the OCF down at around 10pm (we decided to delay until 1:15am).

Update 1/26: Due to routine maintenance before the restructuring, NFS will be down for at least all of Saturday. Total downtime was between 1:15am and 3:00pm.

Update 1/28: NFS file storage maintenance partially completed. We will be taking the OCF down again sometime in the near-ish future to finish the maintenance.

Update 2/11: We will be taking the OCF down on Saturday, February 16 to finish maintenance of the NFS file storage system.

Update 2/13: Maintenance will start on Saturday after 6pm when the lab is closed. Web hosting, email, and SSH/SFTP will be intermittently inaccessible during maintenance.

Update 2/17: The file storage maintenance has been completed successfully. Downtime occurred during the 11:30pm to 12:00am (30 minutes) maintenance window. Note that hard-coded references to /mnt will no longer work.

DNS downtime

DNS and IRC was unavailable between 12:56am and 03:22am due to a networking error. System performance was degraded, unexpected errors may have occurred during that time, and logins may have timed out. We apologize for the inconvenience.

Extended lab hours for RRR and finals

Lab will be open from 9am to 9pm from Tuesday 12/4 to Thursday 12/13 (including weekends) and 9am to 12pm on Friday, 12/14, the last day of finals.

Lab will be closed after finals as we move our servers from Eshleman over winter break. Expect intermittent downtime for web hosting, SSH, email, etc. between December 14 and December 18 (and possibly later).

Possible hard drive failure

At 4am our primary server (hal, which is the hypervisor for most of our servers) reported two uncorrected read errors on one of its hard drives  (in a RAID 10 setup). The presence of uncorrected errors can indicate hard drive failure because it means that the hard drive has run out of spare sectors.

We have a hard drive replacement on hand, but as our servers are in Eshleman (which is under a lockdown) there may be a delay getting physical access.

Update Dec 3: We have taken backups of important data stored on hal, but do not have physical access to our servers at this time.

Eshleman lockdown

UCPD has locked down Eshleman Hall, where our servers are located, until at least Sunday, in response to the Eshleman occupation.

Downtime is possible as we have abruptly lost physical access to our server and networking equipment.

Update 11/28 12:30pm: SSH server was inaccessible between 12:05pm and 12:25pm.