Friday, June 01, 2018

Small amount of downtime today (6/1)

We had a short amount of downtime today with a few of our services between 10:10 PM and 11:19 PM today (6/1/2018) due to some networking issues on one of our hypervisors:

  • OCF website: Down from 10:24 PM to 10:47 PM, intermittently up and down until 11:19 PM.
  • phpMyAdmin: Down from 10:10 PM to 10:48 PM, intermittently up and down until 11:19 PM.
  • OCF mirrors (rsync/ftp/http/https): Down from 10:28 PM to 11:09 PM.
We have fully recovered all affected services and put a couple protections in place to avoid this kind of issue again.

Monday, March 19, 2018

Short downtime tonight (3/19) for security updates

There will be a brief period of downtime tonight (3/19/2018), around 8:15 PM. It will likely last about 30-45 minutes as we restart our servers to apply kernel updates to them.

Edit (8:20 PM): We have started. See you on the flip side!
Edit (8:40 PM): We have rebooted a majority of our public services, they should be coming up soon. Public mirrors are still to come.
Edit (8:50 PM): Our MySQL database did not start correctly, so we have started that now. We are also in the process of rebooting our public software mirrors host.
Edit (9:00 PM): Restarting our servers is fully complete, thanks for your patience!

Friday, February 09, 2018

NFS downtime on 2018-02-09

NFS (containing home directories, cron jobs, and public_html web directories) was down for a short period of time on 2018-02-09 from 4:41 PM to around 5:06 PM. In this time, files stored in user home directories and all hosted websites (including were unavailable. Logins were also affected, both in the OCF lab and from outside the lab to our SSH server. Thank you for your patience while we brought everything back online.

Please contact us if you have any questions or concerns!

Wednesday, January 17, 2018

Downtime on 1/17/18

We're currently recovering from a multi-drive RAID failure on one of our servers. Currently logins, MySQL, printing, account creation, and our internal DNS will all not work. We'll post status updates as we bring things back up.

Downtime was on 1/17 from 13:27 to around 18:00 in total.

Note: MySQL and accounts created are being rolled back to a backup as of 2 AM this morning (1/17), so any accounts created since then or database changes will be rolled back.

Update 1/17 15:25: MySQL, logins, and DNS are all in the process of being restored. Database restoration will likely take around another hour, perhaps more, but logins and DNS are both close to being fixed.

Update 1/17 16:22: Logins (LDAP/Kerberos) and DNS have been restored, MySQL is getting closer to being restored, probably will be about half an hour on restoring that. Printing is still a work in progress, but getting closer. Account creation is still down.

Update 1/17 18:09: Everything except account creation should be back up again. We are working on restoring all accounts created today and then we will re-enable account creation. MySQL databases have been restored from our most recent backup (2 AM this morning). We have noticed some lingering issues with DNS that we will keep checking and update here if it stays a problem. Let us know if you encounter any other problems with our services!

Update 1/17 19:58: Account creation has been re-enabled, although any new accounts made today still need to be re-created, but new accounts made from this point on should work fine. We also fixed a DNSSEC issue that was preventing valid DNS responses in some cases. Contact us if you notice any other problems!

Update 1/18 03:21: All the missing accounts have been re-created again, but they will need to reset their passwords, since we purposefully do not log those for security and they were lost in our Kerberos rollback. These accounts can reset their passwords through our website.

Saturday, December 23, 2017

Downtime scheduled for afternoon of 12/24

Tomorrow afternoon between 3-5pm the OCF expects to undergo a short period of downtime to reboot our physical servers in order to apply security updates. We do not expect our services to be unavailable for longer than 30 minutes. Let us know if you have any questions or concerns. Thank you for flying OCF and have a happy holiday season!

Sunday, October 15, 2017

New: bulk-add vhost mail addresses

Starting 10/15, you can bulk-add forwarding addresses for mail virtual hosting by uploading a CSV file.

Wednesday, August 02, 2017

Expected power outage 8/2

Due to a wildfire, the UC Berkeley campus is expecting a power outage around 2:40 PM today. All OCF services have been powered down in preparation.

We apologize for the inconvenience.

Update (9:02 PM): services have been restored.