Friday, June 14, 2019

Scheduled Maintenance on 2019-06-15

On this Saturday, June 15 2019, between 7-10pm the OCF expects to undergo a short period of downtime to reboot our physical servers in order to apply security updates. We do not expect the maintenance to be longer than an hour. Let us know if you have any questions or concerns. Thanks for flying OCF!

Update (10:30PM): although maintenance is progressing, it is taking slightly longer than expected. We hope to have primary services restored shortly.

Update 2 (12:20AM): All primary services were restored around 11:30pm.

Thursday, May 23, 2019

Downtime on May 21, 22:40 to 22:50

Due to a misconfiguration of OCF's network switches related to changing settings for STP (Spanning Tree Protocol), the OCF experienced brief downtime from 22:40 to 22:50 PST on 2019-05-21.

The issue has since been resolved and we expect no further issues.


We have completed an internal postmortem review of the incident to prevent and mitigate future issues.


Thanks for flying OCF and contact us if you have any questions or concerns!

Thursday, May 02, 2019

Introducing the OCF Mastodon Service!

We’re excited to announce that the OCF is now the proud host of its own Mastodon instance run by our volunteers! Many thanks to our hard-working volunteer staff for getting it set up and working on Kubernetes. Anyone with an OCF account can sign up for our instance with their OCF email (username@ocf.berkeley.edu). If you don’t have an OCF account and are eligible, make one for free!

What is Mastodon? 
Mastodon is a free and open source federated social media platform. What does that mean? It’s kind of like email. Anyone can host their own Mastodon instance (like email servers) and talk to people on other instances (like a GMail user to a Yahoo! Mail user). This allows each instance to have its own moderation (in our case, it’s the OCF volunteer staff) while still being able to talk with the wider community. If you’re looking for an android app to browse Mastodon with, Tusky and Fedilab are both free and open source applications!

Friday, January 04, 2019

Introducing OCF PGP Keyserver Service

The OCF is happy to announce the public release of our PGP keyserver service, available at pgp.ocf.berkeley.edu. We are proud to be a part of the community supporting reliable public key infrastructure!

At present, we are providing HKP and sks-recon on IPv4 and IPv6, as well as HTTP and HTTPS access. We are building out support for HKPS and keyserver pool membership. The following domains are supported, at the accompanying ports.
  • ppg.ocf.berkeley.edu (80, 443, 11370, 11371)
  • keyserver.ocf.berkeley.edu (80, 443, 11370, 11371)
  • pgp.ocf.io (443: HSTS)
  • keyserver.ocf.io (443: HSTS)
We will soon be providing service at pgp.berkeley.edu and keyserver.berkeley.edu as well.

If you would like to peer your SKS server instance with ours, please feel free to submit a pull request against our membership file!

Thursday, December 20, 2018

Downtime on 2018-12-20

The OCF's web, application, and public login servers were unavailable from 10:43 PM PST to 11:21 PM PST tonight (2018-12-20) due to an NFS outage. The issue has since been resolved and we expect no further issues.

Thanks for flying OCF and contact us if you have any questions or concerns!

Tuesday, November 06, 2018

Upgrading user-facing servers to Debian stretch

In the past year and a half we've upgraded our entire infrastructure to Debian stretch, with the notable exception of any user-facing machines.

Edit (2018-11-25): We've now upgraded all of the servers listed below! Please contact help@ocf.berkeley.edu if you have any questions about the upgrade or notice anything broken.

The time to upgrade them is now! We've prepared upgraded versions of each of these servers and will swap them out on Sunday, November 18th, 2018 the evening of Sunday, November 25th, 2018. This has been postponed from the 18th since some apphosting groups did not have their applications ready on the new server to make the migration possible.

The servers that will be upgraded are:
  • tsunami, the public login server
  • werewolves, the apphosting server (we've reached out to groups using this server and will be replacing with a new server named vampires)
  • death, the web server
Most users won't notice the update, except that most software will have newer versions. The one main exception is users who have dynamically-linked binaries somewhere in their home directories.

Because many libraries will be upgrading, most of these kinds of programs will fail to run after the upgrade. The best solution is to recompile the binaries (or find newer, pre-compiled versions).

One specific case is with environment managers like Python's virtualenv, Ruby's rbenv/rvm, and Node's nodeenv/nvm. These often put fully-compiled versions of the interpreter in your home directory, and in most cases, these will fail to work once the server is upgraded. After the upgrade, you'll need to rebuild these to get them to work again. Here are some major versions of programs that will be upgrading:

- Ruby 2.1.5 -> Ruby 2.3.3
- Python 3.4.2 -> Python 3.5.3
- Python 2.7.9 -> Python 2.7.13
- NodeJS 0.10.29 -> NodeJS 4.8.2
- PHP 5.6.36 -> PHP 7.0.30
- Perl 5.20.2 -> Perl 5.24.1

For upgrading any apps using our application hosting, you can find more detailed instructions on our website: https://www.ocf.berkeley.edu/docs/services/webapps/

During the server swap, you should expect a small amount of downtime (about 5-10 minutes) as the new servers are swapped into place of the old servers.

If you have any questions or need assistance feel free to reach out to help@ocf.berkeley.edu.

Thanks for flying OCF!

Friday, October 12, 2018

Account Creation and Password Resets Temporarily Down, October 12

Due to ongoing maintenance, account creation and password resets are down today.

At roughly 6:15PM, we there will be brief NFS downtime as we attempt to fix the issue.

Thanks for flying OCF!

Scheduled maintenance on night of 2018-10-12

The OCF is anticipating a short period of intermittent service unavailability in order to perform some additional maintenance on our hypervisors as a followup to last week's maintenance event. Specifically, we intend to migrate NFS to our new fileserver, reinstall our hypervisors onto new disks, and possibly migrate our mirrors server to new hardware. We are scheduling this event for for low-utilization hours at night to minimize any disruption to our users.

Thanks for flying OCF and send us an email if you have any questions!