Thursday, May 02, 2019

Introducing the OCF Mastodon Service!

We’re excited to announce that the OCF is now the proud host of its own Mastodon instance run by our volunteers! Many thanks to our hard-working volunteer staff for getting it set up and working on Kubernetes. Anyone with an OCF account can sign up for our instance with their OCF email (username@ocf.berkeley.edu). If you don’t have an OCF account and are eligible, make one for free!

What is Mastodon? 
Mastodon is a free and open source federated social media platform. What does that mean? It’s kind of like email. Anyone can host their own Mastodon instance (like email servers) and talk to people on other instances (like a GMail user to a Yahoo! Mail user). This allows each instance to have its own moderation (in our case, it’s the OCF volunteer staff) while still being able to talk with the wider community. If you’re looking for an android app to browse Mastodon with, Tusky and Fedilab are both free and open source applications!

Friday, January 04, 2019

Introducing OCF PGP Keyserver Service

The OCF is happy to announce the public release of our PGP keyserver service, available at pgp.ocf.berkeley.edu. We are proud to be a part of the community supporting reliable public key infrastructure!

At present, we are providing HKP and sks-recon on IPv4 and IPv6, as well as HTTP and HTTPS access. We are building out support for HKPS and keyserver pool membership. The following domains are supported, at the accompanying ports.
  • ppg.ocf.berkeley.edu (80, 443, 11370, 11371)
  • keyserver.ocf.berkeley.edu (80, 443, 11370, 11371)
  • pgp.ocf.io (443: HSTS)
  • keyserver.ocf.io (443: HSTS)
We will soon be providing service at pgp.berkeley.edu and keyserver.berkeley.edu as well.

If you would like to peer your SKS server instance with ours, please feel free to submit a pull request against our membership file!

Thursday, December 20, 2018

Downtime on 2018-12-20

The OCF's web, application, and public login servers were unavailable from 10:43 PM PST to 11:21 PM PST tonight (2018-12-20) due to an NFS outage. The issue has since been resolved and we expect no further issues.

Thanks for flying OCF and contact us if you have any questions or concerns!

Tuesday, November 06, 2018

Upgrading user-facing servers to Debian stretch

In the past year and a half we've upgraded our entire infrastructure to Debian stretch, with the notable exception of any user-facing machines.

Edit (2018-11-25): We've now upgraded all of the servers listed below! Please contact help@ocf.berkeley.edu if you have any questions about the upgrade or notice anything broken.

The time to upgrade them is now! We've prepared upgraded versions of each of these servers and will swap them out on Sunday, November 18th, 2018 the evening of Sunday, November 25th, 2018. This has been postponed from the 18th since some apphosting groups did not have their applications ready on the new server to make the migration possible.

The servers that will be upgraded are:
  • tsunami, the public login server
  • werewolves, the apphosting server (we've reached out to groups using this server and will be replacing with a new server named vampires)
  • death, the web server
Most users won't notice the update, except that most software will have newer versions. The one main exception is users who have dynamically-linked binaries somewhere in their home directories.

Because many libraries will be upgrading, most of these kinds of programs will fail to run after the upgrade. The best solution is to recompile the binaries (or find newer, pre-compiled versions).

One specific case is with environment managers like Python's virtualenv, Ruby's rbenv/rvm, and Node's nodeenv/nvm. These often put fully-compiled versions of the interpreter in your home directory, and in most cases, these will fail to work once the server is upgraded. After the upgrade, you'll need to rebuild these to get them to work again. Here are some major versions of programs that will be upgrading:

- Ruby 2.1.5 -> Ruby 2.3.3
- Python 3.4.2 -> Python 3.5.3
- Python 2.7.9 -> Python 2.7.13
- NodeJS 0.10.29 -> NodeJS 4.8.2
- PHP 5.6.36 -> PHP 7.0.30
- Perl 5.20.2 -> Perl 5.24.1

For upgrading any apps using our application hosting, you can find more detailed instructions on our website: https://www.ocf.berkeley.edu/docs/services/webapps/

During the server swap, you should expect a small amount of downtime (about 5-10 minutes) as the new servers are swapped into place of the old servers.

If you have any questions or need assistance feel free to reach out to help@ocf.berkeley.edu.

Thanks for flying OCF!

Friday, October 12, 2018

Account Creation and Password Resets Temporarily Down, October 12

Due to ongoing maintenance, account creation and password resets are down today.

At roughly 6:15PM, we there will be brief NFS downtime as we attempt to fix the issue.

Thanks for flying OCF!

Scheduled maintenance on night of 2018-10-12

The OCF is anticipating a short period of intermittent service unavailability in order to perform some additional maintenance on our hypervisors as a followup to last week's maintenance event. Specifically, we intend to migrate NFS to our new fileserver, reinstall our hypervisors onto new disks, and possibly migrate our mirrors server to new hardware. We are scheduling this event for for low-utilization hours at night to minimize any disruption to our users.

Thanks for flying OCF and send us an email if you have any questions!

Monday, October 08, 2018

IPv6 Connectivity Issues on October 8

Starting last night, the OCF has been experiencing some connectivity issues to our public SSH server over IPv6. If you are having trouble logging in to ssh.ocf.berkeley.edu, please try using IPv4 to connect. To do this, you can add -4 to your SSH connection command, like so

ssh -4 <OCF username>@ssh.ocf.berkeley.edu

If your SSH client does not support the -4 flag, you can also connect directly to our server's IPv4 address. To do this, just connect to `169.229.226.25` instead of `ssh.ocf.berkeley.edu`.

Some other services such as MySQL may also experience issues over IPv6. If neither IPv6 nor IPv4 is working for you, please let us know.

Thank you for being patient while we restore full connectivity.

UPDATE 2018-10-09 1:45AM: IPv6 connectivity should be restored to all our user-facing services.

Wednesday, October 03, 2018

Downtime on October 6

The OCF will be experiencing downtime, due to scheduled maintenance, on October 6th, from 9PM-12AM. Hosted websites will experience downtime as we briefly reboot the servers to apply critical security updates.  Once our servers are rebooted, users accessing our public login server and our apphosting server will not be able to write files as we are moving NFS to a new host. This read-only period will last no longer than 15 minutes and all operations should behave as normal by midnight at the latest.

Thanks for flying with the OCF!

8:53 Update: We've powered off the servers to do networking hardware and kernel updates.
11:03 Update: We've decided not to do the NFS migration tonight, but networking updates have been performed and services should be back up in the next hour.
11:42 Update: Most of our public servers should be back now, apart from our public mirrors and our own website (vhosts should be fine).
12:17 Update: Our website is back, but our software mirrors are not back yet.
12:51 Update: Everything should be back and working now except our HPC control server, which we are still debugging.
1:35 Update: This is the all clear, everything should be working now! Feel free to let us know by emailing help@ocf.berkeley.edu if you notice anything wrong.