Wednesday, December 10, 2014

Scheduled downtime: Dec 18, Dec 27-28, Jan 3-4

We found out yesterday that, due to construction, Hearst Gym will have no power on Dec 27-28 and Jan 3-4. Unfortunately, all OCF services will be affected by the power outage.

We're looking into ways to reduce the impact, but currently you should expect the following impacts:

  • Web hosting: All web hosting, including student group hosting, will be unavailable. We're working on providing a descriptive error page, rather than simply having requests time out.
  • Email hosting: Email sent to students or to groups with virtually-hosted mail will be delayed until the outage ends. Senders might receive a notice that delivery has been delayed, but you will still receive the messages shortly after the power returns.
These services will be completely unavailable:
  • Database (MySQL) access
  • Shell (SSH/SFTP to tsunami)
  • F/OSS Mirrors (mirrors.ocf.berkeley.edu)
We're working now to try to minimize the impact of the outage, and will post updates here. Please email us if you have any questions.

Update 12/10: We are scheduling downtime during the evening of Thursday, December 18th to test our ability to start all servers and services remotely. Total downtime should be less than 30 minutes.

Update 12/18: Maintenance for tonight is completed. Total downtime was about 45 minutes (instead of the expected 30) due to a problem with a switch after we restored power. The good news is that we caught it now rather than in a week when nobody will be around to fix it. Everything else worked as expected.

Monday, December 01, 2014

WordPress XSS vulnerability; please update!

A vulnerability was recently discovered in WordPress which affects a large number of OCF web hosting users. The vulnerability can potentially allow a malicious person to hijack your session and compromise your website.

All users should update immediately to the latest version of WordPress. Versions 3.9.3, 3.8.5, 3.7.5, 4.0.0, 4.0.1 are unaffected by this vulnerability, but we highly advise to always use the latest version.

Updating WordPress is extremely easy; it's just a single click after logging in to the admin panel.

Recent versions of WordPress come with automatic updates enabled for minor releases, which can help to protect you from future vulnerabilities. We strongly recommend not disabling this feature!

If we've contacted you and you need help updating your site, please don't hesitate to get in touch so that we can help!

Monday, November 03, 2014

Kernel updates Nov. 08

All OCF servers will be restarted Saturday night (11/08) in order to apply security updates. Downtime should be no more than 15 minutes.

Saturday, October 25, 2014

Announcing dedicated hosting for web applications!

We're excited to announce a new OCF service for student groups: dedicated hosting for web applications like Rails, Django, Flask, and Node.js!

Previously, hosting for modern web apps was only available via FastCGI, which was difficult to set up and manage. The new service makes it possible to host any app that can bind to a socket, enabling you to run basically any type of application.

OCF hosting for web apps is a pretty cool choice for student groups compared to hosting off site; it's easy to get a berkeley.edu domain name and to get support from friendly volunteer staffers during staff hours, and now it's easy to set up your app on our powerful servers, all hosted on-campus.

We're opening up the new application hosting on a trial basis, and hoping to work closely with a small number of student group early adopters to work out kinks and improve the service. If you're interested, take a look at the documentation and get in touch!

Friday, October 24, 2014

Moving www.ocf.berkeley.edu to HTTPS-only

On November 22nd, all websites hosted under the www.ocf.berkeley.edu domain will begin using HTTPS instead of plain HTTP. We will redirect all requests to the corresponding HTTPS site.

Virtually-hosted websites will continue to use plain HTTP, so won't be affected. There also shouldn't be any impact on sites which consist of static content.

We recommend to change all of your absolute links to point to the HTTPS version. You can start doing this immediately; it's already supported! Since we will set up a 301 redirect, no links will be broken.

There are a few things which some sites are currently doing which will cause problems under HTTPS:

  • Including resources from non-secure pages. Some browsers will refuse to load this content, while others will load it but display a degraded security icon.
  • POSTing to non-secure URLs. If your website has forms which post to non-secure URLs, you need to update them to use https instead of http. Although we will set up a redirect, browsers will not necessarily follow this redirect when submitting forms. Additionally, browsers may give a warning if submitting a form which sends data to an insecure URL.
If your website does either of the two thing above, you must fix it before November 22nd, or parts of your website may break. If you use WordPress or another CMS, it is usually sufficient to update the URL in the admin panel.

We appreciate that this may be disruptive to many websites, and have not made the decision lightly. There are many reasons to use HTTPS by default, even for websites which don't contain sensitive information or collect passwords. This is an inevitable transition for the OCF at some point, and while it will be painful, it is easier to make it now than in another four years, when even more sites will be affected.

We estimate that there are about 90 OCF users (~0.3% of OCF users) with active websites which will be affected by the change. We'll try to contact them individually about the change. If you need assistance making changes to your website, don't hesitate to come in during staff hours or send us an email.

Thursday, October 09, 2014

mirrors.ocf.berkeley.edu - outdated Debian package repos

Today we noticed that our Debian package mirrors had not receive updates since October 2nd, although syncs had been completing successfully. This was due to an issue with mirrors.kernel.org, our upstream mirror, which had not synced in the past week.

Since our syncs were completing normally, we weren't alerted to the problem until today when apt warned us that our mirror was out-of-date. We didn't really consider the possibility that mirrors.kernel.org, which is a top-tier Debian mirror (and one of several which ftp.us.debian.org can resolve to) would receive no updates for an extended period of time. We'll add additional health checks to make sure that not only are syncs completing, but that we're receiving updates.

We contacted the mirrors.kernel.org admins to report the problem, who were very responsive and fixed the problem within 30 minutes. Our mirror is now up-to-date.

Sunday, August 10, 2014

WordPress xmlrpc exploit

WordPress recently announced a bug in their xmlrpc implementation which can result in denial of service attacks by using large amounts of CPU. Many OCF-hosted sites are running affected versions of WordPress.

In response to activity which took out our webserver for about 15 minutes earlier tonight, we are blocking access to xmlrpc.php files to protect the shared OCF webserver. If you would like to request xmlrpc.php files be unblocked from your site, please contact us.

Now would be a good opportunity to make sure all the software on your website is up-to-date!

Online account tools maintenance

OCF's online account tools will be unavailable for a few days while we perform maintenance and upgrades. During this time, requesting an account and resetting passwords via CalNet will not be possible.

There will be no impact on LDAP or other services, and password changes (assuming you know the old password) can be done via the "passwd" command.

Update 2014-08-18: We've completed the maintenance on our online account tools. Note that the URLs have changed; see the wiki or main website for updated links.