tag:blogger.com,1999:blog-349237662024-03-13T06:25:46.074-07:00Open Computing Facility StatusUpdates for Open Computing Facility servicesKevin Penghttp://www.blogger.com/profile/11883713711554795386noreply@blogger.comBlogger347125tag:blogger.com,1999:blog-34923766.post-40820801084340320642024-02-08T23:28:00.000-08:002024-02-10T21:06:15.535-08:00OCF Mirrors Down, Feb 8 ~11AM - Feb 10 9PM<p><b>Update (2/10 9PM PST) - OCF Mirrors Service is up again, and we are currently syncing from upstreams. </b>OCF Internal APT repos remain unavailable at this moment.<br /></p><p> The OCF Mirrors Service went down due to a campus datacenter power outage. When power was resumed, the server encountered a disk error on the boot disk.</p><p><br /></p><p>ETA of recovery currently unknown as a full OS rebuild seems to be needed.</p><p><br /></p><p>~rjz<br /></p>Jonathanhttp://www.blogger.com/profile/10004598210761796546noreply@blogger.comtag:blogger.com,1999:blog-34923766.post-1593071178940507572023-01-18T14:31:00.002-08:002023-01-18T14:31:50.487-08:00Mirrors maintenance, 1/18 23:00 PST<p>We'll be running some maintenance tasks on the mirrors server (https://mirrors.ocf.berkeley.edu) at 23:00 PST today, 1/18.</p><p>The projected maintenance window is 20 minutes.<br /></p>Jonathanhttp://www.blogger.com/profile/10004598210761796546noreply@blogger.comtag:blogger.com,1999:blog-34923766.post-13397202620955442392022-08-26T23:49:00.005-07:002022-08-26T23:49:45.762-07:00Multiple mirrors out of date<p> We noticed that multiple OCF mirror archives were out of date due to exhausted boot disk space on the mirrors server.</p><p>Affected archives include (this is not an exhaustive list):</p><p>Debian</p><p>Debian-security</p><p>Puppetlabs</p><p>Ubuntu</p><p>Ubuntu-ports</p><p>Kali</p><p><br /></p><p>The last update of the affected archives was on Aug 13. Sync is running but might take a while.</p><p>We sincerely apologize for this. We will take steps to ensure that system monitoring alerts are handled in a timely manner.</p><p><br /></p><p><br /></p>Jonathanhttp://www.blogger.com/profile/10004598210761796546noreply@blogger.comtag:blogger.com,1999:blog-34923766.post-64745589285164358952022-05-13T02:58:00.006-07:002022-05-13T03:22:04.338-07:00Fedora out of sync<p> I noticed that our Fedora mirror was out of sync due to a slow upstream mirror. Working on it.</p><p><br /></p><p>Update: Switched to another upstream mirror. Sync speeds are higher now.</p>Jonathanhttp://www.blogger.com/profile/10004598210761796546noreply@blogger.comtag:blogger.com,1999:blog-34923766.post-90244216032808704562022-04-02T22:09:00.005-07:002022-05-13T03:23:20.137-07:00[RESOLVED] Raspbian out of sync<p> We noticed that our Raspbian mirror is out of sync, likely due to the IP address change. </p><p>10:16 PM - We have temporarily switched our upstream for the Raspbian mirror, and it's currently syncing. Emailed Raspbian. - rjz</p><p>5/13 - Resolved.</p>Jonathanhttp://www.blogger.com/profile/10004598210761796546noreply@blogger.comtag:blogger.com,1999:blog-34923766.post-38059540121126926972022-03-15T12:01:00.001-07:002022-03-15T12:01:36.620-07:00Emergency mirrors server maintenance today<p> The OCF mirrors server will be down for about an hour staring 1300 PDT today due to a required move within the rack and IPv6 provisioning. We apologize for the inconvenience.</p><p><br /></p>Jonathanhttp://www.blogger.com/profile/10004598210761796546noreply@blogger.comtag:blogger.com,1999:blog-34923766.post-6543033293394466692022-03-13T13:43:00.001-07:002022-03-13T13:43:04.271-07:00Emergency maintenance: hypervisors reboot today<p>We will be rebooting the hypervisors this afternoon at about 2 pm to apply the updated kernel that fixes several high-severity vulnerabilities. The downtime will be around 15-30 minutes, and all websites hosted by the OCF as well as the login servers, Matrix server, Slack bridge, mail server, and the IRC server will be unavailable during the reboot.</p>Jonathanhttp://www.blogger.com/profile/10004598210761796546noreply@blogger.comtag:blogger.com,1999:blog-34923766.post-57795664251728739922022-03-11T10:28:00.003-08:002022-03-12T17:40:21.212-08:00[Complete, see notes] Mirrors downtime 3/11/2022 - migration to campus DC<p> migration to campus DC underway, expect downtime for ~2 hours</p><p><br /></p><p>Update 3/12/2022: Migration completed by 14:00 yesterday. Currently our mirrors are IPv4 only, IPv6 ETA 2 weeks. Some Kali packages may not download correctly due to campus firewall settings, so if you experience any broken Kali downloads, please switch to the https version of our mirror. We are in contact with the university to get the issue sorted out as soon as possible.</p>Jonathanhttp://www.blogger.com/profile/10004598210761796546noreply@blogger.comtag:blogger.com,1999:blog-34923766.post-72979751172299109442022-02-14T12:04:00.003-08:002022-02-14T23:15:23.880-08:00Mirrors maintenance tonight at 8<p>We will be installing additional RAM on the mirrors server tonight at 8pm. Expect 15 mins of downtime.</p><p><br /></p><p><br /></p><p>Update: ETA 2130. Apologies, but the process turned out to be more involved than expected.</p><p>Update 2: The mirrors are up as of 21:30.</p>Jonathanhttp://www.blogger.com/profile/10004598210761796546noreply@blogger.comtag:blogger.com,1999:blog-34923766.post-63730178978808554972022-02-08T11:39:00.004-08:002022-02-08T11:41:15.726-08:00Scheduled downtime on Saturday, 2/12<p>We will be upgrading the operating system on our web hosting (death), login (tsunami), and app hosting (vampires) servers this Saturday, 2/12/2022, beginning at 6:00 pm Pacific Time. All OCF-hosted websites and apps will be unavailable during the upgrade. We expect the downtime to last for about 4 hours.</p><p>We apologize for the inconvenience.</p>Jonathanhttp://www.blogger.com/profile/10004598210761796546noreply@blogger.comtag:blogger.com,1999:blog-34923766.post-63735050889302646812022-02-04T19:47:00.007-08:002022-02-06T22:28:00.497-08:00Mirrors upgrade underway, service interruptions expected<p>We are currently performing upgrades on the OCF mirrors server to accommodate more projects and improve the throughput. During this time, you might experience sporadic downtime and reduced performance. Please retry your downloads in a few seconds if you experience failures when downloading from the OCF mirrors.</p><p>We apologize for any inconvenience. We expect the upgrades to be completed by mid-February.</p><p>Later this semester, we will migrate the mirrors server to the campus datacenter to ease up the network load at the OCF lab*. The project is currently in the planning stages, so no definite dates are available. However, we will make a separate post on the status blog before we proceed with the migration. It is likely that the server's IP addresses will fall outside of the OCF subnet as a result of the migration.</p><p><br /></p><p>Planned additions as of 2/6 2230PST (please note, this list is not final and may change without notice):</p><p>Fedora - <i>initial sync underway</i></p><p>OpenSUSE <i>- </i><i>initial sync underway</i></p><p>Gentoo - <b><i>up</i></b></p><p>Ubuntu Ports (releases) - <b><i>up</i></b></p><div>Blender - <b><i>up</i></b></div><div><i><br /></i></div><div>Debian non-free images - <i>initial sync underway</i></div><p>Xfce</p><p>CentOS Stream - <i>initial sync underway</i></p><p>Rocky Linux - <i>initial sync underway</i></p><p>Qubes OS - <b><i>up</i></b></p><p><br /></p><p>Feel free to email help@ocf.berkeley.edu if you have any suggestions :)</p><p><br /></p><p>*: subject to change</p>Jonathanhttp://www.blogger.com/profile/10004598210761796546noreply@blogger.comtag:blogger.com,1999:blog-34923766.post-28620140138778653692022-01-21T10:31:00.000-08:002022-01-21T10:31:23.805-08:00Planned server reboot on 1/22<p>We will be rebooting our servers on Saturday, 1/22 at 2:00 PM PST for kernel updates. We expect the downtime to be around 30 minutes.</p>Jonathanhttp://www.blogger.com/profile/10004598210761796546noreply@blogger.comtag:blogger.com,1999:blog-34923766.post-63268478722748630002021-11-17T19:38:00.005-08:002021-11-17T23:11:08.967-08:00Scheduled downtime on 11/20<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEixmSsDEnBAmbtQM056n77Mwp1vhStbiMCMRhBj0IYmSQ-cnpk8FLASJ-lwxM53sYIG9EyPwdFVqVsL0nixZkuD1nYxFwYcTFGuFUb2iQ7wRZR4iGQbwojlzzprZU8RIEdhCNjWBFEKnRanqdw8DCY_Bwa1Xif9mVCmLvoAPWlUcZDaPSxmAwo=s1280" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1280" data-original-width="1280" height="320" src="https://blogger.googleusercontent.com/img/a/AVvXsEixmSsDEnBAmbtQM056n77Mwp1vhStbiMCMRhBj0IYmSQ-cnpk8FLASJ-lwxM53sYIG9EyPwdFVqVsL0nixZkuD1nYxFwYcTFGuFUb2iQ7wRZR4iGQbwojlzzprZU8RIEdhCNjWBFEKnRanqdw8DCY_Bwa1Xif9mVCmLvoAPWlUcZDaPSxmAwo=s320" width="320" /></a></div><br /><p>Most OCF services will be down on Saturday, 11/20 due to a scheduled hypervisor OS upgrade. Updates will be posted on this page.</p>Jonathanhttp://www.blogger.com/profile/10004598210761796546noreply@blogger.comtag:blogger.com,1999:blog-34923766.post-27359903173848738252021-11-14T05:47:00.000-08:002021-11-14T05:47:03.134-08:00Mirrors scrub in progress<p> Mirrors FS scrub in progress, performance may be impacted. ETA 48 hours.</p>Jonathanhttp://www.blogger.com/profile/10004598210761796546noreply@blogger.comtag:blogger.com,1999:blog-34923766.post-36552779331897767762021-11-01T21:29:00.008-07:002021-11-14T05:41:13.984-08:00[RESOLVED] Scheduled downtime on 11/13<p> We are planning on having some scheduled downtime on Saturday 11/13 starting around 12:00pm PST. We expect to be done before 4pm. We will be rebooting our main hypervisor as well as some other machines. Our website, student group websites, and other OCF services will be unavailable for the duration of this downtime.</p><p>If you have any questions about the downtime please <a href="https://www.ocf.berkeley.edu/docs/contact/">contact us</a>.</p><p><br /></p><p>Update 3:57PM 2021-11-13: We have experienced some issues updating our main production server, so we expect to take at least another hour. We apologize for the inconvenience.</p><p>Update 7:14PM 2021-11-13: We have been told to leave the build which hosts our servers by the Student Union staff, therefore we likely will not be able to bring things back up until Monday.<br /></p><p>Update 4:49AM 2021-11-14: Most services are now up. We are working on bringing scurvy (staff VM, Berkeleytime etc.) up.</p><p>Update 5:28AM 2021-11-14: Scurvy is up, berkeleytime is up, hellfire is up, marking this as RESOLVED?</p><p>Update 5:40AM 2021-11-14: Marking this as RESOLVED</p>Jonathanhttp://www.blogger.com/profile/10004598210761796546noreply@blogger.comtag:blogger.com,1999:blog-34923766.post-88970986821803157222021-10-23T17:31:00.002-07:002021-10-23T17:31:29.173-07:00HTTP/2 is now enabled for web hosting<p> HTTP/2 is enabled for https connections on the web hosting server (death) as of 1730 PDT, Oct. 23.</p><p><br /></p><p>Please email us at help [at] ocf.berkeley.edu (or send a message in #rebuild) if you experience any issues.</p>Jonathanhttp://www.blogger.com/profile/10004598210761796546noreply@blogger.comtag:blogger.com,1999:blog-34923766.post-42294936240080368512021-10-23T13:02:00.003-07:002021-10-23T13:02:54.530-07:00centos-debuginfo mirror decommissioning<p>For years, we have been mirroring the centos-debuginfo project, which hosts the *-debuginfo packages for CentOS. However, our centos-debuginfo mirror is an unofficial one, and is not listed on the CentOS website. </p><p>Our traffic statistics show that the mirror has no to negligible actual usage, with tens to hundreds of megabytes of data transferred daily (almost entirely by bots). </p><p>The project currently occupies 2.7 TiB of disk space, which is the highest of all the projects we mirror, and yet no user is actually using it. As a result, we will decommission the centos-debuginfo mirror on Saturday, October 30, 2021 at around 1200 PDT. The project will continue to be synced in the meantime.</p><p>If you are currently using our centos-debuginfo mirror, or if you have any concerns or questions, please email us as soon as possible at help@ocf.berkeley.edu.</p><p>We expect to have 3 TiB of usable disk space on the mirrors server following the removal of centos-debuginfo, so if you have any suggestions for open source projects to mirror, please also email us at help@ocf.berkeley.edu :) We are currently discussing adding ubuntu-ports. </p><p>rjz</p><p>OCF Staff</p>Jonathanhttp://www.blogger.com/profile/10004598210761796546noreply@blogger.comtag:blogger.com,1999:blog-34923766.post-69189495596463522992021-10-11T00:32:00.005-07:002021-10-12T19:16:40.906-07:00[Investigating] Mirrors disk status<p><b>10/11 0000 PST:</b> We noticed that one disk attribute was abnormal on multiple disks in the mirrors server. All SMART attributes are currently normal, so we currently do not expect any immediate service interruption as we investigate the issue. </p><p>10/12: Scheduled maintenance at 1930 PST. ETA 15 mins.</p><p><b>We will update this post once we have more information on the situation.</b></p>Jonathanhttp://www.blogger.com/profile/10004598210761796546noreply@blogger.comtag:blogger.com,1999:blog-34923766.post-49559083485710588162021-10-08T14:10:00.004-07:002021-10-08T15:31:49.661-07:00Emergency Mirrors Outage 10/8<p>Our mirror server is currently down for maintenance. ETA 60 mins.</p><p><br /></p><p><b>Update: 1530 PST </b>This incident has been resolved.</p>Jonathanhttp://www.blogger.com/profile/10004598210761796546noreply@blogger.comtag:blogger.com,1999:blog-34923766.post-76363156121781870332021-10-08T00:47:00.004-07:002021-10-09T14:51:27.968-07:00 Tanglu, Hyperbola decommissioning<p><b>Update: Maintenance was completed during Friday's downtime. No service interruptions expected today. </b>Tanglu and Hyperbola mirrors have been decommissioned.</p><p><br /></p><p>We are planning to take the mirrors server down for disk array migration from 10am-6pm this Saturday. HTTP, HTTPS, FTP, and Rsync will be unavailable during the maintenance window. The server may come up before 6pm, but please expect degraded performance and/or service interruption throughout the maintenance window.</p><p><b>We are planning to decommission tanglu and tanglu-releases as the project has been defunct for over 2 years. We strongly recommend against the continued use of Tanglu, but if you are still using Tanglu, please change your source.list to another archive mirror. As of Oct 7, 2021, http://ftp-stud.fht-esslingen.de/Mirrors/archive.tanglu.org/tanglu/ is available.</b></p><p><b>We are also planning to decommission Hyperbola. Our Hyperbola mirror has been out of date for quite a while now (almost 3 years!), but please change your configuration accordingly if you are using Hyperbola and still have us as an entry.</b></p><p>We apologize for the inconvenience.</p><p>OCF staff</p>Jonathanhttp://www.blogger.com/profile/10004598210761796546noreply@blogger.comtag:blogger.com,1999:blog-34923766.post-16481461626113925542021-09-22T22:48:00.005-07:002021-09-27T19:06:03.398-07:00Emergency Mirrors Maintenance<p>2021-09-27 19:05 PST: mirrors are back up, this downtime should be resolved. If you notice any issues using our mirrors please reach out to us at help@ocf.berkeley.edu.</p><p><br /></p><p>We are currently (9/22) running emergency maintenance on mirrors.ocf.berkeley.edu. Due to an earlier drive failure we've had to replace the drives. We expect the mirrors server to be down until Monday, but we will keep this blog post updated.<br /></p>ethanhshttp://www.blogger.com/profile/00268153614295180221noreply@blogger.comtag:blogger.com,1999:blog-34923766.post-78455686824751506002021-07-05T15:42:00.005-07:002021-09-16T17:11:02.976-07:00Intermittent service outages during summer break<p>The OCF is currently aware of intermittent service outages during the summer break to our OCF website, web hosting services, mailing services, mirrors, and internal infrastructure. Volunteer staffers have begun an investigation of the issue and are planning to resolve it in the coming weeks. Please expect delays in this process due to the limited number of staff available in the summer and other factors beyond our control. We will update this post once we have more information on the situation.</p><p>If you are a user or represent a group that has been adversely impacted by a recent OCF service outage, please do not hesitate to contact help@ocf.berkeley.edu so we can investigate further.</p><p><b>Update: </b>This has been resolved. </p>Kevin Mohttp://www.blogger.com/profile/00533984642298245914noreply@blogger.comtag:blogger.com,1999:blog-34923766.post-26002647328454924012021-05-31T03:52:00.000-07:002021-05-31T03:52:01.732-07:00Major web service outage<p>The OCF is currently investigating an ongoing service outage that has disrupted web services including our main website (ocfweb) and user-hosted websites. All other services, such as our login servers (tsunami), HPC, and mirrors, should be functioning as normal.<br /></p><p></p><p>Due to the timing of this outage, our response remains limited throughout the early night. We will update this post once we have more information on the situation.</p>Kevin Mohttp://www.blogger.com/profile/00533984642298245914noreply@blogger.comtag:blogger.com,1999:blog-34923766.post-2318731888292835392020-10-03T12:52:00.005-07:002020-10-05T17:03:47.679-07:00Mirrors down while we rebuild the RAID array<p> We are replacing a drive in our mirrors server so it is down. This is expected to be done by 2PM PST 3/10/20.</p><p>In addition, from 12:01AM on 05/10, we will temporarily shut down the mirrors service to allow the RAID array to rebuild properly. This may cause a prolonged outage.</p><p>EDIT: The rebuild is now complete. We're syncing the mirrors live so you might see some outdated packages.<br /></p>jawhttp://www.blogger.com/profile/01168713438480095159noreply@blogger.comtag:blogger.com,1999:blog-34923766.post-68813887412210952162020-09-28T14:12:00.004-07:002020-10-02T01:21:58.681-07:00Mirrors out of date due to server issues<p>We're observing extended issues in syncing a number of our mirrors including Ubuntu and Kali, and so some of them may be very out of date.</p><p>We're currently working on resolving these issues and will update when we have more information.</p><p>Update: the issue has been resolved<br /></p>jawhttp://www.blogger.com/profile/01168713438480095159noreply@blogger.com