Saturday, March 19, 2016

MySQL read-only Saturday 3/19

As part of our work to transition from Percona to MariaDB for our MySQL server, we'll be migrating user data tonight around 9pm.

To do this, we'll put the existing Percona server into read-only mode, then make a final import to the new MariaDB host. We believe this will take about an hour and don't anticipate any issues (we've already tested imports from our regular backups without problems).

Read-only mode is necessary during the import to ensure we get a consistent backup, and so that writes made during the transition are not lost.

Some sites may experience downtime while the server is in read-only mode (if they require writing to the database to show pages). Most sites will experience some level of degradation (e.g. can't log in to admin or edit posts).

Update 10:01pm: We have entered read-only mode.

Update 10:10pm: The backup is complete and is being imported into MariaDB now.

Update 10:14pm: ETA 35 minutes.

Update 10:35pm: The import was interrupted when the new server ran out of memory. We're increasing memory / reducing memory use by mysqld and starting the import again. Still in read-only mode.

Update 10:45pm: ETA 38 minutes.

Update 11:11pm: Import has finished, we're now swapping out MariaDB for Percona (which will involve about 2 minutes of downtime).

Update 11:16pm: We've noticed some issues with the import (views were not correctly copied) so we'll need to re-do the import. Still in read-only mode, expect another hour or two in this state. Sorry for the trouble!

Update 11:44pm: The view problem is fixed, so we're proceeding to move MariaDB into production. Expect about 2-3 minutes of downtime now.

Update 11:55pm: All work is completed and we are now on MariaDB. Total downtime was about 3 minutes, with read-only mode lasting about two hours.