Re-issued site certificates (Heartbleed)

On April 9 the openstreetmap.org and tile.openstreetmap.org web certificates were re-issued as a precautionary step in light of the OpenSSL “Heartbleed” vulnerability

Failing Disk in Shenron replaced (Bytemark)

On April 10 S.M.A.R.T. daemon notified us of a failing system disk (RAID1 member). Grant arranged a disk swap-out with Bytemark.

2 new disks were installed the same day (40mins downtime) and were synced up by April 14th. The old disks were removed on the April 15 with minimal downtime.

Database server outage

Around 0130 BST on Good Friday (April 18) the master database server, ramoth, became unresponsive and as a consequence the web site quickly began to fail

TomH power cycled the server using the remote console at around 0635 BST and the database and web site were up and running by 0645 BST resulting in a total outage of just over five hours.

The replication diffs didn’t restart properly and fixed by Jon Burgess later the same day.