Network outage at IC

Around 19:00 on Saturday 13th December the web site began experiencing problems. TomH became aware of this around 19:20 and after establishing there there appeared to be some sort of network problem that made many machines at IC unreachable the site was manually placed in offline mode as both the database and chef servers were unreachable.

An email was sent to IC just after midnight, as the problem was still unresolved, and things came back to life around 01:30 and the site came back online automatically once chef was available again.

Tile caches

The GeoDNS system used for the tile caches was switched to use gdnsd with the Maxmind database, as the database used in the old system was no longer being updated. This also enabled support for the EDNS client subnet option to allow resolvers to pass on the real address of the client.

Three new tile caches were bought online, in Croatia, London and Oregon and the existing tile caches were all upgraded to Ubunt 14.04.

Planet dumps

The weekly planet dumps were switched to use planet-dump-ng meaning that the planet dump no longer has to query the database directly. In addition the XML and PBF exports are now produced in parallel, along with weekly full history exports.

Nominatim

The original Nominatim server, poldi, which has been out of service for a while was reinstalled with Ubuntu 14.04 on 18th December and is now being setup to share the service load with pummelzacken.

Forum server

The forum server, clifford, which had been down and was not responding to attempts to boot it remotely, was visited and persuaded to boot cleanly. It has now also been upgraded to Ubuntu 14.04 and is ready for forum installation. (See operations issue tracker for details).