Sorry for the downtime folks. I have been running updates on our Facebook and Twitter feed to keep people in the loop and I thought I’d add some here as well now that we are back live again.
We have been organising a move to a new build server from our old box. this was due to have taken place this week. We’d be moving from 3+ year old hardware onto a newer dedicated box of markedly better specs and simply put “newer”. This would bring with it that latest core OS and new versions of our background services and enable me to roll in all our latest software updates. I can then update the site design and all the software we’ve written that runs this site. In short we get a new box and I can make things all shiny and kick some life into the design.
On or around the 6th of October the company behind the server management software that runs our rack pushed out a new version. This new version was quick frankly dreadful for us and crashed a heap of core services, knocking out email and web services. Then after a few days the entire server went down. We run a RAID hard drive set up and also backup regularly off site, however the drives then took that moment to die and due to the core management suite being broken the latest backup was corrupt.
Thus our technicians at the data centre have worked to restore our data and rebuild the hard drives manually so as to prevent any data loss. We could have used old backups but that would loose some forum posts and updates.
At present we are back up on the old box, the new machine is being prepped and shortly we’ll move the IP over to the new machine.
I’m afraid there was little we could do when this all occurred, we run from one dedicated box with no fall back machine if everything goes wrong at once. We are then in the hands of data centre technicians many miles away. Sadly their speed and ability to respond to our needs was slowed down by the number of sites affected by the same software fault.
When there’s more to post up I will add it. Quite simply I’d like to get the new hardware up and running so I can rewrite the site code and design and add all the latest software.
