Well, if anyone tried to access our site over the last few days they may have noticed that we’ve been down quite a bit, so apologies for that. If anyone is interested, the blackout was due to a few reasons:
We attempted to update some software on our VPS so we could replace Hudson with Jenkins. The update caused some serious errors where most system services went offline and refused to start (including Apache).- Upon realising that there was too much damage to put right in a swift amount of time, we decided to restore the VPS snapshot we took immediately before the update. The restore failed. The snapshot was corrupt. The VPS wouldn’t boot.
- Our host (Webfusion) said that a VPS factory reset was pretty much our only option. Fortunately, we take daily backups so we weren’t going to lose much. Problem was, once we got back to where we were, we were still going to need to update the VPS to install Jenkins and potentially break everything again. Since our VPS was due for renewal we decided to purchase a nice new shiny version of our VPS and migrate everything across, this way no updating would be required when things were operational again; so that’s what we did. Unfortunately, once our new VPS was set up we discovered that the backup manager, that’s part of our VPS control panel, wasn’t all it’s cracked up to be. It refused to restore the backup we threw at it, so in the end we had to extract the contents of the backup and move them into place manually… which is still an ongoing task.
Most of the Deadworld stuff is back online and functioning normally. There’s just a few other things, including some personal sites, that still need to be restored. Hopefully, this’ll all be done in the next couple of days.
Good news is that Jenkins is up and running though, so it wasn’t all for nothing!