Charlock said:@RedAgliator might have some more information, possibly.
@RedAgliator might have some more information, possibly.
I have some good news and some incredibility bad news. The good news is that I’m not dead or anything, and the wiki isn’t permanently down. It should be up and running sometime around this upcoming weekend. The bad news that the provider who owns the hardware/server that the wiki was hosted on went bankrupt and had their servers de-racked (essentially decommissioned) without warning by their upstream (datacentre) provider. I had an automated backup strategy running consisting of three different servers, but due to an unfortunate series of events, they are not as recent as anyone would like. In short, the wiki will be back online soon but months of work has been lost. The company in question sent out notices to all their clients on the 6th of August, and at that time, I was on a holiday in Mongolia. I did not have many chances to get secure access to the internet while there but I did manage to view that email around the 12th of August. The company stated that all servers would be taken offline on the 31st of August, and I was due back home by the 20th. I had planned migrate everything on the 22nd and even put up a notice on the wiki about the downtime from DNS changes that it would bring.Unfortunately, the company took all the servers offline on the 21st. That’s 10 days in advance of the shutdown date (31st) that was originally communicated to clients. Even though I run a three-tiered backup system that would normally be considered strong, the ramifications of that event has been incredibly bad. The hard-drive of my primary external backup source failed while I was in Mongolia on the 16th – and the early shutdown had eliminated the provider managed backups out of the possibility. That only left my third back-up source that was taken very irregularly. Here is a quick explanation of the back-up system that I had in place 1. Server hard drive containing the wiki files / database: De-racked and put into storage with hundreds of other drives. It’s impossible to get access to the drive that I rented due to privacy reasons. I even offered to purchase it outright, but that offer was rejected. 2. Primary daily back-up (automated) provided by the server company: These have been pulled from their racks leading the data to become inaccessible. The company is bankrupt so there’s nobody left at the company to spend the money to organise sending backups to clients. 3. Primary daily back-up (automated) on a second server that I rent in a different U.S. datacentre: The hard-drive on this server failed on the 16th of August, and that piece of hardware was immediately replaced by the data center. Unfortunately due to the fact the drive was unconfigured/blank when it was replaced the server wasn't backing up the wiki files/database past this point. 4. Secondary bi-annual backup (automated) on a third server I rent in a different Australian datacentre: This is the only thing that saved the wiki from simply being forever lost. Unfortunately, the last backup on that server dates back to March this year. It’s not even close to acceptable, but it’s better than nothing. Cross-reference this will the time-line and you will start to get a picture of the unfortunate (and very unlikely) sequence of events and why the bi-annual backup is the only option.29 July – Start of Mongolian holiday. 6 August – Hosting provider sends notice stating that all services will be shutdown on the 31st 14 August – Noticed the shutdown email due to coming within range of a CDB. Posted notice on the wiki stating that a planned node migration would occur on the 22nd. 16 August – The hard drive of the second primary daily back-up (automated) server failed. Replaced by the DC on the same day, but data recovery on the drive was not possible. 20 August – Arrive back home late night. I noticed that email about the dead drive, but I was drunk as hell at that moment so I figured I would wait until I migrated the wiki on the 21st to configure the second backup server again. 21 August – Wiki servers de-racked by the upstream (datacentre) provider without warning. Was done early in the morning while I was sleeping. Woke up to a nightmare.
I know how bad this is, and I have lost sleep over this. The fact that there was a window of opportunity on the night of August the 20th has made the guilt I have been feeling this past week incredibly distracting. I could have simply downloaded the backups manually, but I wanted nothing more to plop down in my bed and go to sleep. I didn’t trust myself to fool around with backups when I was drunk either. At that point, I still believed that I had another 10 days until the wiki server (and the remaining backup server) went down. I am really not sure what I can say at this point… I’ve been holding off on posting this the past week because I’ve been desperately trying to find a solution that doesn’t result in months of lost wiki work. Unfortunately, I have to face the fact that it is impossible to get an up-to-date version of the wiki database. I am incredibly sorry to those editors that spent time updating the wiki in the past months. I will bring the wiki back online this weekend. Once that happens it’s likely that I will officially hand over everything (domain, database, files) to somebody that has the capability of running the wiki. I could put blame on the shitty sequence of events, or the server provider and their upstream, but ultimately it was my responsibility to ensure that the community would not lose their work. I failed in that regard. I thought I had covered any possible outcome when I had three servers taking backups of the wiki. I was wrong. PS: It's 2AM as of writing and posting this. I apologies if it's hard to read / understand.
Charlock said:[...] but I am sure we'll get it back to where it was.
[...] but I am sure we'll get it back to where it was.