Posted in: Clan News, Website News
Greetings CMSers! You may have noticed we’ve been down since about Friday, March 23. The situation now is I’ve restored our databases from a backup made March 21, and I am uploading our files from the same backup. For now, only the main site is restored, BBS, coppermine, and others should be restored tomorrow. Also, for future reference, CMS Steam group page is your source for CMS news when our site is down.
March 23, the site went dark on us. We weren’t sure what was going on. On the 24th, we received the following email from our website host, Lunarpages:
Dear Customer,
The Undertow server requires a restore as the server has issues with the raid preventing it from booting up. We have already started restoring the server to a new hardware. Unfortunately, all services will be offline until the restore process is complete. Please note that we do not have any estimated time for completion for the restore process. The best projection would be that the machine will finish migration in 48-72 hours time from now.
You can see the status for this migration at any time in CAP area by logging in with your username and password from our main site, then clicking the Network Status icon.
If you have any questions, please email support@lunarpages.com on this process.
Thank you for your patience and we apologize for any inconvenience this may cause.
Lunarpages System Administrator Team
At this time, constantmayhem was unreachable. The site remained unreachable until Monday evening. When Lunarpages got the site back up, I noticed that much of our data was old, and the forum database had not been restored properly (the database size is ~47MiB, they managed to restore only ~2MiB of it). It seems like Lunarpages’ backup was older than our latest backup!
What I’ve done today is restore all of our databases to as they were on March 21 in the early morning. This process took quite a while, because if I tried to upload the entire forum DB (~47MiB uncompressed, ~10MiB compressed) the restore would timeout and nothing happened. After experimenting with breaking the database dump up into many smaller pieces (with the help of SQLDumpSplitter), I was able to restore it after dividing it into 26 ~2MiB pieces and uploading them one by one.
I am now uploading the backups of BBS, coppermine, and other CMS services. I’d be able to restore them tonight, but my upload pipe is too slow (curse you, Roadrunner!) for me to be able to do that before 1am. I’ll keep you posted.
Permalink
Tags: backup, disaster recovery, lunarpages, mysql, outage, restore, website
