WTF Happened?! A Tale of Extended Downtime

Alright back up! For those that hadn't guessed already UtaForum suffered some extended downtime.

This downtime was caused by the MySQL database that powers this site horribly corrupting after a server crash while the forum was in the middle of a critical read/write operation. The crash was caused by another website on our server node being DDOSed, overloading my server's memory, taking everyone hosted there down with it, UtaForum and many other sites I host included.

With our main database corrupted I have spent the last couple days recovering what I could from our backups. I was able to save the users table and its associated data, the forum structure, the threads, PMS, and even the polls but sadly 'mybb_posts' was the one table I wasn't able to recover.

I will try to see what I can do, but after having spent the last 5 days doing what I can I am not sure what else I can recover. I have even older backups, backups from early April but I am not sure how much good that will do us.

For now UtaForum is tentatively back with myBB but as a shadow of its former self. Myst, the staff, and myself have much to discuss about where we want to go from here. We hope to have an announcement soon. Please check back later.

 

[MystSaphyr]

RE:

I want to apologize deeply for what happened.

Hentai, you've working incredibly hard on recovering everything and I greatly appreciate your help.

If nothing else, we'd be able to make an archive of the aforementioned April backup to go alongside the Forumotion and Zetaboards iterations.

 

[oteto]

RE:

Yiiiiiiii! Utaforum is back! You guys rock!

 

[Ori]

RE:

Ohhh, good luck! I know rollbacks can be a pain.

 

[Aion]

RE:

AHHHHHHHHHHHHHHHH NOO THAT IS TERRIBLE ; O ; I Mean if you guys managed yet to get some threads well there's a few ones with download links which i really needed it that i can't acess i'm sorry for what happened and i also thought it was my PC's problem lol

 

[Hentai]

AHHHHHHHHHHHHHHHH NOO THAT IS TERRIBLE ; O ; I Mean if you guys managed yet to get some threads well there's a few ones with download links which i really needed it that i can't acess i'm sorry for what happened and i also thought it was my PC's problem lol

As I said more backups exist, some older than others, some VERY VERY old.

Depending on the specific threads age it may or may-not be recoverable.

I will keep trying, but what you see today is the culmination of almost a week's work.

More options may open up soon, I will keep you guys updated on the progress.

 

[Kyoren]

RE:

It's very sad to see this, but if there is anything we can do to help. Hopefully, let us know. c:

 

[kimchi-tan]

RE:

[size=-3]I saw this thread this morning but I will make my reply now that others did.[/size]

Hopefully the April backup could do us some good enough. It's really unfortunate for this to happen so suddenly. (Engloids.info seems fine though)

Also, when will the discussion start?

 

[MillyAqualine]

RE:

Happy to see it's back~

 

[Kyoren]

RE:

Most of us are still active in the Skype group chat though, if any of the moderators or Hentai would like to come there and collaborate with us?

 

[moonst4r]

RE:

It's unfortunate what happened, but at least the forum is back now! Thank you Hentai for your hard work you're doing for all of us.

 

[Xuu]

RE:

Well, at least we're back now. o3o
There's a Forum Skype? *wants in*

 

[insanemollum]

RE:

wow o.o i thought it was my pc that wasn't letting me access UTAforum
anyways
i'm glad to see it up and running again! :3

 

[Xuu]

RE:

Might I say that A LOT of threads and posts are missing.. like, 90% of them...

 

[MystSaphyr]

Might I say that A LOT of threads and posts are missing.. like, 90% of them...

See Hentai's first post: The posts table is the one that was corrupted. There are separate tables for topics and posts, so while the topics were recovered, their posts weren't.

 

[Xuu]

RE:

damn, that sucks. No more UST/VSQ thread, all those games...
welp, it's a clean slate I guess, unless we can recover them?

 

[na4a4a]

RE:

Ah HELLa I'm crying!

as long as the forum is back, I think as a community, most of the users would be just fine without their old threads/post!
You have obviously been working your butt off getting things back into working order, even if there wasn't much to be done from this point (assuming that further dated backups don't have what you are looking for) a clean state might be a good thing in some way
(okay, I'm trying to look at the bright side...I'm should shut up)
... ... ...
too bad no one has been downloading the forum AND at least had a mirror of it (you know, for the links and such...though it probably wouldn't be of much use to you guys)
(I totally would of done that, though I didn't because that is probably wrong from some moral standpoint...and it probably is irrational from a technical standpoint...for lack of a better term)
edit: ...oh yeah...and someone downloading the whole forum every week would definitely siphon your allotted bandwidth... (._.)

But even if you have some earlier backups, HAVING SOMETHING could be better than nothing!
I couldn't imagine the work it takes to keep things working smoothly and consistantly, it is apparent that you have be kicking your self a bit getting things back in order- I almost say (short of the old theme) that things are pretty decent!

I'm pretty sure that most would disagree, I'm trying to be an optimist and I'm not good at it...backward opinions, I have.
If it happens that some things can't be recovered, then it can't be helped and people will have to learn to tolerate and understand the situation and the state of fact...
Keep up DAMN GOOD work! I swear I would donate an organ for you guys!...or just have it taken out in a back ally...

 

[Leo]

Most of us are still active in the Skype group chat though, if any of the moderators or Hentai would like to come there and collaborate with us?

I had no idea there was a Skype group chat. I personally would love to join, I think it would be pretty useful!


Hentai really deserves a huge thank you for all the work he's put forth into salvaging what we have of utaforum right now. While we may have lost the most recent archive of posts, Hentai was able to recover everything else which is pretty fantastic.

 

[na4a4a]

RE:

To brighten you day- Here is a recent discovery of mine-
The "MyBB" logo looks like a floating fat person...reaching out for something...just take a look at the top!

and what about old.utaforum.net? how recent is that?

 

[Hentai]

RE:

One you you must realize is UtaForum's database is BIG. I mean ridiculously large in terms of size and scope for something that is pure MySQL. This may get a bit technical so bear with me...

To put things into perspective:
phpMyAdmin, the primary tool for administration of MySQL databases on cPanel servers can import databases a maximum of 50mb in size.
GoDaddy, a popular shared webhosting services while advertising 'unlimited' resources has hard limits on MySQL databases at 100mb due to how unwieldy they become.

UtaForum's 'utaforum_mybb' database, the one that has powered these boards since our SMF days has bloated to over 500mb in size. This is 10x larger than the maximum importable via phpMyAdmin's web interface and 6x GoDaddy hosting accounts of MySQL (if databases could be split like that)

Because administration tasks can't be handled by phpMyAdmin due to its massive size to administer our data I have to SSH into the server and use command line to perform MySQL operations. With only command line as a tool anything involving administering the main DB is an arduous process and must be preformed manually.

I will admit, I backed things up only occasionally, downloading them to my local machine maybe once a month. The staff and I are looking for more sustainable maintenance structure and going forward it will definitely be a priority for us.