Recent Downtime

By WedTM Jun 26, 2011
Category Icon News
58

As many of you are probably aware, the Curse family of sites has been down since early AM (PST) the 22nd. This post is meant to clear up any questions that may arise about this event.

At approximently 7:30AM PST, on June 22nd, our primary SAN controller experienced a catastrophic failure, bringing it offline. This controller was the primary controller for most of the database nodes, and most of the web nodes. Redundant systems failed to come online, even though they had reported as 'ready' before the primary systems failed.

After replacing the failed controller, it begain booting and copying it's configuration from it's peer server. Unfortunately, as soon as the configuration was copied, the secondary controller also died.

After replacing the second failed controller, we began powering the servers that reiled on the SAN for their data - all of the database servers and the network-attached storage (NAS) file servers that store all the media, static content, and most of our web files as well.

This process only took a few hours. The main delay was an extended period (24 hours) of checking the continuty of the data on the disks. This was the majority of the downtime experienced.

As we started pulling our servers online after check was complete, we noticed an issue with the firmware's on the new SAN controllers. The newer firmware versions were conflicting with the storage array, and thus, the controllers couldn't talk to the disks.

The manufacturer told us that this was a known issue, and provided us with a method of repairing it. We then backed up all the data again, and proceeded to apply the firmware patch.

After the patch, we were able to restore the drives, and start booting up the critical systems, followed by the non-critical systems.

We can assure you that at no time during the hardware failure was any of your personal information compromised. We take the sacred trust you put in us with your information VERY seriously.

Thank You

We realize that you rely on the forums as part of your Minecraft experience. Once again, we sincerely apologize for the downtime and hope you'll continue to enjoy the Minecraft Forums.

The Minecraft Forums Team

Comments

King_Korihor
- IRC HACKER
- Join Date: 5/23/2011
- Posts: 1,419
- Member Details
#36 King_Korihor

View User Profile

Send Message

Posted Jun 26, 2011
Quote from Chris.

Well I must say they are pretty dumb, if their backup servers don't work either. Who was checking those things?

Stuff happens. It is called real life and if you had bothered to actually read the OP you would realize that there were multiple failures including stuff that was completely beyond their control. More to the point, once the whole network of servers went down, they didn't dare simply flip the switch to put everything back on right away.

I, for one, am very glad that they took their time to do it right. Had they not done so, posts would have been lost or much more bitching about how various parts of the forums weren't working or pages missing from the wiki and all sorts of other potential problems could have happened.

The only part of the summary that I find a little bit offensive is the claim it only took 24 hours to fix the problems. The site has been going up and down for much longer than just one day. Futhermore, I don't really understand why other parts of the "CURSE Network" came up but the Minecraft stuff didn't. I'm sure some of that is due to the size of the Minecraft community and because so much of the community is "officially" using these servers. Just be honest with us, and don't try to sugar coat what must have been a network technician's worst nightmare come true. Don't try to downplay the extent of the problems.

Also for the kiddies to note, this was also a financial disaster in terms of not just having to replace equipment (which costs $$$) and a whole bunch over technician overtime (even more $$$), but it also cost advertising revenue (still more $$$) not to mention a loss of goodwill (where many in the community may go elsewhere). I'm sure these guys are completely aware of that, but give them a break and don't go rubbing it into their face.

Quote from LJDuckE

Minecraft should stay away from that sh*tty curse network. they got their fingers in their asses all day long, while minecraft is actually attempting to get somewhere...

Ditch CURSE, As a Pre-Paying owner of minecraft, i like to be able to access the forums when i need to....Not when f- curse say so...

For those who think that CURSE "owes" you something because you paid for Minecraft, that is pure and utter BS on your part. You paid for the game, not forum access. Did anybody ever have problems actually playing the game during this time period? I thought not. Your "pre-paying owner of Minecraft" has nothing to do with this whole thing so crawl back under your rock and ditch that holier than thou attitude. Mojang did not promise you anything in terms of forum access when you purchased this game.

You can "Ditch CURSE" if you care, because there are other forum communities out there. Look them up, and I for one would be glad if somebody who is so petty like this would leave the "community" here on these forums on CURSE. Don't let the door slam you in the behind as you leave. I'm not going to miss people with an attitude like this.

Rollback Post to Revision RollBack
CGuangJun
- Gold Miner
- Location: Here, yeah, right here.
- Join Date: 3/3/2011
- Posts: 381
- Minecraft: CGuangJun
- Member Details
#37 CGuangJun

View User Profile

Send Message

Posted Jun 26, 2011
Great work on bringing the servers back up! Now I can continue browsing

Rollback Post to Revision RollBack
DomeOne
- Lapis Lazuli Collector
- Join Date: 5/12/2011
- Posts: 987
- Location: america
- Minecraft: SkullCandyCT
- Xbox: Axiama
- Member Details
#38 DomeOne

View User Profile

Send Message

Posted Jun 26, 2011
lol...

Rollback Post to Revision RollBack
genghaskhan
- Newly Spawned
- Join Date: 6/26/2011
- Posts: 2
- Member Details
#39 genghaskhan

View User Profile

Send Message

Posted Jun 26, 2011
Quote from naruck23

Wow that was unlucky, Double Controller Kill.

hey i just downloaded minecraft and was wanting to get better texture how do look up what mindcraft im on 1.6 or 1.5 where can i find this info

Rollback Post to Revision RollBack
MCFUser332813
- Stone Miner
- Join Date: 4/5/2011
- Posts: 87
- Minecraft: P0rtal
- Member Details
#40 MCFUser332813

View User Profile

Send Message

Posted Jun 26, 2011
Oh,so it was a failer...Me and my buddy on his server thought that the wiki and forums we're down cause of it might being an update to 1.7,so it was just changing...XD!Well it's good to know about what REALLY happend.

~StoneIronPIG! AKA,P0rtal

Rollback Post to Revision RollBack
MCFUser332813
- Stone Miner
- Join Date: 4/5/2011
- Posts: 87
- Minecraft: P0rtal
- Member Details
#41 MCFUser332813

View User Profile

Send Message

Posted Jun 26, 2011
Well,to find out what version you have,just go to the Title Screen (The screen with options and where it says MINECRAFT), and it should be right at the Top Left Corner...

_
/ \
|
| + /
|------
\

and yes the LEFT arrow looks like a tilted Pick-Axe

Rollback Post to Revision RollBack
MCFUser332813
- Stone Miner
- Join Date: 4/5/2011
- Posts: 87
- Minecraft: P0rtal
- Member Details
#42 MCFUser332813

View User Profile

Send Message

Posted Jun 26, 2011
The message above of me is to answer the question of:genghaskhan

Rollback Post to Revision RollBack
kennethtehwh
- Tree Puncher
- Location: Singapore
- Join Date: 3/24/2011
- Posts: 30
- Minecraft: BlueXaphriox
- Member Details
#43 kennethtehwh

View User Profile

Send Message

Posted Jun 26, 2011
Hurrrayyyyyyyy! Forums back up! Gratz on the system recovery, the wait seemed forever .__. FINALLY MINECRAFT TIMEEEE >:DDD

Rollback Post to Revision RollBack
ShatteredLight
- Lapis Lazuli Collector
- Join Date: 3/28/2011
- Posts: 1,130
- Member Details
#44 ShatteredLight

View User Profile

Send Message

Posted Jun 26, 2011
Well, glad the site's back up anyway. My question is, Did lulzsec play a part in this? I read that lulzsec was screwing around with some kind of minecraft-related thing. I am aware that it wasn't just the minecraft areas of Curse that were effected, but I haven't been able to find anything else about Minecraft that recently had downtime.

Rollback Post to Revision RollBack
Xtek
- Out of the Water
- Location:
- Join Date: 4/1/2011
- Posts: 3
- Member Details
#45 Xtek

View User Profile

Send Message

Posted Jun 26, 2011
The reason that Minecraft Forum and Minecraft Wiki were so late to come up is because after a 40 hour check disk (data and hardware integrity check) we noticed our Linux servers (which the MC sites run on) couldn't access the SAN array. In order to give them access we were told by our hardware vendor we needed to update the firmware for our SAN infrastructure. In order to do that we wanted to back everything up off of it just in case something went wrong. This was the 2nd part that took so long.

After doing all of that we were finally able to get our Linux servers access to their data and have them running again.

It wasn't as simple as just having a backup, it wasn't as simple as just copying files. I know from the outside it may seem that way but if it was we would've had the site up long ago.

Rollback Post to Revision RollBack
Alexia
Curse Premium
- Squish Elf
- Join Date: 2/28/2011
- Posts: 345
- Minecraft: Erecka
- Member Details
#46 Alexia

View User Profile

Send Message

Posted Jun 26, 2011
Quote from King Korihor
The only part of the summary that I find a little bit offensive is the claim it only took 24 hours to fix the problems. The site has been going up and down for much longer than just one day. Futhermore, I don't really understand why other parts of the "CURSE Network" came up but the Minecraft stuff didn't. I'm sure some of that is due to the size of the Minecraft community and because so much of the community is "officially" using these servers. Just be honest with us, and don't try to sugar coat what must have been a network technician's worst nightmare come true. Don't try to downplay the extent of the problems.

The 24 hour figure was the 24 hours of Windows running check disk on a many terabyte RAID 50 array.(This was one portion of the disks. See Xtek's post above for the full time.)

After about 48 hours we were able to bring the Windows .NET sites online. They were able to mount their volumes from the SAN. The bug listed was only affecting our Linux based sites which includes MCF.

There are three of our top technicians taking a very long sleep out in Atlanta right now after what certainly was a technician's worst nightmare. Plus the five or so hardware company representatives that were there or on the phone.

I have been awake for nearly 24 hours now.

Last edited by Alexia: Jun 26, 2011

Rollback Post to Revision RollBack
Alexia
Curse Premium
- Squish Elf
- Join Date: 2/28/2011
- Posts: 345
- Minecraft: Erecka
- Member Details
#47 Alexia

View User Profile

Send Message

Posted Jun 26, 2011
Quote from ShatteredLight

Well, glad the site's back up anyway. My question is, Did lulzsec play a part in this? I read that lulzsec was screwing around with some kind of minecraft-related thing. I am aware that it wasn't just the minecraft areas of Curse that were effected, but I haven't been able to find anything else about Minecraft that recently had downtime.

No, they had nothing to do with this. This was entirely a hardware failure.

Rollback Post to Revision RollBack
FuzzykinsAP
- Newly Spawned
- Location:
- Join Date: 6/26/2011
- Posts: 2
- Member Details
#48 FuzzykinsAP

View User Profile

Send Message

Posted Jun 26, 2011
Also, guys, realize that a vast majority of the Curse sites were online in Read-Only mode shortly after the servers went down. However, a site like MinecraftForums can't benefit from a read only mode, as forums write data to disks.
It's just a messy, annoying, and ironic failure. There's nothing anyone could have done to predict it, and they acted as fast as they possibly could to get the site back.
Also, realize this could have ended a lot worse. Say the HDD array failed so bad that we lost data. That would be bad, and chances are, we'd be down significantly longer than we were.

Also, you could still play Minecraft while the forums were down. You paid for the game, not unrestricted access to the forums and website.

Last edited by FuzzykinsAP: Jun 26, 2011

Rollback Post to Revision RollBack
Voxel_Day
- Tree Puncher
- Location: 0, 0, 0
- Join Date: 6/3/2011
- Posts: 35
- Member Details
#49 Voxel_Day

View User Profile

Send Message

Posted Jun 26, 2011
We get what we pay for!

Rollback Post to Revision RollBack
Piejacker875
- Void Walker
- Join Date: 10/12/2010
- Posts: 1,538
- Minecraft: Piejacker875
- Member Details
#50 Piejacker875

View User Profile

Send Message

Posted Jun 26, 2011
Damn curse people trippin' over wires... Kidding. But I had quite a lot of nothing to do while it was all down.

Rollback Post to Revision RollBack
Velimas
- Redstone Miner
- Location: X=-50, Y=14, Z = 35
- Join Date: 9/21/2010
- Posts: 629
- Member Details
#51 Velimas

View User Profile

Send Message

Posted Jun 26, 2011
Quote from BrickVoid

Welcome back guys! I do have a few questions that maybe someone knowing about these server array thingamabobs could answer:

1. How likely is it that this sort out of equipment malfunction would occur in the first place?

2. Why did Curse administrators take the long way around and take out four days of site usability instead of immediately getting working copies of sites up so at least they were usable?

3. Would it be possible to shorten the time needed to restore and reinstall everything if the same hardware failure should occur again?

4. In the foreseeable future is there anything Curse could do about making site backups and restorations less dependent on critical hardware components like the ones which failed?

5. It seems possible from reading about the hardware that failed, Curse could remotely store a copy of their entire Atlanta datacentre at one or more other datacentres. Would they be able to put together some plan to do this on a regular basis, and to switch sites if such a failure occurs again?

6. Is there anything Curse could do to prevent a similar failure from taking out so many sites at once? Having a redundancy system that depends on a few critical components seems a lot like putting too many eggs in one basket to me!

And finally a question for the Minecraft forums staff: Could they please look into having the Minecraft Wiki site hosted on a different set of servers? It takes a lot of time to learn about redstone circuitry and I can't learn much while the wiki is down on the mat for four days!

Cheers ...

BrickVoid

1. Unlikely. Hardware is, though, prone to malfunctioning as well as firmware and other software.
2. Because, if you read the post, the SAN server malfunctioned, backup launched and malfunctioned, then so did the support NAS servers. there was no way they could've hosted it.
3. Depends on the issue, as this time it had much to do with the firmware provided by the provider of said hardware.
4. Most probably not.
5. Curse is, after all, not a company with millions of dollars and hundreds of employees. It's simply financially impossible to do such a thing.
6. Unless they reroute all of their structure, no.

Quote from LJDuckE

Minecraft should stay away from that sh*tty curse network. they got their fingers in their asses all day long, while minecraft is actually attempting to get somewhere...

Ditch CURSE, As a Pre-Paying owner of minecraft, i like to be able to access the forums when i need to....Not when f- curse say so...

I don't have anything to say to this. Mojang has no affiliation with the MinecraftForums. They were origininally set up by Quatroking, as well as the wiki. Curse provides quite a relief for the hosting, and it can handle a LOT more traffic now. If you were here in the early stages of launching you would've known how much better the handling is now. I suggest you actually try to host a site with thousands of visitors daily, and then for 24 sites. Think before you put down **** about a "company" That is willing to provide hosting services for a forum for a game, and incorporating it into their network. This is the first time such a thing has happened.

Quote from ShatteredLight »

Well, glad the site's back up anyway. My question is, Did lulzsec play a part in this? I read that lulzsec was screwing around with some kind of minecraft-related thing. I am aware that it wasn't just the minecraft areas of Curse that were effected, but I haven't been able to find anything else about Minecraft that recently had downtime.

No, lulzsec has other things on their hands as far as I'm concerned.

Last edited by Velimas: Jun 26, 2011

Rollback Post to Revision RollBack
lccpor123
- Carpenter
- Join Date: 2/14/2011
- Posts: 56
- Minecraft: lccpor123
- Member Details
#53 lccpor123

View User Profile

Send Message

Posted Jun 26, 2011
Thank god its back and no accounts are lost or any threads are lost (I hope )

Rollback Post to Revision RollBack
Emcitement
- Stuck at Home
- Join Date: 6/15/2011
- Posts: 498
- Member Details
#55 Emcitement

View User Profile

Send Message

Posted Jun 26, 2011
Quote from EpicFishFingers

Damn, you don't know what you have until it's gone...

I must have visited the forum about 10 times while it was down, and it was only down for about a week...

That's it? I refreshed my page only to see that curse error screen like 100 times within the 4 days.

Rollback Post to Revision RollBack
Velimas
- Redstone Miner
- Location: X=-50, Y=14, Z = 35
- Join Date: 9/21/2010
- Posts: 629
- Member Details
#56 Velimas

View User Profile

Send Message

Posted Jun 26, 2011
Quote from infam0use

God , im bloody GLAD to be back on the forums .... what i would do without them , i dont know!

Actually play minecraft/APB/whatever (Play APB.)

Rollback Post to Revision RollBack
Skribbles_JR
- Coal Miner
- Location: Florida
- Join Date: 6/20/2011
- Posts: 110
- Minecraft: Skribbles_JR
- Member Details
#57 Skribbles_JR

View User Profile

Send Message

Posted Jun 26, 2011
Sounds like you guys worked hard to get everthing back up. I'm gald you guys got i back up though, I can't live with Minecraft without the Forums and the Wiki.
Thank god you guys are super computer geniuses.

Rollback Post to Revision RollBack
To post a comment, please login.

Minecraft Forums

Recent Downtime

Comments

Social Media

Services

Resources

Our Communities