Server upgrade complete!

Welcome to the upgraded server! If you notice anything wrong with the board, please let Jeffrey know!Free packets for study/practiceAnnounced tournament dates

Unexpected server downtime

READ THESE THREADS FIRST: Official announcements from board staff can be found here.
Post Reply
Online
User avatar
Jeffrey Hill
Posts: 6358
Joined: Thu Apr 29, 2004 11:00 pm
Location: In between the bright lights and the far unlit unknown (aka Johnson County, KS)
Contact:

Unexpected server downtime

Post by Jeffrey Hill » Thu Dec 23, 2010 8:13 am

Just curious... did anyone try to visit the board between 10:30 and 11:15 last night? If so, was the board down? I've gotten a couple emails that the server that the board is hosted on had problems and so they had to swap out hardware.

In any case, slicehost seems to be very professional about notifying their customers about downtime and getting everything back up in a timely manner. Also, this is the first time in the 13 months I've been with them that the server has had any problems to my knowledge... I know a couple of months ago I looked at the uptime and the server had been running for around 300 days straight. I'm pleased :)

User avatar
Charlie Dees
Posts: 4134
Joined: Tue Jul 25, 2006 11:00 pm
Location: Columbia, MO
Contact:

Re: Was the board down last night?

Post by Charlie Dees » Thu Dec 23, 2010 12:38 pm

Yes it was down.

Online
User avatar
Jeffrey Hill
Posts: 6358
Joined: Thu Apr 29, 2004 11:00 pm
Location: In between the bright lights and the far unlit unknown (aka Johnson County, KS)
Contact:

Re: Was the board down last night?

Post by Jeffrey Hill » Wed Jan 05, 2011 7:21 am

Don't know what happened but it was down this morning... I ended up having to force a hard reboot..... this had better stop happening...

Online
User avatar
Jeffrey Hill
Posts: 6358
Joined: Thu Apr 29, 2004 11:00 pm
Location: In between the bright lights and the far unlit unknown (aka Johnson County, KS)
Contact:

Re: Was the board down last night?

Post by Jeffrey Hill » Sun Mar 20, 2011 12:30 pm

Yeah, it was down again earlier today... I've probably got something set up wrong that's causing this to occasionally happen... :?

Online
User avatar
Jeffrey Hill
Posts: 6358
Joined: Thu Apr 29, 2004 11:00 pm
Location: In between the bright lights and the far unlit unknown (aka Johnson County, KS)
Contact:

Re: Was the board down last night?

Post by Jeffrey Hill » Wed Nov 23, 2011 5:24 pm

:evil: Conveniently enough, it looks like the server stopped responding again this afternoon...... and this time a table was apparently corrupt when I got it back up and had to be repaired. Let me know if you notice anything weird on the board (or the MOQBA or MACA websites).

Online
User avatar
Jeffrey Hill
Posts: 6358
Joined: Thu Apr 29, 2004 11:00 pm
Location: In between the bright lights and the far unlit unknown (aka Johnson County, KS)
Contact:

Re: Was the board down last night?

Post by Jeffrey Hill » Mon Dec 19, 2011 8:01 am

Woohoo, it happened again... at least this time I might have a lead on what the problem might be.

Online
User avatar
Jeffrey Hill
Posts: 6358
Joined: Thu Apr 29, 2004 11:00 pm
Location: In between the bright lights and the far unlit unknown (aka Johnson County, KS)
Contact:

Re: Was the board down last night?

Post by Jeffrey Hill » Fri Jan 27, 2012 9:04 am

I was thinking to myself this morning before getting on my computer that it would be inconvenient for the server to go down this weekend... and it looks like the server had been down since around 3:40 this morning. I'm going to have to start manually restarting the server about once every 2-3 weeks as a workaround to keep this from happening since it looks like it's happening about once a month now... I apologize for the repeated downtime.

Online
User avatar
Jeffrey Hill
Posts: 6358
Joined: Thu Apr 29, 2004 11:00 pm
Location: In between the bright lights and the far unlit unknown (aka Johnson County, KS)
Contact:

Re: Unexpected server downtime

Post by Jeffrey Hill » Sun Feb 19, 2012 7:57 am

Server crashed sometime after 2:30 this morning. Guess I'll make it every other week on the planned reboots. My hosting plan is supposed to be going through a transition to a different pricing model soon so once that happens I'll hopefully be able to upgrade to a better plan that I think will help with this recurring problem.

Online
User avatar
Jeffrey Hill
Posts: 6358
Joined: Thu Apr 29, 2004 11:00 pm
Location: In between the bright lights and the far unlit unknown (aka Johnson County, KS)
Contact:

Re: Unexpected server downtime

Post by Jeffrey Hill » Mon Feb 20, 2012 1:28 pm

REALLY... less than 36 hours?! I don't know what has changed that is suddenly causing the server to crash so frequently but I'm definitely going to have to do something about it soon. :evil: :evil: :evil: :evil: :evil: :evil: :evil: :evil: :evil:

Online
User avatar
Jeffrey Hill
Posts: 6358
Joined: Thu Apr 29, 2004 11:00 pm
Location: In between the bright lights and the far unlit unknown (aka Johnson County, KS)
Contact:

Re: Unexpected server downtime

Post by Jeffrey Hill » Wed Mar 21, 2012 6:51 am

The server has been acting weird this morning so I wouldn't be totally shocked if the site becomes unresponsive at some point today... :(

Online
User avatar
Jeffrey Hill
Posts: 6358
Joined: Thu Apr 29, 2004 11:00 pm
Location: In between the bright lights and the far unlit unknown (aka Johnson County, KS)
Contact:

Re: Unexpected server downtime

Post by Jeffrey Hill » Sun Apr 22, 2012 5:15 pm

Site went down sometime between 2 and 3:45 pm...

Online
User avatar
Jeffrey Hill
Posts: 6358
Joined: Thu Apr 29, 2004 11:00 pm
Location: In between the bright lights and the far unlit unknown (aka Johnson County, KS)
Contact:

Re: Unexpected server downtime

Post by Jeffrey Hill » Wed Apr 25, 2012 9:27 pm

OK, so it's clear that the server isn't able to handle an apparent spike in traffic ("Most users ever online was 608 on Sun Apr 22, 2012 13:34" was right about the time it went down on Sunday)... so I'm going to upgrade this weekend and hopefully the extra capacity will solve this problem, so there will be some downtime probably Friday evening. Sorry once again for the inconvenience.

Online
User avatar
L-Town Expatriate
Posts: 5771
Joined: Thu Apr 22, 2004 11:00 pm
Location: Riding a Mule down the Katy Trail to the State Fair

Re: Unexpected server downtime

Post by L-Town Expatriate » Thu Apr 26, 2012 11:00 pm

U. Lou Sthagaim wrote:OK, so it's clear that the server isn't able to handle an apparent spike in traffic ("Most users ever online was 608 on Sun Apr 22, 2012 13:34" was right about the time it went down on Sunday)... so I'm going to upgrade this weekend and hopefully the extra capacity will solve this problem, so there will be some downtime probably Friday evening. Sorry once again for the inconvenience.
How the heck did we get 608 users? Sounds like a spambot barrage.

Online
User avatar
Jeffrey Hill
Posts: 6358
Joined: Thu Apr 29, 2004 11:00 pm
Location: In between the bright lights and the far unlit unknown (aka Johnson County, KS)
Contact:

Re: Unexpected server downtime

Post by Jeffrey Hill » Tue Oct 01, 2013 7:49 pm

Short version: I've made some more configuration changes to hopefully completely fix the out of memory instances that have caused the server to go down intermittently. Let me know if you notice the site getting significantly slower or anything like that.

Long, semi-technical version: Today when I got home from work the site wasn't responding, but I was able to log in remotely, which usually didn't work when the site went down in the past (although it was incredibly slow). I dug around trying to figure out what was going on, and all of a sudden the server rebooted. It turns out that the server was in such a bad state that it was constantly swapping memory to and from disk that it was adversely affecting the servers of other customers running on the same physical host server, and so the host initiated a hard reboot (which I would have done immediately had I not been able to log in remotely). Needless to say, I got a warning from my host to figure out what caused this.

I think I had the server configured to accept way too many simultaneous connections and some bot tried to make tons of requests at once. (The access logs appear to indicate this was the case, and I have blocked the offending IP.) A while back I had already reduced the maximum number of clients, but it apparently wasn't enough. I think that reduction may have actually contributed to the server going into a constant swap state rather than crashing entirely. So, I have slashed the number of simultaneous clients even further. Hopefully this won't adversely affect your experience on this board, moaca.org, and moqba.org, but if you notice that the sites are significantly slower or anything else doesn't seem to be working right, let me know.

I hope this finally fixes the problem, because if this exact situation happens again it sure sounds like it's going to be a lot more urgent for me to solve it once and for all..

Post Reply