Team Missouri A finishes sixth in all-star tournament!
Congratulations to Team Missouri A for earning sixth place out of twenty-nine teams in the 2018 National All-Star Academic Tournament! Team Missouri B finished in nineteenth place.
View unanswered posts | View active topics It is currently Tue Jun 19, 2018 2:49 am



Post new topic Reply to topic  [ 14 posts ] 
Unexpected server downtime 
Author Message
User avatar

Joined: Thu Apr 29, 2004
Posts: 6280
Location: In between the bright lights and the far unlit unknown (aka Johnson County, KS)
Post Unexpected server downtime
Just curious... did anyone try to visit the board between 10:30 and 11:15 last night? If so, was the board down? I've gotten a couple emails that the server that the board is hosted on had problems and so they had to swap out hardware.

In any case, slicehost seems to be very professional about notifying their customers about downtime and getting everything back up in a timely manner. Also, this is the first time in the 13 months I've been with them that the server has had any problems to my knowledge... I know a couple of months ago I looked at the uptime and the server had been running for around 300 days straight. I'm pleased :)


Thu Dec 23, 2010 8:13 am
Profile WWW
User avatar

Joined: Tue Jul 25, 2006
Posts: 4134
Location: Columbia, MO
Post Re: Was the board down last night?
Yes it was down.


Thu Dec 23, 2010 12:38 pm
Profile
User avatar

Joined: Thu Apr 29, 2004
Posts: 6280
Location: In between the bright lights and the far unlit unknown (aka Johnson County, KS)
Post Re: Was the board down last night?
Don't know what happened but it was down this morning... I ended up having to force a hard reboot..... this had better stop happening...


Wed Jan 05, 2011 7:21 am
Profile WWW
User avatar

Joined: Thu Apr 29, 2004
Posts: 6280
Location: In between the bright lights and the far unlit unknown (aka Johnson County, KS)
Post Re: Was the board down last night?
Yeah, it was down again earlier today... I've probably got something set up wrong that's causing this to occasionally happen... :?


Sun Mar 20, 2011 12:30 pm
Profile WWW
User avatar

Joined: Thu Apr 29, 2004
Posts: 6280
Location: In between the bright lights and the far unlit unknown (aka Johnson County, KS)
Post Re: Was the board down last night?
:evil: Conveniently enough, it looks like the server stopped responding again this afternoon...... and this time a table was apparently corrupt when I got it back up and had to be repaired. Let me know if you notice anything weird on the board (or the MOQBA or MACA websites).


Wed Nov 23, 2011 5:24 pm
Profile WWW
User avatar

Joined: Thu Apr 29, 2004
Posts: 6280
Location: In between the bright lights and the far unlit unknown (aka Johnson County, KS)
Post Re: Was the board down last night?
Woohoo, it happened again... at least this time I might have a lead on what the problem might be.


Mon Dec 19, 2011 8:01 am
Profile WWW
User avatar

Joined: Thu Apr 29, 2004
Posts: 6280
Location: In between the bright lights and the far unlit unknown (aka Johnson County, KS)
Post Re: Was the board down last night?
I was thinking to myself this morning before getting on my computer that it would be inconvenient for the server to go down this weekend... and it looks like the server had been down since around 3:40 this morning. I'm going to have to start manually restarting the server about once every 2-3 weeks as a workaround to keep this from happening since it looks like it's happening about once a month now... I apologize for the repeated downtime.


Fri Jan 27, 2012 9:04 am
Profile WWW
User avatar

Joined: Thu Apr 29, 2004
Posts: 6280
Location: In between the bright lights and the far unlit unknown (aka Johnson County, KS)
Post Re: Unexpected server downtime
Server crashed sometime after 2:30 this morning. Guess I'll make it every other week on the planned reboots. My hosting plan is supposed to be going through a transition to a different pricing model soon so once that happens I'll hopefully be able to upgrade to a better plan that I think will help with this recurring problem.


Sun Feb 19, 2012 7:57 am
Profile WWW
User avatar

Joined: Thu Apr 29, 2004
Posts: 6280
Location: In between the bright lights and the far unlit unknown (aka Johnson County, KS)
Post Re: Unexpected server downtime
REALLY... less than 36 hours?! I don't know what has changed that is suddenly causing the server to crash so frequently but I'm definitely going to have to do something about it soon. :evil: :evil: :evil: :evil: :evil: :evil: :evil: :evil: :evil:


Mon Feb 20, 2012 1:28 pm
Profile WWW
User avatar

Joined: Thu Apr 29, 2004
Posts: 6280
Location: In between the bright lights and the far unlit unknown (aka Johnson County, KS)
Post Re: Unexpected server downtime
The server has been acting weird this morning so I wouldn't be totally shocked if the site becomes unresponsive at some point today... :(


Wed Mar 21, 2012 6:51 am
Profile WWW
User avatar

Joined: Thu Apr 29, 2004
Posts: 6280
Location: In between the bright lights and the far unlit unknown (aka Johnson County, KS)
Post Re: Unexpected server downtime
Site went down sometime between 2 and 3:45 pm...


Sun Apr 22, 2012 5:15 pm
Profile WWW
User avatar

Joined: Thu Apr 29, 2004
Posts: 6280
Location: In between the bright lights and the far unlit unknown (aka Johnson County, KS)
Post Re: Unexpected server downtime
OK, so it's clear that the server isn't able to handle an apparent spike in traffic ("Most users ever online was 608 on Sun Apr 22, 2012 13:34" was right about the time it went down on Sunday)... so I'm going to upgrade this weekend and hopefully the extra capacity will solve this problem, so there will be some downtime probably Friday evening. Sorry once again for the inconvenience.


Wed Apr 25, 2012 9:27 pm
Profile WWW
User avatar

Joined: Thu Apr 22, 2004
Posts: 5577
Location: Riding a Mule down the Katy Trail to the State Fair
Post Re: Unexpected server downtime
U. Lou Sthagaim wrote:
OK, so it's clear that the server isn't able to handle an apparent spike in traffic ("Most users ever online was 608 on Sun Apr 22, 2012 13:34" was right about the time it went down on Sunday)... so I'm going to upgrade this weekend and hopefully the extra capacity will solve this problem, so there will be some downtime probably Friday evening. Sorry once again for the inconvenience.

How the heck did we get 608 users? Sounds like a spambot barrage.


Thu Apr 26, 2012 11:00 pm
Profile
User avatar

Joined: Thu Apr 29, 2004
Posts: 6280
Location: In between the bright lights and the far unlit unknown (aka Johnson County, KS)
Post Re: Unexpected server downtime
Short version: I've made some more configuration changes to hopefully completely fix the out of memory instances that have caused the server to go down intermittently. Let me know if you notice the site getting significantly slower or anything like that.

Long, semi-technical version: Today when I got home from work the site wasn't responding, but I was able to log in remotely, which usually didn't work when the site went down in the past (although it was incredibly slow). I dug around trying to figure out what was going on, and all of a sudden the server rebooted. It turns out that the server was in such a bad state that it was constantly swapping memory to and from disk that it was adversely affecting the servers of other customers running on the same physical host server, and so the host initiated a hard reboot (which I would have done immediately had I not been able to log in remotely). Needless to say, I got a warning from my host to figure out what caused this.

I think I had the server configured to accept way too many simultaneous connections and some bot tried to make tons of requests at once. (The access logs appear to indicate this was the case, and I have blocked the offending IP.) A while back I had already reduced the maximum number of clients, but it apparently wasn't enough. I think that reduction may have actually contributed to the server going into a constant swap state rather than crashing entirely. So, I have slashed the number of simultaneous clients even further. Hopefully this won't adversely affect your experience on this board, moaca.org, and moqba.org, but if you notice that the sites are significantly slower or anything else doesn't seem to be working right, let me know.

I hope this finally fixes the problem, because if this exact situation happens again it sure sounds like it's going to be a lot more urgent for me to solve it once and for all..


Tue Oct 01, 2013 7:49 pm
Profile WWW
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 14 posts ] 


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Forum style by STSoftware for PTF.
Translated by MaĆ«l Soucaze © 2010 phpBB.fr