MySQL problems

[Arnie]

We have (as I've posted in many many places all over the sop and the news pages) got issues on the new machine regarding the database (mysql).

 

We are on the new box right now which, when mysql hasn't crashed, runs wonderfully. At the moment there is an as yet undiagnosed fault with is causing mysql to fail.

 

People are looking into it and trying to figure out the issue, however at this time no solution or evidence to what the problem even is has been found.

 

This is being looked into. As soon as we get this fixed, we'll be on 99.99% uptime. As things are, we're waiting on techs at the datacentre to fix the issue as it's is beyond a site admin problem.

 

PLEASE do not post about database connection problems, search facilities failing, or anything else that has "mysql" or "database" in it. We know there's a problem and are fixing it.

[Arnie]

This is an update in relation to the problems we've been having with mysql lately (the database behind the site). The technicians can't pinpoint the problem, but it seems that the most likely problem is a faulty memory card or a problem with the hard drives.

 

With this in mind I've asked them to do these checks for us. At the moment I've asked them to pull the rack around 0200 GMT in the morning. This will result in the server being shutdown safely, physically pulled out of the shelf and then diagnosed by a technician. Hopefully we'll find it's a memory card that simply needs replacing.

 

I will update with some news here as soon as I know more.

 

 

(as posted on the news pages)

[Arnie]

Downtime 0200-0400 GMT confirmed for maintenance. Server will be offline and disconnected for this period for extended tests.

[Arnie]

Fortunately/unfortunately hardware tests drew a blank and no problems were found.. work continues.

[Arnie]

Complete data rebuild of post table (using repair table use_frm option) now complete. Work continues.

 

I know that this is largely irrelevant to everyone, however I wanted to post updates to let you guys know that we are working on the issue.

[Arnie]

Topics table repaired in the same fashion. Minor downtime would have occured as mysql was disabled to backup and back the data up elsewhere before running the repair.

 

Now goin to leave thing and watch the logs and see how that's affected things.

[Arnie]

Okay, so far over an hour of uptime on mysql and not a single error thrown up. It looks good.

[ferretkiller]

Hey Man, Thanks for all your hard work. I understand how hard it is to keep sites this big running effeciently. Keep up the good work.

[Chris North]

So, if I didn't miss something, Arnies was down a whole lot for the past week? Every time I tried to connect, I either got a web browser error page or that error screen. I was almost sure it was something on my end, but it was happening to everyone, yes?

 

Edit: Okay, it was definitely something with the site. Thanks for working on fixing it!

Edited April 27, 2007 by ChrisNorthInGer

[Arnie]

8 hours of no errors now since last fix, still looks good, no errors or even a hint of a slowdown.

[Graeme]

As soon as we get this fixed, we'll be on 99.99% uptime

<{POST_SNAPBACK}>

 

What about the other 00.01 %

 

Seriosuly though thanks for updating and letting me know, I too thought it was a problem at my end. Thanks for the hard work.

 

EDIT Also thanks for changing my name mate. Only just noticed

Edited April 27, 2007 by Graeme

[PlasticMag]

I just opened 32 wndows, no errors.

[Arnie]

21 hours, no errors or issues so far. Good to hear that it isn't slowing down at all

[Arnie]

So far so good, all looks pretty cosy at the moment. The recent fixes appear to have fixed all the wierd issues we did have.

 

We're now serving around 12-18Gb of data per day at the moment from the new server. Page views are in the region of 140,000 per day and increasing after the move (we had a lull while the data moved servers and all the problems were fixed). In short.. the new box is streaming stuff wonderfully!