Last days was painful. Not only to you, when you can not access opensubtitles – page was not loaded, you got overload error or…, but also to me, when I was doing everything to get things in right order. Now we can say – it was worth of it. Too bad, it wasn’t ready 3 days ago, during weekend, when are peak times…
So what’s happened? We are using mysql in our database servers, and one of the “good” things in mysql 5.1 is, it sometimes hangs up, when is query_cache enabled. This is not-so-known “feature”, when running on FreeBSD servers. So we run our servers without query_cache, which is really important, so we have to move our cached data to memcached.
Website was slow, because “slow database”, it just don’t have enough speed to answer around 1000-2000 queries per second. Problem with databases (after optimizing SQL queries, indexes, setup of mysql…) is usually access time to a data. We upgraded storage to SAS 15k raid1, downgraded mysql to 5.0, and added some CPU power.
Now it seems, website is loading in lightning speed, so I hope this will last at least next couple of months. Next server side thing is – we will upgrade FreeBSD to 8.0, which should give more power to running applications.
Now I can more focus on coding new features 🙂
UPDATE: Now database server is running FreeBSD 8.0, after installing it didn’t boot, so it takes couple of hours to fix this issue. Hopefully there won’t be anymore downtime like this.