June 12th, 2005


Bumpy Ride

I've been upgrading our online kitchen-sink box to FreeBSD 5. This will be the hardest one of the four (I hope!).

When Rackspace says 'four hour maintenence window' they mean 4 hours. Which is cool, but it was a bit unexpected.

Then they only had FreeBSD 5.3, not 5.4 like I wanted. It would have been nice to know that ahead of time, but that's just a couple extra hours.

So, Friday morning I have a working FreeBSD 5.4 up and running with an SMP kernel. It was very smooth. Then I installed all the packages it needed.

Next step was copying the machine's replacement configs over, which was not terribly difficult (yay version control!).

Then I spent several hours trying to figure out why apache wasn't logging. (Apache buffered logs + chronolog == suck).

mysqld decided it wanted to corrupt its user table for some reason, but eventually that sorted itself out.

Moving all our internal tools over was the hardest. Some of our stuff hasn't been touched/prettied in months. I found we had 3.2G of session logs, not because we had that much traffic, but because we had 3 copies of them! That is something to be remedied in the future. Now I'm down to the last step, wating for DNS to resolve and getting awstats caught back up.

awstats sucks at IP resolution

So, the biggest problem we have with awstats is IP address resolution. awstats has a script that merges and (allegedly) resolves IP addresses, but it appears it does this in a single process, so any IP address that blocks takes forever.

I've written two lovely scripts to handle this, one that merges two logs, and one that resolves IPs "quickly". The merging one is simple, it just opens N filenames (from ARGV), parses out the date, and sticks the line into an Array. Then I pull out the smallest time and shove the line onto STDOUT.

The input logs must be sorted for it to work, but I awstats can already take care of that stuff (or so it says).

The DNS resolver uses Ruby's resolv.rb and spawns 100 threads to do DNS lookup. Instead of awstats taking 4-5 hours to do DNS lookups, my multithreaded resolver takes about 40 minutes.

The problem with DNS lookups isn't really that its slow to look up names, its that its slow to lookup names that don't exist. Instead of caching hits, I should cache the misses, that way I can avoid looking up failed names over and over.

Cutting down DNS resolution time would make access logs interesting to be processed on an hourly instead of daily basis. Its been an idea of mine to see how traffic for the site changes on a shorter-term basis than monthly, which is what most stats packages give you.