February 1st, 2005


Helping the little people

Today I found an email with the following headers (and more!) in my mailbox:
Return-Path: <AOShinyStar@aol.com>
X-Original-To: drbrain@segment7.net
Delivered-To: drbrain@magnesium.net
Received: from imo-m26.mx.aol.com (imo-m26.mx.aol.com [])
	by toxic.magnesium.net (Postfix) with ESMTP id 4E5D4DA8AA
	for <drbrain@segment7.net>; Mon, 31 Jan 2005 22:52:50 -0800 (PST)
Received: from AOShinyStar@aol.com
	by imo-m26.mx.aol.com (mail_out_v37_r3.8.) id o.142.3e5fa89a (4459)
	 for <drbrain@segment7.net>; Tue, 1 Feb 2005 01:52:44 -0500 (EST)
From: AOShinyStar@aol.com
Date: Tue, 1 Feb 2005 01:52:44 EST
Subject: could you help me with something?
To: drbrain@segment7.net
Since it seemed to be a real AOLer, I gave the following reply:
On 31 Jan 2005, at 22:52, AOShinyStar@aol.com wrote:
> Hello, my name is Alina.

Hello Alina.

> I have an essay to do which called "a persuasive essay" for social=20
> studies.

The first thing you have succeeded in doing is persuading me to help=20
you, however minorly, in your essay writing.

> The teacher asks us to choose an amendment out of the 27 amendments,

> and write our position of it.(whether we against it or for it).

That's great!

> So I was wondering which amendments are the most controversial,
> because I can't choose.

Well, you can ask a whole lot of people that question, and they'll all

give you different answers.  So my answer to you is to look here:


You can find all sorts of Supreme Court decisions about all kinds of
things.  My answer for which one is most controversial will be
different from what you think is most controversial.

If you can't find an amendment you (dis)like, try reading the actual
constitution again.

>  Please answer this quastion.


> and sorry for the bad grammar.

Your grammar is fine, but your spelling could use some improvement.

Eric Hodel - drbrain@segment7.net - http://segment7.net
FEC2 57F1 D465 EB15 5D6E  7C11 332A 551C 796C 9F04


What we did wrong

Today we were down for at least 25 minutes (according to Nagios, actual time was probably longer).

There were three contributing factors to our down-time:

  • Connecting to the production DB from development
  • Using Rails' dependency reloading in a persistent process
  • Using Rails' multiple-connection feature with a class under development

Rails does not clean up ActiveRecord connections in child classes if they are automatically reloaded. Instead the connections just 'leak' and a new connection is created.

Because we were connecting to the production DB, we ended up using more connections than the DB could provide (we actually need to trim this), it wants to use 1.4G with 250 connections, which is too much and too many.

The DB ran out of something (the logs were not verbose enough to tell us what) and when we tried to kill it, it wedged itself trying to shut down. A forced restart brought things back up, and restarting Apache brought us back online.

Fortunately, we will no longer be attempting to connect to the production DB from development machines (this was a not-so-subtle hint that this is a Bad Thing™), eliminating all of the above factors as possible future failure points.

The price to pay will simply be one of developer ease-of-use, which we'll have to live with. (I've been 100% comfortable with it all the time.) It also means that I won't ever need an internet connection to do useful work on the site.