So today I set up a test environment using the FreeBSD 5 database box so I can figure out what went wrong.
The most notable difference is that the FreeBSD 4 box has linuxthreads, while the FreeBSD 5 is using libpthread.
The objectionable behavior is an insane amount of context switches and syscalls. Normal operation (3 crawlers) gives between 8-40k context switches, and 4-20k syscalls. Misbehavior is 100k+ context switches and 40k+ syscalls.