Eric Hodel (drbrain) wrote,
Eric Hodel
drbrain

MSNBot sucks

Favorite checkin of the week!

Log:
Tell msnbot not to be a dick

Revision:	4847
Author:		eric
Date:		2005-08-24T13:41:56-0700

Total Changed:	+ 3	- 0
WWW Changed:	+ 3	- 0
Test Changed:	+ 0	- 0

Files Modified: 1
	hugster/trunk/public/robots.txt

Modified: hugster/trunk/public/robots.txt
===================================================================
--- hugster/trunk/public/robots.txt	2005-08-24 20:07:57 UTC (rev 4846)
+++ hugster/trunk/public/robots.txt	2005-08-24 20:41:56 UTC (rev 4847)
@@ -1,6 +1,9 @@
 # robots.txt for www.43things.com
 # 

+User-agent: msnbot
+Crawl-delay: 20
+
 User-agent: *
 Disallow: /admin
 Disallow: /auth


It dropped 10 queries in 1 second and 10 seconds later proceeded with 35 queries in 3 seconds.
Subscribe

  • I love drunks

    Its 2:30 local time, and a pair of drunks are wrestling in the hotel lobby. Well, kinda, mostly they're falling down on each other, but I saved the…

  • Invisible Fight Club

    Today we watched a man fight with an invisible man from our office. I think the invisible man was winning.

  • Here comes the loser

    "I wasn't expecting it" "I was drunk, I fell down" He got punched once in the head and knocked over. Not even any blood! Now he's going on about…

  • Post a new comment

    Error

    default userpic

    Your reply will be screened

    Your IP address will be recorded 

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.
  • 6 comments