Eric Hodel (drbrain) wrote,
Eric Hodel
drbrain

MSNBot sucks

Favorite checkin of the week!

Log:
Tell msnbot not to be a dick

Revision:	4847
Author:		eric
Date:		2005-08-24T13:41:56-0700

Total Changed:	+ 3	- 0
WWW Changed:	+ 3	- 0
Test Changed:	+ 0	- 0

Files Modified: 1
	hugster/trunk/public/robots.txt

Modified: hugster/trunk/public/robots.txt
===================================================================
--- hugster/trunk/public/robots.txt	2005-08-24 20:07:57 UTC (rev 4846)
+++ hugster/trunk/public/robots.txt	2005-08-24 20:41:56 UTC (rev 4847)
@@ -1,6 +1,9 @@
 # robots.txt for www.43things.com
 # 

+User-agent: msnbot
+Crawl-delay: 20
+
 User-agent: *
 Disallow: /admin
 Disallow: /auth


It dropped 10 queries in 1 second and 10 seconds later proceeded with 35 queries in 3 seconds.
Subscribe
  • Post a new comment

    Error

    default userpic

    Your reply will be screened

    Your IP address will be recorded 

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.
  • 6 comments