Eric Hodel (drbrain) wrote,
Eric Hodel
drbrain

MSNBot sucks

Favorite checkin of the week!

Log:
Tell msnbot not to be a dick

Revision:	4847
Author:		eric
Date:		2005-08-24T13:41:56-0700

Total Changed:	+ 3	- 0
WWW Changed:	+ 3	- 0
Test Changed:	+ 0	- 0

Files Modified: 1
	hugster/trunk/public/robots.txt

Modified: hugster/trunk/public/robots.txt
===================================================================
--- hugster/trunk/public/robots.txt	2005-08-24 20:07:57 UTC (rev 4846)
+++ hugster/trunk/public/robots.txt	2005-08-24 20:41:56 UTC (rev 4847)
@@ -1,6 +1,9 @@
 # robots.txt for www.43things.com
 # 

+User-agent: msnbot
+Crawl-delay: 20
+
 User-agent: *
 Disallow: /admin
 Disallow: /auth


It dropped 10 queries in 1 second and 10 seconds later proceeded with 35 queries in 3 seconds.
Subscribe
  • Post a new comment

    Error

    default userpic

    Your reply will be screened

    Your IP address will be recorded 

  • 6 comments