Log: Tell msnbot not to be a dick Revision: 4847 Author: eric Date: 2005-08-24T13:41:56-0700 Total Changed: + 3 - 0 WWW Changed: + 3 - 0 Test Changed: + 0 - 0 Files Modified: 1 hugster/trunk/public/robots.txt Modified: hugster/trunk/public/robots.txt =================================================================== --- hugster/trunk/public/robots.txt 2005-08-24 20:07:57 UTC (rev 4846) +++ hugster/trunk/public/robots.txt 2005-08-24 20:41:56 UTC (rev 4847) @@ -1,6 +1,9 @@ # robots.txt for www.43things.com # +User-agent: msnbot +Crawl-delay: 20 + User-agent: * Disallow: /admin Disallow: /auth
It dropped 10 queries in 1 second and 10 seconds later proceeded with 35 queries in 3 seconds.
