On Thu, Feb 23, 2012 at 2:07 PM, Baksik, Corinna M. <corinna_baksik_at_harvard.edu> wrote: > We don't allow crawlers because it has caused serious performance issues in the past. There are simple solutions that may help for the performance problem. You can use Crawl-delay: http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive You can set Google's crawl rate in Webmaster Tools as well. JasonReceived on Thu Feb 23 2012 - 15:05:59 EST