On Mon, Jul 29, 2013 at 6:48 AM, Peter Schlumpf <pschlumpf_at_gmail.com> wrote:
> And yet, most of the world seems to go to Google for their information
> anymore.
>
I agree hence my catalogue is on the web in a form google can crawl
unlike some who think exposing their search box is all they need to do.
I have a number of entry points all linked from the front page.
maincat all the items
fondcat all the collections
and a few targeted searches where google will find other links
Yes it means google (and other engines) will find a lot of urls
meaning it will take a lot of pages a day
it is important you can handle that, although google seems to have its
own regulating mechanism
depending on rates your site can handle (never seen documentation but
I do watch what is going on with webmaster tools).
I am a small site and self host on an ADSL line so one would think I
would be overwhelmed by google and the other search engines but seem
to be ok, just have to make my search pages fast enough.
It sees a change to the site and ups its crawl rate then drops back to normal
http://www.collection.archivist.info/Screenshot-google.png
Dave Caroline
Received on Mon Jul 29 2013 - 03:28:10 EDT
