Googling

11 July 2004

I see that Amit has an interesting comment on Google. He says that:

Key words “armi reflective” appears number two in Google amazing, “reflective practitioner” comes up about 6th … amazing! I’m a little surprise as I’ve been having problems getting google to trawl the site correctly. Although Google comes to http://www.thereflectivepractitioner.org/ every day or so, it never crawls through the whole web site. It just reads the robots.txt file and the main index of The reflectivepractitioner and then stops. I can’t figure out what’s going-on.

The robot exclusion file, that is to say robots.txt, shouldn’t stop it crawling. The meta data in the index.html page, i.e.

<meta name="robots" content="index,follow,noarchive" \> 

shouldn’t stop it—so what’s happening? I emailed the folks at Google and got a very general reply back. I should try emailing them again. In the meantime, if anyone has an idea why the site isn’t being crawled—please let me know.