11 July 2004

I see that Amit has an interesting comment on Google. He says that:

Key words “armi reflective” appears number two in Google amazing, “reflective practitioner” comes up about 6th … amazing! I’m a little surprise as I’ve been having problems getting google to trawl the site correctly. Although Google comes to every day or so, it never crawls through the whole web site. It just reads the robots.txt file and the main index of The reflectivepractitioner and then stops. I can’t figure out what’s going-on.

The robot exclusion file, that is to say robots.txt, shouldn’t stop it crawling. The meta data in the index.html page, i.e.

<meta name="robots" content="index,follow,noarchive" \> 

shouldn’t stop it—so what’s happening? I emailed the folks at Google and got a very general reply back. I should try emailing them again. In the meantime, if anyone has an idea why the site isn’t being crawled—please let me know.