Archive for February, 2010

Create a robots.txt file for your website

After you’ve created an xml sitemap, create a new text file in notepad and add the following to guide the google bot to your sitmap. this robots file also disallows the indexing of paths to the pdf and word docs on your site as well – this helps stop out of date links appearing in Google.

User-agent: *
disallow: /*.pdf$

disallow: /*.DOC$

User-agent: Googlebot

disallow: /*.pdf$

disallow: /*.DOC$

SITEMAP:  http://www.domainname.co.uk/sitemap.xml