Create a robots.txt file for your website
After you’ve created an xml sitemap, create a new text file in notepad and add the following to guide the google bot to your sitmap. this robots file also disallows the indexing of paths to the pdf and word docs on your site as well – this helps stop out of date links appearing in Google.
User-agent: *
disallow: /*.pdf$
disallow: /*.DOC$
User-agent: Googlebot
disallow: /*.pdf$
disallow: /*.DOC$
SITEMAP: http://www.domainname.co.uk/sitemap.xml
