Search Engine Positioning
|
|
|
 |
Guide Search Engines With This Industry Standard Web File - Robots.txt
When search engine robots or spiders come to visit your web pages, they first look for a web file called robots.txt.
The robots.txt file gives them important guidance on what pages of your website you would like to have cataloged for public viewing and other pages that you do not want reported to the public.
Your Home Page would be a good example of a page you WANT cataloged and made public. However, admin, members only, orders, contactus.txt and pricesheet.pdf represent areas you very well might NOT WANT to be catalogued and publicly displayed.
Robots.txt files, properly created, will guide search engines to the pages you want to be public and help them avoid the pages you want to be private. You'd be surprised how many websites do not even have a robots.txt file: does yours?
|
