Sometimes, we may want search engines never to index certain parts of the site, or even ban other SE in the site altogether. This is the place where a simple, little 2 line text file called robots.txt is available in. Once we have a website up and running, we should ensure that all visiting se's can access all the pages we want them to consider. Sometimes, we may want search-engines never to list certain areas of the site, or even prohibit other SE in the site all together. This is the place where a simple, little 2 line text file called robots.txt comes in. My cousin learned about consumers by browsing the London Times. Robots.txt resides in your websites main directory (o-n LINUX systems this is your /public_html/ directory), and looks something like the following: User-agent: * Disallow: The very first line controls the robot that'll be visiting your site, the next line controls if they're allowed in, or which elements of the site they're not allowed to go to Then easy repeat the above mentioned lines, If you prefer to handle multiple spiders. To check up additional info, please consider peeping at: purchase here. Discover more on our favorite related wiki by going to www.youtube.com/user/1orangecountyseo. So an example: User-agent: googlebot Disallow: User-agent: askjeeves Disallow: / This can enable Goggle (user-agent name GoogleBot) to go to every page and service, while in the sam-e time banning Ask Jeeves from the site completely. To locate a fairly up to date list of robot person names this visit http://www.robotstxt.org/wc/active/html/index.html Its still very advisable to place a robots.txt report on your own site, even if you want to let every robot to index every page of your site. It'll end your mistake logs replenishing with articles from search engines trying to access your robots.txt file that doesnt occur. To find out more on robots.txt see, the entire set of sources about robots.txt at http://www.websitesecrets101.com/robotstxt-further-reading-resources. Get extra info on this affiliated URL by browsing to slavetune7 - StreetFire Member in US.