what is the Robots.txt?

[Danielnash]

Hi everybody,


tell me about Robots.txt file?


Thanks

[justinmarch]

See this The Web Robots Pages

[RDragos]

Robots.txt tells the crawlers (search engines) which part of the site is available for them and which not, and also tells which search engines are allowed to index you website.

[Alfredwinston]

Google it dude and you can find lots of info regarding it. Why making a thread for these simple things. Anyhow let me say, Robots.txr is the instructor for search engine bots that crawl ur site. It says which page to be indexed, which to be skipped etc..,

[airforce1]

Simply speaking, robots.txt is a file under your site root for guiding search engine crawlers how to crawl your sites. But I think it might also a security risk for misusing this file.

[RDragos]

Simply speaking, robots.txt is a file under your site root for guiding search engine crawlers how to crawl your sites. But I think it might also a security risk for misusing this file.

Why do you consider it a security risk ?

[extomas]

The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code.

[shannonbburns]

If you want to hide your site from search engine or you want to hide particular directory or file of your site from search engines crawler or want to give permission to particular search engine for indexing or link follow... then you need to use Robots.txt file.

[vizion]

In short: A file that contains a list of pages that the webmaster doesn’t want indexed by a search engine spider.

[ranacseruet]

it is used to restrict search engine robots what to crawl and what not to... such as you might not want search engine to crawl your admin panel page's secret information.....