Robot Exclusion Protocol

[temi]

Web publishers often ask us how they can maximize their visibility on the web. Much of this has to do with search engine optimization -- making sure a publisher's content shows up on all the search engines.

full post

[searchtools]

I'm writing an article on the changes in the Robots Exclusion Protocol for infotoday.com, and would very much like to know what SEO people think about them. Is it going to make your lives a lot easier, to have a protocol to write to for all three big crawlers, and I'd guess most of the small ones too? Are there any downsides you can see? Is it just ho-hum business as usual? I can't find a lot of discussion, so am asking here. Please note whether you would or would not like to be quoted.

[temi]

I think people are probably not that excited because there is nothing new or exciting robot exclusion will do that robots.txt does not do at the moment or is there?
I also know that some rouge robots do not follow robots.txt, will the robots exclusion protocol address that?

Thanks and good luck with you article, when its own, let me know and I will be happy to draw some attention to it via this forum and my blogs.

Originally Posted by searchtools

I'm writing an article on the changes in the Robots Exclusion Protocol for infotoday.com, and would very much like to know what SEO people think about them. Is it going to make your lives a lot easier, to have a protocol to write to for all three big crawlers, and I'd guess most of the small ones too? Are there any downsides you can see? Is it just ho-hum business as usual? I can't find a lot of discussion, so am asking here. Please note whether you would or would not like to be quoted.

[searchtools]

Not excited is a useful response as well. Would you like to be quoted? private message or email is fine.

And no, the robots.txt cannot do anything about a rogue robot except help identify it. If you have a page that's disallowed by robots.txt, and a bot goes there, then you know it's a bad one. I suppose you could automate a system to ban that IP address, at least for a while.

[temi]

You are welcomed to quote me searchtools, I can send my quot to you via PM

Thanks

Originally Posted by searchtools

Not excited is a useful response as well. Would you like to be quoted? private message or email is fine.

And no, the robots.txt cannot do anything about a rogue robot except help identify it. If you have a page that's disallowed by robots.txt, and a bot goes there, then you know it's a bad one. I suppose you could automate a system to ban that IP address, at least for a while.