hide your robots.txt from visitors and show it only for validated robots

[Bagi Zoltán]

After some hours searching and hacking i have finally found everything to build a solution which makes possible to hide the content of your robots.txt file from visitors but make it display ONLY for validated user agents such as googlebot, Yahoo Slurp and msnbot.

You may find the whole thing very strange why would somebody hide that content? My answer for this question is the following:
That content (folder structure of the core script files) is a private information, and don't want to share it with every script kiddies to make the possible to hurt my site.

How to execute this hack? I will guide through..

1. As first step you need to add these lines to your .htaccess file, or if you don't have create one and upload it to the root domain folder.

RewriteEngine On
RewriteCond %{http_user_agent} !(googlebot|Msnbot|Slurp) [NC]
RewriteRule ^robots\.txt$ http://seo.i-connector.com/  [R,NE,L]
AddHandler application/x-httpd-php .txt

I think i don't have to explain the first row, the second and the thirs says that if you are not one of the three big search engines and want to reach the robots.txt file you will be redirected to the main domain. It is very handy since a lot of people set their homepage as the landing page of 404 errors, so the cloacking won't be recognised. (will talk about the cloacking a bit later as well)
The fourth row make possible that your robots.txt file behave as a php script.

Now you are ready with the first step, lets see what else you need to do.

2. Open a text editor or your favourite web editor application and insert the code below into a new file save as reversedns.php and upload it to your root folder.

PHP Code:

<?php
$ua = $_SERVER['HTTP_USER_AGENT'];
if(stristr($ua, 'msnbot') || stristr($ua, 'googlebot') || stristr($ua, 'yahoo slurp')){
$ip = $_SERVER['REMOTE_ADDR'];
$hostname = gethostbyaddr($ip);
if(!preg_match("/\.googlebot\.com$/", $hostname) &&!preg_match("/search\.live\.com$/", $hostname) &&!preg_match("/crawl\.yahoo\.net$/", $hostname)) {
$block = TRUE;
$URL="/";
header ("Location: $URL");
exit;
} else {
$real_ip = gethostbyname($hostname);
if($ip!= $real_ip){
$block = TRUE;
$URL="/";
header ("Location: $URL");
exit;
} else {
$block = FALSE;
}
}
}
?>

This script can be famaliar for many of you. This is a hacked version of the reversedns.php file which was presented some months ago. According to the hack if the robot can not be validated the script will redirect it to your main domain. So i return back for a min to the cloaking or not cloaking issue. I had to recognise that google are not capable to protect my rankings from exploits, so i have to defend myself, hence i belice it is not a bad cloacking only a protection solution. If somebody mask him/herself as googlebot he/she will fail during this robot valadiation so will be redirected to the main domain via php. No way to recognise the cloacking!

3. And as the last step
Open the robots.txt file you would like to protect and insert the code below to the first line.

PHP Code:

<?php include("reversedns.php"); ?>

You are done, and your robots.txt file is in safe!

Thanks!

[Midlandi]

Nice work Bagi....

[Bagi Zoltán]

Thank you Midi, it took my afternoon.

[temi]

This is brilliant Bagi, thanks for sharing, I know what you created this for originally

Fellow UK WW members, please digg this post

[pow-wow]

Nice post! this is great

[gkd_uk]

Great post - Dugg and rep added

[Bagi Zoltán]

Thank you guys

[temi]

Originally Posted by pow-wow

Nice post! this is great

Biodun, did you digg the article?

[SkinnerW]

Bagi,

Do you mind if we refer to your article in UKWW blog?

Digged, stumbled and rep added

[Bagi Zoltán]

No Skinner, that is absolutely no problem. Thanks for the digg the rep and the stumble