Site Crawlers - Webmaster Forums UK SEO SEM Webmaster Community Forum

#1 (**permalink**) 05-15-2005, 05:17 PM

How do I know if my sites been crawled? I've heard Duke mention seeing that he was crawled or being crawled, now that I have one of my sites up and the other going up this week, I'd like to start keeping track of this.

temi · #2 (**permalink**) 05-15-2005, 06:29 PM

There are lots of ways to find out if your has been crawled but the two I favour are:
1. Look in the raw logs. Raw log look something like this,

it shows the search engines that has been "crawling" your site. But because you have just put your site online you may not see your first visit for a day or so unless you have links from very strong sites.

2. If your hosting package come with Webalize or other graphical stats package you can also see visits from crawlers, see attached image for example

Thanks

#3 (**permalink**) 05-15-2005, 06:58 PM

I do that but I also go into the site admin panel, click on forums and it shows me the IP of everyone who's online. Click on the IP and it will take you to Network Tools.com and already have that IP entered. Click submit and it tells you who's at the other end.

temi · #4 (**permalink**) 05-15-2005, 07:31 PM

Are you talking about phpnuke Ken?

#5 (**permalink**) 05-15-2005, 11:15 PM

Ya it's the admin panel in all phpnuke websites.