How to get my page crawled - Webmaster Forums UK SEO SEM Webmaster Community Forum

rhyswynne · #1 (**permalink**) 06-10-2008, 10:29 PM

Hi everybody

One of my pages hasn't been crawled for at least a month now, and I've added a lot of content to it. It has a few promenant links on it (in my blog, on a few websites i own, squidoo lenses etc.), but no indexion. I think it is not indexed because my sitemap and .htaccess conflicted with each other, but would like to get the bots to pass over the site just to make sure.

But they haven't crawled it (according to Google Webmaster Tools).

Is that the figure in GWT the last time google CRAWLED (i.e. successfully passed over the site) or VISITED the site? Or does anybody know for sure?

OldWelshGuy · #2 (**permalink**) 06-10-2008, 10:38 PM

google spiders far more pages than it adds to the index. have you submitted a new sitemap? Are you positive that there is nothing blocking the page from being spidered?

rhyswynne · #3 (**permalink**) 06-10-2008, 10:54 PM

Quote:

Originally Posted by OldWelshGuy

google spiders far more pages than it adds to the index. have you submitted a new sitemap? Are you positive that there is nothing blocking the page from being spidered?

The sitemap is being generated by a script linked to the site, will try another format. I have deleted the .htaccess and the robots.txt and the site still works, so I'm a bit confused by it.

It just the "last crawled on the Apr 26th 2008" line that's worrying me. I could be in the sandbox (it is an affiliate site, but with good information, certainly a lot more content compared to a lot of my other sites!), but I just don't think it's getting crawled.

Will continue to look into it, thanks OWG

OldWelshGuy · #4 (**permalink**) 06-11-2008, 07:22 AM

Apr 26 th is an age ago, how often was it getting spidered? Have you checked for canonical issues www. non www. etc? maybe the wrong site is getting spidered. Have you got more than one domain pointing at the same folder on the server?

jitendraag · #5 (**permalink**) 06-11-2008, 07:28 AM

Check your robots.txt too.

If nothing works try to put that page in forum signature and it will get indexed quickly

OldWelshGuy · #6 (**permalink**) 06-11-2008, 08:08 AM

Quote:

Originally Posted by rhyswynne

The sitemap is being generated by a script linked to the site, will try another format. I have deleted the .htaccess and the robots.txt and the site still works, so I'm a bit confused by it.

It just the "last crawled on the Apr 26th 2008" line that's worrying me. I could be in the sandbox (it is an affiliate site, but with good information, certainly a lot more content compared to a lot of my other sites!), but I just don't think it's getting crawled.

Will continue to look into it, thanks OWG

he said he has deleted the robots TXT already, so that can't be it.

have a look at the page load times of the home page, make sure your internal links are pointing to domain/ make sure that your index page is returning a 200OK header response, and also that domain.com/ is doing the same.

it could be a server setup thing

zurk · #7 (**permalink**) 06-11-2008, 01:45 PM

Link building might help

Garcia Coder · #8 (**permalink**) 06-11-2008, 02:46 PM

well as for the knowledge I have I can say that link building.

bochgoch · #9 (**permalink**) 06-11-2008, 06:19 PM

What page is it? What links do you have to the page?

...I've had problems with pages that are linked to using PHP parameters ... i.e http://www .vvvvvv.com?P=page1 - G doesn't like ?'s in URL's

By the way Dating Site For Bloggers in your sig!!!! That sounds like a scary site

... bloggers shouldn't be allowed to procreate!

pow-wow · #10 (**permalink**) 06-13-2008, 05:47 PM

I suggest you build more backlinks on dofollow sites/blogs/forums/etc. If the backlinks are from frequently crawled sites - the bots will find your backlinks and crawl your site as well.