Not crawling all my pages
« on: August 16, 2006, 01:47:11 PM »
Hi,

Just a quick question, I have found that only 1515 pages are being crawled?? I have 2800 + pages.

What can I do get it to crawl all 2800+ page, I have notices it only crawls to a link depth of 2.

Any ideas?? (please) or advise.

Thanks a million
James
[ External links are visible to forum administrators only ]
Re: Not crawling all my pages
« Reply #1 on: August 16, 2006, 09:53:07 PM »
Hello James,

please make sure that all your pages are linked together, i.e. it's possible to find every page starting from homepage. If it's not a problem in your case, please send me a private message with your generator instance URL/login and example link that is not included in sitemap.
Re: Not crawling all my pages
« Reply #2 on: August 16, 2006, 10:47:20 PM »
I too am having that problem. I have four html pages but the first page is blank. I have it set to record 250 pages each
Re: Not crawling all my pages
« Reply #3 on: August 17, 2006, 11:16:43 PM »
Hello,

do you mean that first page of HTML sitemap is empty in your case? Please ensure that you have correct permissions set for this file (0666).
Re: Not crawling all my pages
« Reply #4 on: August 22, 2006, 10:54:06 AM »
Hi all,

Just a quick question, will javascript links be followed.

I.E. <a href="javascript:printpopups('../../print_page/mp_s_kruger/bongani_mountain_lodge.htm');">
      PRINT PAGE</a>

Just a thought as to why some pages are not getting scanned.

cheers
james
Re: Not crawling all my pages
« Reply #5 on: August 22, 2006, 12:27:03 PM »
Search engine robots can have problems with JavaScript links. You can check to see what links are picked up on different pages of your site with our search engine robot simulator at https://www.xml-sitemaps.com/se-bot-simulator.html

Philip Nicosia