Sitemap Not Crawling Through All The Pages
« on: September 22, 2008, 12:14:14 PM »
Hello,

I have set up the configuration so many different ways, I have increased max_execution_time=500, memory_limit=256M. My website is a forum with about 70,000 pages, but it keeps coming up with about 2100 pages. Why does it not go through all the pages. I get no errors. It says on there that there are many more pages to add to the sitemap. It just doesn't add them, and then it says Congratulations, and returns back to the main menu.

Please help.
Re: Sitemap Not Crawling Through All The Pages
« Reply #2 on: April 29, 2009, 02:41:38 PM »
I have the same problem, but with a smaller site, crawler produces a sitemap, but this is mapping far less pages than we're expecting. help please!!!

Hello,

I have set up the configuration so many different ways, I have increased max_execution_time=500, memory_limit=256M. My website is a forum with about 70,000 pages, but it keeps coming up with about 2100 pages. Why does it not go through all the pages. I get no errors. It says on there that there are many more pages to add to the sitemap. It just doesn't add them, and then it says Congratulations, and returns back to the main menu.

Please help.

Re: Sitemap Not Crawling Through All The Pages
« Reply #3 on: April 29, 2009, 08:05:35 PM »
Hello,

could you please PM me your generator URL and an example URL that is not included in sitemap and how it can be reached starting  from homepage?