Sitemap Generator Forum

Hi Oleg,

Thank you for all your help in our emails so far. Very good.

As you know I have purchased the Standalone Unlimited version some days ago. I thought I would post here hopefully to get this problem sorted hopefully sooner as emails are way too slow and because I have a large site without any sitemaps as i've removed the previous ones to get XML-sitemap generator going. I thought I would update here so if anyone has the same issue the solution might help.

Url to sitemap generator = [ External links are visible to forum administrators only ]

The big problem is the generator will not index more than 5200 pages. My site has over 58,000 pages running joomla 1.5.x and Moset's tree 2.2 with 30597 listings and 20944 categories just in mosets.

I know the memory limit and execution time are not at fault as the memory used never goes above 26 meg for this issue.

I have my server php.ini set to,
memory_limit = 256M
max_execution_time = 9000

Regardless of xml, ror, html pages being created for a sitemap it appears 5200 is the max limit of pages it will index.

Request date:
2 July 2011, 11:57
Processing time:
0:44:51s
Pages indexed:
5197  <<<<<<<<<< Never goes above 5197 pages indexed.
Sitemap files:
1 <<< For a site of 58,000 pages this should be 2 files.
Pages size:
120.99Mb

During the crawling process it hits 5197 and starts the dump to write to files.

Links depth: 3
Current page: australia/western-australia/municipalities/upper-gascoyne-shire.html
Pages added to sitemap: 5197
Pages scanned: 5180 (123,288.7 KB)
Pages left: 3412 (+ 12261 queued for the next depth level)
Time passed: 0:42:05
Time left: 0:27:43  <<< This reduces to zero in a matter of seconds.
Memory usage: 25,370.7 Kb  <<< Note: Max mem used is only 26 meg.

No matter how many times or what settings I change I cannot get the sitemap generator to index more than 5200 pages.

Thank you for your continued help but until I can get this sitemap generator to index more than 5200 pages i'm having to continue to use Gsitecrawler which manages to index the whole 58,000 pages without issue. I would much rather this use XML-sitemap generator as it is what I paid for and has the features perfect for my site that i'm looking for.

What else can be done to get this XML-Sitemap generator to index more than 5200 pages?

Thank you for your help.

In anticipation of your next question here is a path of urls where it is indexing to the final path which includes pages not indexed.

[ External links are visible to forum administrators only ]    << indexed

[ External links are visible to forum administrators only ]    << indexed

[ External links are visible to forum administrators only ]    << indexed

[ External links are visible to forum administrators only ]   << indexed

[ External links are visible to forum administrators only ]   << indexed

[ External links are visible to forum administrators only ]   << indexed

[ External links are visible to forum administrators only ]    << Not indexed

[ External links are visible to forum administrators only ] << Not indexed

[ External links are visible to forum administrators only ]    << Not indexed

[ External links are visible to forum administrators only ]    << Not indexed


Hope this helps to see our problem. Please let me know if you need more info.

It really feels like it is limited but by what I do not know. Please help before I completely pull all my hair out.

Best wishes and regards

To give more information. My website uses cookies and setting

<option name="xs_no_cookies">1</option>

in the default.conf has allowed the crawler to pass 5200 pages.

Thank you very much Oleg for your excellent support during the initial setup and trouble shooting.

This product now does everything I expect of it and am very happy.

Recommended.

Best wishes and regards

You are welcome! I'm glad we got it working