Sitemap Generator Forum

My sitemap generation is strucking at 10570 pages ...

after that when re run the interrupted session its running for 30sec with no progress and stopping....

this is very bad....

in my server log i didn't see any process running...

even i have tried cron job but its happening same

Here is the state of sitemap generation::::::::

Run in background
Do not interrupt the script even after closing the browser window until the crawling is complete
Resume last session
Continue the interrupted session (2008-10-23 09:44:13, URLs added: 10570, estimated URLs left: 27811)
Click button below to start crawl manually:

in the crawling screen if i try to run  again its running for 30 sec and no progress after 30 sec it is terminating and giving option to re-run...

Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.

there is no problem with site config file coz i had given 256mb and max execution time of 180....  when i spoke with my hosting company they told me that they dont have any issue with my config...

whats wrong is going on my sitemap is [ External links are visible to forum administrators only ]

Please PM me your generator login details to check that.

Please PM me your generator login details to check that.

i have PM'ed details...

Hi- did you get this fixed ?- I have exactly the same issue!!

Hi,
new to this today and I have the same problem as this - and noticed its also reported in many other posts.
The answer always seems to be a private PM to the customer.

Is there a public fix for this that we can all share?

My problem is that it stops at only 784 pages, and when I kick it in the guts again it restarts about 30 less than the number it stopped at, works its way up to that number and then stops again.
I solved it by resetting the number of pages to 784 and got it to write a sitemap for me but this is no good as I need to get up to 50,000.

Also , whilst on this subject, is there a way - or could there be a way, of getting this to write the sitemap after every 50 or 100 pages so that when the thing kicks the bucket we at least have a sitemap up to the last 50 pages? Seems a lot of waste of time and resources to go through this only to find you stop at a certain point and have nothing to show for it.

regards
ET

Hello,

the issue can be resolved by increasing memory_limit and max_execution_time settings in PHP configuration. Also, it is possible to optimize crawler settings with "do not parse"/"exclude urls" options depending on specific website structure.
It's not possible to create sitemap "every X URLs", but you can define "Maximum URLs" settings at any moment to let it create sitemap at specified amount.

i have same problem


my forum contain 1004 topic


i don't why get only 495 pages in sitemap.htm


and i can't find alot Url for some topic in sitemap.xml


plz help

when i add this lines in Exclude URLs


p=
mode=
mark=
start=
order=
highlight=
profile.php
privmsg.php
posting.php
view=previous
view=next
search.php
faq.php
ucp.php
memberlist.php
view=print
/download


it's stop on pages number 495

but when i delete Exclude URLs it's make alot of 5000 pagws why??

Hello,

because there is a lot of "noise content" on your site, i.e. the URLs that should NOT be indexed. So, as a result you have only the *valuable* content pages included in sitemap.

I know that sir


but he do'nt add all my good pages stop only in 495...


i want it to add all my forum topic

but he didin't my forum contain alot of 1015 topic....and the script add only 495 topic url...

Hello,

could you please PM me your generator URL and an example URL that is not included in sitemap and how it can be reached starting  from homepage?

Hi,
new to this today and I have the same problem as this - and noticed its also reported in many other posts.
The answer always seems to be a private PM to the customer.

Is there a public fix for this that we can all share?

My problem is that it stops at only 784 pages, and when I kick it in the guts again it restarts about 30 less than the number it stopped at, works its way up to that number and then stops again.
I solved it by resetting the number of pages to 784 and got it to write a sitemap for me but this is no good as I need to get up to 50,000.

Also , whilst on this subject, is there a way - or could there be a way, of getting this to write the sitemap after every 50 or 100 pages so that when the thing kicks the bucket we at least have a sitemap up to the last 50 pages? Seems a lot of waste of time and resources to go through this only to find you stop at a certain point and have nothing to show for it.

regards
ET

I'm also having a similar problem. The crawler adds around 42,000 pages to the sitemap with over 200,000 pending in the q, the crawler stops around that point. when i go to restart it, it starts from the beginning. Yes, I selected all options when I restarted everything. I've done this a number of times before in the past and I never had this problem before.

php.ini file settings

max_execution_time = 9000     ; Maximum execution time of each script, in seconds
max_input_time = -1   ; Maximum amount of time each script may spend parsing request data
memory_limit = 1024M      ; Maximum amount of memory a script may consume (8MB)

progress state storage type: var_export

Save the script state, every X seconds: 180

Please PM me your generator URL/login to check that.