Sitemap Generator Forum

I have just brought Unlimited PHP Sitemap Generator. When I started Crawling it worked fine for 8K pages but suddenly it got struck there. I tired it to start again but after few secs it gets stuck again.

What can be done to fix this and for fast crawling.

Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
 

We have increased memory_limit and max_execution_time settings but the issue is still there. The script stops after sometime.

Hello,
in this case I would recommend to run generator in command line if you have ssh access to your server.

i have made changes in Configuration settings and crawling is going on now but it's very slow and my site have large number of pages. Can you share what are the best Configuration settings for quick crawling of all the pages.

Hello,

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

I'm facing one another problem with the Crawling option. Few hours ago I interrupted the crawling and when I tried to start crawling again there is no response from the page. I'm not able to access the Crawling page when I try to open that page nothing happens.

All the other options are working fine.

It means that connection is still open to the website. Please try to close browser completely or open generator page in another browser (not new tab though).

In case if the issue is only on Crawling page though, you need to increase memory_limit setting again as it's exceeded when generator tries to load current session state.

Is there any option to create partial sitemap. There are lot of pages added to sitemap and lot of pages are queued for the next depth level.

In the meantime can the sitemap be created for pages added to sitemap till now.

Hello,

yes, you can use  "Maximum pages" setting to limit the number of URLs to index.

Can Crawling be restarted from that point by again resetting the maximum number of pages to be crawled

You can save a copy of "generator/data/crawl_dump.log" file which contains current crawling session, create sitemap, then restore that file from a copy, change settings if needed and run generator again from that point.

I'm not able to access Crawling option since last 2-3 days. I interrupted crawling few day back but when I try to resume crawling nothing happens and that page doesn't open. It has been so long i'm trying to generate Sitemap but nothing is happening.

Hello,

you need to increase memory_limit setting in PHP configuration since currently it's not sufficient to read crawl_dump file.