Competitor Analysis Tool: Analyze rival websites and gain valuable SEO and PPC insights
« previous next »
Pages: 1

Tech-Mentor

*
  • *
  • 2
I can not get the sitemap to crawl my website.
« on: May 25, 2007, 11:23:08 PM »
I have database driven dynamic website. I can not get the site-map to work. The most I got was about 14000 pages. I got up to 14k by restarting from the last saved point several times. But the site-map never saved that file. All my other attempts have been successful for about 2 - 3k. Thats it.  The crawler gets stuck. I have left it running overnight. I have experimented with setting different limits and other available options. I always get the same results. It just stops working. I even reinstalled the site-map on a different server.

Right now I am very frustrated with this product. I think I am doing everything I suppose to.

Can someone please explain the entire process to crawl a large website?
Logged

Tech-Mentor

*
  • *
  • 2
Re: I can not get the sitemap to crawl my website.
« Reply #1 on: May 27, 2007, 07:22:57 PM »
I wanted to update my latest attempt to create a site map. The crawler seems to be stuck at 3560 pages and it is in a suspended state where I do see that the browser is trying to get the information and the status bar shows transferring data from the website, but nothing else is changing. No error messages to report. Its just stuck. Its been like this for the last hour or so. Stuck in one place much like how I feel using this program, stuck in one place.
« Last Edit: May 27, 2007, 07:24:32 PM by Tech-Mentor »
Logged

XML-Sitemaps Support

*
  • *****
  • 11782
Re: I can not get the sitemap to crawl my website.
« Reply #2 on: May 28, 2007, 09:50:25 PM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
Logged
Pages: 1
« previous next »