Tools for Any SEO Challenge: Get more traffic to your website with versatile solutions for every step of your SEO journey.
« previous next »
Pages: 1

ventas11

*
  • *
  • 5
PROBLEM WHITH CRAWLING 5.0
« on: March 14, 2012, 05:14:47 PM »
Hi,
we haven't got crawl more than about 20.000 links. We resume every time but it don't crawl more than 100 links each time and interrupt.

could you help me.

Thanks.
Logged

ventas11

*
  • *
  • 5
Re: PROBLEM WHITH CRAWLING 5.0
« Reply #1 on: March 15, 2012, 04:19:35 PM »
allways finich whith something like that:

Thu Mar 15 17:17:30 UTC+0100 2012: resuming generator (120 seconds with no response)
Thu Mar 15 17:15:30 UTC+0100 2012: resuming generator (120 seconds with no response)
Thu Mar 15 17:13:30 UTC+0100 2012: resuming generator (121 seconds with no response)

and ..... finish. Resume and happens the same each time.
Logged

XML-Sitemaps Support

*
  • *****
  • 11782
Re: PROBLEM WHITH CRAWLING 5.0
« Reply #2 on: March 16, 2012, 06:51:55 AM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
Logged

ventas11

*
  • *
  • 5
Re: PROBLEM WHITH CRAWLING 5.0
« Reply #3 on: March 16, 2012, 10:12:20 AM »
no, it isn't the problem because i have set no time limit and no memory limit.
Logged

XML-Sitemaps Support

*
  • *****
  • 11782
Re: PROBLEM WHITH CRAWLING 5.0
« Reply #4 on: March 16, 2012, 01:24:56 PM »
It's defined in php.ini on server side, not in generator configuration.
Logged

ventas11

*
  • *
  • 5
Re: PROBLEM WHITH CRAWLING 5.0
« Reply #5 on: March 17, 2012, 04:59:23 PM »
yes, i know. But i'm trying to say you that in php.ini i set the time limit and the memory limit to infinite. And it continue failing.

Thanks
Logged

ventas11

*
  • *
  • 5
Re: PROBLEM WHITH CRAWLING 5.0
« Reply #6 on: March 18, 2012, 08:45:55 AM »
hello,
well i think that the memory and time limit memory is solved, but the crawling is finished and only crawls 11.000 pags and i have got more than 330.000 products.

Please, i need help.

Thanks.
Logged

XML-Sitemaps Support

*
  • *****
  • 11782
Re: PROBLEM WHITH CRAWLING 5.0
« Reply #7 on: March 18, 2012, 01:18:12 PM »
Hello,

could you please PM me your generator URL and an example URL that is not included in sitemap and how it can be reached starting from homepage?
Logged
Pages: 1
« previous next »