Competitor Analysis Tool: Analyze rival websites and gain valuable SEO and PPC insights
« previous next »
Pages: 1

Tiryaki

*
  • *
  • 22
Problem with Crawling Website
« on: November 18, 2014, 10:03:01 PM »
Hello,

the crawling of my website stops after a few minutes.
I have tried and changed some settings, but have still the problem.

Thanks in advance,
Tiryaki
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: Problem with Crawling Website
« Reply #1 on: November 19, 2014, 10:21:43 AM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
 
Logged

Tiryaki

*
  • *
  • 22
Re: Problem with Crawling Website
« Reply #2 on: November 19, 2014, 11:27:23 AM »
Hello,

max_execution_time is 60
memory_limit is 90

isn't that enough?

BR,
Tiryaki
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: Problem with Crawling Website
« Reply #3 on: November 20, 2014, 05:20:38 AM »
Hello,

please try to increase it.
Logged

Tiryaki

*
  • *
  • 22
Re: Problem with Crawling Website
« Reply #4 on: November 20, 2014, 11:19:38 AM »
Hello,

can't increase it, that is the maximum what my hosting can provide.
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: Problem with Crawling Website
« Reply #5 on: November 20, 2014, 11:46:27 AM »
in this case I would recommend to run generator in command line if you have ssh access to your server.
Logged

Tiryaki

*
  • *
  • 22
Re: Problem with Crawling Website
« Reply #6 on: November 20, 2014, 03:48:57 PM »
No, I have not.
Isn't there any alternative method?
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: Problem with Crawling Website
« Reply #7 on: November 21, 2014, 02:42:51 PM »
Otherwise you would need to run generator multiple times, resuming the process until it finishes.
Logged

Tiryaki

*
  • *
  • 22
Re: Problem with Crawling Website
« Reply #8 on: November 21, 2014, 02:57:32 PM »
Is it not possible that the script makes some brakes between the crawling or do it slowlier?
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: Problem with Crawling Website
« Reply #9 on: November 22, 2014, 02:02:14 PM »
Hello,

there is a setting "Make delay for X seconds after each Y request" in generator configuration to slow down crawling process if it's needed.
Logged

Tiryaki

*
  • *
  • 22
Re: Problem with Crawling Website
« Reply #10 on: November 23, 2014, 08:37:15 PM »
I have tried several settings without success, can you please have a look at it.
I will send you the link per pm.

Thanks in Advance.
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: Problem with Crawling Website
« Reply #11 on: November 25, 2014, 07:59:50 AM »
Hello,

replied to your PM.
Logged
Pages: 1
« previous next »