mhz

*
  • *
  • 3
Crawler is not working at all! - serious problem
« on: March 01, 2011, 02:36:31 PM »
I installed it successfully and tried crawling my big web site. after hours crawling faced problem with memory.

Now I can access crawling interface at all. Half of the page loads and the crawling button don't show at all.

Please help me urgently

Screen shot was attached .
Re: Crawler is not working at all! - serious problem
« Reply #1 on: March 01, 2011, 02:48:36 PM »
Hello,

that happens when memory limit is exceeded when generator is trying to read the crawl dump file, you will need to increase memory_limit setting in PHP configuration to resolve that.

mhz

*
  • *
  • 3
Re: Crawler is not working at all! - serious problem
« Reply #2 on: March 01, 2011, 03:15:45 PM »
1-How can I do that?
2-Is it possible to limit total memory used by crawler?
Re: Crawler is not working at all! - serious problem
« Reply #3 on: March 01, 2011, 08:09:46 PM »
You should modify php.ini file on your server or contact your hosting support for that.
You can set "Maximum URLs" setting in generator cofniguration to keep smaller sitemap and correspondingly less memory usage.

mhz

*
  • *
  • 3
Re: Crawler is not working at all! - serious problem
« Reply #4 on: March 04, 2011, 08:36:51 AM »
I raised the memory to 512 MB but now the program stop responding at all.

There are many free tools which capable of producing 50,000 pages site map, I paid you to have a better tool. but it seems it can't work even as good as free software.

I can't generate a site map with more than 42,000 pages. it just crashes. I use a dedicated server with 4GB Ram, so there shouldn't be any problem with CPU or Ram.
Re: Crawler is not working at all! - serious problem
« Reply #6 on: May 02, 2011, 07:18:59 AM »
I have exactly the same problem.

I have set the memory limit on the server to unlimited and I still have the problem.
Re: Crawler is not working at all! - serious problem
« Reply #8 on: May 03, 2011, 08:30:40 PM »
Hi Oleg,  Message sent