Sitemap Generator Forum

The generator will run.  Then will stop running and I can continue from where it saved, then it will run and stop, and I can continue where it saved.  Eventually when I try to continue from were it saved it comes back with this in like 2 seconds:

Links depth: -
Current page: -
Pages added to sitemap: -
Pages scanned: - (- Kb)
Pages left: - (+ - queued for the next depth level)
Time passed: -
Time left: -
Memory usage: -
Resuming the last session (last updated: 2006-10-03 15:39:34)

And of course is stopped.  Can't get any further, says its only written 1,200 pages, which it hasn't written any.  Max_execution is huge, memory size is about 32 Mb, Don't understand should only be about 5,000 pages

Okay, figured out the problem.  When you are running it from the command line it ACTUALLY produces error messages.  Running it in your browser doesn't produce any error messages.  But the error message says "Allowed memory size of 33554432 bytes exhausted"  which I only have the memory limit set to 32 Mb.  Afterall I'm on a shared host, much more than that and my hosting company would kill me.

I take it that this software really only runs well on a dedicated server.  I really wish I would have known that before purchasing it.

Is there anyway to run this software from a dedicated server that can handle the processing, but have it processing and spidering a different website that is on a shared host?  That looks like the only way to get it to work but I have a feeling this was a wasted purchase.

Well, I guess getting a response on here is out of the question.

So, from what I can tell:

1) This is designed to run only on Dedicated Servers with PLENTY of available Memory.  (It's a pretty big hog)

2) But it's also not possible to run it on a separate server, and hit a different target website.

So unless you have a monster server, don't even attempt to run this.

Is this a correct assumption?  Do I have any other options that I am missing?

I'm in the same boat. lack of replies to problems toboot.

---------
Links depth: 2
Current page: Article/Diagnosis-Hemorrhoids/8016
Pages added to sitemap: 4611
Pages scanned: 6840 (139,946.4 Kb)
Pages left: 94 (+ 28007 queued for the next depth level)
Time passed: 12:53
Time left: 0:10
Memory usage: -
----------

I have an article directory, and it stops at this point. then nothing afterwards.
I'm on a shared host too.

Would be nice if the Admin could actually look after their customers, maybe it's because the software was $19.00 or so, that support isn't in the package when problems arise?
This seems to be a never ending circle that I run into when I buy products.

lozpaypal,

yeah I am getting the same feeling.  I am thinking that the way this software pulls and stores the record is very sloppy as far as it's use of files.  I'm thinking that it wouldn't be near the memory hog if it did a lot more of the transactions between a mysql database, I'm going to see what I can do on that myself.

Hello,

sitemap generator script runs successfully in different environments (including shared hosting). The amount of memory required depends on the total number of pages the site has (as it's specified in system requirements).
To deal with memory limit problem you can:
- increase the memory_limit setting in php.ini
- limit the number of pages included in sitemap at Configuration page
- exclude a part of you pages from indexing with "Do not parse URLs" option

Also disabling the "HTML sitemap generation" and "Changelog calculation" reduces the memory usage a bit.

lozpaypal, I've replied to your private message.

The problem is that I did increase the memory_limit, I did exclude the pages, I did HTML sitemap generation, and at the end of the day 32 Mb of memory is only enough to crawl about 2000 pages.   The default on our shared host is 8Mb.  Of course if there wasn't this monster data file being created, and left open while everything is in process this might not be an issue.  This software ended up being rather useless.  But it wouldn't be so if I was on a dedicated server with a couple of Gigs of RAM, but don't know how to afford that, just because I bought a $15 script. 

Hello,

sitemap generator script runs successfully in different environments (including shared hosting). The amount of memory required depends on the total number of pages the site has (as it's specified in system requirements).
To deal with memory limit problem you can:
- increase the memory_limit setting in php.ini
- limit the number of pages included in sitemap at Configuration page
- exclude a part of you pages from indexing with "Do not parse URLs" option

thank you admin for replying...
But you need to understand my situation fully, as this may be the case with others too.

My host can not increase the memory limit just for me, because it will increse it for all the people on the shared server.
I have a reseller account, but even still, I can not have the memory limit increased.
My website became paralised for about 1hr, because your sitemap generator used up over 550MB of bandwidth in 30 or so minutes of each request of starting again to get the sitemap generator to work.

Also disabling the "HTML sitemap generation" and "Changelog calculation" reduces the memory usage a bit.

lozpaypal, I've replied to your private message.

Yes, thank you for replying to my private message, I have replied because the problem persists.
I'm at a loss why the problem arises saying it finds a page that is not linked up when there is no such link on the page it says.

I tried changing the html sitemap gen and changelog, but it still paralised my website.

Surely there has to be another solution here?

Thank you.

Hello,

you should use "Save state" option to avoid restarting crawling from the scratch. In case if sitemap generator takes too much cpu resources, I suggest to use "Make a delay between requests" feature to make it less intensive (although that will increase generation time).

Yes, thank you for replying to my private message, I have replied because the problem persists.
I'm at a loss why the problem arises saying it finds a page that is not linked up when there is no such link on the page it says.

Replied via PM again - I need access to your generator instance to check the broken  links issue.

Thank you for your feedback!