Sitemap Generator Forum

HI,

I installed the generator last night.  I installed it in mydomain.com/generator and left all the default paths.  Problem is crawling my site is using alot of bandwidth, i though as the generator is on the server i wouldn't.  My host uses cpanel and although the bandwidth isn't a problem at the moment if i run the generator every day like i was planning i would become one.

So have i done something wrong when setting this up or is this the way it's suppose to be?

TIA

Hello,

the bandwidth calculation depends on the hosting platform and you may want to contact your hosting support regarding this. Sitemap generator only creates network connections to the domain you specified in the configuration.
On the other hand, it's not a must to generate sitemap daily. A weekly (or even monthly) generation should work fine as well.

ok no problem thanks for your reply.  Love the script by the way, i am recommending it to friends alreay!

Great, thanks!

I am having the same problem. Just installed the script and get this in my log:

           No Date/Time           Total pages Proc.time Bandwidth New URLs Removed URLs Broken links
             1 2007-09-09 02:29 1,768           33.74s        49.47         1000           0                        64

50 mb for less than 2000 pages??

Hello,

yes, why not - it's only 25Kb per page, which is normal for the html page.

I'm very worried about using this prog as it has shot my band width through the roof. My monthly usage is about 18GB however is only day 5 of this month and its already over 14GB!!!! and I've only run the prog about 4 times so far, and those were tests as its still listing urls that I though I'd set it to exclude.

I have gone to my host about this programme and they said

When the sitemap generator goes through your site - generally it will download/view everything it fines - thus logging in your webstats as traffic (as the webserver served the request).

I'd be careful with those - they can cause load issues on the server that may cause your site to be suspended or even worse, cause the server to become unresponsive and crash.

The 'unresponsive' and 'crashing' is very worrying.

Is there any way to limit its effect on band width?

Hello,

what are the bandwidth stats included in sitemap details report? ("Pages size" value)
Sitemap Generator only downloads html documents, so it should not cause any issues normally.

Presumably its these you want

No    Date/Time    Total pages    Proc.time    Bandwidth    New URLs    Removed URLs    Broken links
26    2007-11-02 22:30    12,558    4,445.80s     300.71       1000                884    0
27    2007-11-03 00:55    12,532    4,233.30s     299.55        202                228    0
28    2007-11-06 02:39    13,502    45,562.00s    650.25           0                   0    1

Hello,

the bandwidth used by generator in a single session is 300Mb for you 12,500+ URLs. I suggest you to generate the sitemap less frequently to keep total usage lower (weekly or even monthly sitemap refreshing is enough in most cases).