Sitemap Generator Forum

I just bought and installed this Unlimited PHP Sitemap Generator.

All instructions on permissions have been followed.

I have put the generator software under the directory "generator" in the root directory (public_html)

It is now crawling our website.  It shows an error now:

"The webpage at [ External links are visible to forum administrators only ] might be temporarily down or it may have moved permanently to a new web address.

Error 324 (net::ERR_EMPTY_RESPONSE): Unknown error."

The status says - "Crawling already in progress.
Last log access time: 2011-01-15 09:53:14
Click here to interrupt it."

I don't know if it is still working, or stopped.  From the message, it seems like it's still working. 

Can Support shed some light on this issue ?  I did not expect any errors of course, like I have no problems with the free version.

Thanks

Well, I resume, it goes on, comes back later with the same error, so I resume again, and so on ... been doing this the whole day ...

Anyway, I just downloaded the free GSiteCrawler and so far so good, no interruptions, almost halfway done in just a couple of hours ... let's see who comes out with the better sitemap ...

Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.

Hi,

Thanks for the reply !  I am using Bluehost, and I just found out the following:

"The program or script you run on your site may require a process to run longer than 5 minutes.
On a shared IP address, the process will terminate at the 5 minute mark (300 seconds).
With a dedicated IP address, processes are allowed to run for more than 5 minutes as long as they aren't consuming too much of the server's memory, which would result in account being throttled. See CEO's Blog.
Account throttling can occur both on a shared and a dedicated IP address.
Certain voice chat programs require a dedicated IP address before they can be setup and used.
If you have a program that sends out emails every so many seconds, such as from a mailing list program like DaDa Mail, then you would need to get a dedicated IP address, if it's going to take more than 5 minutes to send out the emails.
A dedicated IP address will help when processes from scripts or programs, which don't consume too much CPU, need to run in the background for longer than 5 minutes."

What a bummer. 

Anyway, I don't want to move our websites out of Bluehost, too much work just to get a sitemap.  I can however probably sign up with another service provider which has no such limitations.  Question is :  Can I install the generator in the new server, and run it from there pointing to the current Bluehost server ? 

Thanks

Hello,

yes, you can install sitemap generator on another host and create sitemap for your main website.

Ok, thanks.  I have now put this xml program under Gatorhost, as I have read that it is one that does not have throttling.   Your website should really tell buyers which hosting works with your software, for creating sitemaps for large sites.  Otherwise, it will just lead to your customer dissatisfaction.

Well, almost.  It still stops once everytime I ran the script.  No problem, just resume.  Well, so we have to babysit this script !!  Anyway, I guess I can't expect much from $20 worth of software ...

Now, the latest problem, a message from Google:
"Your Sitemap exceeds the maximum file size limit. Please create multiple Sitemaps containing fewer URLs to meet the file size limit."

The sitemap is about 10mb+ (less than 11mb).  So, Google is no longer looking at just the 50k limit for number of urls, but filesize as well.  I guess you did not know this ?

My sitemap ballooned to this size after I selected the option to add Image Information under the Configuration.  I thought it would be a good idea to include images.

So, will you suggest I don't add Image Information anymore ?  But you should be aware of this file size limitation as well, which I find no information about in your documentation.

Please help with suggestions, thank you.

-->>> Some information : [ External links are visible to forum administrators only ]

Hello,

you can keep the images information in sitemap, but to avoid growing filesize over 10Mb limit you'll need to decrease the "Number of URLs per xml file" in generator configuration and recreate sitemap, it will split sitemap on multiple files (sitemap.xml + sitemap1.xml, sitemap2.xml etc)

Ok, thanks.

Right now, it is 40,000 URLs I think by default.

So, with this new rule of Google of 10mb file size, I will try 2500 URLs per xml.

That is very far from default 40,000 ... I think you need to update your documentation/Help to inform people about this new Google rule.

Unfortunately, Google rules as top search engine, so we just follow the rules of Google even if they don't follow the rules of the others ...

Hello,

it's not necessary to set it to 2500, if you get filesize 11Mb with 40000 URLs, then you will be well under 10Mb with say 30000 per file limit.

Our website has many images.  For 2500 URLS, our sitemap1.xlm is about 9MB. 

In total, our URLs are only between 3000 to 3500, but it exceeds 10mb if I don't split it up.

So it all depends on what a website contains.

Anyway, I have tried Bluehost hosting, Unlimited PHP Sitemap Generator cannot work there.  Now, I am on HostGator, it's ok, it only fails once everytime, and we just resume.

Can you recommend a host that will not have any issues with bigger websites ?

I know I can use other Sitemap generators, but somehow I like yours.  Especially as we used the free version when our website was smaller, and we are always in the front page of Google for our stuff even then.  SEO techniques are important of course, but it also means XML Sitemap is a good helper.


Thanks

Hello,

normally you can get sitemap generator working on any host, it resumes the process automatically if you keep the windows open. Also, you can setup a scheduled task to run generator, which in many cases runs longer (and also auto-resumes the process).

Well, on both BlueHost and HostGator, they don't resume automatically even with windows open. 

Anyway, I will try the Cron job, thanks ...