Sitemap Generator Forum

I've been having a problem with the Unlimited XML Sitemap generator since I paid for it. Since I've been using it I've been getting relatively small sitemaps of 100 or 200 kb that doesn't seem right for the size of my site.

However in the past no error log was ever generated most likely because I had the directory password protected through cpanel and .htaccess.

I just decided last night to upload fresh files, reconfigure the generator and remove the password restriction on the directory and just use a username/password to access the interface.

After running the script I was once again given a fairly small sitemap file, so I decided to check if any error logs were generated and sure enough they were. There was about a 4 kb error_log file and about five 17 - 18 MB core dump files.

I have attached the error log below. I should make note that the file it claims is missing, is actually in the right directory, I even tried uploading it again to be sure it wasn't the problem despite what the error_log claimed.

I should also mention that I have set the proper permissions on the files as the install documentation listed.

I would really appreciate it if someone can take a look at the error log, figure out what the problem is so I can fix this problem and be able to use this script without a problem on my website.

Thank You.

Sorry to double post, just a small update:

I managed to fix the errors/warnings at least for now..

However the sitemap I get upon completion is too small, at least much smaller than it should be.

It seems the script will index some of the pages, quit and then claim it completed successfully/fully.

I had over a 1500 page count, but this is only indexing between 300 - 600, it quits and claims it completed indexing all pages/url's successfully.

I know the sitemap should be larger than 100 - 200 kb, more around 3 - 8 MB. Using the free (limited) generator which only indexes 500 pages/addresses gave me a 3 - 5 MB sitemap, before I purchased the unlimited version. However for the life of me, I can not get this script to work without issues.

Hello,

could you please PM me your generator URL and an example URL that is not included in sitemap and how it can be reached starting  from homepage?

Did you get my message?

Update: the issue resolved via PM.

Issue hasn't been resolved...

This script isn't indexing all the URL's, it seems to time out or quit working after indexing so many URL's and then say it completed running.

I know it isn't indexing the Url's because I tried playing around by trying different settings and was able to get it to index more Url's, however it still wasn't all the url's/links my site contains.

Since this seems to be a server/host/settings issue with indexing Url's/links, I need to know the appropriate settings to use for the sitemap generator and my host's configuration.

Can you help me with the correct settings?

Hello,

please let me know an example URL that is not included in sitemap and how it can be reached, starting from homepage.

Please note that XML sitemap protocol doesn't allow to include URLs from different (sub)domains in the same sitemap (Google will not accept such a sitemap), so you should create a separate sitemap for a subdomain.

I don't know what URL's they are, but if I fiddle with the settings a bit the script crawls/indexes more Urls/Links..

I only have one subdomain, it's the main root domain/directories that all the links aren't being indexed..

I have several thousand links on my forum alone, not counting my portal page. That should generate a 3 - 5 MB sitemap file. Out of the several thousand links, the sitemap script is only indexing between 500 - 1000 links at most..

I need  a proper configuration/settings for the server/host that I am hosted on..

I don't think I have to be much clearer on this, I wish for an actual solution rather than a rinse & repeat of my last support request which did no good..

Hello,

I've modified your crawler settings for better performance.
Sitemap has been created:
Request date:
27 January 2009, 18:05
Processing time:
1204.90s
Pages indexed:
805


You have about 770 topics on your forum, so it covers that number (note that you should count topics/ not messages).

I need help configuring the sitemap generator again..

I changed the url structure of my forum by installing Community Seo which makes the urls search engine friendly (Furl)..

Ever since I installed this modification, the sitemap generator is now indexing less urls. It was indexing up to 1500+ urls per run before installing the IPB modification to my forum, and now it's only indexing a max of 600 - 700 urls per run on my forum after installing the modification.

So I need help with properly reconfiguring the sitemap generator for the server/host that I am on. If you need server details and specifications, I can gladly pm them to you so we can settle with the correct configuration..

Hello,

could you please PM me your generator URL and an example URL that is not included in sitemap and how it can be reached starting  from homepage?

As I've mentioned before I am unable to determine what urls aren't indexed due to the amount of urls on my website. I can however provide an estimate on how many urls the generator was indexing before this change and how many urls it is indexing now after the change..

Digging through the sitemap to determine what urls are not being indexed would be too time consuming and difficult to do..

I was hoping if I provide the specifications or details of my hosting account, that maybe you would be able to help determine the best configuration for the sitemap generator..

I manually initiated a crawl, while monitoring the crawl I noticed that the sitemap generator seems to choke or stall during a certain point while indexing..

Because it stalls or freezes at that certain point while indexing, the script thinks it has finished indexing and sends the email.

This script has been causing me a lot of frustration trying to get it to work right. I can not provide you any of the urls it doesn't index as there is too many on my site and it's time consuming to try and analyze the site map to figure out what isn't being indexed.

All I know is about a minute after the indexing starts, the script just chokes out or stops indexing, so it never finishes indexing all of the urls present on my website(s). When it freezes/stops indexing, the script of course can't tell that everything wasn't indexed, thus sends the email thinking that the job completed even though it didn't..

Please PM me your generator URL/login.

Sorry it took me a while to respond, I've been extremely busy.

I was able to get it to index slightly more urls by removing everything in the "Exclude Urls" and "Do not Parse Urls" boxes. However it's still not indexing all the urls it should index on my site, so I am led to believe that at some point crawling is breaking and then the script stop indexing and sends the email.

Sending you the login info right now via pm.