Sitemap Generator Forum

Hello,

in our shop [ External links are visible to forum administrators only ] the online tool indexes all articles like it should be, i.e. 61 pages with Xlayer products. But the stand-alone generator only indexes a fraction of those (10 pages with Xlayer products). How many pages the stand-alone indexes varies little and I can not see why one product page is indexed and the other is not.

I already tried to add a pause of 1 second between each request (under "fine tuning") to make sure that the stand-alone generator is not faster than the apache-server, but this made no difference (except the crawling time of course).

What could be the reason?

Best regards,
Axel

Meanwhile i have found that the pages are skipped because the server seems to respond  "301 moved permanently" although the page is available. I guess it is some timing problem with sitemap generator and the seo url rewrite of xt-commerce.

I just tried out: indeed if I disable the search engine friendly url of the webshop all pages are indexed. But I think the se-friendly url are useful because they contain the product name in the url, so I want to keep them.

Any hint for a solution? There should be one, because the online tool works also with se-friendly urls.

Best regards,
Axel Booltink

Hello,

 in case if server gets overloaded when running generator, you can use "Make delay for X seconds after each Y requests" setting in generator configuration - that will slow down the crawling, reducing the load.

Dear Oleg,

I already tried this without any success (see my first post).

Best regards,
Axel Booltink

You could try to increase the delay in this case.

Dear Oleg,

I already tried before with 5s after every 10 pages, but right now I am testing with 5s after every page and the result is the same. I am looking at debug.log (with tail -f) so I can easily see as soon as it goes wrong. Do you want me to send you the debug log?

I do not think that further increase will help so the reason seems to be something else.

Best regards,
Axel Booltink

Hello,

perhaps the site blocks access to it after certain number of requests.

Hi,

No, it the server is not configured to block after a certain number of requests. By the way: if that would be the case it would also happen if the URL-rewrite is disabled. But if I disable SE-friendly URLs, so that the URL is not rewritten all pages are indexed.

And also: the online-tool indexes allway all pages.

Best regards,
Axel

Hi,

too make sure it is not the fault of my outdated Debian, I just installed a brand-new server (I needed to migrate the shop anyway).
So, now I have a stand-alone server with only a fresh Debian installation and our newly installed webshop under IP 195.225.198.69.
If I do not enabled Search Engine Optimalization in the webshop everything is ok.
If I use SEO (which works with the rewrite engine of apache2) the online tool still indexes all pages but the stand-alone generator skips a lot of them.

What can I do?

Best regards,
Axel

Hello,

please let me know your generator URL/login in private message to check this.