Sitemap Generator Forum

Hi,
My site has over 14000 pages and the sitemap generated shows only ~300+ links in it. I noticed that the pages are being skipped. When the crawler is executing, I can see all those pages being crawled, however they are not getting added in the sitemap.

Issue #2:
I want the crawler to ignore a certain type of query. I have tried blocking Disallow: /? and also tried the generator exclude url. Both works fine. However, the crawled URL's are now even less.

Would appreciate if you could help me fix/ look into what is going wrong in my case.

Hello,

could you please PM me your generator URL and an example URL that is not included in sitemap and how it can be reached starting from homepage?
 

Hi,
I just sent you a PM with all the information. Please check.

Hi,
Please also note that the images and videos are not crawled at all. The images are stored in AWS S3.

Replied to your PM.

Yes. The url can be reached. Please see my reply too in your PM

Hi,
Have you figured out why the sitemaps are not generated? Awaiting for your response and fix for this issue.

Hi,
I just noticed that when I use your free sitemap tool for the same site I use your unlimited sitemap generator, all the skipped url's are perfectly being crawled and added to the sitemap. I'm wondering why your unlimited generator is skipping my URL's when the free tool can generate properly.

Sorry..it's the same.. I looked at the wrong counter.

Hello,

looks like all "pin/" links are loaded with ajax request which is not visible to bots. It must comply with "ajax crawling" specifications: [ External links are visible to logged in users only ]