Sitemap Generator Forum

The version I purchased crawls only half the URLs on the page, but the online version crawls all the URLS.

The URLs that it crawls are like [ External links are visible to forum administrators only ] and the ones that it does not crawl are like [ External links are visible to forum administrators only ]e where the only difference is that the ones not crawled have an additional /Some-name-here.

For XML-sitemaps admin: This is one of the URL where the crawling problem occurs: [ External links are visible to forum administrators only ] . I set the URL to start crawling from to be [ External links are visible to forum administrators only ].

How can I fix this? I will PM you the login details

Hello,

I've updated your crawler settings, please check now.

Thanks, its crawling those now. Can you explain to me why adding the substring like the way you did will cause it to crawl these pages?

I also notice that those pages (../view/some-number/some-string) that were not crawled and are not being crawled appears in the generated sitemap after all those with only (../view/some-number) have been listed. Just wondering why both of them are not listed one after the other, like (../view/some-number-1/some-string1) with (../view/some-number-2/some-string2).

They are added directly in sitemap if specified in "Do not parse" setting.