Sitemap Generator Forum

We tried to crawl our site and seems its ok with the URL with filename like [ External links are visible to forum administrators only ]

However our site does .htaccess rewrite and thus some of the URL would be just the directory name like

[ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]

which we found those could not be crawled.

May I know if there is any setting to fix this behavior?

Hi, anyone can help here?

My site is [ External links are visible to forum administrators only ] and all the URLs end with .html are ok but those just the directory name cannot be crawled as shown below:

[ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]
etc.

Please help and thanks in advance.

Hello,

those links are not followed since they have rel="nofollow" attribute:
https://www.xml-sitemaps.com/se-bot-simulator.html?op=se-bot-simulator&go=1&pageurl=http%3A%2F%2Fwww.mobimega.com%2F&se=googlebot&submit=Start