Hello,
Love the software, its been great for our site.
Just one problem: we recently added a feature to our site with about 40 new URLs and found that (for some reason) these new URLs are not crawled by the generator... even though there are links to the new content throughout the site.
I noticed that if I instruct the script to start with the feature URL as the initial URL, the pages are all crawled correctly and a site map with the 40 URLs is created. However, starting from the root of the site, the script glances over these URLs and does not visit, parse or add them.
The links pointing the new feature do not currently use redirects (I understand the script does not follow http redirects) and there are no exclusion rules set that would apply to the new URLs.
Any other ideas about why this might be happening?
Guest