Can you do a "reverse exclude"?
« on: November 15, 2007, 12:07:41 PM »
I am having problems getting the xml-sitemap generator to crawl all my site, let me explain:

I have a site full with business listings divided into separate categories I would like to be indexed. Whitin each category these business listings are shown 10 at a time, and if there is more than 10, it obviously get divided into several pages with a page-navigator at the bottom (like with the html-sitemap xml-sitemaps generates). The problem is that this "page counter" only shows 10 page links at a time, and then a next and last link, which if pressed shows another set of ten page links, holding links to 10 business listings each. You get the picture.

What this seems to do for XML-sitemap is this: Say I in this case have +1000 listings in a category, there are only links generated in the sitemap to the first 100. Now, this problem is not entirely consistant cause I have seen xml-generations taking more links into the sitemap (but still not all), however what I am sure of is that a problem only occurs when there are more than 100 business listings within one category.

Is there a way around this problem? In my case I would really like XML-sitemap be dead serious on prioritizing and making sure of following up every url with the string "limit=10" which is a part of every url when a category is divided into multiple pages. Is there a way to do this kind of "reverse exclude" rule? Or is there anything else you would recommend me to do?
Re: Can you do a "reverse exclude"?
« Reply #1 on: November 16, 2007, 12:17:33 AM »
Hello,

Quote
Is there a way to do this kind of "reverse exclude" rule? Or is there anything else you would recommend me to do?
All links must be included in sitemap, unless they are affected by exclusion option OR pages are excluded with robots meta tag/robots.txt file.
Please let me know your generator URL and an example URL that is not included in sitemap and how that URL can be reached starting from homepage.
Re: Can you do a "reverse exclude"?
« Reply #2 on: November 16, 2007, 12:03:52 PM »
Thank you for the reply, you will receive a PM with all the details.

I have now revised my earlier claims somewhat. As it seems I was correct only for when the level is deeper than 3.

As it is now, this level of depth works fine, meaning all urls are included, even if there are more than 100:
[ External links are visible to forum administrators only ] -> Level 1 -> Level 2 -> Level 3 -> 100s of Urls separated by 10 links per page

but what doesn't seem to work is when another level is added:
[ External links are visible to forum administrators only ] -> Level 1 -> Level 2 -> Level 3 -> Level 4 -> 100 of Urls separate by 10 links per page