Sitemap Generator Forum

Hi,
first of all I would like to say that the script I purchased up to now fulfills my expectations.

Nevertheless I feel that I'll have to do some customization in order to optimize it for my site.
This is my situation:
I have a large number of ways do do site searches (with different parameters) which all of them are indexed.
Unfortunately, most of the searches give an empty result.
Obviously, there would be no need for robots to index them.
But how can that be achieved?
One more obstacle is that the search parameters are repeated on the search result page, even if no results are found.
In other words, all the result pages are different.

Having this in mind can you please anwer my questions:
1. if two different links point to the same page (same HTML content), will they both show up in the sitemap?
2. if the answer to 1 is positive, where can I find that piece of code in the sources?
3. which methods / best practises are in place to write addons for the script (for example an addon that finds the string "0 search results" on the target page and then discards the link)?

Any hint where to start looking into to solve this would be appreciated.

Regards,
marketplace

Hello,

1. yes, both pages will be included in sitemap.
2. source code cannot be modified according to license agreement. Anyway, that won't be an easy task - the content of all pages will need to be stored in this case and cross-compared, and many cases there will be *slight* changes on the page (like date/time in footer or some ad code etc).
3. I would recommend you to perform this change on *your site*, by adding rel="nofollow" attributes to links pointing to pages with no results, for instance. or adding robots meta tag with "noindex/nofollow" content in it to avoid indexing those pages. It's definitely a task that should be performed on the site rather than in generator - that will also help search engines to crawl your site more efficiently.

Hi Oleg,

thanks for your reply.
I can fully accept your answer to 1 and 2.

Concernig 3 I would like to suggest you could add to future version a feature like "discard link if it points to a page that contains a certain string". Maybe that could also help other people to discard links that they don't want.

Another suggestion would be to allow the use of regular expressions for the selection of the links for higher flexibility.



All the best,
marketplace.

> Concernig 3 I would like to suggest you could add to future version a feature like "discard link if it points to a page that contains a certain string". Maybe that could also help other people to discard links that they don't want.

Do you mean a string containing in the *URL* or in page text? Can you please provide examples so that I can understand this request better?

The idea was to discard links that point to meaningless or duplicate pages.
For example search result pages. If the string "0 search results" appears on the target page I would like to discard the link that points to it.
Otherwise I could be penalized by Google because of duplicate content.

You would need to add robots noindex meta tag on those pages, otherwise googlebot will be able to find them without sitemap.

mmm, that would prevent all searches from being indexed.
Obviously, that's not what I want.
The point is that you cannot predict if a search will have (meaningfull) results or not. So it depends on the content of the target page if I want the link to be indexed or not.

Regards,
marketplace.

That's why it should be done in your site code.