Unclear reason of removing URLs from sitemap
« on: April 28, 2014, 02:45:28 PM »
Dear developers,

Could you help me to understand why some URLs was removed from a sitemap, please?

All removed URLs are correct:
[ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]

Thank you in advance for your help,
Vlad
Re: Unclear reason of removing URLs from sitemap
« Reply #1 on: April 29, 2014, 07:05:31 AM »
Hello,

the typical reason would be that there are no links pointing to those pages anymore and generator bot cannot find them as a result.
Re: Unclear reason of removing URLs from sitemap
« Reply #2 on: May 02, 2014, 11:42:25 PM »
Dear Oleg,

But these URLs was included into the block "Skipped URLs - crawled but not added in sitemap".
If I correctly understood (based on the block description) these pages was found/crawled at the last run of a sitemap script, but was skipped. And for the each URL after symbol "-" must be specified reason of exclusion ("404 Not Found", "text/xml", "lu", "301 Moved Permanently" or other).
If my point of view is not correct could you please explain difference between blocks "Removed URLs" and "Skipped URLs - crawled but not added in sitemap".

All my pages are static and on the next run of the sitemap script these pages was found and included into sitemap.xml (please see attached image #1).

I have faced with this problem again, but with other URL (please see attached image #2):
[ External links are visible to forum administrators only ]

This URL included into the list of URLs of the page [ External links are visible to forum administrators only ] (please see attached image #3)

URL [ External links are visible to forum administrators only ] included into sitemap.xml (please see attached picture #4)

May be problem in my configuration of the Sitemap Generator?

Thank you in advance for your help,
Vlad
Re: Unclear reason of removing URLs from sitemap
« Reply #3 on: May 03, 2014, 03:20:17 PM »
Hello,

it means that generator bot got an error when trying to fetch those pages (one of the reasons could be that website server was overloaded at that moment and couldn't handle the request).
Re: Unclear reason of removing URLs from sitemap
« Reply #4 on: May 03, 2014, 06:37:12 PM »
Dear Oleg,

Is there ability definitely receive reason of the error?
Re: Unclear reason of removing URLs from sitemap
« Reply #5 on: May 05, 2014, 08:04:32 AM »
Hello,

you can try to check your server error logs for details. From the generator's crawler side it appears as "connection failed" attempt.