Sitemap Generator Forum

Hi,

I've been trying to get the site I'm working on to crawl correctly however it keeps stopping on a certain page. I've tried using the tools on your site also: the headers return 200 correctly, however the SE tool returns nothing for Google and Yahoo but works correctly for MSN.
The link to the page stopping progress is: [ External links are visible to forum administrators only ]. This comes off this page: [ External links are visible to forum administrators only ] which works correctly.

Any ideas? Any help would be immensely appreciated!

SE simulator reports no issues for that page as I see:
https://www.xml-sitemaps.com/se-bot-simulator.html?op=se-bot-simulator&go=1&pageurl=http%3A%2F%2Fwww.workboot.co.nz%2Fbrowse%2Fcategory%2FArchitectural_Designers%2F189.html&se=msnbot&submit=Start

Hi,

It appears to work for MSNBot but now Google. Why would this be?

Thanks for your help.

Perhaps your server blocks requests from Google IPs.

Hi,

I don't think that is the case as [ External links are visible to forum administrators only ] crawls correctly.
I suspect that this page is the reason that the sitemap generator does not generate a full sitemap.
What algorithm do you use for the sitemap crawler to extract links? Is it a regular expression? I can modify the page to suit however I simply need to find out what to change it to. It is currently XHTML compliant [ External links are visible to forum administrators only ] so the HTML is valid.
Any further ideas?

You can use our SE bot simulator to find out which link on page are found by sitemap generator:
https://www.xml-sitemaps.com/se-bot-simulator.html?op=se-bot-simulator&go=1&pageurl=http%3A%2F%2Fwww.workboot.co.nz%2Fbrowse%2Findex.html&se=googlebot&submit=Start

Sorry; I don't think you quite understand:

[ External links are visible to forum administrators only ] works perfectly. The SE bot simulator works on this page with all bot types, and so does the sitemap generator.

[ External links are visible to forum administrators only ] or similar does not work correctly. This doesn't work with the SE bot simulator for Google and Yahoo bots however DOES work for the MSN bot. It also does not work correctly with the Sitemap generation tool; that is no pages below that depth are added to the sitemap.
What I don't understand is what is different between pages. What is different between the Google bot and the MSN bot? The page is XHTML compliant so what is it on the page that is stopping the Google bot from finding any links?

Thanks.

Hello,

that page doesn't load completely when googlebot is specified for user-agent. You can try that with "user agent switched" firefox add-on and opening the page in browser identified as googlebot.

Thanks for that!
Turned out to be the exact issue because of a bug in the ASP.NET framework for the new Google user agent string.

You are welcome!