Not crawling whole site
« on: July 03, 2007, 08:49:55 PM »
I have the stand alone which you just installed...it is not doing my whole site. THe settings look right but only does 3943 pages I have or urls...I have over 35,000 .....I need whole site indexed. When it crawled it looked like it was getting all but the at end same number as before.
Re: Not crawling whole site
« Reply #1 on: July 05, 2007, 04:41:14 PM »
I have the same problem.

I have installed the standalone registered version.

I have giving the starting URL as [ External links are visible to forum administrators only ]. It is doing all pages for this but it is not including my sub-domain at [ External links are visible to forum administrators only ].

When I try to give the starting URL as [ External links are visible to forum administrators only ] and click the button for crawling, it does not continue and comes back to the configuration page.

Please advice how to crawl the entire site including sub-domains.

Thanks
Regards
Raj



Cheers
Raj
Re: Not crawling whole site
« Reply #2 on: July 05, 2007, 09:26:17 PM »
I would like to add to my post with it did do more of site but still not close....it does about 15,000 pages...I noticed it says link level 2....this needs to be increased as my site has many folders with many galleries...so I think this may be the problem. Where do I change settings for this?
Thank you
Re: Not crawling whole site
« Reply #3 on: July 05, 2007, 10:48:35 PM »
Hello,

please let me know (in private message) an example URL that is not included in sitemap and how it can be reached from homepage.

Raj,
Quote
I have installed the standalone registered version.

I have giving the starting URL as  It is doing all pages for this but it is not including my sub-domain at
According to XML sitemap protocol, it is not allowed to include URLs from different domains or subdomains in the same sitemap, so you will have to create a separate sitemap for every domain.
Re: Not crawling whole site
« Reply #4 on: July 06, 2007, 02:53:37 AM »
Hi Oleg,

If I cannot include my sub-domains as part of my main domain for site-mapping, does it mean that I have to install XML-Stiemap software again separately in the sub-domain folder?

Please advice.

Thanks
Regards
Raj




Cheers
Raj
Re: Not crawling whole site
« Reply #5 on: July 06, 2007, 08:17:13 PM »
Hello,

yes, you should install separate instances of Sitemap Generator per (sub)domain.