Keyword suggestion tool: Pick the right keywords to target in your SEO and PPC campaigns
« previous next »
Pages: 1

wilf86

*
  • *
  • 2
Does not crawl all the pages
« on: December 17, 2015, 05:31:17 AM »
We have CMS website (developed using DotNetNuke). Whenever we run the crawler it generates sitemap.xml for 500+ pages.

We expect the pages to be more than 9000+

What configuration should be done, please help.
Logged

XML-Sitemaps Support

*
  • *****
  • 11783
Re: Does not crawl all the pages
« Reply #1 on: December 17, 2015, 06:24:35 AM »
Hello,

could you please PM me your generator URL and an example URL that is not included in sitemap and how it can be reached starting from homepage?
 
Logged

wilf86

*
  • *
  • 2
Re: Does not crawl all the pages
« Reply #2 on: December 17, 2015, 07:39:24 AM »
example url
[ External links are visible to forum administrators only ]

We are using standalone sitemap generator configured in our server. using the credentials we login in and then go to Crawling tab (attached screen grab) and click run

starting from home page
1. [ External links are visible to forum administrators only ]
2. Click on Get Ideas from the top menu
3. In the find - select make -> Porsche , model -> Cayenne then click GO
4. Click on the first picture AudioMobile will reach you to the above page.

Logged

XML-Sitemaps Support

*
  • *****
  • 11783
Re: Does not crawl all the pages
« Reply #3 on: December 18, 2015, 07:12:04 AM »
Hello,

there must be a way to reach the pages by clicking links, so that crawler bot can find them. In this case it's required submitting the form (selecting item from dropdown list) hence it's not visible to a bot.
Logged
Pages: 1
« previous next »