Tools for Any SEO Challenge: Get more traffic to your website with versatile solutions for every step of your SEO journey.
« previous next »
Pages: 1

kevin18

*
  • *
  • 1
Issues crawling our site
« on: November 22, 2016, 07:08:28 PM »
We have a Xencart site, the system has around 70k active items but the sitemap generator only found around 58k pages I would expect it to find significantly more that that.

It also takes 14 hours+ to run the crawl, how do we speed that up?
-Kevin
Logged

XML-Sitemaps Support

*
  • *****
  • 11780
Re: Issues crawling our site
« Reply #1 on: November 23, 2016, 06:40:50 AM »
Hello,

could you please PM me your generator URL and an example URL that is not included in sitemap and how it can be reached starting from homepage?
 
The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.

You can use "Exclude URLs"/ "Add directly in sitemap" settings to increase crawling speed.
Logged
Pages: 1
« previous next »