jr5

*
  • *
  • 8
script not crawling deeply enough
« on: May 17, 2019, 05:15:12 AM »
I have a page that has potentially 50,000 listings on it, and I want to be able to sitemap and index them all.  I had some sub-pages with a lot of links set on the home page, and it was crawling those and I would have a sitemap of about 1500 pages.  So to help get a sitemap with all of the pages, I experimented with a footer link with links of links to all the pages.  Since doing this, I have only gotten less than 300 pages on a crawl.

I checked the configuration and it says "maximum depth level", and it is set to 0, which says "unlimited".  However, it is not doing this.

How do I get it to crawl every single link on every single page?
Re: script not crawling deeply enough
« Reply #1 on: May 17, 2019, 07:21:38 PM »
Hello,

could you please PM me your generator URL and an example URL that is not included in sitemap and how it can be reached starting from homepage?
 

jr5

*
  • *
  • 8
Re: script not crawling deeply enough
« Reply #2 on: May 22, 2019, 10:33:25 PM »
Hi, I sent the PM's to you last Friday, but I haven't heard back yet.  Can you please look at it and respond to me?  Thank you.

jr5

*
  • *
  • 8
Re: script not crawling deeply enough
« Reply #3 on: May 22, 2019, 10:35:53 PM »
Hi sorry my email didn't notify me that you had privately responded already, I see the message, thank you for your help.