Competitor Analysis Tool: Analyze rival websites and gain valuable SEO and PPC insights
« previous next »
Pages: 1

matthew

*
  • *
  • 6
I can't get the crawler to crawl my entire website
« on: December 31, 2006, 04:06:55 PM »
I do not know how many pages I have in the entire site, but when I do a search for a certain category number it isn't showing so I know it is not reaching my entire site.

Anyways to fix this issue?

I have had it on 0 for unlimited still nothing.
Logged

matthew

*
  • *
  • 6
Re: I can't get the crawler to crawl my entire website
« Reply #1 on: December 31, 2006, 08:11:25 PM »
Update;

What I did was create a sitemap that had every single category for the site, and since then it was able to fully see the site and report it to the xml.

Thanks
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: I can't get the crawler to crawl my entire website
« Reply #2 on: January 02, 2007, 09:11:34 PM »
Great!

I confirm that you should have links to all you category pages so that sitemap generator crawler is able to find your pages starting from homepage.
Logged
Pages: 1
« previous next »