cds

*
  • *
  • 3
The whole site is not getting crawled
« on: January 30, 2006, 01:46:35 AM »

Hi guys,

I've just bought the standalone site generator and I'm having trouble trying to gent completely crawled.

For example:

[ External links are visible to forum administrators only ]

Has about 21 links, and only these are in the urllist.txt file:

[ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]

Also, there are some URL's that I need to know where are they coming from (referer). Is there a way to know it? I haven't been able to find it.

Best regards,

César Enrique

Re: The whole site is not getting crawled
« Reply #1 on: January 30, 2006, 02:01:43 AM »
Hello,

please try to redownload the generator again (using the same link) and install it.
Let me know if that helps.

cds

*
  • *
  • 3
Re: The whole site is not getting crawled
« Reply #2 on: January 30, 2006, 02:56:39 AM »

Hello,

It  worked!. Thanks a lot.

I'll let you know how's everytghing going. So far it's ok.

Regards

César Enrique

cds

*
  • *
  • 3
Re: The whole site is not getting crawled (Date Problem)
« Reply #3 on: January 30, 2006, 01:25:09 PM »
Hello Again,

I'm checking my the xml generated sitemap and I see some dates like this:

- <url>
  <loc>[ External links are visible to forum administrators only ]</loc>
  <priority>0.9</priority>
  <lastmod>2008-05-01T00:00:00+00:00</lastmod>
  <changefreq>weekly</changefreq>
  </url>
- <url>
- <url>
  <loc>[ External links are visible to forum administrators only ]</loc>
  <priority>0.9</priority>
  <lastmod>2008-05-01T00:00:00+00:00</lastmod>
  <changefreq>weekly</changefreq>
  </url>
+ <url>

My server date is ok, and actually, the file "listados" is the same for both url, so I think it should be a problem in the scripts.

Please give me a feedback. Best regards,

César Enrique
Re: The whole site is not getting crawled
« Reply #5 on: February 06, 2006, 02:54:14 PM »
I am having the same problem. I've redownloaded the script, reinstalled it, run it, and getting the same results.

I've checked my root folder, no robot.txt in sight...

Any ideas?

Thanks ???
Re: The whole site is not getting crawled
« Reply #6 on: February 06, 2006, 08:14:58 PM »
Hello,


please PM me the access details for your Sitemap Generator instance and let me know the examples of URLs that are not included in sitemap.

Thank you!
Re: The whole site is not getting crawled
« Reply #7 on: February 06, 2006, 10:34:39 PM »
Actually, I figured it out on my own....  Nothing wrong with the script, it was the moron using it!  LOL :D

Thanks though,
Anne