Elanor

*
  • *
  • 6
  • Web Developer
Full Crawl but no new Sitemap
« on: May 29, 2008, 04:04:53 PM »
I've been having problems with permissions and files ever since we installed this. It works off and on, and I don't understand why. The main problem is it takes so frickin long to generate a site-map I can't just see if its working immediately (our site is rather large, only about 40k links in the sitemap, but about 100k pages to sort through - takes about a day)

I recieve the email after the cron job completes, and I get to the very bottom of the list and this is what I see:
Code: [Select]
<h4>Completed</h4>Total pages indexed: 38832<br>Calculating changelog...PHP Warning:  Invalid argument supplied for foreach() in public_html/sitemap/pages/class.templates.inc.php(2) : eval()'d code on line 75

Warning: Invalid argument supplied for foreach() in public_html/sitemap/pages/class.templates.inc.php(2) : eval()'d code on line 75
Elanor
Web Developer
Test Equipment Connection
Re: Full Crawl but no new Sitemap
« Reply #1 on: May 29, 2008, 11:08:32 PM »
Hello,

do you mean that sitemap is not updated after that and you have no new entries on Changelog page?

Elanor

*
  • *
  • 6
  • Web Developer
Re: Full Crawl but no new Sitemap
« Reply #2 on: May 30, 2008, 01:50:23 PM »
Yes. When it ran on the 27th, it ran all the way through the site, got to the end said completed - and yet the sitemap is still dated May 15. I ran it again manually yesterday and again, the sitemap is still dated the 15th and no sitemap is shown to be in progress.
I was ignoring this problem before because half the time it would give me this error, yet the sitemap would still be updated.

What's wrong? ???
Elanor
Web Developer
Test Equipment Connection
Re: Full Crawl but no new Sitemap
« Reply #3 on: May 30, 2008, 07:24:58 PM »
Make sure that you have 0666 permissions for sitemap.xml file and 0777 for generator/data/ folder.

Elanor

*
  • *
  • 6
  • Web Developer
Re: Full Crawl but no new Sitemap
« Reply #4 on: June 02, 2008, 05:23:52 PM »
Make sure that you have 0666 permissions for sitemap.xml file and 0777 for generator/data/ folder.

Yes, both are set. However, I do recall having to change the owner of these folders in order to change their permissions via FTP to the user of the html account.. should they be owned by a different user? (this is on unix)
Elanor
Web Developer
Test Equipment Connection
Re: Full Crawl but no new Sitemap
« Reply #5 on: June 02, 2008, 11:04:56 PM »
In case if you have 0666/0777 permissions, it should work with any user id.
Please try to reinstall generator in a new folder and recheck it.

Elanor

*
  • *
  • 6
  • Web Developer
Re: Full Crawl but no new Sitemap
« Reply #6 on: June 05, 2008, 04:52:09 PM »
I've reuploaded the generator into a new folder and its running right now. Hopefully it works. Thanks for your help so far. I'll let you know the results.
 :-\
Elanor
Web Developer
Test Equipment Connection

Elanor

*
  • *
  • 6
  • Web Developer
Re: Full Crawl but no new Sitemap
« Reply #7 on: June 06, 2008, 07:10:52 PM »
Same thing happened with new sitemap installation. The sitemap generator ran to the end, but did not generate a new sitemap.
If I click on the "Crawling" Tab it shows 39,000 links added to the sitemap, 0 left - but thats where it stopped, and its waiting to be restarted.

In my data/ folder I have crawl_dump.log and crawl_state.log but no sitemap log like usual.

What am I doing wrong?
Elanor
Web Developer
Test Equipment Connection
Re: Full Crawl but no new Sitemap
« Reply #8 on: June 07, 2008, 01:53:08 AM »
Perhaps you have an endless loop of links on your site so the crawler cannot finish the process. Please let me know your generator URL/login in PM so that I can check that.

Elanor

*
  • *
  • 6
  • Web Developer
Re: Full Crawl but no new Sitemap
« Reply #9 on: June 09, 2008, 01:34:57 PM »
Why would it give me a PHP error then? PM'd to you.
Elanor
Web Developer
Test Equipment Connection