Odd - two problems:
(1) I can't run the crawler using standard ssh login. I tried executing the generator.php using the php command and now it won't run.
(2) A while ago the script stopped running. I upgraded to V 6.0, cleared out all the files and that doesn't run with cron either:
/usr/local/bin/php
/home/mydir/public_html/generator/runcrawl.php
Perhaps there is something that has changed? I can't figure it out after reading through the docs and these forums.