Competitor Analysis Tool: Analyze rival websites and gain valuable SEO and PPC insights
« previous next »
Pages: 1

glimbeek

*
  • **
  • 15
cronjob
« on: October 26, 2009, 12:05:44 PM »
I create a cronjob on my server using Cron-5min.php (there's no cpanel or something along those lines)
What do I put into the Cron-5min.php file so that my site gets crawled?
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: cronjob
« Reply #1 on: October 27, 2009, 04:02:07 PM »
Hello,

try to put this:
Code: [Select]
system('/usr/bin/php /path/to/your/generator/runcrawl.php');
Logged

glimbeek

*
  • **
  • 15
Re: cronjob
« Reply #2 on: October 28, 2009, 10:20:14 AM »
Thanks for the reply,

Just to be sure I'm doing it correctly...

I got my file cron-5min.php
in this file I put the following:

Code: [Select]
<?phpsystem('/usr/bin/php /path/to/my/generator/runcrawl.php');?>

Because when I look at my generator tool in the crawling tab it says the command line should be the following:
Code: [Select]
/usr/local/bin/php /path/to/my/generator/runcrawl.php
Now I tried both but the tool isn't crawling.

The difference is in the 'local".
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: cronjob
« Reply #3 on: October 28, 2009, 09:02:57 PM »
It's possible that php is located in a different folder in your case, you can ask your hosting support to find it out exactly.
Logged

glimbeek

*
  • **
  • 15
Re: cronjob
« Reply #4 on: October 29, 2009, 01:02:42 PM »
Yeah I figured as much so I already mailed them. They seem to be finding it hard to grasp the concept of a command line instead of using curl... I'll get there though :)
Logged

glimbeek

*
  • **
  • 15
Re: cronjob
« Reply #5 on: November 02, 2009, 08:44:49 AM »
After a fair amount of mailing they told me to put the following in the cronjob file, cron-5min.php:
Code: [Select]
<?phpsystem('wget -O /dev/null http://www.my-domain.com/location/of/generator-folder/runcrawl.php');?>

If I browse to my cronjob file it returns the following:
Malformed header from CGI script: => `/dev/null' Resolving [ External links are visible to forum administrators only ]... **.**.**.** Connecting to [ External links are visible to forum administrators only ]|**.**.**.**|:80... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] 0K 4.86 MB/s 09:33:22 (4.86 MB/s) - `/dev/null' saved [51] X-Powered-By: PHP/5.2.6 Content-type: text/html

But the sitemap isn't updated nor is the "Request date" changed in the generator tool.

**EDIT**

After some more mailing, my host did some testing and they came up with the following:

Code: [Select]
<?phpsystem('wget --delete-after http://www.my-domain.com/path/to/generator-tool/runcrawl.php');?>

Which results into:
"I'd like to inform that the source of inconvenience seems to be in runcrawl.php file which cannot provide proper headers to php interpreter.
I afford to modify Your cron file, and replace actual wget command parameter to --delete-after. This action shows that header is malformed by runcrawl.php file.
Unfortunately, mentioned file is encrypted, so we cannot analyze its content.
We suggest You to contact author of the script to get more information concerning Your request."
« Last Edit: November 02, 2009, 09:42:27 AM by glimbeek »
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: cronjob
« Reply #6 on: November 02, 2009, 09:41:42 PM »
You can replace runcrawl.php file with:
Code: [Select]
<?php	$op = $_REQUEST['op'] = 'crawlproc';	$cmdlined = true;	chdir(dirname(__FILE__));	$_REQUEST['bg'] = true;	$_REQUEST['resume'] = true;	include './index.php';?>
Logged

glimbeek

*
  • **
  • 15
Re: cronjob
« Reply #7 on: November 03, 2009, 07:05:55 AM »
Seems to be working. Thanks for the fast reply!
Logged
Pages: 1
« previous next »