admin317

*
  • *
  • 6
  • http://www.dartthornton.com
How to configure the generator to crawl secure https pages?
« on: October 23, 2009, 01:40:38 PM »
I have just downloaded the latest version of  Unlimited PHP Sitemap Generator but cannot work out how to tell it to crawl the secure pages on my e-commerce website.
I have looked all over the documentation and the forums but seem to have missed the instructions.
Should I simply add 's' to the 'http' prefix in my site's URL?
Or is it something else?

« Last Edit: October 23, 2009, 01:45:53 PM by admin317 »

Re: How to configure the generator to crawl secure https pages?
« Reply #1 on: October 23, 2009, 01:55:27 PM »
Hello,

yes, you should simple specify Starting URL as [ External links are visible to logged in users only ]

admin317

*
  • *
  • 6
  • http://www.dartthornton.com
Re: How to configure the generator to crawl secure https pages?
« Reply #2 on: October 24, 2009, 12:13:59 AM »
Thank you, Admin, for your quick reply.
I have tried that method, however the Sitemap Generator only crawls one single page and then gives me the following message -
Request date:
23 October 2009, 16:02
Processing time:
0:00:01s
Pages indexed:
1
Sitemap files:
1
Pages size:
0.00Mb

I must have configured something wrongly, but I cannot work out what it is.
BTW, OpenSSL *is* enabled on my server
« Last Edit: October 24, 2009, 12:18:20 AM by admin317 »

admin317

*
  • *
  • 6
  • http://www.dartthornton.com
Re: How to configure the generator to crawl secure https pages?
« Reply #3 on: October 24, 2009, 10:59:47 AM »
Just some additional information that might help solve the problem:

1) ERROR MESSAGE
An error occured: There was an error while retrieving the URL specified: [ External links are visible to forum administrators only ]
HTTP headers follow:
Sat, 24 Oct 2009 09:47:16 GMT
Apache/2.2.3 (CentOS)
[ External links are visible to forum administrators only ]

2) ERROR LOG
My server's error log generated the following error lists the last couple of times I tried to get Sitemap Generator to crawl my site:

[Sat Oct 24 01:51:52 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/webmasters, referer: [ External links are visible to forum administrators only ]


[Sat Oct 24 01:51:52 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/SiteExplorerService, referer: [ External links are visible to forum administrators only ]


[Sat Oct 24 01:51:52 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/ping, referer: [ External links are visible to forum administrators only ]


[Sat Oct 24 01:51:52 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/ping, referer: [ External links are visible to forum administrators only ]


[Sat Oct 24 01:51:52 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/ping.aspx, referer: [ External links are visible to forum administrators only ]


[Sat Oct 24 01:51:52 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/rpc, referer: [ External links are visible to forum administrators only ]


[Sat Oct 24 02:46:45 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/robots.txt, referer: [ External links are visible to forum administrators only ]


[Sat Oct 24 02:46:46 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/webmasters, referer: [ External links are visible to forum administrators only ]


[Sat Oct 24 02:46:46 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/SiteExplorerService, referer: [ External links are visible to forum administrators only ]


[Sat Oct 24 02:46:46 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/ping, referer: [ External links are visible to forum administrators only ]


[Sat Oct 24 02:46:46 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/ping, referer: [ External links are visible to forum administrators only ]


[Sat Oct 24 02:46:46 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/ping.aspx, referer: [ External links are visible to forum administrators only ]


[Sat Oct 24 02:46:46 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/rpc, referer: [ External links are visible to forum administrators only ]
« Last Edit: October 24, 2009, 11:07:39 AM by admin317 »

Re: How to configure the generator to crawl secure https pages?
« Reply #4 on: October 25, 2009, 07:34:55 AM »
Hello,

[ External links are visible to logged in users only ] shows hosting control panel login page for me, no site content there.

admin317

*
  • *
  • 6
  • http://www.dartthornton.com
Re: How to configure the generator to crawl secure https pages?
« Reply #5 on: October 26, 2009, 05:07:46 AM »
Thanks for your reply, Oleg!
My apologies, I overlooked that fact that I'm using a *shared* SSL certificate.
The URL should be  [ External links are visible to forum administrators only ]
But this URL still gives me the same problem - only one page gets crawled:

Request date:
25 October 2009, 22:02
Processing time:
0:00:02s
Pages indexed:
1
Sitemap files:
1
Pages size:
0.02Mb

« Last Edit: October 26, 2009, 05:11:02 AM by admin317 »

admin317

*
  • *
  • 6
  • http://www.dartthornton.com
Re: How to configure the generator to crawl secure https pages?
« Reply #6 on: October 26, 2009, 05:22:01 AM »
Server Error Log Messages when I try to run the generator:

[Sun Oct 25 22:02:08 2009] [error] [client 220.237.137.184] File does not exist: /home/virtual/site199/fst/var/www/html/favicon.ico

[Sun Oct 25 22:02:11 2009] [error] [client 220.237.137.184] File does not exist: /home/virtual/site199/fst/var/www/html/favicon.ico

[Sun Oct 25 22:02:12 2009] [error] [client 220.237.137.184] File does not exist: /home/virtual/site199/fst/var/www/html/favicon.ico

[Sun Oct 25 22:02:29 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/webmasters, referer: [ External links are visible to forum administrators only ]

[Sun Oct 25 22:02:29 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/SiteExplorerService, referer: [ External links are visible to forum administrators only ]

[Sun Oct 25 22:02:29 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/ping, referer: [ External links are visible to forum administrators only ]

[Sun Oct 25 22:02:29 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/ping, referer: [ External links are visible to forum administrators only ]

[Sun Oct 25 22:02:29 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/ping.aspx, referer: [ External links are visible to forum administrators only ]

[Sun Oct 25 22:02:29 2009] [error] [client 209.151.28.209] File does not exist: /home/virtual/site199/fst/var/www/html/rpc, referer: [ External links are visible to forum administrators only ]
 ???

Re: How to configure the generator to crawl secure https pages?
« Reply #7 on: October 27, 2009, 04:12:06 PM »
Hello,

it looks like all shop links from inner pages are pointing to your main domain anyway, so they are autoamtically excluded from sitemap (only pages from the same domain as Starting URL can be included in sitemap).

Entries from the error log do not seem to be related to sitemap generator.

admin317

*
  • *
  • 6
  • http://www.dartthornton.com
Re: How to configure the generator to crawl secure https pages?
« Reply #8 on: October 28, 2009, 01:23:58 AM »
Okay, thanks Oleg. I shall try to figure out how to fix this...

Best wishes.