Sitemap Generator Forum

Hello, I have been using Generator for years and just recently, on one of the sites, Generator stopped indexing beyond the index.php page. ie I had sitemaps, (ror,xml,txt etc) but with only 1 page indexed instead of the usual 350-plus pages, either via cron or manually. As far as I can gather, the script does not error-out, it simply stops. My Apache host has upgraded to PHP 5.4.21 recently. I don't have .htaccess in the root as I am trying to resolve this issue and robots.txt is not the problem. Can you advise please? Thanks, Nick.

Will an upgrade to your latest version fix this? I just installed Standalone Sitemap Generator (PHP) v6.1, 2013-11-19 and no change.  

FOR INFO:
To add, just now, via a cron job, and at another php site and built by a different php developer but hosted at the same hosting - the Generator failed to index some 250+ pages. Just the index.php and index.html were counted in the sitemaps. Sitemaps were generated, but only 2 pages were indexed.

I really don't know if it is the hosting's php config. or the Generator script? Incidentally, this evening, another Generator script worked perfectly on another site (same hosting) that was all html (no php).

Hope this helps in your diagnosis !

Regards, Nicky.   

Hello,

please try our search engine bot simulator tool to check if the site is "crawlable":
https://www.xml-sitemaps.com/se-bot-simulator.html

SE Bot Simulation Results
[ External links are visible to forum administrators only ]
Page layout as seen by the robots
Proxy access not allowed

SE Bot Simulation Results
[ External links are visible to forum administrators only ]

Page layout as seen by the robots
Proxy access not allowed
Raw HTTP Headers »»
Internal Links Found
none found
Restricted with robots.txt
none found
Restricted with rel="nofollow" attribute
none found
External URLs
none found

There is no .htaccess in the root ?!?

ISSUE RESOLVED

I found this in the index.php which prevented crawling:

<?php if(@fsockopen($_SERVER['REMOTE_ADDR'], 80, $errstr, $errno, 1))
die("Proxy access not allowed"); ?>

Thanks,
Nicky

I'm glad you got it resolved!