Competitor Analysis Tool: Analyze rival websites and gain valuable SEO and PPC insights
« previous next »
Pages: 1 2

support49

*
  • *
  • 43
Sitemap page limit
« on: March 24, 2008, 03:13:20 PM »
The sitemap generator has been working very well. I've broken the sitemap into 25,000 pages each and they all seem to be updating correctly. The one problem I've just started having however is that the number of pages indexed seems to be limited at 227,015. I've increased the time limit, but still have the same problem. I know the number of pages is constantly being changed so I don't know why the generator would stop at this number of pages.
Logged

XML-Sitemaps Support

*
  • *****
  • 11782
Re: Sitemap page limit
« Reply #1 on: March 25, 2008, 01:05:35 AM »
Hello,

do you re-generate sitemap from the scratch?
Logged

support49

*
  • *
  • 43
Re: Sitemap page limit
« Reply #2 on: March 25, 2008, 02:04:59 AM »
Yes, I believe that the sitemap is generated from scratch each evening.
Logged

support49

*
  • *
  • 43
Re: Sitemap page limit
« Reply #3 on: March 25, 2008, 12:39:02 PM »
Here's a daily changelog:

X-Powered-By: PHP/5.2.5
Set-Cookie: PHPSESSID=58d2bc9bd81ba4bd6a9168371fe8b6dd; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Content-type: text/html

<html>
<head>
<title>XML Sitemaps - Generation</title> <meta http-equiv="Content-type" content="text/html;charset=iso-8859-15" /> <link rel=stylesheet type="text/css" href="pages/style.css"> </head> <body> Resuming the last session (last updated: 1970-01-01 00:00:00)1 | 591 | 237.2 | 0:06 | 66:14 | 1 | 1,780.3 Kb | 1 | 0 | 1780 20 | 572 | 1,256.5 | 0:16 | 7:50 | 1 | 1,102.4 Kb | 20 | 621 | -678 40 | 552 | 1,585.4 | 0:21 | 5:01 | 1 | 819.6 Kb | 40 | 655 | -283 60 | 532 | 1,916.1 | 0:27 | 4:02 | 1 | 843.4 Kb | 60 | 701 | 24 80 | 512 | 2,246.7 | 0:32 | 3:29 | 1 | 873.4 Kb | 80 | 723 | 30 100 | 492 | 2,581.0 | 0:38 | 3:10 | 1 | 887.7 Kb | 100 | 735 | 14 120 | 472 | 2,906.5 | 0:45 | 2:57 | 1 | 897.5 Kb | 120 | 747 | 10 140 | 452 | 3,224.2 | 0:50 | 2:42 | 1 | 908.6 Kb | 140 | 757 | 11 160 | 432 | 3,577.4 | 0:56 | 2:31 | 1 | 961.7 Kb | 160 | 805 | 53 180 | 412 | 3,895.8 | 1:01 | 2:20 | 1 | 952.0 Kb | 180 | 826 | -9 200 | 392 | 4,227.1 | 1:06 | 2:11 | 1 | 1,001.0 Kb | 200 | 841 | 49 220 | 372 | 4,553.4 | 1:12 | 2:02 | 1 | 1,006.5 Kb | 220 | 860 | 5 240 | 352 | 4,882.8 | 1:17 | 1:53 | 1 | 1,011.2 Kb | 240 | 877 | 5 260 | 332 | 5,215.1 | 1:23 | 1:46 | 1 | 1,037.0 Kb | 260 | 886 | 26 280 | 312 | 5,558.8 | 1:28 | 1:38 | 1 | 1,062.9 Kb | 280 | 900 | 25 300 | 292 | 5,884.6 | 1:33 | 1:31 | 1 | 1,057.7 Kb | 300 | 905 | -5 320 | 272 | 6,215.7 | 1:39 | 1:24 | 1 | 1,088.5 Kb | 320 | 914 | 31 340 | 252 | 6,552.0 | 1:44 | 1:17 | 1 | 1,088.3 Kb | 340 | 920 | 0 360 | 232 | 6,879.3 | 1:50 | 1:11 | 1 | 1,106.3 Kb | 360 | 926 | 18 380 | 212 | 7,203.7 | 1:55 | 1:04 | 1 | 1,118.8 Kb | 380 | 931 | 12 400 | 192 | 7,529.5 | 2:01 | 0:58 | 1 | 1,148.6 Kb | 400 | 941 | 30 420 | 172 | 7,854.1 | 2:07 | 0:52 | 1 | 1,157.1 Kb | 420 | 950 | 9 440 | 152 | 8,179.1 | 2:12 | 0:45 | 1 | 1,172.4 Kb | 440 | 953 | 15 460 | 132 | 8,506.1 | 2:17 | 0:39 | 1 | 1,205.4 Kb | 460 | 960 | 33 480 | 112 | 8,832.9 | 2:23 | 0:33 | 1 | 1,202.5 Kb | 480 | 965 | -3 500 | 92 | 9,153.9 | 2:28 | 0:27 | 1 | 1,216.7 Kb | 500 | 965 | 14 520 | 72 | 9,502.3 | 2:34 | 0:21 | 1 | 1,242.5 Kb | 520 | 972 | 26 540 | 52 | 9,820.8 | 2:39 | 0:15 | 1 | 1,245.5 Kb | 540 | 972 | 3 560 | 32 | 10,152.3 | 2:45 | 0:09 | 1 | 1,257.0 Kb | 560 | 973 | 12 580 | 12 | 10,476.4 | 2:50 | 0:03 | 1 | 1,279.1 Kb | 580 | 974 | 22 600 | 379 | 11,194.5 | 2:58 | 1:52 | 2 | 2,141.2 Kb | 591 | 597 | 862 620 | 359 | 13,263.6 | 3:12 | 1:51 | 2 | 2,943.0 Kb | 611 | 645 | 802 640 | 339 | 15,863.6 | 3:30 | 1:51 | 2 | 3,224.9 Kb | 631 | 705 | 281 660 | 319 | 18,437.2 | 3:48 | 1:50 | 2 | 3,203.3 Kb | 651 | 765 | -21 680 | 299 | 21,012.5 | 4:05 | 1:48 | 2 | 3,251.7 Kb | 671 | 825 | 48 700 | 279 | 23,567.5 | 4:23 | 1:45 | 2 | 3,283.6 Kb | 691 | 885 | 32 720 | 259 | 26,137.1 | 4:40 | 1:41 | 2 | 3,329.4 Kb | 711 | 945 | 46 740 | 239 | 28,706.5 | 4:58 | 1:36 | 2 | 3,341.3 Kb | 731 | 1005 | 12 760 | 219 | 31,274.6 | 5:16 | 1:31 | 2 | 3,403.1 Kb | 751 | 1065 | 62 780 | 199 | 33,874.3 | 5:34 | 1:25 | 2 | 3,434.8 Kb | 771 | 1125 | 31 800 | 179 | 36,468.0 | 5:52 | 1:18 | 2 | 3,458.3 Kb | 791 | 1185 | 24 820 | 159 | 39,035.5 | 6:10 | 1:11 | 2 | 3,490.6 Kb | 811 | 1246 | 32 840 | 139 | 41,596.8 | 6:27 | 1:04 | 2 | 3,504.0 Kb | 831 | 1309 | 14 860 | 119 | 44,132.2 | 6:45 | 0:56 | 2 | 3,546.4 Kb | 851 | 1369 | 42 880 | 99 | 46,685.9 | 7:03 | 0:47 | 2 | 3,584.4 Kb | 871 | 1430 | 38 900 | 79 | 49,274.7 | 7:21 | 0:38 | 2 | 3,620.1 Kb | 891 | 1493 | 36 920 | 59 | 51,842.5 | 7:38 | 0:29 | 2 | 3,672.5 Kb | 911 | 1554 | 52 940 | 39 | 54,407.7 | 7:56 | 0:19 | 2 | 3,738.0 Kb | 931 | 1614 | 66 960 | 19 | 56,950.5 | 8:14 | 0:09 | 2 | 3,730.2 Kb | 951 | 1674 | -8 980 | 760 | 59,919.2 | 8:32 | 6:37 | 3 | 1,637.4 Kb | 971 | 3 | -2093 1000 | 740 | 59,940.1 | 8:36 | 6:21 | 3 | 1,659.9 Kb | 982 | 13 | 22 1020 | 720 | 59,959.1 | 8:40 | 6:07 | 3 | 1,678.2 Kb | 992 | 23 | 19 1040 | 700 | 59,978.2 | 8:44 | 5:52 | 3 | 1,697.9 Kb | 1002 | 33 | 19 1060 | 680 | 59,997.1 | 8:47 | 5:38 | 3 | 1,715.7 Kb | 1012 | 43 | 18 1080 | 660 | 60,016.2 | 8:51 | 5:24 | 3 | 1,734.5 Kb | 1022 | 52 | 19 1100 | 640 | 60,035.2 | 8:55 | 5:11 | 3 | 1,757.0 Kb | 1032 | 62 | 23 1120 | 620 | 60,054.2 | 8:59 | 4:58 | 3 | 1,775.5 Kb | 1042 | 72 | 18 1140 | 600 | 60,073.2 | 9:03 | 4:46 | 3 | 1,794.1 Kb | 1052 | 82 | 19 1160 | 580 | 60,092.2 | 9:08 | 4:34 | 3 | 1,813.0 Kb | 1062 | 92 | 19 1180 | 560 | 60,111.1 | 9:12 | 4:22 | 3 | 1,830.9 Kb | 1072 | 102 | 17 1200 | 540 | 60,130.1 | 9:16 | 4:10 | 3 | 1,850.1 Kb | 1082 | 112 | 20 1220 | 520 | 60,149.1 | 9:20 | 3:58 | 3 | 1,869.3 Kb | 1092 | 122 | 19 1240 | 500 | 60,168.2 | 9:24 | 3:47 | 3 | 1,889.8 Kb | 1102 | 132 | 20 1260 | 480 | 60,187.2 | 9:28 | 3:36 | 3 | 1,908.1 Kb | 1112 | 142 | 19 1280 | 460 | 60,206.2 | 9:32 | 3:25 | 3 | 1,926.0 Kb | 1122 | 152 | 18 1300 | 440 | 60,225.2 | 9:36 | 3:14 | 3 | 1,943.8 Kb | 1132 | 162 | 17 1320 | 420 | 60,244.3 | 9:39 | 3:04 | 3 | 1,964.1 Kb | 1142 | 172 | 21 1340 | 400 | 60,263.4 | 9:44 | 2:54 | 3 | 1,983.2 Kb | 1152 | 182 | 19 1360 | 380 | 60,282.4 | 9:48 | 2:44 | 3 | 2,002.6 Kb | 1162 | 192 | 19 1380 | 360 | 60,301.6 | 9:52 | 2:34 | 3 | 2,022.9 Kb | 1172 | 202 | 20 1400 | 340 | 60,318.7 | 9:56 | 2:24 | 3 | 2,038.5 Kb | 1181 | 212 | 16 1420 | 320 | 60,337.7 | 10:00 | 2:15 | 3 | 2,057.1 Kb | 1191 | 222 | 19 1440 | 300 | 60,356.7 | 10:03 | 2:05 | 3 | 2,079.0 Kb | 1202 | 231 | 22 1460 | 280 | 60,373.8 | 10:07 | 1:56 | 3 | 2,096.1 Kb | 1211 | 240 | 17 1480 | 260 | 60,392.7 | 10:11 | 1:47 | 3 | 2,113.4 Kb | 1221 | 250 | 17 1500 | 240 | 60,411.7 | 10:15 | 1:38 | 3 | 2,132.5 Kb | 1231 | 260 | 19 1520 | 220 | 60,428.7 | 10:21 | 1:30 | 3 | 2,146.5 Kb | 1240 | 270 | 14 1540 | 200 | 60,447.7 | 10:26 | 1:21 | 3 | 2,164.5 Kb | 1250 | 280 | 18 1560 | 180 | 60,466.6 | 10:30 | 1:12 | 3 | 2,183.3 Kb | 1260 | 290 | 19 1580 | 160 | 60,483.8 | 10:34 | 1:04 | 3 | 2,213.0 Kb | 1269 | 299 | 30 1600 | 140 | 60,501.0 | 10:38 | 0:55 | 3 | 2,220.1 Kb | 1278 | 308 | 7 1620 | 120 | 60,519.9 | 10:42 | 0:47 | 3 | 2,237.9 Kb | 1288 | 318 | 17 1640 | 100 | 60,539.0 | 10:46 | 0:39 | 3 | 2,257.2 Kb | 1298 | 328 | 20 1660 | 80 | 60,557.9 | 10:50 | 0:31 | 3 | 2,275.5 Kb | 1308 | 338 | 18 1680 | 60 | 60,576.9 | 10:55 | 0:23 | 3 | 2,293.5 Kb | 1318 | 348 | 18 1700 | 40 | 60,595.9 | 10:59 | 0:15 | 3 | 2,311.2 Kb | 1328 | 358 | 18 1720 | 20 | 60,614.8 | 11:03 | 0:07 | 3 | 2,329.0 Kb | 1338 | 368 | 18 1740 | 375 | 60,633.8 | 11:08 | 2:24 | 4 | 2,139.0 Kb | 1348 | 0 | -190 1760 | 355 | 65,142.8 | 11:29 | 2:18 | 4 | 4,618.6 Kb | 1368 | 594 | 2479 1780 | 335 | 69,887.3 | 11:51 | 2:13 | 4 | 4,771.3 Kb | 1388 | 594 | 153 1800 | 315 | 74,631.7 | 12:13 | 2:08 | 4 | 4,791.2 Kb | 1408 | 594 | 20 1820 | 295 | 79,376.1 | 12:35 | 2:02 | 4 | 4,800.3 Kb | 1428 | 594 | 9 1840 | 275 | 84,120.6 | 12:56 | 1:56 | 4 | 4,819.9 Kb | 1448 | 594 | 19 1860 | 255 | 88,865.0 | 13:17 | 1:49 | 4 | 4,838.1 Kb | 1468 | 594 | 19 1880 | 235 | 93,609.4 | 13:38 | 1:42 | 4 | 4,857.1 Kb | 1488 | 594 | 19 1900 | 215 | 98,353.9 | 13:59 | 1:35 | 4 | 4,866.3 Kb | 1508 | 594 | 9 1920 | 195 | 103,098.3 | 14:21 | 1:27 | 4 | 4,877.9 Kb | 1528 | 594 | 11 1940 | 175 | 107,842.7 | 14:42 | 1:19 | 4 | 4,890.8 Kb | 1548 | 594 | 13 1960 | 155 | 112,587.2 | 15:07 | 1:11 | 4 | 4,915.0 Kb | 1568 | 594 | 25 1980 | 135 | 117,331.6 | 15:28 | 1:03 | 4 | 4,930.2 Kb | 1588 | 594 | 15 2000 | 115 | 122,076.0 | 15:50 | 0:54 | 4 | 4,944.6 Kb | 1608 | 594 | 14 2020 | 95 | 126,820.5 | 16:11 | 0:45 | 4 | 4,957.3 Kb | 1628 | 594 | 13 2040 | 75 | 131,564.9 | 16:34 | 0:36 | 4 | 4,979.1 Kb | 1648 | 594 | 22 2060 | 55 | 136,309.3 | 16:55 | 0:27 | 4 | 4,998.1 Kb | 1668 | 594 | 19 2080 | 35 | 141,053.8 | 17:15 | 0:17 | 4 | 5,010.4 Kb | 1688 | 594 | 12 2100 | 15 | 145,798.2 | 17:36 | 0:07 | 4 | 5,028.3 Kb | 1708 | 594 | 18
2115 | 0 | 149,356.5 | 17:52 | 0:00 | 5 | 3,624.0 Kb | 1723 | 0 | -1404 <h4>Completed</h4>Total pages indexed: 1723<br>Calculating changelog...<br>Creating HTML sitemap...<script> top.location = 'index.php?op=view'
</script>
Logged

XML-Sitemaps Support

*
  • *****
  • 11782
Re: Sitemap page limit
« Reply #4 on: March 26, 2008, 02:00:38 AM »
Can you provide an example URL that has not been included in sitemap?
Logged

support49

*
  • *
  • 43
Re: Sitemap page limit
« Reply #5 on: March 26, 2008, 03:06:39 AM »
I'm not sure how to go about that, but I do know that their are thousands of pages that get updated each evening based on available properties for sale in different markets around the country. The odds of the number of pages being the same over an extended period of time is very low, yet the number of pages indexed has remained constant for over a month.
Logged

XML-Sitemaps Support

*
  • *****
  • 11782
Re: Sitemap page limit
« Reply #6 on: March 26, 2008, 11:06:06 PM »
Try to remove all sitemap files from generator/data/ folder and recreate sitemap.
Logged

support49

*
  • *
  • 43
Re: Sitemap page limit
« Reply #7 on: March 30, 2008, 04:43:33 AM »
Now I see that I've been getting errors in the generator configuration:

Sitemap file is not writable: /home/ihsrea5/public_html/sitemap11.xml
Sitemap file is not writable: /home/ihsrea5/public_html/sitemap12.xml
Sitemap file is not writable: /home/ihsrea5/public_html/sitemap13.xml
Sitemap file is not writable: /home/ihsrea5/public_html/sitemap14.xml
Sitemap file is not writable: /home/ihsrea5/public_html/sitemap15.xml
Sitemap file is not writable: /home/ihsrea5/public_html/sitemap16.xml
Sitemap file is not writable: /home/ihsrea5/public_html/sitemap17.xml
Sitemap file is not writable: /home/ihsrea5/public_html/sitemap18.xml
Sitemap file is not writable: /home/ihsrea5/public_html/sitemap19.xml
Sitemap file is not writable: /home/ihsrea5/public_html/sitemap110.xml
Logged

support49

*
  • *
  • 43
Re: Sitemap page limit
« Reply #8 on: March 30, 2008, 05:51:28 AM »
I've upgraded to 2.8 and now I get this error:

Warning: include(/home/ihsrea5/public_html/generator/pages/page-login.inc.php) [function.include]: failed to open stream: Permission denied in /home/ihsrea5/public_html/generator/index.php(2) : eval()'d code on line 60

Warning: include() [function.include]: Failed opening '/home/ihsrea5/public_html/generator/pages/page-login.inc.php' for inclusion (include_path='.:/usr/lib/php:/usr/local/lib/php.;pages/;/home/ihsrea5/public_html/generator\pages') in /home/ihsrea5/public_html/generator/index.php(2) : eval()'d code on line 60


index.php is set to 0644
« Last Edit: March 30, 2008, 05:53:43 AM by support49 »
Logged

XML-Sitemaps Support

*
  • *****
  • 11782
Re: Sitemap page limit
« Reply #9 on: March 30, 2008, 09:09:39 PM »
Try to reupload all sitemap generator files.
As for sitemaps issue, you should manually create empty sitemap2(3,4,5,..).xml files and set permissions to 0666 for them.
Logged

support49

*
  • *
  • 43
Re: Sitemap page limit
« Reply #10 on: March 31, 2008, 11:49:10 AM »
The script has been running for about a day now and no sitemap files have been created yet. The crawl_dump.log file is at 5,510kb and I got the exact same email as above in the post at 11:19pm and this email at 4:01am:

X-Powered-By: PHP/5.2.2
Set-Cookie: PHPSESSID=bbf5e3001bcd57e981b623c94a17910b; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Content-type: text/html

<html>
<head>
<title>XML Sitemaps - Generation</title> <meta http-equiv="Content-type" content="text/html;charset=iso-8859-15" /> <link rel=stylesheet type="text/css" href="pages/style.css"> </head> <body> <h4>Already in progress. Current process state is displayed:</h4><div id="glog"> Links depth: <b><span id="llevel">-</span></b> <br> Current page: <span id="cpage">-</span> <br> Pages added to sitemap: <span id="compno">-</span> <br> Pages scanned: <span id="pdone">-</span> (<span id="bdone">-</span> Kb) <br> Pages left: <span id="pleft">-</span> (+ <span id="l2">-</span> queued for the next depth level) <br> Time passed: <span id="tdone">-</span> <br> Time left: <span id="tleft">-</span> <br> Memory usage: <span id="musage">-</span> </div> <script language="Javascript"> function mRzUw6eeyzk(id,txt) { el = document.all[id] el.innerHTML = txt } function ouBX34uQh_Fu3VD(txt1,txt2,txt3,txt4,txt5,txt6,txt7,txt8,txt9,txt10)
{
mRzUw6eeyzk('cpage',txt1)
mRzUw6eeyzk('pleft',txt2)
mRzUw6eeyzk('pdone',txt3)
mRzUw6eeyzk('bdone',txt4)
mRzUw6eeyzk('tdone',txt5)
mRzUw6eeyzk('tleft',txt6)
mRzUw6eeyzk('llevel',txt7)
mRzUw6eeyzk('musage',txt8)
mRzUw6eeyzk('compno',txt9)
mRzUw6eeyzk('l2',txt10)
}
</script>
<script>ouBX34uQh_Fu3VD('preview_homes.php?MLSID=5268729',
'322',
'28500',
'1,037,920.7',
'1439:29',
'16:15',
'14',
'14,190.2 Kb',
'12188',
'1983'
);</script>
« Last Edit: March 31, 2008, 11:53:31 AM by support49 »
Logged

XML-Sitemaps Support

*
  • *****
  • 11782
Re: Sitemap page limit
« Reply #11 on: April 01, 2008, 01:55:11 AM »
Hello,

it looks like your server configuration doesn't allow to run the script long enough to create full sitemap. Please try to increase memory_limit and max_execution_time settings in php configuration at your host (php.ini file) or contact hosting support regarding this.
Logged

support49

*
  • *
  • 43
Re: Sitemap page limit
« Reply #12 on: April 01, 2008, 02:55:49 AM »
I think it may actually still running, the crawl_dump.log file keeps getting bigger and bigger. It's at 7,843kb now and growing.

The max_execution_time is set at 15000 and memory_limit is at 128M.

I've reuploaded all of the generator files from 2.8. and still get:

Warning: include(/home/ihsrea5/public_html/generator/pages/page-login.inc.php) [function.include]: failed to open stream: Permission denied in /home/ihsrea5/public_html/generator/index.php(2) : eval()'d code on line 60

Warning: include() [function.include]: Failed opening '/home/ihsrea5/public_html/generator/pages/page-login.inc.php' for inclusion (include_path='.:/usr/lib/php:/usr/local/lib/php.;pages/;/home/ihsrea5/public_html/generator\pages') in /home/ihsrea5/public_html/generator/index.php(2) : eval()'d code on line 60
« Last Edit: April 01, 2008, 03:53:52 AM by support49 »
Logged

support49

*
  • *
  • 43
Re: Sitemap page limit
« Reply #13 on: April 01, 2008, 08:34:51 PM »
crawl_dump.log is now at 11MB
Logged

XML-Sitemaps Support

*
  • *****
  • 11782
Re: Sitemap page limit
« Reply #14 on: April 01, 2008, 10:39:43 PM »
Please PM me your generator URL.
Logged
Pages: 1 2
« previous next »