Sitemap Generator Forum

Hello!

I Install the  site map  via:

php -q /home/Username/public_html/generator/runcrawl.php
(username been the Server username).

he run for near 20 minutes but I see that  under the
[ External links are visible to forum administrators only ]

Links depth: 2
Current page: cliente/announcements/119/Promocao-de-Maio---35-desconto-vitalicio.html
Pages added to sitemap: 128
Pages scanned: 160 (6,245.7 KB)
Pages left: 27 (+ 225 queued for the next depth level)
Time passed: 0:00:43
Time left: 0:00:07
Memory usage: 1,873.0 Kb

Seams  that he blocks,

Am runing  php 5.6


AM not very happy with the service.

gets  stuck on:




Last log access time: 2016-08-15 19:20:43
Is 19.22 now

Already in progress. Current process state is displayed:

Links depth: 2
Current page: cliente/announcements/119/Promocao-de-Maio---35-desconto-vitalicio.html
Pages added to sitemap: 128
Pages scanned: 160 (6,245.7 KB)
Pages left: 27 (+ 225 queued for the next depth level)
Time passed: 0:00:38
Time left: 0:00:06
Memory usage: 3,280.0 Kb

Hello,

what is the output you get in command line when running "php -q /..." command?

hello!

Via  ssh command, it finish how even  this is a dedicated server   only one website the traffic is Very low (for the server never gets 0.4 0.5 Load)

Created on: 15 August 2016, 23:07
Processing time: 1:49:45s
Pages indexed: 937

How can i speed up the Crawling rate? 

Hello,

what is the output you get after running the command though (it should display running progress in command line).

are you talking about this?


1 | 29 | 52.4 | 0:00:01 | 0:00:39 | 1 | 1,455.5 Kb | 1 | 29 | 1455
20 | 10 | 717.4 | 0:00:02 | 0:00:01 | 1 | 1,530.4 Kb | 19 | 108 | 1530
40 | 145 | 1,191.8 | 0:00:05 | 0:00:21 | 2 | 1,543.7 Kb | 35 | 23 | 1543
45 | 140 | 1,246.5 | 0:00:11 | 0:00:34 | 2 | 1,614.1 Kb | 38 | 29 | 1614
58 | 127 | 1,833.1 | 0:00:16 | 0:00:35 | 2 | 1,746.7 Kb | 50 | 72 | 1746
60 | 125 | 1,919.2 | 0:00:16 | 0:00:34 | 2 | 1,749.4 Kb | 52 | 76 | 1749
72 | 113 | 2,425.6 | 0:00:21 | 0:00:34 | 2 | 1,788.4 Kb | 64 | 93 | 1788
80 | 105 | 2,771.7 | 0:00:24 | 0:00:32 | 2 | 1,803.4 Kb | 72 | 103 | 1803
96 | 89 | 3,844.5 | 0:00:29 | 0:00:27 | 2 | 2,030.6 Kb | 88 | 127 | 2030
100 | 85 | 4,238.4 | 0:00:30 | 0:00:25 | 2 | 2,044.3 Kb | 92 | 135 | 2044
120 | 65 | 5,407.1 | 0:00:34 | 0:00:18 | 2 | 2,101.5 Kb | 104 | 167 | 2101
140 | 45 | 5,665.9 | 0:00:37 | 0:00:11 | 2 | 1,990.3 Kb | 111 | 180 | 1990
160 | 25 | 6,266.5 | 0:00:40 | 0:00:06 | 2 | 2,027.6 Kb | 130 | 229 | 2027
180 | 5 | 6,755.2 | 0:00:43 | 0:00:01 | 2 | 2,099.3 Kb | 150 | 298 | 2099
182 | 3 | 6,805.1 | 0:34:43 | 0:00:34 | 2 | 15,973.8 Kb | 152 | 303 | 15973
183 | 2 | 20,696.8 | 1:08:43 | 0:00:45 | 2 | 43,706.7 Kb | 153 | 303 | 43706
184 | 1 | 34,588.4 | 1:42:43 | 0:00:33 | 2 | 43,708.7 Kb | 154 | 303 | 43708
200 | 288 | 49,115.7 | 1:42:46 | 2:28:00 | 3 | 2,120.0 Kb | 170 | 49 | 2120
217 | 271 | 49,794.5 | 1:42:52 | 2:08:27 | 3 | 2,181.5 Kb | 185 | 83 | 2181
220 | 268 | 49,853.8 | 1:42:52 | 2:05:19 | 3 | 2,135.8 Kb | 186 | 88 | 2135
224 | 264 | 49,853.8 | 1:42:57 | 2:01:20 | 3 | 2,152.1 Kb | 187 | 88 | 2152
232 | 256 | 49,915.3 | 1:43:02 | 1:53:42 | 3 | 2,273.3 Kb | 189 | 88 | 2273
240 | 248 | 50,017.9 | 1:43:05 | 1:46:32 | 3 | 2,182.0 Kb | 192 | 90 | 2182
253 | 235 | 50,437.1 | 1:43:10 | 1:35:50 | 3 | 2,377.6 Kb | 201 | 124 | 2377
255 | 233 | 50,487.3 | 1:43:16 | 1:34:21 | 3 | 2,342.9 Kb | 202 | 126 | 2342
256 | 232 | 50,487.3 | 1:43:21 | 1:33:40 | 3 | 2,350.4 Kb | 202 | 126 | 2350
258 | 230 | 50,487.3 | 1:43:26 | 1:32:12 | 3 | 2,355.6 Kb | 202 | 126 | 2355
260 | 228 | 50,595.3 | 1:43:27 | 1:30:43 | 3 | 2,436.2 Kb | 204 | 136 | 2436
261 | 227 | 50,595.3 | 1:43:33 | 1:30:03 | 3 | 2,387.9 Kb | 204 | 136 | 2387
263 | 225 | 50,644.0 | 1:43:38 | 1:28:40 | 3 | 2,387.4 Kb | 205 | 141 | 2387
267 | 221 | 50,760.1 | 1:43:44 | 1:25:52 | 3 | 2,419.5 Kb | 207 | 152 | 2419
269 | 219 | 50,815.6 | 1:43:50 | 1:24:32 | 3 | 2,432.7 Kb | 208 | 158 | 2432
270 | 218 | 50,815.6 | 1:43:56 | 1:23:55 | 3 | 2,439.7 Kb | 208 | 158 | 2439
272 | 216 | 50,815.6 | 1:44:01 | 1:22:36 | 3 | 2,495.1 Kb | 208 | 158 | 2495
274 | 214 | 50,864.0 | 1:44:07 | 1:21:19 | 3 | 2,446.0 Kb | 209 | 163 | 2446
275 | 213 | 50,864.0 | 1:44:12 | 1:20:42 | 3 | 2,452.5 Kb | 209 | 163 | 2452
277 | 211 | 50,864.0 | 1:44:17 | 1:19:26 | 3 | 2,460.3 Kb | 209 | 163 | 2460
278 | 210 | 50,864.0 | 1:44:22 | 1:18:51 | 3 | 2,467.1 Kb | 209 | 163 | 2467
280 | 208 | 50,908.0 | 1:44:25 | 1:17:34 | 3 | 2,507.5 Kb | 210 | 163 | 2507
281 | 207 | 50,908.0 | 1:44:31 | 1:16:59 | 3 | 2,471.0 Kb | 210 | 163 | 2471
282 | 206 | 50,908.0 | 1:44:36 | 1:16:25 | 3 | 2,478.7 Kb | 210 | 163 | 2478
285 | 203 | 50,959.8 | 1:44:42 | 1:14:34 | 3 | 2,505.1 Kb | 211 | 168 | 2505
286 | 202 | 50,959.8 | 1:44:47 | 1:14:00 | 3 | 2,514.5 Kb | 211 | 168 | 2514
287 | 201 | 50,959.8 | 1:44:52 | 1:13:27 | 3 | 2,521.8 Kb | 211 | 168 | 2521
289 | 199 | 50,959.8 | 1:44:58 | 1:12:16 | 3 | 2,528.4 Kb | 211 | 168 | 2528
290 | 198 | 50,959.8 | 1:45:03 | 1:11:43 | 3 | 2,536.2 Kb | 211 | 168 | 2536
291 | 197 | 50,959.8 | 1:45:08 | 1:11:10 | 3 | 2,542.8 Kb | 211 | 168 | 2542
293 | 195 | 50,959.8 | 1:45:14 | 1:10:02 | 3 | 2,550.4 Kb | 211 | 168 | 2550
294 | 194 | 50,959.8 | 1:45:19 | 1:09:29 | 3 | 2,557.3 Kb | 211 | 168 | 2557
295 | 193 | 50,959.8 | 1:45:24 | 1:08:57 | 3 | 2,565.0 Kb | 211 | 168 | 2565
298 | 190 | 51,010.7 | 1:45:30 | 1:07:15 | 3 | 2,573.9 Kb | 212 | 173 | 2573
300 | 188 | 51,209.4 | 1:45:30 | 1:06:07 | 3 | 2,775.4 Kb | 214 | 173 | 2775
320 | 168 | 53,191.0 | 1:45:34 | 0:55:25 | 3 | 2,814.2 Kb | 234 | 173 | 2814
340 | 148 | 54,943.7 | 1:45:38 | 0:45:59 | 3 | 2,554.6 Kb | 252 | 173 | 2554
360 | 128 | 55,722.3 | 1:45:42 | 0:37:35 | 3 | 2,598.0 Kb | 260 | 173 | 2598
380 | 108 | 56,179.2 | 1:45:45 | 0:30:03 | 3 | 2,740.8 Kb | 275 | 193 | 2740
400 | 88 | 56,788.0 | 1:45:48 | 0:23:16 | 3 | 2,800.1 Kb | 295 | 221 | 2800
420 | 68 | 57,411.6 | 1:45:52 | 0:17:08 | 3 | 2,843.5 Kb | 315 | 221 | 2843
440 | 48 | 57,991.3 | 1:45:56 | 0:11:33 | 3 | 2,873.0 Kb | 335 | 261 | 2873
460 | 28 | 58,419.9 | 1:46:00 | 0:06:27 | 3 | 2,942.0 Kb | 355 | 277 | 2942
480 | 8 | 58,919.0 | 1:46:04 | 0:01:46 | 3 | 3,012.7 Kb | 375 | 310 | 3012
500 | 302 | 60,043.0 | 1:46:08 | 1:04:06 | 4 | 2,998.1 Kb | 393 | 16 | 2998
520 | 282 | 61,452.1 | 1:46:13 | 0:57:36 | 4 | 2,939.0 Kb | 412 | 40 | 2939
540 | 262 | 62,101.8 | 1:46:17 | 0:51:34 | 4 | 3,207.2 Kb | 427 | 40 | 3207
559 | 243 | 62,490.4 | 1:46:22 | 0:46:14 | 4 | 3,026.7 Kb | 435 | 41 | 3026
560 | 242 | 62,490.4 | 1:46:22 | 0:45:58 | 4 | 3,020.4 Kb | 435 | 41 | -6
572 | 230 | 62,598.8 | 1:46:27 | 0:42:48 | 4 | 3,152.1 Kb | 439 | 46 | 3152
577 | 225 | 62,701.2 | 1:46:33 | 0:41:32 | 4 | 3,164.2 Kb | 441 | 47 | 3164
580 | 222 | 62,701.2 | 1:46:35 | 0:40:48 | 4 | 3,169.8 Kb | 441 | 47 | 3169
598 | 204 | 62,794.5 | 1:46:41 | 0:36:23 | 4 | 3,157.9 Kb | 447 | 49 | 3157
600 | 202 | 62,846.5 | 1:46:41 | 0:35:55 | 4 | 3,111.4 Kb | 448 | 51 | 3111
605 | 197 | 62,846.5 | 1:46:46 | 0:34:46 | 4 | 3,124.0 Kb | 449 | 51 | 3124
614 | 188 | 62,894.2 | 1:46:52 | 0:32:43 | 4 | 3,237.6 Kb | 451 | 51 | 3237
620 | 182 | 62,894.9 | 1:46:53 | 0:31:22 | 4 | 3,157.0 Kb | 452 | 51 | 3157
640 | 162 | 62,939.9 | 1:46:58 | 0:27:04 | 4 | 3,194.9 Kb | 457 | 52 | 3194
660 | 142 | 62,984.3 | 1:47:03 | 0:23:01 | 4 | 3,236.0 Kb | 462 | 53 | 3236
680 | 122 | 63,552.3 | 1:47:07 | 0:19:13 | 4 | 3,383.2 Kb | 481 | 75 | 3383
700 | 102 | 64,157.5 | 1:47:11 | 0:15:37 | 4 | 3,439.8 Kb | 501 | 98 | 3439
720 | 82 | 64,764.6 | 1:47:15 | 0:12:12 | 4 | 3,488.2 Kb | 521 | 108 | 3488
740 | 62 | 65,420.4 | 1:47:19 | 0:08:59 | 4 | 3,548.6 Kb | 541 | 136 | 3548
760 | 42 | 65,879.5 | 1:47:23 | 0:05:56 | 4 | 3,576.0 Kb | 561 | 149 | 3576
780 | 22 | 66,359.9 | 1:47:27 | 0:03:01 | 4 | 3,621.6 Kb | 581 | 155 | 3621
800 | 2 | 66,858.2 | 1:47:31 | 0:00:16 | 4 | 3,676.3 Kb | 601 | 171 | 3676
820 | 153 | 68,694.0 | 1:47:35 | 0:20:04 | 5 | 3,798.7 Kb | 621 | 0 | 3798
840 | 133 | 70,644.5 | 1:47:40 | 0:17:02 | 5 | 3,838.1 Kb | 641 | 0 | 3838
850 | 123 | 70,840.4 | 1:47:46 | 0:15:35 | 5 | 3,569.6 Kb | 644 | 0 | 3569
852 | 121 | 70,881.6 | 1:47:52 | 0:15:19 | 5 | 3,662.0 Kb | 645 | 0 | 3662
853 | 120 | 70,881.6 | 1:47:57 | 0:15:11 | 5 | 3,668.9 Kb | 645 | 0 | 3668
855 | 118 | 70,881.6 | 1:48:02 | 0:14:54 | 5 | 3,707.7 Kb | 645 | 0 | 3707
860 | 113 | 71,034.4 | 1:48:03 | 0:14:11 | 5 | 3,693.6 Kb | 650 | 3 | 3693
880 | 93 | 71,633.8 | 1:48:07 | 0:11:25 | 5 | 3,741.0 Kb | 670 | 15 | 3741
900 | 73 | 72,235.9 | 1:48:11 | 0:08:46 | 5 | 3,790.0 Kb | 690 | 27 | 3790
920 | 53 | 72,836.1 | 1:48:15 | 0:06:14 | 5 | 3,837.4 Kb | 710 | 33 | 3837
940 | 33 | 73,464.8 | 1:48:19 | 0:03:48 | 5 | 3,913.4 Kb | 730 | 36 | 3913
960 | 13 | 74,039.2 | 1:48:23 | 0:01:28 | 5 | 3,927.8 Kb | 750 | 69 | 3927
980 | 78 | 74,578.3 | 1:48:28 | 0:08:37 | 6 | 3,928.1 Kb | 770 | 6 | 3928
1000 | 58 | 75,168.8 | 1:48:31 | 0:06:17 | 6 | 3,982.9 Kb | 790 | 27 | 3982
1020 | 38 | 75,772.0 | 1:48:35 | 0:04:02 | 6 | 4,029.1 Kb | 810 | 33 | 4029
1040 | 18 | 76,415.6 | 1:48:39 | 0:01:52 | 6 | 4,082.2 Kb | 830 | 51 | 4082
1060 | 57 | 76,962.8 | 1:48:44 | 0:05:50 | 7 | 4,089.2 Kb | 850 | 0 | 4089
1080 | 37 | 77,551.6 | 1:48:48 | 0:03:43 | 7 | 4,139.2 Kb | 870 | 14 | 4139
1100 | 17 | 78,150.2 | 1:48:52 | 0:01:40 | 7 | 4,187.4 Kb | 890 | 22 | 4187
1120 | 27 | 78,766.0 | 1:48:56 | 0:02:37 | 8 | 4,203.8 Kb | 910 | 0 | 4203
1140 | 7 | 79,352.9 | 1:49:00 | 0:00:40 | 8 | 4,246.5 Kb | 930 | 0 | 4246
1147 | 0 | 79,565.1 | 1:49:02 | 0:00:00 | 9 | 4,245.0 Kb | 937 | 0 | 4245
<h4>Completed</h4>Total pages indexed: 937
<br>Creating sitemaps...
 and calculating changelog...
<div id="percprog"></div>
Creating HTML sitemap...<div id="percprog2"></div> |  | 0.0 | 0:00:00 | 0:00:00 |  |  |  |  | 0
 |  | 0.0 | 0:00:00 | 0:00:00 |  |  |  |  | 0
 |  | 0.0 | 0:00:00 | 0:00:00 |  |  |  |  | 0
<br>Sending email notification... |  | 0.0 | 0:00:00 | 0:00:00 |  |  |  |  | 0
<br />Done, redirecting to sitemap view page. <script> top.location = 'index.php?op=view' </script>

Hello,

yes. From what I see the process has been completed successfully.

The crawling time itself depends on the website page generation time mainly, since it crawls the site similar to search engine bots.
For instance, if it it takes 1 second to retrieve every page, then 1000 pages will be crawled in about 16 minutes.
 

I am getting 130000 pages in 23 minus. Used to having 130000 pages in days.
Hint was in Narrow Indexed Pages Set. Try Exclusion preset. I am now using this software without a hassle.