Tools for Any SEO Challenge: Get more traffic to your website with versatile solutions for every step of your SEO journey.
« previous next »
Pages: 1

FlipediS

*
  • **
  • 10
Question
« on: December 22, 2007, 03:08:13 AM »
Aloha,  I had a question,  I been trying to run the site map on my page but is taking forever.. been going at it for two days,  it goes for a few hours and stops... and the last two days so far it has scanned 2400 and has over 15,000 more to go.  and keeps finding more... is there any tricks to make this faster or any settings or anything? 
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: Question
« Reply #1 on: December 22, 2007, 07:47:25 AM »
Hello,

the most useful way to increase crawling speed is to configure "Do not parse" and "Exclude URLs" options. The first one defines which pages should not be *fetched* from website by the crawler (which is the most time consuming task), but they still *will* be included in sitemap. The second option will disallows including URLs in sitemap.
Logged

FlipediS

*
  • **
  • 10
Re: Question
« Reply #2 on: December 23, 2007, 08:20:42 PM »
Sorry im a but new so,  what do you mean fetched but still included?
I have also noticed when its running it adds a bunch to a shopping cart.  Is there a way to stop this?  and when i use exclude do i use the whole path?  lets say i have the starting patch at [ External links are visible to forum administrators only ]  and i was to exclude  lets say [ External links are visible to forum administrators only ]  would i place it in the exclude like that?
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: Question
« Reply #3 on: December 23, 2007, 09:05:06 PM »
1. Let me provide an example to illustrate that option.
Let's say the site structure is as following:

1. homepage:
domain.com/
2. homepage contains links to categories like:
domain.com/cat-1.html
domain.com/cat-2.html
etc.
3. every category page contains links to "entries" like:
domain.com/entry-1.html
domain.com/entry-2.html
domain.com/entry-3.html

Sitemap Generator will try to crawl ALL the pages it finds (all listed above). "Crawl" means request them from the server via http, similar to how the browser requests them.

However, you can add this to the "Do not parse" option:
Code: [Select]
entry-That will tell sitemap generator NOT to request entry pages. The entry URLs will still be included in sitemap since "category" links are crawled and they contain links to entries.

Does it make sense?

2. You can exclude any SET of URLs by adding a *substring* that is common to those URLs, i.e. add this *both* to "Do not parse" and "Exclude URLs" options:
Code: [Select]
Shop/That will exclude all urls in Shop/ folder.
Logged

FlipediS

*
  • **
  • 10
Re: Question
« Reply #4 on: December 23, 2007, 11:34:09 PM »
So what you mean by using the code entry in the do not phase would actually be the product name instead?  If so that would take even longer because there are like 16k products.  or am i miss understand? 
My root directory is [ External links are visible to forum administrators only ]  ,  What im trying to do is build a good site map for google and other search engines
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: Question
« Reply #5 on: December 25, 2007, 11:31:14 PM »
No, you should add the string that is common to all product URLs, in your case this should work:
Code: [Select]
.htmlalso, you can exclude review pages with:
Code: [Select]
reviews/
Logged

FlipediS

*
  • **
  • 10
Re: Question
« Reply #6 on: December 27, 2007, 05:16:19 AM »
Thanks,  Now im running in to another problem.. it seems to run for a while then stop working... then i try to contnuie  and it seems to stop at the same place every time.. so then i ran in in PuTTy to try it there and it stoped at the same place and gave me this error
Fatal error: Out of memory (allocated 534773760) (tried to allocate 1228658 byte              s) in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2              ) : eval()'d code on line 6
  why am i getting this error?
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: Question
« Reply #7 on: December 27, 2007, 07:33:35 PM »
How many URLs do you have crawled at that point? How does the memory usage change when crawler is in progress?
Logged

FlipediS

*
  • **
  • 10
Re: Question
« Reply #8 on: December 28, 2007, 01:39:48 AM »
It seems to stop at diffreent amount of pages each time a start a fresh scan from the begining.  but i after it dose stop and i continue an previous scan that scan never makes it past what one it prevoulsy stoped on.  As for the memory.  How would i be abel to tell this?  sorry not famliar with that. 
this is what it said today when i ran it
<head>
<title>XML Sitemaps - Generation</title>
<meta http-equiv="Content-type" content="text/html;charset=iso-8859-15" />
<link rel=stylesheet type="text/css" href="pages/style.css">
</head>
<body>
Resuming the last session (last updated: 2007-12-26 21:46:24)200 | 60 | 7,288.6 | 9:25 | 2:49 | 2 | - | 163 | 797 | 0
220 | 40 | 8,547.3 | 10:39 | 1:56 | 2 | - | 183 | 1095 | 0
240 | 20 | 9,800.9 | 11:51 | 0:59 | 2 | - | 203 | 1414 | 0

Warning: fopen(/home/FlipediS/TBOStore.com/www/html/generator/data/crawl_dump.log): failed to open stream: Permission denied in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 32

Warning: fwrite(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 33

Warning: fclose(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 34
260 | 1324 | 10,976.8 | 12:58 | 66:03 | 3 | - | 223 | 0 | 0
280 | 1304 | 11,536.2 | 13:38 | 63:33 | 3 | - | 237 | 279 | 0
300 | 1284 | 11,536.2 | 13:47 | 58:59 | 3 | - | 248 | 279 | 0
320 | 1264 | 11,536.2 | 13:53 | 54:52 | 3 | - | 259 | 279 | 0
340 | 1244 | 11,536.2 | 13:59 | 51:11 | 3 | - | 270 | 279 | 0
360 | 1224 | 11,536.2 | 14:04 | 47:51 | 3 | - | 280 | 279 | 0
380 | 1204 | 11,536.2 | 14:08 | 44:49 | 3 | - | 292 | 279 | 0
400 | 1184 | 11,536.2 | 14:13 | 42:06 | 3 | - | 302 | 279 | 0
420 | 1164 | 11,536.2 | 14:19 | 39:41 | 3 | - | 314 | 279 | 0
440 | 1144 | 11,536.2 | 14:23 | 37:24 | 3 | - | 326 | 279 | 0
460 | 1124 | 13,369.2 | 17:31 | 42:49 | 3 | - | 346 | 829 | 0

Warning: fopen(/home/FlipediS/TBOStore.com/www/html/generator/data/crawl_dump.log): failed to open stream: Permission denied in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 32

Warning: fwrite(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 33

Warning: fclose(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 34
480 | 1104 | 15,255.8 | 20:46 | 47:46 | 3 | - | 366 | 1195 | 0

Warning: fopen(/home/FlipediS/TBOStore.com/www/html/generator/data/crawl_dump.log): failed to open stream: Permission denied in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 32

Warning: fwrite(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 33

Warning: fclose(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 34
500 | 1084 | 17,246.0 | 24:20 | 52:47 | 3 | - | 386 | 1745 | 0

Warning: fopen(/home/FlipediS/TBOStore.com/www/html/generator/data/crawl_dump.log): failed to open stream: Permission denied in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 32

Warning: fwrite(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 33

Warning: fclose(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 34
520 | 1064 | 19,141.7 | 27:48 | 56:53 | 3 | - | 406 | 2125 | 0

Warning: fopen(/home/FlipediS/TBOStore.com/www/html/generator/data/crawl_dump.log): failed to open stream: Permission denied in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 32

Warning: fwrite(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 33

Warning: fclose(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 34
540 | 1044 | 21,033.1 | 31:20 | 60:36 | 3 | - | 426 | 2480 | 0

Warning: fopen(/home/FlipediS/TBOStore.com/www/html/generator/data/crawl_dump.log): failed to open stream: Permission denied in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 32

Warning: fwrite(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 33

Warning: fclose(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 34
560 | 1024 | 23,003.5 | 35:25 | 64:47 | 3 | - | 446 | 2895 | 0

Warning: fopen(/home/FlipediS/TBOStore.com/www/html/generator/data/crawl_dump.log): failed to open stream: Permission denied in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 32

Warning: fwrite(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 33

Warning: fclose(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 34
580 | 1004 | 24,942.7 | 38:59 | 67:30 | 3 | - | 466 | 3309 | 0

Warning: fopen(/home/FlipediS/TBOStore.com/www/html/generator/data/crawl_dump.log): failed to open stream: Permission denied in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 32

Warning: fwrite(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 33

Warning: fclose(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 34
600 | 984 | 26,938.1 | 42:41 | 70:00 | 3 | - | 486 | 3913 | 0

Warning: fopen(/home/FlipediS/TBOStore.com/www/html/generator/data/crawl_dump.log): failed to open stream: Permission denied in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 32

Warning: fwrite(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 33

Warning: fclose(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 34
620 | 964 | 28,871.6 | 46:01 | 71:33 | 3 | - | 506 | 4185 | 0

Warning: fopen(/home/FlipediS/TBOStore.com/www/html/generator/data/crawl_dump.log): failed to open stream: Permission denied in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 32

Warning: fwrite(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 33

Warning: fclose(): supplied argument is not a valid stream resource in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 34
640 | 944 | 30,891.5 | 49:30 | 73:01 | 3 | - | 526 | 4606 | 0

Fatal error: Out of memory (allocated 533200896) (tried to allocate 1218213 bytes) in /home/FlipediS/TBOStore.com/www/html/generator/pages/class.utils.inc.php(2) : eval()'d code on line 6
[FlipediS@mgmt-01 ~]$
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: Question
« Reply #9 on: December 28, 2007, 03:14:45 AM »
Hello,

does it also happen via web interface?
Logged

FlipediS

*
  • **
  • 10
Re: Question
« Reply #10 on: December 28, 2007, 05:41:42 AM »
Yes it dose.  i don't get any error messages.  Just runs and seems to get stuck after a while...  thats why i decied to try it in PuTTy
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: Question
« Reply #11 on: December 29, 2007, 12:45:03 AM »
Can you provide access to your host for troubleshooting?
https://www.xml-sitemaps.com/contact.html
Logged

FlipediS

*
  • **
  • 10
Re: Question
« Reply #12 on: December 30, 2007, 05:56:22 AM »
did you get my e-mail?
Logged

XML-Sitemaps Support

*
  • *****
  • 11781
Re: Question
« Reply #13 on: December 30, 2007, 11:40:44 PM »
No, I can't find it. Please use forums username/password that was sent to you in automated email with download link after purchase and send me details via private message here on the forum.

Thank you.
Logged
Pages: 1
« previous next »