Sitemap Generator Forum

I'm crawling one of my sites, and there are considerably more pages scanned then added to the sitemap... why?

Hello,

the number of "scanned" pages includes ALL links found on the pages, but some of them are NOT includes into sitemap depending on the configuration settings ("excludes pages", "exclude extensions").

Once the crawling is done, is there a way to add certain missed pages to the sitemap afterwards w/o recrawling?

Hello,

it can be done only by manual modification of generated xml file (in any text editor).

Oh... (yuck).

Still waiting for LyricsLane to finish crawling...


Has anyone else noticed that it stops every once and a while?  I leave it running, and occassionally, I have to go back in and start it up again...?

Yes, it can be stopped depending on your php configuration (memory_limit/max_execution_time settings).

Is there a way to fix that?

Hello,

yes, increase the values for the options mentioned above (in php.ini).

ok, that's something my host would have to do, I guess.

Now, when I go in to continue with the crawling, it won't even start... it goes to page cannot be displayed. 

Hello,

answered in email.

Ok, it has finally finished crawling the site.  But now it gives me the following error:

Sitemap file is not writable: /usr/home/public_html/sitemap1.xml
Sitemap file is not writable: /usr/home/public_html/sitemap2.xml
Sitemap file is not writable: /usr/home/public_html/sitemap3.xml
Sitemap file is not writable: /usr/home/public_html/sitemap4.xml
Sitemap file is not writable: /usr/home/public_html/sitemap5.xml

I have these filles created and they are writeable.

Great!

You should either have /usr/home/public_html/ folder writable with 0766 permissions (which is not recommended) or CREATE empty files sitemap1.xml...sitemap5.xml and set their permissions to 0666.

Now you can simply copy these files from the data/ folder to  /usr/home/public_html/ and set permissions.

I have already created those files, and set the permission @ 666.

So I copy the files from the data folder to public_html ?

Yes. And set 0666 permissions for them in public_html for further sitemap generations.