Sitemap not adding all urls
« on: May 13, 2011, 07:43:48 PM »
We are running a site with Joomla and Mosets Tree. We have over 2000 directory listings right now and I'm only getting 799 urls in the sitemap. I have trolled the forum looking for answers and most of the answers end with pm me. I have tried a couple things already, like adding ini_set() to add more execution time. Our memory is already set to a more than adequate number. I tried adding a second between urls.

It is still executing and here is a sample of what it is showing:

Code: [Select]
Links depth: 3
Current page: index.php?option=com_mtree&task=viewlink&link_id=866&Itemid=2
Pages added to sitemap: 799
Pages scanned: 1340 (42,432.8 KB)
Pages left: 1695 (+ 2849 queued for the next depth level)
Time passed: 0:37:29
Time left: 0:47:24
Memory usage: 4,700.7 Kb

The part of the url above in the code (link_id=866) is different for every page and we need all the pages to show in the sitemap. Do you have any suggestions? I have run this about 8 times since yesterday trying to get it to work correctly. I will be happy to pm you our info if you can help.

One other issue I came across - with Mosets tree, I had it set to view only 20 items per page and found that the sitemap didn't seem to get to other pages (there is a 'next' button on the bottom for the next 20 results in the site.)
« Last Edit: May 13, 2011, 07:46:46 PM by info1728 »
Re: Sitemap not adding all urls
« Reply #1 on: May 13, 2011, 09:46:27 PM »
Hello,

could you please PM me your generator URL and an example URL that is not included in sitemap and how it can be reached starting from homepage?
Re: Sitemap not adding all urls
« Reply #2 on: May 16, 2011, 03:42:32 PM »
Oh -  I see you ran it.
I guess I'm not understanding the 'parse' term for this. I see you added "task=viewlink" to the box "do not parse urls". Can you explain what that does?
« Last Edit: May 16, 2011, 03:47:49 PM by info1728 »
Re: Sitemap not adding all urls
« Reply #3 on: May 16, 2011, 08:46:09 PM »
It means that those pages are added directly in sitemap, without fetching from server - that allows to complete crawling faster.
Re: Sitemap not adding all urls
« Reply #4 on: May 16, 2011, 09:42:32 PM »
Thank-you!
Re: Sitemap not adding all urls
« Reply #5 on: May 30, 2011, 12:33:27 PM »
Hope you can help me out here.
My website is [ External links are visible to forum administrators only ]
Generator is placed at [ External links are visible to forum administrators only ]
The generator follows the path from [ External links are visible to forum administrators only ]
(including others) it will come to --> [ External links are visible to forum administrators only ], there are a lot of links there, but the generator does not include any of them.
All links goto [ External links are visible to forum administrators only ] Where 420 is variable here.
This is the primairy content of the site and is the most inportant to be indexed.
Unfortunately it does not include it these links in te sitemap.xml

Kind regards
(I included them manualy now, but for testing you can overwrite it.)
Thanks for your answer.
John
username and password on request
« Last Edit: May 30, 2011, 01:14:36 PM by info1746 »
Re: Sitemap not adding all urls
« Reply #6 on: May 31, 2011, 06:32:07 AM »
Hello,

you should remove "SID" from generator configuration->"Advanced settings" -> "Remove session ID from URLs" option.
Re: Sitemap not adding all urls
« Reply #7 on: May 31, 2011, 08:13:27 AM »
Wonderfull, that  did the trick

Thanks
John