Problem: not all the site is indexed
« on: February 15, 2007, 11:22:07 AM »
Ciao,
 I've just bought your product.
Installation is esy and it's ok.

Now, I don't understand how to index all the site.

I mean, the script create a sitemap only for all the links it finds on the homepage.
I have a lot of subdirectories not linked to the homepage.
How can I index also all the others contents?

Example:

[ External links are visible to forum administrators only ] ---> the script generate a sitemap only for the links it finds on the homepage
but
I have also this structure:
[ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]
[ External links are visible to forum administrators only ]
and so on

All the "sub" folders are not indexed. All these folders are not linked by the home page.
I want to generate a sitemap for all the site. If I have to put, handly, all the links to the sub folders into the homepage... it means I'm doing a sitemap by myself!

What can I do?

Please, let me know.

Thanks,

take care,

Rick
« Last Edit: February 15, 2007, 12:02:38 PM by r.porta1 »
Re: Problem: not all the site is indexed
« Reply #1 on: February 15, 2007, 02:31:22 PM »
Hello,

sitemap generator crawls your site similar to search engine bots and normal visitors, i.e. it only finds the pages that can be reached from homepage (not necessary directly linked from homepage, but there should be a chain of links that allow to find the page).
Re: Problem: not all the site is indexed
« Reply #2 on: February 15, 2007, 02:37:40 PM »
Ciao,
 thanks for your quick replay.

I understand the problem. But in this way... why I have to use your sitemap generator?
I 've bought it thinking it could generate an entire sitemap. If I have, in some way, tell to the script where there are all my subfolders...it's a big work.

What do you suggest? Is there any way to allow quickly to the script to index all the site? Any trick?

Thanks

R.
Re: Problem: not all the site is indexed
« Reply #3 on: February 15, 2007, 02:46:59 PM »
Hello,

Yes I have the same question. Most other sitemap generators will index the whole site not just the root, you have to add exclusions for directories you don't want indexed. Some sites of mine have many sub directories which relate to the same site and some which don't ie I have my main site but also I have sub directories which relate to pages of a different nature and are not linked from the homepage or through it, I don't want them ignored when generating a sitemap.

The only way to do it as far as I can see is to make several .xml files and then edit them together - nightmare.

This is a major failing of this product and it should be made clearer before purchasing. Can I get a refund?

Disappointed
Re: Problem: not all the site is indexed
« Reply #4 on: February 15, 2007, 02:57:13 PM »
Ciao Robert,
 I hope that the Admin will able to find a solution. I can't think that this script is so limited.
Ciao

R.
Re: Problem: not all the site is indexed
« Reply #5 on: February 16, 2007, 08:48:57 AM »
Hi,
 any news?
If no, I would like to be refund. A script like this has a very good interface but it's not useful.

Le me know as soon as you can,

thanks,

R.
Re: Problem: not all the site is indexed
« Reply #6 on: February 16, 2007, 05:22:50 PM »
The script is designed to crawl websites and add links it finds just as a search engine crawler does. Without links it won't find your pages.

It is unusual to have a site structure such as yours where you don't have links to other areas of the site and treat the sub directories as different sites. Normally if this is done sub-domains rather than directories would be used to differentiate between them and not confuse search engine algorithms of the theme of your site.

If you want to include all these pages within 1 sitemap then I would suggest as an option that you create a page on your site listing the index files of the sub directories so that the crawler can find them.


Philip Nicosia
Re: Problem: not all the site is indexed
« Reply #7 on: February 16, 2007, 05:31:51 PM »
Ciao,
 so... why is useful your script? How can it helps the webmasters? If I have all my site linked it's quit easy to generate a sitemap and is really easy for the search engine to spider it.
Ok, I've totally wasted my money.

That's it.

R.
Re: Problem: not all the site is indexed
« Reply #8 on: February 16, 2007, 10:49:08 PM »
Hello,

it allows you to automatically create sitemaps (XML, Text, HTML) for the site, which helps search engine bots to crawl your site more efficiently. More details about sitemaps.
Re: Problem: not all the site is indexed
« Reply #9 on: February 26, 2007, 07:36:30 AM »
I'm having the same problem.  My entire site isn't being indexed.  The free online generator finds more than 500 pages and stops at 500 but the standalone generator is only finding 147 pages.

It's indexing the SAME url.  Is this normal?  Did I miss an obvious setting?  The online generator says the link depth is 4 but the standalone doesn't go past a link depth of 2.

James...