my index.htm starts with a frame. the index.htm loads top.htm and main.htm.
in the noframe area i have some links to follow such as other htm-urls and the main site in a subfolder with index.php.
the generator will only visit the index, top and main.htm. no subfolders and other pages are crawled.
do you have any idea why the whole site is not crawled ?
when I run the script in the subfolder it works almost perfect but i want the whole site to be crawled-
Guest