I apologize the frustrated tone of my message in advance. I have a Wordpress website that I simply want to generate an XML sitemap for. The site has about 300 main pages that need to be crawled, but there are three unnecessary directories that have over 700 pages that DO NOT need to be crawled. If I use the Free Online Sitemap Generator, it crawls EVERYTHING, maxing out at 500 while skipping the other remaining pages that needed to be crawled. There’s no option to exclude directories with the free version.
So I reluctantly (I say reluctantly because I had a funny feeling it wasn’t going to work very well) purchased the Unlimited Sitemap Generator. I installed it with no problem, followed the instructions, and added the directories that DO NOT need to be crawled under the “Excluded URLs” section. I used the following format for the excluded directories:
excludedirectory1/
excludedirectory2/
I believe this is correct, right?
Anyway, I started the Crawling process TWO DAYS ago, and it’s STILL running! Why is this taking so darn long for only 300 pages?!?! The Free Generator processed 500 pages in less than 5 minutes, while the paid version has only made it to 193 in two whole days. After several hours, I would switch tabs and go back to resume the process, but it only inches up a few more pages each time. It seems to stall after every few pages.
Why on God’s green earth is the process taking so long? I wish I could use the free version because it’s light years faster, however I can’t exclude directories however
Guest