Hello, Im a new happy customer of XML Sitemaps, just got this yesterday and created my first sitemap today.
Everything went just fine but I have a couple of questions.
after the scan I reviewed the broken/404 links and most of them are not really "broken" I believe the robots.txt file is blocking them, because I can open them when I click on them.
I ran XML Sitemap on my IPB site, and I wanted to know if there are exclusions presets for IPB, I searched the forums but didnt see any.
if not, this is my robots.txt, can I create them from the robot.txt so they get excluded and dont show up as broken links?
Thank You for the great product!
User-agent: *
Disallow: /admin/
Disallow: /public/js/
Disallow: /public/style_captcha/
Disallow: /public/style_css/
Disallow: /index.php?app=core&module=task
Disallow: /index.php?app=core&module=usercp&tab=forums&area=forumsubs
Disallow: /index.php?app=core&module=usercp&tab=forums&area=watch&watch=topic
Disallow: /index.php?app=forums&module=extras§ion=forward
Disallow: /index.php?app=members&module=messaging
Disallow: /index.php?app=members&module=chat
Disallow: /index.php?app=members&module=search
Disallow: /index.php?app=members&module=search&do=active
Disallow: /index.php?&unlockUserAgent=1
Disallow: /*app=core&module=usercp
Disallow: /*app=core&module=usercp
Disallow: /*app=members&module=messaging
Disallow: /*&p=
Disallow: /*&pid=
Disallow: /*&hl=
Disallow: /*&start=
Disallow: /*page__*$
Disallow: /*view__*$
Disallow: /*&view=*$
Disallow: /calendar/
Sitemap: http:// mysite/sitemap_index.xml.gz