Exclude url in wordpress and others
« on: January 29, 2012, 03:27:26 PM »
Hello!

1.
What is the excluded urls in wordpress since the there is no settings?

2. If i want exclude domain.com/t/ but not domain.com/t/hello
how do i set that?


changing this setting will automatically prepopulate the options below with preset data
Exclude URLs:
Re: Exclude url in wordpress and others
« Reply #1 on: January 30, 2012, 12:10:02 PM »
Hello,

for wordpress it's recommended to exclude:
Code: [Select]
trackbackdo not parse:
Code: [Select]
rss
feed
tag/

2. You can use this:
Code: [Select]
^t*/$
Re: Exclude url in wordpress and others
« Reply #2 on: February 27, 2012, 03:17:00 AM »
Hello, should i use for example the [ External links are visible to forum administrators only ] or can i just put rss?
Re: Exclude url in wordpress and others
« Reply #4 on: April 01, 2012, 08:41:06 AM »
Will this make the code not crawl the /v/ but crawl the html links in /t/ but nothing else

exclude:

/video/t/*.html$
/video/v/

parse:

/video/t/*.html$
/video/v/
Re: Exclude url in wordpress and others
« Reply #5 on: April 02, 2012, 01:35:06 PM »
Anything that matches the "Exclude URLs" will not be crawled or added in sitemap.
Re: Exclude url in wordpress and others
« Reply #6 on: April 03, 2012, 05:28:10 PM »
Ok i am trying to include the LINKS but the problem is the script generator start crawling the whole YOUTUBE. So is this correct? You can check here for example bangkokevents.org but this is in root.
Re: Exclude url in wordpress and others
« Reply #8 on: April 04, 2012, 01:05:39 AM »
This is a video script. xml-sitemaps make a sitemap for YOUTUBE since my content is api from YOUTUBE. I need to stop it from crawling ALL YOUTUBE.com just the links on the left side of the page and not any other VIDEO in search.