I have 100s of pages I do not want crawled. I am having a hard time figuring out how to set this up to work properly. Here are some examples of what I want crawled and below are the some I dont. Please help.
Want crawled:
Base URL/idirectory/inspectors.php?st=va
Base URL/idirectory/inspectors.php?st=md
And all the rest of the states
Dont want crawled
All the listings below exsist for each member of our site. That is 100s Each listing pulls the information out of the database.
Base URL/idirectory/view.php?id=142&page=0&cat=1
Base URL/idirectory/out.php?ID=142
Based URL/idirectory/mail2.php?id=142&cat=1&page=0
Base URL/idirectory/review.php?id=142&cat=1&page=0
Guest