Sitemap Generator Forum

Hi, long time listen first time caller...

I have a phpbb3 board with several forums in it but only one is the main forum with about 1200 posts/replies.  Titles are set to be hyperlinks too so I know I have a lot of links.

I also have a portal.php page that has some topics displayed, other links like members, search, etc. It also lists all the forums so someone can pick a forum and go to it.

I exclude all those other links so it doesn't go through all the members, etc. No problem there.

My problem is when I start at the main page [ External links are visible to forum administrators only ] and go unlimited depth/pages it's taken more than 12 hours and still is not done.  I've checked and have 27000+ links. (Unfortunately, I didn't check out the sitemap that was created so far so don't know what the links were.)

When I limit the depth and pages it gets done in a reasonable amount of time but:

• At the beginning of the sitemap those topics listed on the portal page are correctly indexed. (There's only 5 of them.)
  All the other links in the sitemap are the same with a one up serialization and they all don't exist and when you try one it just goes to the portal page

.

Example:
          [ External links are visible to forum administrators only ]
          [ External links are visible to forum administrators only ]
          [ External links are visible to forum administrators only ]
          [ External links are visible to forum administrators only ]
          [ External links are visible to forum administrators only ]  --
          [ External links are visible to forum administrators only ]         |
          [ External links are visible to forum administrators only ]          --- Correctly Indexed
          [ External links are visible to forum administrators only ]         |
          [ External links are visible to forum administrators only ]  --
          [ External links are visible to forum administrators only ]
          [ External links are visible to forum administrators only ]
          [ External links are visible to forum administrators only ]
          [ External links are visible to forum administrators only ]
          [ External links are visible to forum administrators only ]
          [ External links are visible to forum administrators only ]
          [ External links are visible to forum administrators only ]
          [ External links are visible to forum administrators only ]
          [ External links are visible to forum administrators only ]
          [ External links are visible to forum administrators only ]
          [ External links are visible to forum administrators only ]
          [ External links are visible to forum administrators only ]
          [ External links are visible to forum administrators only ]
          [ External links are visible to forum administrators only ]
          [ External links are visible to forum administrators only ]
          [ External links are visible to forum administrators only ]
          [ External links are visible to forum administrators only ]
          [ External links are visible to forum administrators only ]
             *
             *
             *
             *
          [ External links are visible to forum administrators only ]
          [ External links are visible to forum administrators only ]
          [ External links are visible to forum administrators only ]
          [ External links are visible to forum administrators only ]

It is not indexing any of the forums and so google says my top search word is "total" at 16.  If I start at the main forum page the sitemap is still the same as above.

          1.  Am I not waiting long enough in the unlimited depth/pages indexing?

          2.  Do I need to tweek something?

          3.  Am I not starting on the correct page?

          4.  Is it my deodorant?

Any help would be appreciated. 

Thanks,
Diveguy

Hello,

those links seem to be form the calendar widget on your portal page, you can just add this in "Exclude URLs" setting:
portal.php?m=and that should help I thnk.

I figured that I could block the M= and now when I run the scan I get only 9 pages read.

Here is what I am blocking:

• m=
  profile.php
  privmsg.php
  search.php
  faq.php
  board3.de
  installphpbb
  paypal.php
  ucp.php
  acp.php
  mcp.php
  memberlist.php
  viewprofile.php
  hostmonster.php
  phpbb.com
  simplescripts.com

And that now produces:

     [ External links are visible to forum administrators only ]
     [ External links are visible to forum administrators only ]
     [ External links are visible to forum administrators only ]
     [ External links are visible to forum administrators only ]
     [ External links are visible to forum administrators only ]
     [ External links are visible to forum administrators only ]
     [ External links are visible to forum administrators only ]
     [ External links are visible to forum administrators only ]
     [ External links are visible to forum administrators only ]

It is like it doesn't see anything past that widget.

I ran the bot simulator on the main page, ratethatprovider.com/portal.php, and here is what it sees:

---

Welcome to Rate That Provider - St. Louis. Click here to register    ATTENTION GUESTS: Please register to view all sections    Rate That Provider - St. Louis Local Hobbyiests Review Local St. Louis escorts and Visiting escorts and Discuss Hobbying Skip to content  Board index Change font size  FAQ  Register  Login    Menu Content Board index  Register  The team  Help FAQ  BBCode FAQ  Terms of use  Privacy policy   Birthdays No birthdays today In the next 7 days No members have a birthday within this period of time.  Clock  Random member GrayGhost2  Ensign Join:29.Aug.2010 Posts:0  Peak posters Username Posts Diveguy  405  pappabear  25  Cunninglinguist  24  myluvsunny  17  ike88hd  15  Cfkr69  14  xcobrax  12  Splitdawg  10   Newest members Username Joined henryj501  09 Oct montyr@insightbb.com  09 Oct J_Graham  09 Oct jackfoxx16  09 Oct soulblink  09 Oct atesta  09 Oct Sheik Yerbouti  09 Oct spiritedfun  08 Oct  Link to us Please feel free to link to Rate That Provider - St. Louis. Use the following HTML: St. Louis' Best Escort Reviews We are St. Louis' premier escort review board. We offer the largest selection of St. Louis escorts for you to determine if one is right for you.. A sexy, sophisticated female escort by your side at that important function, an attentive lady for a quiet evening, a fantasy date with a sensual, or passionate St. Louis escort to pamper you with her sensuality. Our 5-star St. Louis escorts get the best reviews. No need to wonder if that escort you're meeting is going to be a disappointment. No need to guess which escort will match her photo. Search for an escort's last four digits of her phone number to find out if she is a 5-star or is a ripoff. Use our St. Louis escort photo page to quickly find an escort that intrigues you, check her rating and review count. If she seem good so far click on her photo and read her reviews. Read the reviews, choose your St. Louis escort, then provide your own review. Welcome Have fun browsing. If you have information that could help your peers then please post that information. Who is online In total there are 4 users online :: 2 registered, 0 hidden and 2 guests (based on users active over the past 5 minutes)Most users ever online was 19 on Thu Sep 16, 2010 7:03 pm Registered users: Diveguy, Splitdawg  Legend: Administrators, Global moderators, Newly registered users  Recent Recent announcements Welcome To The New Members  Ask Sunny  Confirmed Provider Info.......Please Read  Rank Icons  Terms of Use Latest global announcements No global announcements

Latest news No news

Click Here to Search for Providers by Photo  (Only St. Louis Providers.)

This board has no forums.

PayPal donations Rate That Provider - St. Louis is a group supplying services with no intention of any monetary profit. Your donations are welcome so that the cost of our server, domain name, etc. can be covered. U.S. Dollars (USD) Australian Dollars (AUD) Canadian Dollars (CAD) Czech Koruna (CZK) Danish Kroner (DKK) Euros (EUR) Hong Kong Dollars (HKD) Hungarian Forint (HUF) New Zealand Dollars (NZD) Norwegian Kroner (NOK) Polish Zlotych (PLN) British Pounds (GBP) Singapore Dollars (SGD) Swedish Kronor (SEK) Swiss Francs (CHF) Japanese Yen (JPY) Mexican Pesos (MXN) Israeli New Shekels (ILS) Please use a decimal point (not a comma) as the separator, e.g. 3.50 board3 Portal - based on phpBB3 Portal   Login  Username: Password: Remember me  Statistics Totals Total posts 1029 Total topics 385 Total Announcements: 14 Total Stickies: 24 Total Attachments: 4 Topics per day: 2 Posts per day: 5 Users per day: 2 Topics per user: 1 Posts per user: 3 Posts per topic: 3 Total members 397 Our newest member henryj501  Calendar  Oct. 2010  Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31  The team Administrators BossyLady  Diveguy  Emergency  Moderators bazinga_69  myluvsunny   Last 3 visited bots Google [Bot] Sun Oct 10, 2010 9:15 am MSN [Bot] Sun Oct 10, 2010 7:36 am Yahoo [Bot] Sun Oct 10, 2010 6:48 am  Links BackPage  TER  Portal » Board index  The team • Delete all board cookies • All times are UTC - 6 hours [ DST ] Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group  Install phpBB web hosting

---

It does find the 5 links that are in one section of the portal.php page. They're listed above with the viewtopic in the links. (Highlighted in blue.)

The section in red that I highlighted above says there is no news but there are 5 links of news displayed.  So it finds 5 links in one section and then cannot find the next 5 links in another section.



It also says there are no forums but this is where the list of forum links are located.  They are in between the Search for Providers by Photo link and the Paypal link.



I've tried using [ External links are visible to forum administrators only ] which has just the forum links and it produces the same sitemap.

I cannot get past that issue.  Any other thoughts?

Thanks,
Russ

Hello,

it looks like your forum requires registration and is not visible to guests, that's why sitemap generator bot is unable to access it.

Thanks Oleg,

Was kind of thinking the same thing but I couldn't find any reference to a generator bot to give it access.  Do you have any ideas how I might do that.  I mean I allow google bot and other bots.

Thanks again,

Russ

You would need to modify your forum settings to allow guest visitors to view topics.

So basically I paid $20 for nothing.  You want me to open up my whole site to guests just to allow your bot?  Kind of defeats the purpose of having a pay site.

Guess you're not going to refund my money.

Hello,

it's not just about sitemap generator bot - search engine bots will be unable to crawl your site as well (since they work in the same way as generator crawler), and correspondingly they cannot index your site content.

That's not correct.  Bots have permission to crawl the site.  For instance, Google(Bot) crawls everyday.  But I have the names of those bots so I can give them permission and not open up my whole site.

I do not have a name for your bot so I cannot.

Hello,

the user-agent name of sitemap generator bot is:
Mozilla/5.0 (compatible; XML Sitemaps Generator; https://www.xml-sitemaps.com) Gecko XML-Sitemaps/1.0

Thank you very much.  That solved the issue.