Robot.txt

**risala** · 01-01-2010, 05:14 AM

Hello,

What is the best robot.txt file so a search engine find and index the urls of a vbulletin forum quickly?

Gr

**Brandon Sheley** · 01-01-2010, 09:08 AM

Well, this is what we use if you want to copy it.
http://forum.vbulletinsetup.com/robots.txt

**GeminiFilly** · 01-01-2010, 09:26 AM

Quote:

Originally Posted by Brandon Sheley

Well, this is what we use if you want to copy it.
http://forum.vbulletinsetup.com/robots.txt

ditto..........

**risala** · 01-02-2010, 02:19 AM

my robot.txt is like this at this moment

# robots.txt for Risala Community- De bron voor koran en sounah, hadith en anasheed

# ALL BOTS
User-agent: Mediapartners-Google*
Disallow:

User-agent: *
Disallow: /admin/
Disallow: /ajax.php
Disallow: /attachments/
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /images/
Disallow: /includes/
Disallow: /mod/
Disallow: /customavatars/
Disallow: /customprofilepics/
Disallow: /announcement.php
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /faq.php
Disallow: /frm_attach
Disallow: /image.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /payment_gateway.php
Disallow: /payments.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /showpost.php
Disallow: /subscription.php
Disallow: /usercp.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php

Sitemap: http://forum.risala.nl/sitemap_index.xml.gz
Sitemap: http://www.risala.nl/sitemap.xml

Is this one good or not?

**Mikey** · 01-09-2010, 08:18 AM

Looks good to me

**Dreamgun** · 01-18-2010, 02:03 AM

Hello,

I have a few questions about robots.txt and SEO redirection.

I read somewhere if you disallow a php file in robots.txt, it also blocks the redirected url generated by SEO software used to access pages. Is this correct?

For example:
Disallow: /printthread.php

It would take care of all "site.com/forum/category/123whatever-title-print.html" URLs.

Please confirm if that is correct, or if instead I would need someone having to do with:
Disallow: /*-print/ or Disallow: /*/*-print/

Second, Is it completely necessary as I see above, to block certain .php files that crawl bots wont be clicking to? For example maybe; cron, ajax, images, inlinemod ... unless maybe I guess a URL was posted directly into someones post.

Thank you.

Gotta be someone smart around here! Been a couple days... BUMP!!

**Brandon Sheley** · 01-21-2010, 05:50 AM

yes, if you blocked your printthread.php page, any page that uses that file will also be blocked.

**Dreamgun** · 01-21-2010, 07:06 AM

Thanks and...

Quote:

Originally Posted by Dreamgun

Second, Is it completely necessary as I see above, to block certain .php files that crawl bots wont be clicking to? For example maybe; cron, ajax, images, inlinemod ... unless maybe I guess a URL was posted directly into someones post.

**Brandon Sheley** · 01-21-2010, 08:56 AM

Quote:

Originally Posted by Dreamgun

Thanks and...

you can see what we block from the example above

**Dreamgun** · 01-21-2010, 05:14 PM

Yes, I can also see from visiting http://forums.vbulletinsetup.com/robots.txt but that doesnt exactly answer if its necessary to block certain php files which bots shouldnt be clicking links to.

**BBR-APBT** · 01-21-2010, 06:16 PM

Quote:

Originally Posted by Dreamgun

Yes, I can also see from visiting http://forums.vbulletinsetup.com/robots.txt but that doesnt exactly answer if its necessary to block certain php files which bots shouldnt be clicking links to.

It is not necessary, but it will help keep down on your server load down from all the bots hitting all them files.

**ArnyVee** · 02-21-2010, 04:10 PM

Just to confirm.....if your vBulletin install is not in the root of the site, then you'd have to add the /forums/ before the files you'd like to disallow, right?

meri0098 · 03-02-2010, 01:11 AM

It is really very helpful!!