Connect with Facebook
Go Back   vBulletin Setup > vBulletinSetup Information > vBulletin Hacks & Questions


Please Register to get full access to the forums.
Reply
 
LinkBack Thread Tools Display Modes
Old 12-03-2008, 03:05 PM   #1
vBulletin Owner
 
Nick R's Avatar
 
Join Date: May 2008
Location: Florida
Posts: 403
Nick R is a jewel in the rough
Help with my robots.txt...

I've never had a problem with my robots.txt file until now.

I recently got an e-mail from AdSense stating that it cannot reach some of my site's pages that are currently serving ads due to robots.txt restrictions.
They request that I give access to their AdSense bot (Mediapartners-Google*) to the pages that are usually blocked.

How, in my robots.txt file, can I grant access only to Mediapartners-Google* but block everything else?

Do I just simply add
Code:
User-agent: Mediapartners-Google*
Disallow:
below my current code, which is:
Code:
User-agent: *
Disallow: certain pages 
?

Thanks in advance for any assistance.

.....

...And if anybody is interested, here is a copy of the e-mail that AdSense sent:
Quote:
Hello,

While reviewing your ad implementation, we noticed that your robots.txt file is currently preventing our AdSense crawler from reaching a significant number of pages with ads in your account.

In order to serve targeted, paid ads to your sites, our crawler needs to visit your sites’ pages to determine their content. Please update your robots.txt file to allow the AdSense crawler to access all pages showing Google ads. You can allow the AdSense crawler access to your sites by adding the following lines to your robots.txt file:

User-agent: Mediapartners-Google*
Disallow:

Thanks for helping enable us to serve the most relevant ads to your sites. Please note that in the future, if we can't crawl some of your pages, we may disable ad serving to those pages.

For more information, please visit: https://www.google.com/adsense/suppo...y?answer=37091

We appreciate your understanding.

Yours sincerely,

The Google AdSense Team

Google, Inc.
1600 Amphitheatre Parkway
Mountain View, CA 94043, USA

Last edited by Nick R; 12-03-2008 at 03:08 PM.
Nick R is offline   Reply With Quote
Old 12-03-2008, 05:59 PM   #2
entrepreneur
 
Join Date: Oct 2006
Posts: 170
Crow will become famous soon enough
Re: Help with my robots.txt...

Why would you want to block a search engine from crawling your site in the first place?


Putting these two lines on the top two lines in your robot.txt is said to work.
(per googles own site)
Code:
User-agent: Mediapartners-Google* 
Disallow:
__________________
I whine to much
Crow is offline   Reply With Quote
Old 12-03-2008, 06:24 PM   #3
vBulletin Owner
 
Nick R's Avatar
 
Join Date: May 2008
Location: Florida
Posts: 403
Nick R is a jewel in the rough
Re: Help with my robots.txt...

Quote:
Originally Posted by Crow View Post
Why would you want to block a search engine from crawling your site in the first place?


Putting these two lines on the top two lines in your robot.txt is said to work.
(per googles own site)
Code:
User-agent: Mediapartners-Google* 
Disallow:
I'm not blocking my entire site from being crawled; just certain pages. Everybody has pages they don't want/need crawled.

Yes, I know that code is said to allow the AdSense bot to crawl my site. However, my question is, will that code override my current code which disables all crawlers from crawling my specified content?
Nick R is offline   Reply With Quote
Old 12-03-2008, 06:30 PM   #4
Community Manager
vBulletin Owner
 
Rocket 442's Avatar
 
Join Date: Nov 2007
Location: Buffalo, NY
Posts: 1,002
Blog Entries: 1
Rocket 442 is just really niceRocket 442 is just really niceRocket 442 is just really nice
Send a message via AIM to Rocket 442
Re: Help with my robots.txt...

That should only allow googles mediapartner spiders to crawl it. Google has separate crawlers for adsense and their actual search engine

This shouldn't make any other crawler index these pages at all since its giving special directions to mediapartners-google crawler, and not others.
__________________
Andy / vBulletinSetup Staff
Check the Newsletter & Marketplace for the latest deals.
Looking for vBRecipe or a place to Support vBulletinSetup!
Submit your Forum and other Quality Websites.


Need a Custom Wordpress or vBulletin 4.0 Design? Staple Web Design
Camaro Forums & Firebird Forums
Rocket 442 is offline   Reply With Quote
Old 12-03-2008, 08:04 PM   #5
vBulletin Owner
vBSetup Owner
 
Brandon Sheley's Avatar
 
Join Date: Jul 2006
Location: Topeka, KS
Posts: 12,639
Recipes: 4
Blog Entries: 35
Brandon Sheley is a glorious beacon of lightBrandon Sheley is a glorious beacon of lightBrandon Sheley is a glorious beacon of lightBrandon Sheley is a glorious beacon of light
Send a message via AIM to Brandon Sheley Send a message via MSN to Brandon Sheley Send a message via Yahoo to Brandon Sheley
Re: Help with my robots.txt...

I got the same email from google
this is what I did
http://forum.vbulletinsetup.com/robots.txt
__________________
Brandon Sheley / vBulletinSetup Staff
Check the Newsletter & Marketplace for the latest deals.
Looking for vBRecipe or a place to Support vBulletinSetup!
Submit your Forum and other Quality Websites.

Add our Facebook Blog, Page and LinkedIn group.. & Don't forget to read the-> Forum Rules
Brandon Sheley is offline   Reply With Quote
Old 12-03-2008, 08:39 PM   #6
vBulletin Owner
 
popowich's Avatar
 
Join Date: Jul 2007
Location: Rochester, New York
Posts: 359
popowich is a jewel in the roughpopowich is a jewel in the rough
Re: Help with my robots.txt...

I got the same e-mail yesterday but following their directions my robots.txt looks different like this.

-Raymond
__________________
New York Forum | Email Help
popowich is offline   Reply With Quote
Old 12-03-2008, 09:15 PM   #7
vBulletin Owner
vBSetup Owner
 
Brandon Sheley's Avatar
 
Join Date: Jul 2006
Location: Topeka, KS
Posts: 12,639
Recipes: 4
Blog Entries: 35
Brandon Sheley is a glorious beacon of lightBrandon Sheley is a glorious beacon of lightBrandon Sheley is a glorious beacon of lightBrandon Sheley is a glorious beacon of light
Send a message via AIM to Brandon Sheley Send a message via MSN to Brandon Sheley Send a message via Yahoo to Brandon Sheley
Re: Help with my robots.txt...

I wasn't sure how it should go, I changed mine thanks
__________________
Brandon Sheley / vBulletinSetup Staff
Check the Newsletter & Marketplace for the latest deals.
Looking for vBRecipe or a place to Support vBulletinSetup!
Submit your Forum and other Quality Websites.

Add our Facebook Blog, Page and LinkedIn group.. & Don't forget to read the-> Forum Rules
Brandon Sheley is offline   Reply With Quote
Old 12-04-2008, 02:52 AM   #8
vBulletin Owner
 
valdet's Avatar
 
Join Date: Jan 2008
Location: Kosova
Posts: 290
valdet has a spectacular aura about
Send a message via Yahoo to valdet
Re: Help with my robots.txt...

Here is how I have it

Code:
User-agent: Mediapartners-Google*
Allow: /

User-agent: *
Disallow: /forum/admincp/
Disallow: /forum/archive/
Disallow: /forum/clientscript/
. .
Good or bad... ?

I'm lost
valdet is offline   Reply With Quote
Old 12-05-2008, 07:01 AM   #9
vBulletin Owner
 
popowich's Avatar
 
Join Date: Jul 2007
Location: Rochester, New York
Posts: 359
popowich is a jewel in the roughpopowich is a jewel in the rough
Re: Help with my robots.txt...

I'm not sure if this is good or bad, but I'd rather not screw around with google.

I noticed in the webmaster tools I still have a warning for google not being allowed per the robots.txt.

I changed mine to be this:

Quote:
User-agent: *
Disallow:
Sitemap: http://www.emailquestions.com/sitemap_index.xml.gz

Allow everything, and if they run into permissions errors so be it. If that's what makes google happy...

-Raymond
__________________
New York Forum | Email Help
popowich is offline   Reply With Quote
Old 12-05-2008, 09:29 AM   #10
Supporters
vBulletin Owner
 
Cerberus's Avatar
 
Join Date: Mar 2008
Posts: 1,321
Cerberus is just really niceCerberus is just really niceCerberus is just really nice
Re: Help with my robots.txt...
Recent Blog: Rip

Well here is one I use for most sites...it works fairly well

Sitemap: http://yoursite.com/sitemap_index.xml.gz

User-agent: *
Disallow: /admincp/
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /customavatars/
Disallow: /customprofilepics/
Disallow: /images/
Disallow: /modcp/
Disallow: /ajax.php
Disallow: /arcade.php
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /global.php
Disallow: /image.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /showpost.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php


Works fine and never get errors..Though I have showpost removed because using vbseo..If you do not have vbseo..You may want to take that out
__________________
Cerberus is offline   Reply With Quote
Old 12-05-2008, 09:36 AM   #11
vBulletin Owner
 
popowich's Avatar
 
Join Date: Jul 2007
Location: Rochester, New York
Posts: 359
popowich is a jewel in the roughpopowich is a jewel in the rough
Re: Help with my robots.txt...

Stupid pointer, but just in case, you don't actually have yoursite.com in the robots.txt file, right?

-Raymond
__________________
New York Forum | Email Help
popowich is offline   Reply With Quote
Old 12-05-2008, 09:41 AM   #12
Supporters
vBulletin Owner
 
Cerberus's Avatar
 
Join Date: Mar 2008
Posts: 1,321
Cerberus is just really niceCerberus is just really niceCerberus is just really nice
Re: Help with my robots.txt...
Recent Blog: Rip

Quote:
Originally Posted by popowich View Post
Stupid pointer, but just in case, you don't actually have yoursite.com in the robots.txt file, right?

-Raymond
Yeah replace that url with the one to your sitemap...I just put yoursite as an example
__________________
Cerberus is offline   Reply With Quote
Old 12-05-2008, 12:31 PM   #13
vBulletin Owner
 
valdet's Avatar
 
Join Date: Jan 2008
Location: Kosova
Posts: 290
valdet has a spectacular aura about
Send a message via Yahoo to valdet
Re: Help with my robots.txt...

It is confusing, because in Google Webmaster Tools, under the Analyze robots.txt, if you want to allow a crawler it brings up

User-agent: Mediapartners-Google*
Allow: /

while that quoted email says the opposite.

User-agent: Mediapartners-Google*
Disallow:
valdet is offline   Reply With Quote
Old 12-05-2008, 01:35 PM   #14
vBulletin Owner
 
Nick R's Avatar
 
Join Date: May 2008
Location: Florida
Posts: 403
Nick R is a jewel in the rough
Re: Help with my robots.txt...

Quote:
Originally Posted by valdet View Post
It is confusing, because in Google Webmaster Tools, under the Analyze robots.txt, if you want to allow a crawler it brings up

User-agent: Mediapartners-Google*
Allow: /

while that quoted email says the opposite.

User-agent: Mediapartners-Google*
Disallow:
They both mean the same thing. "Allow: /" says that the spiders can crawl everything on the domain past "/", which obviously is everything on your site.
"Disallow:" is saying to disallow nothing, since there is nothing placed after the word "Disallow".
Nick R is offline   Reply With Quote
Reply

Tags
robotstxt

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Tags, Archive, robots.txt and duplicate content nfn vBulletin SEO Tips and SEO Questions 6 01-12-2009 01:17 PM
Google releases a robots.txt generator! News Search Engine News 0 04-06-2008 09:49 PM
Preventing people reading your robots.txt file? Ekka vBulletin Hacks & Questions 19 10-28-2007 01:06 AM
Can I disallow visitors from showing the threads and allow robots? THE X FILES vBulletin SEO Tips and SEO Questions 3 12-13-2006 01:25 AM


All times are GMT -6. The time now is 06:26 PM.

vBulletin Setup, vBulletin Setup Forums, vBulletin Services, vBulletin Blogs, vBulletin SEO, vBulletin Questions
vBulletin Skins, Styles, Templates, vBulletin Monetization, Blogs, vBulletin Link Directory,Quality Link Directory