I've never had a problem with my robots.txt file until now.

I recently got an e-mail from AdSense stating that it cannot reach some of my site's pages that are currently serving ads due to robots.txt restrictions.
They request that I give access to their AdSense bot (Mediapartners-Google*) to the pages that are usually blocked.

How, in my robots.txt file, can I grant access only to Mediapartners-Google* but block everything else?

Do I just simply add below my current code, which is: ?

Thanks in advance for any assistance.

.....

...And if anybody is interested, here is a copy of the e-mail that AdSense sent:

Why would you want to block a search engine from crawling your site in the first place?


Putting these two lines on the top two lines in your robot.txt is said to work.
(per googles own site)

--------------------
A Dollar Link - Dollar Link!
Bid A Link - Place your websites link starting at $1.00
ShareUrLink - ShareUrLink for $1!

I'm not blocking my entire site from being crawled; just certain pages. Everybody has pages they don't want/need crawled.

Yes, I know that code is said to allow the AdSense bot to crawl my site. However, my question is, will that code override my current code which disables all crawlers from crawling my specified content?

That should only allow googles mediapartner spiders to crawl it. Google has separate crawlers for adsense and their actual search engine

This shouldn't make any other crawler index these pages at all since its giving special directions to mediapartners-google crawler, and not others.

--------------------


Need a Custom Wordpress or Business Design? Staple Web Design
Camaro Forums & Firebird Forums

I got the same email from google
this is what I did
http://forum.vbulletinsetup.com/robots.txt

--------------------

Post your URL and a link to your vb.com username here to get full access.

Add our Facebook Blog, Group and LinkedIn group.. Thanks

Stay up to date by installing our Tool Bar

I got the same e-mail yesterday but following their directions my robots.txt looks different like this.

-Raymond

--------------------
New York Forum | Email Help

I wasn't sure how it should go, I changed mine thanks

--------------------

Post your URL and a link to your vb.com username here to get full access.

Add our Facebook Blog, Group and LinkedIn group.. Thanks

Stay up to date by installing our Tool Bar

Here is how I have it

Good or bad... ?

I'm lost

I'm not sure if this is good or bad, but I'd rather not screw around with google.

I noticed in the webmaster tools I still have a warning for google not being allowed per the robots.txt.

I changed mine to be this:


Allow everything, and if they run into permissions errors so be it. If that's what makes google happy...

-Raymond

--------------------
New York Forum | Email Help

Well here is one I use for most sites...it works fairly well

Sitemap: http://yoursite.com/sitemap_index.xml.gz

User-agent: *
Disallow: /admincp/
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /customavatars/
Disallow: /customprofilepics/
Disallow: /images/
Disallow: /modcp/
Disallow: /ajax.php
Disallow: /arcade.php
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /global.php
Disallow: /image.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /showpost.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php


Works fine and never get errors..Though I have showpost removed because using vbseo..If you do not have vbseo..You may want to take that out

--------------------