The top 4 search engines now support Sitemaps auto submission

**News** · 04-20-2007, 10:38 PM

In November 2006, Google, Yahoo and Microsoft agreed to support the Sitemap protocol. The Sitemap protocol is a standard that can help you to submit all of your web pages to search engines.
Last week, Ask.com joined the team so that now all four major search engines support the Sitemap protocol. In addition, search engines can now automatically detect your Sitemaps file.

What are sitemaps and what is the official Sitemaps protocol?

Sitemaps are an easy way for webmasters to inform search engines about pages on their sites that are available for crawling. In its simplest form, a sitemap is an XML file that lists URLs for a site along with additional metadata about each URL:
when it was last updated

how often it usually changes

how important it is, relative to other URLs in the site

That information helps search engines to more intelligently crawl your site. The Sitemaps news is a standard that makes it easier to create a sitemap that can be parsed by all search engines.

How can the Sitemaps protocol help your website?

Web crawlers usually discover pages from links within the site and from other sites. Sitemaps supplement this data to allow crawlers that support sitemaps to pick up all URLs in the sitemap and learn about those URLs using the associated metadata.
The sitemap allows you to inform search engines about the pages on your website. Depending on how your website is structured, it can be easier for search engines to find all of your web pages if you use a sitemap.

How do search engines find my Sitemaps file? What's new?

The big four search engines Google, Yahoo, Live/MSN and Ask will now automatically index your Sitemaps file if you add a link to your Sitemaps file in your robots.txt file.
All the major search engines regularly check the robots.txt of a website. Just add the following line to your robots.txt file:
Sitemap: http:/www.yoursite.com/sitemap.xml
Of course, you can also actively inform Google and Yahoo about your Sitemap file. This is not possible with Live/MSN and Ask.

Does a Sitemaps file improve your search engine rankings?

Using the Sitemap protocol does not guarantee that your web pages are included in search engines, but provides hints for web crawlers to do a better job of crawling your site.
A sitemap helps search engines to find your web pages but it does not help you to get higher search engine rankings.
When search engines index your web pages, they must find optimized web page content and good inbound links. Search engines will only give your web pages high rankings if they have both.

The Sitemaps protocol can help you to inform search engines about your web pages. Further information about the new sitemaps standard can be found on the official sitemaps web site.

[Source..]

**eric** · 04-20-2007, 11:21 PM

ok, I already ping Google and Yahoo, how to ping both others .. someone knows ?

**Brandon Sheley** · 04-21-2007, 01:23 PM

good question, I'm not sure

**peterpigman** · 04-22-2007, 01:47 AM

How would you write the xml file?

**AzzidReign** · 04-22-2007, 12:16 PM

Great article! The only thing I'm wondering is how to point to these sitemap files when using vBSEO. With vBSEO, you get a ton of sitemap files (if you have a large site...I'm up to around 16 sitemap files I think)...do I just add the index file...or what?

**Brandon Sheley** · 04-22-2007, 12:36 PM

Quote:

Originally Posted by AzzidReign

Great article! The only thing I'm wondering is how to point to these sitemap files when using vBSEO. With vBSEO, you get a ton of sitemap files (if you have a large site...I'm up to around 16 sitemap files I think)...do I just add the index file...or what?

another good question..bummer, stumped 2 times in a row..lol

well, I don't have that large of sitemap, but I've seen questions like this at vbseo.com.. I'll have to run over and check things out and reply here later.

-Bs

**AzzidReign** · 04-22-2007, 12:45 PM

Well, if you want to, send me the link to your thread over there

I like to keep tract of progress that goes on at vbseo.

**Brandon Sheley** · 04-22-2007, 01:15 PM

well I haven't made it over to look for any threads. This is the url I send to google http://forum.vbulletinsetup.com/sitemap_index.xml.gz

**Capper** · 04-23-2007, 05:20 PM

Where would i add this in my sitemap? Also, would i put a separate one for my forums and my main site?

http:/www.yoursite.com/sitemap.xml

Code:

User-agent: *
Disallow: /cowadmin/
Disallow: /archive/
Disallow: /templates/
Disallow: /dmenu/
Disallow: /tpllib/
Disallow: /data/
Disallow: /print.php
Disallow: /pdf.php
Disallow: /xml.php
Disallow: /txt.php
Disallow: /mailto.php
Disallow: /forums/ajax.php
Disallow: /forums/attachment.php
Disallow: /forums/calendar.php
Disallow: /forums/cron.php
Disallow: /forums/editpost.php
Disallow: /forums/global.php
Disallow: /forums/image.php
Disallow: /forums/inlinemod.php
Disallow: /forums/joinrequests.php
Disallow: /forums/login.php
Disallow: /forums/member.php
Disallow: /forums/memberlist.php
Disallow: /forums/misc.php
Disallow: /forums/moderator.php
Disallow: /forums/newattachment.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/poll.php
Disallow: /forums/postings.php
Disallow: /forums/printthread.php
Disallow: /forums/private.php
Disallow: /forums/profile.php
Disallow: /forums/register.php
Disallow: /forums/report.php
Disallow: /forums/reputation.php
Disallow: /forums/search.php
Disallow: /forums/sendmessage.php
Disallow: /forums/showgroups.php
Disallow: /forums/subscription.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php
Disallow: /forums/usernote.php

**eric** · 04-24-2007, 12:48 AM

this is what you have to set in robots.txt in highest level of your account

this is to tell search robots not to go in those areas. According me nothing to do with sitemap

**Capper** · 04-24-2007, 03:26 AM

Say again? Your post makes absolutely no sense to me. The above is my robots.txt file in my site root.....It keeps robots from accessing all those files listed......my question is....where would I put the the sitemap information...."http:/www.MYsite.com/sitemap.xml" in the above file?

**AzzidReign** · 04-24-2007, 06:27 AM

Reading this from the original post would help:

Quote:

How do search engines find my Sitemaps file? What's new?
The big four search engines Google, Yahoo, Live/MSN and Ask will now automatically index your Sitemaps file if you add a link to your Sitemaps file in your robots.txt file.
All the major search engines regularly check the robots.txt of a website. Just add the following line to your robots.txt file:
Sitemap: http:/www.yoursite.com/sitemap.xml
Of course, you can also actively inform Google and Yahoo about your Sitemap file. This is not possible with Live/MSN and Ask.

So just put it anywhere in your robots.txt. I would put it right before the first disallow.

**Capper** · 04-27-2007, 03:57 AM

well, I read the original post, which is why I asked the question.......

Quote:

Reading this from the original post would help:

you'd think maybe answering questions without rudeness would make for a better experience for everyone......maybe some of us aren't quite the programming expert you are. I wasn't sure where exactly to put it.

**AzzidReign** · 04-28-2007, 10:51 AM

Quote:

Originally Posted by Capper

well, I read the original post, which is why I asked the question.......

you'd think maybe answering questions without rudeness would make for a better experience for everyone......maybe some of us aren't quite the programming expert you are. I wasn't sure where exactly to put it.

I'm no programming expert...I just use common sense.

**Capper** · 05-02-2007, 12:12 AM

Common sense? Like maybe not posting when you don't have anything to add to a discussion?

People like you baffle me

**eric** · 05-02-2007, 12:32 AM

Quote:

Originally Posted by Capper

Say again? Your post makes absolutely no sense to me. The above is my robots.txt file in my site root.....It keeps robots from accessing all those files listed......my question is....where would I put the the sitemap information...."http:/www.MYsite.com/sitemap.xml" in the above file?

ok, now everybody cools down or I close the thread. We're here to help each other and not to yell, if you feel the urge, well go elsewhere.

Regarding the original question. It is clear I understood wrong.

First remark, why want to hide your sidemap to spiders ? Basically it is made for them and via via even to submit that directly to Google, Yahoo etc...

If you really want to hide it..

Disallow: /sitemap.xml

should do the trick

**Capper** · 05-02-2007, 01:08 AM

I don't want to hide my sitemap.......that is my robots.txt file......I wanted to know exactly where to place the sitemap to make sure it ISN'T hidden.

**eric** · 05-02-2007, 01:23 AM

in that case just don't do anything, it'll be read now
as you can see in your robots.txt there are onlyu disallow lines, clearly not allowed to read those files, if it isn't listed then it will be read.

just don't put it in the directories (lines ending with /) that are disallowed completely

**AzzidReign** · 05-02-2007, 08:00 AM

I think you may have misunderstood eric:

Quote:

How do search engines find my Sitemaps file? What's new?

The big four search engines Google, Yahoo, Live/MSN and Ask will now automatically index your Sitemaps file if you add a link to your Sitemaps file in your robots.txt file.
All the major search engines regularly check the robots.txt of a website. Just add the following line to your robots.txt file:
Sitemap: http:/www.yoursite.com/sitemap.xml
Of course, you can also actively inform Google and Yahoo about your Sitemap file. This is not possible with Live/MSN and Ask.

It in fact does say to add it to your robots.txt file. Instead of putting Disallow: /whatever/, it says to put Sitemap: /whatever/. This will help the 4 big search engines to find your sitemap; even though you can submit your sitemaps automatically, this, to me, would be more sufficient because spiders ALWAYS read the robots.txt file.

Quote:

Common sense? Like maybe not posting when you don't have anything to add to a discussion?

I do believe I did add to the discussion telling you exactly what you have to do and where to put it, yet you still are having troubles...no wonder I baffle you.

**Brandon Sheley** · 05-10-2007, 02:07 PM

This is what I'm adding to top of my robot.txt page

Code:

Sitemap:http://forum.vbulletinsetup.com/sitemap_1.xml.gz

I think I'll add the link to the blogs sitemap and the directories sitemap and I'll put it in all 3 robot.txt files