The information provided here is intended to help you
in the identification of the search engine spiders (also
known as robots,
bots or crawlers) that visit
your site. Details are based on information obtainable
by viewing your site's visitor log reports.
There are other search engine spiders, of course, but they
either get their results from one of the search engines listed
here, or are too small to bother with.
| Company |
|
Ask |
| User Agent |
|
Mozilla/2.0 (compatible; Ask Jeeves/Teoma;
+http://about.ask.com/en/docs/about/webmasters.shtml) |
| Robots.txt Identifier |
|
User-agent: Teoma |
| Details |
|
http://about.ask.com/en/docs/about/webmasters.shtml |
|
| Company |
|
Google |
| User Agent |
|
Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html) |
| Robots.txt Identifier |
|
User-agent: Googlebot |
| Details |
|
http://www.google.com/bot.html |
|
| Company |
|
Google |
| User Agent |
|
Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0
(compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) |
| Robots.txt Identifier |
|
User-agent: Googlebot-Mobile |
| Details |
|
http://www.google.com/support/webmasters/bin/answer.py?answer=35308 |
|
| Company |
|
Google Images |
| User Agent |
|
Googlebot-Image/1.0 |
| Robots.txt Identifier |
|
User-agent: Googlebot-Image |
| Details |
|
http://www.google.com/support/webmasters/bin/answer.py?answer=35308 |
|
| Company |
|
Google Adsense |
| User Agent |
|
Mediapartners-Google/2.1 |
| Robots.txt Identifier |
|
User-agent: Mediapartners-Google |
| Details |
|
http://www.google.com/bot.html Note:
This and other Googlebots share crawled pages with each
other to reduce the amount of crawling Google is required
to do for their different services. |
|
| Company |
|
Microsoft / MSN |
| User Agent |
|
msnbot/1.0 (+http://search.msn.com/msnbot.htm)
msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)
msnbot-news/1.0 (+http://search.msn.com/msnbot.htm)
msnbot-products/1.0 (+http://search.msn.com/msnbot.htm) |
| Robots.txt Identifier |
|
User-agent: msnbot
User-agent: msnbot-media
User-agent: msnbot-news
User-agent: msnbot-products |
| Details |
|
MSN has not updated their page http://search.msn.com/msnbot.htm yet
to officially address the specific bots they are using.
If you wish to block a specific bot in your robots.txt
file use the associated bot specific name as the User-agent |
|
| Company |
|
Yahoo |
| User Agent |
|
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) |
| Robots.txt Identifier |
|
User-agent: Slurp |
| Details |
|
http://help.yahoo.com/help/us/ysearch/slurp/ |
|
| Company |
|
Yahoo |
| User Agent |
|
Nokia6682/2.0 (3.01.1) SymbianOS/8.0 Series60/2.6
Profile/MIDP-2.0 configuration/CLDC-1.1 UP.Link/6.3.0.0.0
(compatible;YahooSeeker/M1A1-R2D2; http://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html) |
| Robots.txt Identifier |
|
User-agent: YahooSeeker/M1A1-R2D2 |
| Details |
|
http://help.yahoo.com/help/us/ysearch/crawling/crawling-01.html |
