如何通过UserAgent识别谷歌爬虫

2016-04-23 15:12:00
admin
原创
4086
摘要:谷歌爬虫,googlebot,googlebot_useragent

如何通过UserAgent识别谷歌爬虫?


我们可以通过UserAgent来查看那些谷歌蜘蛛在爬取我们的网站。


“爬虫”是一个通用术语,用于任何程序(如机器人或蜘蛛),用于自动发现和扫描网站,通过以下链接从一个网页到另一个。谷歌的爬虫叫做Googlebot。下表列出了常见的谷歌爬虫,您可能会看到您的参考日志信息,以及他们应该如何指定在robots.txt文件,robots meta标签,和x-robots-tag HTTP指令。


谷歌爬虫 Useragent标识 完整的UserAgent (在网站日志中看的的)
Googlebot (Google Web search) Googlebot Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
or
(rarely used): Googlebot/2.1 (+http://www.google.com/bot.html)
Googlebot News Googlebot-News
(Googlebot)
Googlebot-News
Googlebot Images Googlebot-Image
(Googlebot)
Googlebot-Image/1.0
Googlebot Video Googlebot-Video
(Googlebot)
Googlebot-Video/1.0
Google Mobile (feature phone) Googlebot-Mobile
  • SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
  • DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
Google Smartphone Googlebot

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Google Mobile AdSense Mediapartners-Google
or
Mediapartners
(Googlebot)
[various mobile device types] (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)
Google AdSense Mediapartners-Google
Mediapartners
(Googlebot)
Mediapartners-Google
Google AdsBot landing page quality check AdsBot-Google AdsBot-Google (+http://www.google.com/adsbot.html)

Google app crawler

(Used to fetch resources for mobile apps, obeys AdsBot-Google robots rules.)

AdsBot-Google-Mobile-Apps AdsBot-Google-Mobile-Apps

发表评论
评论通过审核后显示。
 
在线咨询
售前咨询
售后支持
电话咨询