About Services
Product
Crawl Analysis AI bots Bot authenticity Crawl like a bot Policy & coverage Setup & integrations Sample dashboard
Tracking Systems
Googlebot AI Chatbot Bingbot Facebookbot Twitterbot Yahoo Slurp DuckDuckBot Baiduspider Yandexbot
Pricing Contact
EN TR DE FR
Login Sign Up

Crawler Directory

Explore crawler types in the NMOS.NET catalog. Each has a dedicated tracking page.

Start free Open setup guide

AI Bots

Family: Allen Institute for AI

Ai2Bot-Dolma Caution AI2Bot Caution

Family: Anthropic

Claude-SearchBot Safe anthropic-ai Caution Claude-User Safe Claude-Code Safe Claude-Web Safe ClaudeBot Caution

Family: Big Sur AI

Big Sur AI Caution BigSurAI Caution

Family: Cohere

Cohere-Training-Data-Crawler Caution cohere-ai Caution

DeepSeek

DeepSeekBot DeepSeek Danger

DigitalOcean

DigitalOceanGenAI-Crawler DigitalOcean Caution

DuckDuckGo

DuckAssistBot DuckDuckGo Safe

Family: Mistral

MistralAI-User/1.0 Safe MistralAI-User Safe

Family: OpenAI

ChatGPT-User/2.0 Safe ChatGPT Operator Safe OAI-SearchBot Safe ChatGPT-User Safe GPTBot Caution

Family: Perplexity

Perplexity-User/1.0 Safe Perplexity-User Safe PerplexityBot Caution

QuillBot

QuillBot QuillBot Caution

Family: xAI

GrokBot Caution Grok Caution

You.com

YouBot You.com Caution
MyCentralAIScraperBot Danger SBIntuitionsBot Caution img2dataset Danger Liner-Bot Caution PanguBot Caution AI Chatbot Caution

Cloud Services

Alibaba

AliyunSecBot Alibaba Caution

Family: Amazon

Amazonbot Caution amzn-user Caution

Google Bots

Family: Google

Google-InspectionTool Safe Google-CloudVertexBot Caution Gemini-Deep-Research Caution Googlebot-Discovery Safe GoogleOther-Image Safe GoogleOther-Video Safe Googlebot-Image Safe Googlebot-Video Safe Storebot-Google Safe AdsBot-Google-Mobile Safe AdsBot-Google Safe Mediapartners-Google Safe Google-Extended Caution Googlebot-News Safe GoogleOther Safe Googlebot Safe

Other Agents

Family: Internet Archive

ia_archiver-web.archive.org Safe archive.org_bot Safe ia_archiver Safe

Pingdom

Pingdom Pingdom Safe

UptimeRobot

UptimeRobot UptimeRobot Safe
ViennaTinyBot Caution

SEO Tools

Ahrefs

AhrefsBot Ahrefs Caution

Family: Awario

AwarioSmartBot Caution AwarioRssBot Caution

Brandwatch

Brandwatch Brandwatch Caution

DataForSEO

DataForSeoBot DataForSEO Caution

Family: Google

Google-PageSpeed-Insights Safe Chrome-Lighthouse Safe PageSpeed Safe

Majestic

MJ12bot Majestic Caution

MegaIndex

MegaIndex MegaIndex Caution

Meltwater

Meltwater Meltwater Caution

Moz

DotBot Moz Caution

Family: Peer39

Peer39 Crawler/1.0 Caution Peer39 Crawler Caution

Family: Screaming Frog

Screaming Frog SEO Spider Caution Screaming Frog Caution

Searchmetrics

SearchmetricsBot Searchmetrics Caution

Family: Semrush

SemrushBot-OCOB Caution SemrushBot-SWA Caution SemrushBot Caution

TaranGroup

TaranGroup Intelligent Bot TaranGroup Caution
Sidetrade Indexer Bot Caution AudigentAdBot Caution SiteAuditBot Caution NeticlaBot Caution NeticleBot Caution Jetslide Caution Netvibes Caution

Scrapers

Amazon

Amazon Kendra Amazon Caution

ByteDance

Bytespider ByteDance Danger

Common Crawl

CCBot Common Crawl Caution

Diffbot

DiffBot Diffbot Caution

FactSet

Factset-SpyderBot FactSet Caution

Firecrawl

Firecrawl Firecrawl Danger

Family: Turnitin

TurnitinBot Caution Turnitin Caution

Webmeup

BLEXBot Webmeup Caution
Poseidon Research Crawler Caution NetEstate Imprint Crawler Caution VelenPublicWebCrawler Danger Arquivo-web-crawler Caution ISSCyberRiskCrawler Caution iMediaEthics.org Caution FriendlyCrawler Caution Webzio-Extended Caution iAskSpider/2.0 Caution magpie-crawler Danger ImagesiftBot Caution Kangaroo Bot Caution QualifiedBot Caution EchoBot-Bot Caution ICC-Crawler Caution SeekportBot Caution news-please Caution Barkrowler Caution Crawlspace Danger EchoBoxBot Caution JenkersBot Danger LivelapBot Caution iAskSpider Caution MoodleBot Caution crawler4j Danger Hada News Caution omgilibot Caution Crawl4AI Danger Cotoyogi Caution SeekrBot Caution Timpibot Caution imgproxy Caution Bravest Danger MauiBot Danger NewsNow Caution NovaAct Caution yacybot Caution Scrapy Danger omgili Caution Seekr Caution Yacy Caution

Search Engines

Amazon

Amazon-SearchBot Amazon Safe

Family: Apple

Applebot-Extended Caution Applebot Safe

Ask

Teoma Ask Safe

Baidu

Baiduspider Baidu Safe

Coc Coc

coccocbot-web Coc Coc Safe

Daum

Daumoa Daum Safe

DuckDuckGo

DuckDuckBot DuckDuckGo Safe

Family: Huawei

AspiegelBot Safe PetalBot Caution

Family: Microsoft

BingVideoPreview Safe MicrosoftPreview Safe BingPreview Safe adidxbot Safe Bingbot Safe MSNBot Safe

Family: Mojeek

MojeekBot Safe Mojeek Safe

Family: Naver

NaverBot Safe Yeti Safe

Family: Seznam

SeznamHomepageCrawler Safe SeznamBot Safe

Sogou

Sogou Sogou Caution

Family: Yahoo

Yahoo-FeedSeeker Safe Yahoo-MMCrawler Safe Yahoo! Slurp Safe Yahoo-Blogs Safe YahooSeeker Safe Slurp Safe Yahoo Safe

Family: Yandex

YandexAdditionalBot Safe YandexAdditional Safe YandexBot Safe Yandex Safe

Social Bots

LinkedIn

LinkedInBot LinkedIn Safe

Family: Meta

meta-externalfetcher Safe FacebookExternalHit Safe meta-externalagent Safe meta-externalads Safe FacebookCatalog Safe meta-webindexer Safe FacebookBot Safe WhatsApp Safe

Pinterest

PinterestBot Pinterest Safe

Quora

Quora-Bot Quora Caution

Slack

Slackbot Slack Safe

X

Twitterbot X Safe