About Services
Product
Crawl Analysis AI bots Bot authenticity Crawl like a bot Policy & coverage Setup & integrations Sample dashboard
Tracking Systems
Googlebot AI Chatbot Bingbot Facebookbot Twitterbot Yahoo Slurp DuckDuckBot Baiduspider Yandexbot
Pricing Contact
EN TR DE FR
Login Sign Up

Crawler Directory

Explore crawler types in the NMOS.NET catalog. Each has a dedicated tracking page.

Start free Open setup guide

AI Bots

Family: Allen Institute for AI

Ai2Bot-Dolma Caution AI2Bot Caution

Family: Anthropic

anthropic-ai Caution ClaudeBot Caution

Family: Big Sur AI

Big Sur AI Caution BigSurAI Caution

Family: Cohere

Cohere-Training-Data-Crawler Caution cohere-ai Caution

DigitalOcean

DigitalOceanGenAI-Crawler DigitalOcean Caution

OpenAI

GPTBot OpenAI Caution

Perplexity

PerplexityBot Perplexity Caution

QuillBot

QuillBot QuillBot Caution

Family: xAI

GrokBot Caution Grok Caution

You.com

YouBot You.com Caution
SBIntuitionsBot Caution Liner-Bot Caution PanguBot Caution AI Chatbot Caution

Cloud Services

Alibaba

AliyunSecBot Alibaba Caution

Family: Amazon

Amazonbot Caution amzn-user Caution

Google Bots

Family: Google

Google-CloudVertexBot Caution Gemini-Deep-Research Caution Google-Extended Caution

Other Agents

ViennaTinyBot Caution

SEO Tools

Ahrefs

AhrefsBot Ahrefs Caution

Family: Awario

AwarioSmartBot Caution AwarioRssBot Caution

Brandwatch

Brandwatch Brandwatch Caution

DataForSEO

DataForSeoBot DataForSEO Caution

Majestic

MJ12bot Majestic Caution

MegaIndex

MegaIndex MegaIndex Caution

Meltwater

Meltwater Meltwater Caution

Moz

DotBot Moz Caution

Family: Peer39

Peer39 Crawler/1.0 Caution Peer39 Crawler Caution

Family: Screaming Frog

Screaming Frog SEO Spider Caution Screaming Frog Caution

Searchmetrics

SearchmetricsBot Searchmetrics Caution

Family: Semrush

SemrushBot-OCOB Caution SemrushBot-SWA Caution SemrushBot Caution

TaranGroup

TaranGroup Intelligent Bot TaranGroup Caution
Sidetrade Indexer Bot Caution AudigentAdBot Caution SiteAuditBot Caution NeticlaBot Caution NeticleBot Caution Jetslide Caution Netvibes Caution

Scrapers

Amazon

Amazon Kendra Amazon Caution

Common Crawl

CCBot Common Crawl Caution

Diffbot

DiffBot Diffbot Caution

FactSet

Factset-SpyderBot FactSet Caution

Family: Turnitin

TurnitinBot Caution Turnitin Caution

Webmeup

BLEXBot Webmeup Caution
Poseidon Research Crawler Caution NetEstate Imprint Crawler Caution Arquivo-web-crawler Caution ISSCyberRiskCrawler Caution iMediaEthics.org Caution FriendlyCrawler Caution Webzio-Extended Caution iAskSpider/2.0 Caution ImagesiftBot Caution Kangaroo Bot Caution QualifiedBot Caution EchoBot-Bot Caution ICC-Crawler Caution SeekportBot Caution news-please Caution Barkrowler Caution EchoBoxBot Caution LivelapBot Caution iAskSpider Caution MoodleBot Caution Hada News Caution omgilibot Caution Cotoyogi Caution SeekrBot Caution Timpibot Caution imgproxy Caution NewsNow Caution NovaAct Caution yacybot Caution omgili Caution Seekr Caution Yacy Caution

Search Engines

Apple

Applebot-Extended Apple Caution

Huawei

PetalBot Huawei Caution

Sogou

Sogou Sogou Caution

Social Bots

Quora

Quora-Bot Quora Caution