robots.txt
Block/open status, AI/scraper rule suggestions, diff history, and post-block visit flags
Our Services
See which search and AI crawlers hit your pages, catch policy and coverage gaps, and install tracking with meta, JS, beacon, or Worker.
What you get in the dashboard
Technical file and policy checks from the same tools your panel uses—not blocking promises.
Crawl like a bot
Simulate Googlebot/GPTBot fetches: robots.txt checks, index signals, and HTML preview (no JS render)
Meta / X-Robots-Tag
Sample noindex/nofollow on homepage or top crawled URL; conflict with robots Allow
llms.txt
Check llms.txt and download a Markdown template
sitemap.xml
Reachability, freshness, and sitemap × crawl gap map
Site readiness
One score across robots, sitemap, llms, and tracking
What you see in the panel
Concrete crawl workflows beyond basic visit lists
Crawl journeys
Follow a bot’s estimated path across pages (near-IP + 30‑minute window)
HTTP errors for bots
See 404/5xx paths bots hit, with last seen and visit counts
Bot timeline
First/last seen, New/Active/Quiet status, jump into matching visits
llms.txt × AI visits
Which URLs in llms.txt were visited by AI crawlers — and which were not (visit records, not model training proof)
Today’s priorities
Up to three issues across your sites that need attention first
Install methods
Meta, JS, no-JS beacon, WordPress, GTM, and Cloudflare Worker — pick what your stack supports
Policy & technical file checks
robots.txt, meta robots, llms.txt, sitemap.xml, and readiness—exactly as in the dashboard.
robots.txt analysis
See which caution/danger crawlers are open or blocked, filter AI or scraper suggestions, compare current vs suggested rules (diff), keep version history from rescans and daily checks, and save a block date to flag visits that return afterward.
Crawl like a bot
On your own sites, run a crawler simulation with the selected bot user-agent. See whether robots.txt would allow the path, what title/canonical/noindex signals appear, and the HTML response — without executing JavaScript. Included on Pro.
Meta robots / X-Robots-Tag
Read meta robots and X-Robots-Tag from the homepage or your most crawled URL. Flag when robots.txt allows a path but meta/header blocks indexing (noindex/nofollow conflict).
llms.txt
Check whether /llms.txt exists. If missing, copy a Markdown template from the panel. Crawler blocks belong in robots.txt.
sitemap.xml & crawl gap
Check sitemap reachability, URL count, and lastmod freshness. Compare sitemap URLs with crawl hits (overlap + gap map) to find pages never crawled or crawled without a sitemap entry.
Site readiness score
One score across robots.txt, sitemap.xml, llms.txt, and tracking—plus a short history so you see what improved and what to fix next.
Verification & risk
See who is real, which AI crawlers hit you, where budget is wasted, and how risky each crawler is.
Bot authenticity
rDNS verification, verified rate, and spoof suspects
AI crawler insights
30-day AI visits, unique bots, caution/danger hits, and next steps
Crawl budget waste
Paths hit by risky/scraper crawlers with path-level Disallow suggestions
Safety levels
Caution and danger badges across the crawler catalog and visit filters
Authenticity, AI risk, and crawl waste
The same verification tools from overview, bot authenticity, and crawl analysis.
rDNS / bot authenticity
Verify known crawlers with reverse DNS. See verified rate, spoof suspects, and hostname mix so fake User-Agents stand out.
AI crawler insights
Last 30 days of AI crawler visits: total hits, unique AI bots, caution/danger volume, and recommendations such as updating llms.txt or reviewing unblocked AI crawlers in robots.txt.
Crawl budget waste
See which paths caution/danger and scraper crawlers hit most in the last 30 days, then copy path-specific Disallow suggestions to cut wasted crawl budget.
Safety badges & risk levels
Every crawler type carries a clear safe / caution / danger level in the catalog and visit filters—so risky traffic is visible without blocking promises.
Traffic, analytics & export
See how traffic breaks down, which sources convert to real visits, and export the rows you need.
Traffic
Devices, browsers, OS, and human vs crawler mix
Referral sources
Referrers that produced real visits, with CSV export
Crawl Analysis
Filter and search visits by page, crawler, UA, or IP
CSV export
Export crawl visits and referral sources as CSV
Period comparison
Compare the current period with the previous window
Advanced analytics
Deeper reports on Pro and higher plans
Crawl analysis and reporting
The same traffic, referrer, bots, and export tools from the dashboard.
Traffic breakdown
Review devices, browsers, operating systems, and the human vs crawler split so you can separate real visitors from bot load.
Referral sources
See which referrers produced real visits—not just raw hits—and export the list as CSV when you need it offline.
Crawl Analysis
Search and filter crawl visits by page URL, crawler name, User-Agent, or IP. Empty states and safety badges stay in sync with the catalog.
CSV export
Export crawl visits and referral sources to CSV from the panel whenever you need a snapshot for sheets or audits.
Period comparison
Compare the current monitoring window with the previous period to spot crawl volume and risk shifts without leaving the dashboard.
Advanced analytics
Pro and higher plans unlock deeper reporting beyond basic visit totals—aligned with your plan limits in billing.
Setup & tracking channels
Cookieless tracking plus script, no-JS beacon, and install verification.
Cookieless tracking
No tracking cookies — privacy-friendly capture for data and analytics teams
tracking.js
Cookieless script install alongside the meta tag for crawl capture
No-JS beacon
Cookieless CSS beacon ping when JavaScript is blocked or unavailable
Install Check
Confirm meta tag and tracking.js are live on your site
How install works
The same channels and checks from My Websites and the setup guide.
Cookieless tracking
Customer-site capture does not set tracking cookies. Data teams can instrument crawl and traffic signals without adding cookie consent burden for NMOS tracking itself.
tracking.js
Copy meta tag plus tracking.js from My Websites. The cookieless script reports crawler visits so analytics and policy tools have live data.
No-JS beacon
Add the optional cookieless beacon stylesheet for environments where JavaScript is limited. The panel shows channel status and last beacon ping.
Install Check
Use Check in My Websites to confirm the meta tag and tracking.js are live. Tracking can still start before verification completes.
Site management & collaboration
Invite teammates, grow sites within plan limits, and see quota warnings early.
Site share
Email invites with view or manage roles
Multi-site & quotas
Plan site limits with a soft wall when you hit the cap
Billing & quota alerts
Panel and email warnings at 80% and when quota is full
Share access and stay within plan limits
The same share, site-limit, and quota tools from My Websites and Billing.
Site share
Invite someone by email to view or manage a site. Pending invites, accept/decline, and revoke stay in My Websites—so collaboration does not need shared logins.
Multi-site & plan quotas
Each plan caps how many sites you can track. When the site limit is reached, the panel shows a soft wall and points you to upgrade instead of failing silently.
Billing & quota alerts
Monthly crawl quotas reset per billing period. You get panel and optional email alerts around 80% usage and when the quota is full so new visits stop being logged.
Monitoring, alerts & notifications
Health signals, robots change tracking, in-panel alerts, push, home-screen app, and email digests.
Site health
SSL expiry, 48h+ crawl quiet, 7d+ tracking stale
robots.txt changes
Diff timeline plus optional email when content changes
Violation emails
Alert when a blocked crawler visits again after your block date
Panel notifications
Bell inbox for quota, health, and policy events
Web Push
Device alerts for quota, SSL, quiet—not crawl spam
PWA / home screen
Install the dashboard like an app from Integrations
Activity summary email
Optional digest of crawl activity by email
How we keep you informed
The same health, notification, push, and PWA tools from overview and Integrations.
Site health
Overview flags SSL certificates nearing expiry or already expired, sites with no crawls for 48+ hours, and installs with no tracking ping for 7+ days—so quiet failures surface early.
robots.txt change tracking
When robots.txt content changes, rescans and daily checks keep a diff timeline. Optional email preference notifies you when the file changes.
robots violation emails
After you save a block date for a risky crawler, visits that return afterward can trigger a violation email—useful for spoof or non-compliant bots.
Panel notifications
The notification bell lists quota, health, robots, and similar events in one inbox—mark all read without leaving the dashboard.
Web Push
Enable push on supported browsers (install to Home Screen first on iOS) for quota, SSL, and crawl-quiet style alerts. Crawl visit spam is excluded.
PWA / home screen app
From Integrations, add NMOS to the home screen for faster panel access. Crawl data still needs a connection—nothing is stored offline.
Activity summary email
Optional activity digests summarize crawl activity so you do not have to open the panel every day. Preferences live with your notification settings.
Account & growth
Secure login, invite rewards, and help expand the crawler catalog.
2FA (TOTP)
Authenticator apps like Authy or Google Authenticator
Referral program
+15 days on your plan when an invitee upgrades free→paid
Catalog suggest
Propose a missing crawler type for the team to review
Security and growth tools
The same 2FA, referral, and catalog suggest tools from account settings and the crawler directory.
Two-factor authentication
Enable TOTP in account settings with Authy, Google Authenticator, or a similar app. A warning stays visible until 2FA is on.
Referral program
Share your referral code or invite link. When someone you invited upgrades from free to paid for the first time, your plan is extended by 15 days. Reward status stays in the referral panel.
Crawler catalog suggest
If a crawler type is missing from the directory, send a suggestion from the panel. The team reviews submissions—this expands detection coverage over time.
How It Works?
Start tracking crawls in 3 simple steps
1
Sign Up
Create a free account and choose your package. Activate your account with email verification.
2
Install tracking
Add the meta tag, tracking.js, and optional no-JS beacon to your site head. Copy the full snippet from My Websites.
3
Track & review
Click Check to verify the install, follow the onboarding tour if you want, then review crawls and readiness.
Tracking Systems
Dedicated pages for major search and AI crawlers