Are AI agents talking about your site?

OpenAI ChatGPT, Anthropic Claude, Perplexity, Google Gemini, and Microsoft Copilot collectively fetch billions of pages each month on behalf of users. When answering a question like "what is X company's return policy", they don't do what a browser does — execute JavaScript, wait for hydration, lazy-load most ad scripts. They fetch HTML, extract content, and if extraction fails the user gets "I couldn't access the source" and the agent moves on to a different site.

The result: sites that are not amenable to agentic readers are invisible in answer panels, even when they have the most relevant content. This invisibility is not a bug — it's a protocol mismatch. AIDE audits these mismatches automatically, producing a concrete report: which check failed, which standard is missing, which fix is the priority.

What matters is that the agentic ecosystem is expanding fast. Cohere, xAI, DeepSeek, Mistral are spinning up their own crawlers; each interprets robots.txt directives differently, uses a different User-Agent, prefers different meta tags. A static "Allow all bots" approach no longer covers the modern reader profile; each family of crawler expects its own User-Agent block.

Bots fall into three classes: training, user-initiated, and search-indexing. Training bots (GPTBot, ClaudeBot, CCBot, Bytespider, Amazonbot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Cohere-AI) collect your content for the next generation of foundation models; if you allow them, your brand enters the model's implicit representation. If you refuse, your brand may still leak in via cross-references but your canonical content does not.

User-initiated bots (ChatGPT-User, Claude-User, Perplexity-User, Gemini-User) fetch a page in response to a human question. Blocking them usually backfires: the agent says "I couldn't read this source", picks a different one, and your URL is invisible to that conversation. You can refuse training and allow user-initiated; that macro decision shows up as an explicit rule in the AIDE score.

Search-indexing bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Cohere-AI) build indexes for AI-powered search. ChatGPT Search, Perplexity Pages, Claude's embedded search are all dependent on these indexes. Block them, and you exit the channel — neither hurt nor helped, but you've opted out of a growing, intentful surface.

Classical robots.txt is RFC 9309 (2022), built for the single-crawler model — Googlebot, Bingbot. In the modern AI era a new User-Agent appears every month and its operator wants to apply different policies. AIDE's bot taxonomy is the manifesto for this multi-policy need: instead of User-agent: * for everything, every family gets its own block, every purpose its own rule.

AIDE's robots-search-index, robots-user-initiated, and robots-training each demand ≥ 2 explicit User-Agent blocks. Passing through with a single wildcard does not count as deliberate policy; readers that value the structure are designed to prefer site owners who've made an explicit choice.

Cloudflare's Content-Signal directive introduced in 2024 (format: search=yes, ai-input=yes, ai-train=no) is a natural extension of the protocol. AIDE treats it as an optional bonus signal: not having it does not cost score, having it earns extra points because site authority is more sharply expressed.

Model Context Protocol (MCP) is a standard from Anthropic; it lets an AI agent invoke a tool directly. Publishing /.well-known/mcp.json is the agent-facing equivalent of "I'm here, I expose these capabilities, call me via JSON-RPC". AIDE's proto-mcp-server check looks for this document; it validates protocolVersion, name, capabilities, endpoints.

MCP's value: an agent, with user consent, can use your API directly. "Get the product catalog from this site", "fill out this form" — complex actions that are the agent's job. For that the agent needs to know your tool exists; the discovery surface is mcp.json. Sites that don't publish it are absent from the agent's tool pool.

Three adoption levels: (1) publish only the manifest — agents discover. (2) manifest + running MCP server — agents actually call. (3) add OAuth 2.1 client registration — agents act on the user's behalf. AIDE today audits (1) and (2); the OAuth registration check ships in Q2 2026.

Google's A2A protocol emerged for agent-to-agent communication: one agent can discover and delegate work to another. /.well-known/agent.json is the foundation of that discovery; the agent card declares the agent's skills, input/output modalities, and contact information.

AIDE's proto-a2a check schema-validates agent.json + requires at least two declared skills. A single-skill or skill-less manifest counts as an "empty card". This is not Google's spec but AIDE's practical observation: a single-skill manifest is usually demo/test only.

A2A's ecosystem effect: LangGraph, AutoGen, CrewAI default to agent.json discovery. Joining the workflows they orchestrate is as simple as publishing your card; the agents wire up the integration themselves.

In the 2010s the web migrated from RDFa to microdata to JSON-LD; the SEO impact remains relevant. In the AI era there's an additional importance: instead of observing HTML and reasoning over it, an agentic reader can read the schema.org JSON-LD block and extract clean structure — Organization, name=X, address=Y. The noise floor on NLP-model input drops.

AIDE's sd-jsonld check audits the presence and validity of schema.org types like Organization, WebSite, BreadcrumbList, Article, FAQPage, Product. It does not just look for schema-org content — it does a "consistent with the type's definition" semantic check: an Organization without a contactPoint, an Organization without ≥ 2 sameAs links, an Organization without a logo all count as incomplete.

OpenGraph + Twitter Cards are the social side-arm of structured data. Agents use these meta tags as "URL preview cards"; missing or broken og:image means the link card looks broken. AIDE's sd-og-twitter check expects og:image to return 200 + image/* content-type via HTTP HEAD; for SSRF reasons it does not GET, just verifies presence.

A fast site matters not just for the user but for the agent. The agent's budget is finite: producing one answer requires 5–15 page fetches; if each has a median TTFB of 800ms, network wait alone burns 4–12 seconds. The speed budget is exceeded, the agent moves to other sources.

AIDE's perf-ttfb check expects the median of five cold-cache requests under 600ms. The cache header check looks for a revalidator (max-age or no-cache) in Cache-Control + ETag or Last-Modified. Vary: Accept-Encoding is required for the gzip variant; without it the CDN tries to cache the same binary for both gzip and brotli in a single snapshot, behaving inconsistently.

CDN hit ratio: if the site is behind Cloudflare/Akamai/Fastly/Bunny, static assets returning HIT matters. Constant MISS means edge cache rules are missing; every request hits origin; extra latency for the AI agent. AIDE's perf-cdn-hit checks two probes of the homepage and expects HIT or a positive Age on the second.

When an AI agent parses HTML, it strips script + style + tag overhead; what remains is visible text. If the page's real content ratio is below 15%, the agent likely classifies the page as empty and does not use it for an answer. AIDE's text-html-ratio check audits this threshold.

Boilerplate-heavy sites (10+ tracking scripts, hydration payload, lazy-loaded data) struggle to clear the threshold. Solutions: server-side rendering, genuinely meaningful server-rendered content, moving non-critical inline JSON to external files, defer or lazy-load non-critical scripts.

Markdown content negotiation is a modern trend: when an agent sends Accept: text/markdown, the site returns a markdown rendition. The agent skips HTML parsing — markdown is already agent-friendly. AIDE's markdown-negotiation check expects one of three paths: Accept negotiation (most correct), .md suffix, ?format=md fallback. A Vary: Accept header earns extra points.

When an AI agent crawls your site, all your public content enters the agent's context. Combine that with the EU data-protection definition of "personal data" and: published employee profiles, customer reviews, references in old blog posts — all of it enters the model. KVKK / GDPR compliance is no longer about forms; it's about web publishing discipline.

AIDE itself is KVKK-compliant: account deletion completes within 30 days, audit log retained 90 days, scan history can be wiped only by the domain owner. Data lives inside the EU: ClickHouse + Redis in Germany, edge cache in Cloudflare's EU pops. No cross-border transfer; nothing written to US or APAC pops without opt-in.

The agent ecosystem complicates the right-to-be-forgotten obligation: a deleted page is not deleted from training data. AIDE shows the site owner the question — "you allow training bots here, are you deliberate about it?". The choice is yours; we test only deliberateness.