Web Bot Auth: Agent Identity via HTTP Message Signatures

User-Agent strings stopped being a credible bot identity — anyone can spoof them. Web Bot Auth (Cloudflare + IETF, 2025), built on RFC 9421 HTTP Message Signatures, gives cryptographically verifiable bot identity. AIDE's bots-web-bot-auth and web-bot-auth-detected checks test that your site both verifies and reacts.

What AIDE actually checks

bots-web-bot-auth runs three probes:

Does a known bot UA + valid signature get a 200?
Does the same UA + invalid signature get blocked (403)?
Does Cloudflare directory key fetch work for the bot in question?

web-bot-auth-detected further verifies that the site uses the mechanism (not just lets it pass) — does cache or logging behavior actually change for signed bot traffic?

Anatomy of a bot signature

The Cloudflare bot network (Googlebot 2.0, ChatGPT-User, ClaudeBot v2, etc.) attaches four headers:

GET /robots.txt HTTP/2
Host: aide.tr
User-Agent: ClaudeBot/1.0
Signature-Agent: claude-bot
Signature-Input: sig1=("@authority" "@method" "@target-uri" "signature-agent");created=1735689600;keyid="cloudflare-claude-1";alg="ed25519"
Signature: sig1=:abc123...:

Four critical fields:

| Field | Meaning | |---|---| | Signature-Agent | Which bot — registered name in the Cloudflare directory | | Signature-Input | What was signed (method, URI, etc.) and which key | | Signature | Ed25519 signature (base64) | | User-Agent | Cosmetic — real identity comes from Signature-Agent |

Server-side verification

In Python, with the http-message-signatures package:

# bot_verifier.py
import httpx
from http_message_signatures import HTTPMessageVerifier, algorithms
from http_message_signatures.resolvers import HTTPSignatureKeyResolver
import base64
from nacl.signing import VerifyKey


CF_DIRECTORY = "https://radar.cloudflare.com/api/v1/bot-management/keys"


class CloudflareKeyResolver(HTTPSignatureKeyResolver):
    """Fetches public keys from the Cloudflare bot directory."""

    async def resolve_public_key(self, key_id: str):
        async with httpx.AsyncClient() as client:
            resp = await client.get(f"{CF_DIRECTORY}/{key_id}")
            resp.raise_for_status()
            key_pem = resp.json()["public_key"]
        return VerifyKey(base64.b64decode(key_pem))


async def verify_bot(request) -> str | None:
    """Returns the signature_agent if verified, otherwise None."""
    if "Signature" not in request.headers:
        return None

    verifier = HTTPMessageVerifier(
        signature_algorithm=algorithms.ED25519,
        key_resolver=CloudflareKeyResolver(),
    )
    try:
        await verifier.verify(request)
    except Exception:
        return None

    return request.headers.get("Signature-Agent")

Caller:

from fastapi import FastAPI, Request

app = FastAPI()

@app.middleware("http")
async def bot_auth_middleware(request: Request, call_next):
    bot_name = await verify_bot(request)
    request.state.verified_bot = bot_name
    return await call_next(request)


@app.get("/")
async def root(request: Request):
    if request.state.verified_bot:
        return {"hello": f"verified bot {request.state.verified_bot}"}
    return {"hello": "browser"}

What to do with a verified bot

Just letting them through isn't enough — web-bot-auth-detected looks for behavior differentiation:

Rate limit relaxation: verified bots get a higher QPS ceiling
Cache headers: Cache-Control: public, max-age=300 for bots, strict for browsers
Logging: route bot_traffic to a separate table (keep analytics clean)
Robots.txt override: verified bots can bypass softer restrictions

Example:

@app.get("/articles/{slug}")
async def article(slug: str, request: Request):
    bot = request.state.verified_bot
    response = JSONResponse(load_article(slug))

    if bot in ("googlebot", "claude-bot"):
        # Verified crawler: full content + aggressive cache
        response.headers["Cache-Control"] = "public, max-age=86400"
        response.headers["X-Bot-Allowed"] = bot
    elif bot:
        # Verified but not whitelisted
        response.headers["Cache-Control"] = "public, max-age=300"
    else:
        # Human or unverified
        response.headers["Cache-Control"] = "private, max-age=60"

    return response

Common mistakes

| Mistake | Symptom | Fix | |---|---|---| | Only UA check, no signature | AIDE WARNING — "User-Agent spoofing not detected" | Verify UA + signature together | | Key fetch hits the network on every request | Latency >500ms | Cache public keys for 24 h (pubkey rotation is rare) | | No created timestamp check | Replay attack possible | Reject if now - created > 300 s | | Soft rate limit applied to unverified bots too | Spoofers can starve real admins | Default to strict, only relax for verified signatures |

Without Cloudflare — your own key directory

If you want to publish your own bot — say AIDE's own scan bot:

// /.well-known/http-message-signatures/keys/aide-bot-1.json
{
  "kty": "OKP",
  "crv": "Ed25519",
  "x": "ABC123...base64...",
  "kid": "aide-bot-1",
  "alg": "EdDSA"
}

Submit to the Cloudflare directory:

curl -X POST https://radar.cloudflare.com/api/v1/bot-submissions \
  -H "Authorization: Bearer $CF_API_TOKEN" \
  -d '{
    "name": "AIDE Scan Bot",
    "agent_string": "AIDEScanBot/1.0",
    "key_url": "https://aide.tr/.well-known/http-message-signatures/keys/aide-bot-1.json",
    "purpose": "compliance auditing",
    "rate_limit": "10rps"
  }'

After manual review (~2 weeks) you're listed; AIDE-aware sites grant your bot the higher QPS ceiling.

Testing — what AIDE does

Send a manual bot signature:

# Valid signature — using Cloudflare's test key
curl -i https://your-site.com/ \
  -H 'User-Agent: ClaudeBot/1.0' \
  -H 'Signature-Agent: claude-bot' \
  -H 'Signature-Input: sig1=("@authority");created=1735689600;keyid="cloudflare-claude-1";alg="ed25519"' \
  -H 'Signature: sig1=:VALID_BASE64_SIG:'

Expected: 200 + bot-relaxed cache headers.

# Invalid signature — corrupted
curl -i https://your-site.com/ \
  -H 'User-Agent: ClaudeBot/1.0' \
  -H 'Signature: sig1=:CORRUPTED:'

Expected: 403, or browser-strict headers.

If both behave correctly, bots-web-bot-auth reports PASS in an AIDE scan.

Production hardening

Key rotation: Rotate public keys every 90 days. Old key IDs must remain valid for 30 days (bots may have cached them).
Failure mode: What if the Cloudflare directory is down? Either degrade gracefully (treat as browser) or use a cached key. Never return 5xx.
Telemetry: Emit bot_signature_invalid_total and bot_signature_valid_total to Prometheus — early warning for attacks.

Related resources

RFC 9421 — HTTP Message Signatures
Cloudflare Web Bot Auth
Cloudflare Bot Directory
http-message-signatures (Python)
AIDE check details: /learn/bots-web-bot-auth, /learn/web-bot-auth-detected