Your rate limiter was designed for humans

At 03:14, the origin latency graph bent upward and stayed there. No deploy had gone out. No dependency had degraded. The WAF dashboard was green, the DDoS tab was empty and the per-user rate limiter had not issued a single 429 in six hours.

The traffic was from one agent. It had been given a research task, it had hit a paginated endpoint, it had misparsed a response as retryable and it had been retrying ever since. Sixty requests a minute, patiently, for six hours. Fifty thousand requests. Every one of them a cache miss, because the agent walked unique URLs and never looked back. Every one of them a cold round trip to a database that had been sized for a workload where people get bored and close the tab.

The limiter was configured at sixty requests per minute. The agent never exceeded it. There was nothing to alert on, because nothing was wrong according to the only question the system knew how to ask.

That question: how many requests did this caller make in the last minute?
is a human era artifact. It encodes an assumption that stopped being true sometime in the last eighteen months: that a client which behaves politely per minute is a client behaving reasonably overall. Humans get tired. Humans hit a wall of text and give up. Humans close the laptop. An agent has none of those properties and the entire request-handling stack was built on the quiet expectation that it would.

Same ceiling, two populations. The one that never idles never trips the throttle.

---

The throttle held. That was the failure.

Request counting measures a rate. Agent traffic does not fail on rate. It fails on persistence, on cost variance and on cache behaviour - three dimensions that a request-per-minute counter is structurally blind to.

Persistence is the obvious one. A limiter with a one-minute window has no memory of the fifty-nine minutes before it. An agent stuck in a loop is, from the limiter's point of view, fifty-nine consecutive well-behaved minutes followed by another one.

Cost variance is the one that gets expensive fastest. If you front an LLM, two requests to the same endpoint can differ by two orders of magnitude in what they actually consume. A fifty-token prompt and a forty-thousand-token prompt are one request each. Counting requests when the cost distribution is that wide is like billing a data centre by the number of power cables.

Cache behaviour is the one nobody sees coming. Rate limiting and caching are usually owned by different people, so the interaction goes unnoticed until the origin graph moves. Human traffic is Zipf-shaped: a small number of popular paths absorb most of the volume, the edge serves them and the origin sees a fraction. Crawler traffic is the inverse. It enumerates. It touches each URL once. Your hit rate does not degrade gracefully; it goes to zero and the origin starts receiving the full unfiltered stream it has never once been load-tested against.

Then there is the retry pathology, which deserves its own sentence because it is so avoidable. If your API returns 200 OK with an error object in the body - a pattern that is everywhere in internal services - an agent parsing for HTTP status sees success, sees a body it cannot use and tries again. Forever. You have built a perfect retry engine and handed it the keys.

---

Four assumptions break at the same moment

The reason this is hard to diagnose is that it does not present as four problems. It presents as one: the origin is slow and the bill is up. You go fix throttling and the failure relocates to caching. You go fix caching and it relocates to cost.

Each layer holds an assumption about the caller. Agent traffic violates all four at the same time.

They relocate because they are not four independent bugs. They are four symptoms of one missing primitive.

---

Nobody actually knows who is calling

Every one of those layers is trying to make a decision about a client and every one of them is using evidence that does not survive contact with 2026.

User-Agent is a free-text field. It has always been a free-text field. We built two decades of bot policy on a string the client chooses and it worked only because most automated traffic had no incentive to lie. That incentive now exists: being blocked has a cost so the string is worthless as an input to anything that matters.

IP allowlisting fails for a subtler reason. It does not fail because IPs are forgeable; it fails because they are rented. Agent platforms egress through pools that rotate faster than any operator refreshes their reverse-DNS map. A CIDR-based rule you wrote in 2024 is not wrong so much as it is describing a network topology that no longer exists.

And reverse DNS - the one method that actually was cryptographically defensible, in the loose sense that it required control of a DNS zone; only ever covered the crawlers that bothered to publish PTR records, which is a shrinking fraction of the traffic and a set that does not include most agent platforms.

So the honest statement of the problem is not our rate limiter is misconfigured. It is: we are applying policy to an anonymous caller and calling it identity.

---

What the CDNs shipped while nobody was looking

The fix that the infrastructure layer converged on is the boring,
correct one: make the caller prove who it is with a key.

Web Bot Auth is a profile on top of RFC 9421 HTTP Message Signatures, which has been a Proposed Standard since February 2024. The mechanics are deliberately unexciting. An operator generates an Ed25519 keypair. It publishes the public key as a JWKS at a fixed well-known path on a domain it controls. It signs every outbound request over a small set of components. You fetch the directory once, cache it and verify.

RFC 9421 signatures, one Ed25519 key per operator, one directory fetch per rotation window.

What makes it operationally interesting is not the cryptography. It is that the keyid - the RFC 8037 JWK thumbprint of the operator's public key is a stable, unforgeable identifier you can key a bucket on. Every broken layer in the previous section was broken because it had no such identifier. This is the primitive they were all missing.

Cloudflare activated verification at its edge in March 2026. Akamai, AWS and Vercel followed. In May, Shopify moved from supporting it to pricing it: unsigned bots and agents hitting the Storefront API were dropped to the strictest rate-limit tier and signing became the path to a higher one. That is the moment the thing stopped being a standards conversation and became a procurement one.

Generating and publishing the key material

The operator side is three commands and a static file.

# 1. Ed25519 private key — the only key algorithm Cloudflare accepts today
openssl genpkey -algorithm ed25519 -out private-key.pem

# 2. Public half
openssl pkey -in private-key.pem -pubout -out public-key.pem

# 3. Convert to JWK (jwker, or WebCrypto's generateKey, or any JOSE library)
go install github.com/jphastings/jwker/cmd/jwker@latest
jwker public-key.pem public-key.jwk

The directory itself has one requirement that trips almost everyone on the first attempt: the directory response must itself be signed. Otherwise anyone can mirror your JWKS and register as you.

# directory.py — serves /.well-known/http-message-signatures-directory
# The response is signed with tag="http-message-signatures-directory"
# over ("@authority";req), per draft-meunier-http-message-signatures-directory.
import base64, hashlib, json, secrets, time
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
from cryptography.hazmat.primitives.serialization import Encoding, PublicFormat

CONTENT_TYPE = "application/http-message-signatures-directory+json"


def b64u(raw: bytes) -> str:
    """base64url, no padding — the encoding used throughout JOSE."""
    return base64.urlsafe_b64encode(raw).decode().rstrip("=")


def jwk_thumbprint(pub_raw: bytes) -> str:
    """RFC 8037 §A.3 — canonical JSON, lexicographic keys, SHA-256."""
    jwk = {"crv": "Ed25519", "kty": "OKP", "x": b64u(pub_raw)}
    canonical = json.dumps(jwk, separators=(",", ":"), sort_keys=True).encode()
    return b64u(hashlib.sha256(canonical).digest())


def build_directory(key: Ed25519PrivateKey, authority: str, ttl: int = 86400):
    pub = key.public_key().public_bytes(Encoding.Raw, PublicFormat.Raw)
    kid = jwk_thumbprint(pub)

    body = json.dumps({
        "keys": [{"kty": "OKP", "crv": "Ed25519", "x": b64u(pub)}]
    }, separators=(",", ":"))

    created = int(time.time())
    expires = created + 60
    params = (
        '("@authority";req)'
        f";created={created};expires={expires}"
        f';keyid="{kid}";alg="ed25519"'
        f';nonce="{b64u(secrets.token_bytes(32))}"'
        ';tag="http-message-signatures-directory"'
    )

    # RFC 9421 signature base: one line per component, no trailing newline.
    sig_base = "\n".join([
        f'"@authority";req: {authority}',
        f'"@signature-params": {params}',
    ])
    sig = key.sign(sig_base.encode())

    headers = {
        "Content-Type": CONTENT_TYPE,
        "Cache-Control": f"max-age={ttl}",
        "Signature-Input": f"sig1={params}",
        "Signature": f"sig1=:{base64.b64encode(sig).decode()}:",
    }
    return headers, body

Cloudflare ships a http-signature-directory CLI that validates a directory before you submit it. Use it; the failure modes here are silent.

Signing a request

# sign.py — attach Web Bot Auth headers to an outbound request
import base64, secrets, time
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
from cryptography.hazmat.primitives.serialization import Encoding, PublicFormat

COVERED = '("@authority" "signature-agent")'


def sign(key: Ed25519PrivateKey, authority: str, agent_url: str,
         label: str = "sig2", ttl: int = 60) -> dict:
    """
    authority : the Host you are calling, e.g. "api.example.com"
    agent_url : https URL of YOUR directory, e.g. "https://agent.example"
    ttl       : keep this short. A minute is plenty and limits replay.
    """
    pub = key.public_key().public_bytes(Encoding.Raw, PublicFormat.Raw)
    kid = jwk_thumbprint(pub)

    created = int(time.time())
    expires = created + ttl
    params = (
        f"{COVERED}"
        f";created={created};expires={expires}"
        f';keyid="{kid}";alg="ed25519"'
        f';nonce="{b64u(secrets.token_bytes(32))}"'
        ';tag="web-bot-auth"'
    )

    signature_agent = f'"{agent_url}"'          # structured string, quoted
    sig_base = "\n".join([
        f'"@authority": {authority}',
        f'"signature-agent": {signature_agent}',
        f'"@signature-params": {params}',
    ])
    sig = key.sign(sig_base.encode())

    return {
        "Signature-Agent": signature_agent,
        "Signature-Input": f"{label}={params}",
        "Signature": f"{label}=:{base64.b64encode(sig).decode()}:",
    }

Four things fail verification at Cloudflare and are worth internalising before you debug them the hard way. The Signature-Agent value must be an https:// URL in double quotes — it is a structured field, not a bare string and not the dictionary form (sig2="https://…") that appears in later drafts. signature-agent must appear in the covered component list, not just in the headers. expires must still be in the future when the request lands, which means your ttl has to absorb network latency and any clock skew. And several RFC 9421 features are simply not supported: @query-param, @status and the sf, bs, key, req and name parameters on header components will all fail.

Verifying at your own edge

If you are single-CDN you can read the verified-bot flag your provider sets and move on. Most people are not single-CDN and origin-side services frequently need the decision too. It is thirty lines.

# verify.py — origin-side or edge-side verification
import base64, time, httpx
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PublicKey
from cryptography.exceptions import InvalidSignature

_dir_cache: dict[str, tuple[float, dict[str, Ed25519PublicKey]]] = {}


async def load_directory(agent_url: str) -> dict[str, Ed25519PublicKey]:
    """Fetch + cache a JWKS directory. One fetch per rotation window."""
    now = time.time()
    hit = _dir_cache.get(agent_url)
    if hit and hit[0] > now:
        return hit[1]

    async with httpx.AsyncClient(timeout=2.0) as c:
        r = await c.get(f"{agent_url}/.well-known/http-message-signatures-directory",
                        headers={"Accept": CONTENT_TYPE})
    r.raise_for_status()

    keys = {}
    for jwk in r.json().get("keys", []):
        if jwk.get("kty") == "OKP" and jwk.get("crv") == "Ed25519":
            raw = base64.urlsafe_b64decode(jwk["x"] + "==")
            keys[jwk_thumbprint(raw)] = Ed25519PublicKey.from_public_bytes(raw)

    ttl = _max_age(r.headers.get("Cache-Control", "")) or 3600
    _dir_cache[agent_url] = (now + ttl, keys)
    return keys


async def verify(headers: dict, authority: str) -> str | None:
    """Returns the verified keyid, or None. Never raises on untrusted input."""
    try:
        agent = headers["Signature-Agent"].strip('"')
        if not agent.startswith("https://"):
            return None

        label, params = headers["Signature-Input"].split("=", 1)
        meta = _parse_params(params)                    # created/expires/keyid/tag
        if meta.get("tag") != "web-bot-auth":
            return None
        if int(meta["expires"]) < time.time():          # replay window closed
            return None

        raw_sig = base64.b64decode(
            headers["Signature"].split("=", 1)[1].strip(":")
        )
        sig_base = "\n".join([
            f'"@authority": {authority}',
            f'"signature-agent": "{agent}"',
            f'"@signature-params": {params}',
        ])

        keys = await load_directory(agent)
        pub = keys.get(meta["keyid"])
        if pub is None:
            return None

        pub.verify(raw_sig, sig_base.encode())
        return meta["keyid"]
    except (KeyError, ValueError, InvalidSignature, httpx.HTTPError):
        return None

Note what the function returns: an identifier, or nothing. It does not return allow or deny. That distinction is the whole design.

---

The rule that keeps you from breaking the web

Here is where most rollouts go wrong and it is worth being blunt about it.

Adoption is early. Google publishes keys for its AI-browsing agent, but Googlebot proper still does not sign. A large share of legitimate crawler traffic including traffic you want from search engines that send you customers will be unsigned for at least another year. If you deploy verify-or-block, you have not hardened your edge. You have written an outage with a compliance story attached to it.

The correct posture is dual-path. Signature verification runs first. Reverse DNS runs as a fallback for the legacy crawlers that never got a key. Everything else lands in a default tier that is throttled, not rejected. Verification assigns a tier. The tier chooses the budget. Nothing about the identity layer ever issues a verdict on its own.

Identity resolution is an ordered fallback that always terminates in a tier, never in a block.

---

Rate limiting on identity instead of IP

Once you have a keyid, the limiter finally has something worth counting against. Four changes follow and they are independent - ship them in any order.

Key the bucket on identity, not the network. IP rotation stops mattering the moment the bucket key is a thumbprint.

-- token_bucket.lua — atomic refill-and-consume, keyed on trust identity
-- KEYS[1] : "rl:{tier}:{keyid}"
-- ARGV    : capacity, refill_per_sec, now_ms, cost
local state    = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local capacity = tonumber(ARGV[1])
local rate     = tonumber(ARGV[2])
local now      = tonumber(ARGV[3])
local cost     = tonumber(ARGV[4])

local tokens = tonumber(state[1])
local ts     = tonumber(state[2])
if tokens == nil then tokens = capacity; ts = now end

-- refill by elapsed time, clamped to capacity
tokens = math.min(capacity, tokens + math.max(0, now - ts) / 1000.0 * rate)

local allowed = 0
if tokens >= cost then
  tokens  = tokens - cost
  allowed = 1
end

redis.call('HSET', KEYS[1], 'tokens', tokens, 'ts', now)
redis.call('PEXPIRE', KEYS[1], math.ceil(capacity / rate * 1000) + 1000)

local retry_after = 0
if allowed == 0 then retry_after = math.ceil((cost - tokens) / rate) end
return { allowed, math.floor(tokens), retry_after }

Charge cost, not calls. Pass a real cost into that script. For an LLM-backed route, debit estimated tokens and reconcile against actuals after the response. For a database-backed route, debit rows scanned. The unit does not matter; what matters is that the expensive request costs more than the cheap one.

Give every identity a retry budget. This is the control that would have caught the incident at the top of this article. Rate is per minute; a retry budget is per outcome and it notices a loop that a rate limit cannot.

# retry_budget.py — sheds a caller whose failure ratio says it is looping
RETRY_BUDGET = {"T0": 0.08, "T1": 0.03, "T2": 0.01}   # retries per request


async def admit(redis, tier: str, keyid: str, was_retry: bool) -> bool:
    win = f"rb:{tier}:{keyid}:{int(time.time()) // 60}"
    pipe = redis.pipeline()
    pipe.hincrby(win, "total", 1)
    if was_retry:
        pipe.hincrby(win, "retries", 1)
    pipe.expire(win, 180)
    res = await pipe.execute()

    total = res[0]
    if total < 50:                       # not enough signal yet
        return True
    retries = int(await redis.hget(win, "retries") or 0)
    return (retries / total) <= RETRY_BUDGET[tier]

Shed loudly. An agent cannot infer backpressure from a 200 containing an apology. Return a real status, a real Retry-After and the RateLimit fields so a well-built client can self-regulate instead of hammering.

HTTP/1.1 429 Too Many Requests
Retry-After: 42
RateLimit-Limit: 600
RateLimit-Remaining: 0
RateLimit-Reset: 42
Content-Type: application/problem+json

{"type":"https://example.com/probs/rate-limit",
 "title":"Budget exhausted for this identity",
 "detail":"Signed agents get 600 req/min. Unsigned traffic gets 20.",
 "trust_tier":"T2"}

That detail string is not decoration. It is the cheapest possible documentation of the incentive: sign your requests and this number goes up thirty-fold.

Illustrative starting values. Calibrate against your own p99 and your own cost-per-request before you enforce anything.

The operational payoff of tiering on identity rather than network is the thing that is easy to miss on a diagram: a verified operator that misbehaves loses its budget without losing its identity. You can email them. You are no longer guessing at a CIDR block and hoping you did not just blackhole a customer's office.

---

The part that is not on the vendor slide

Now the uncomfortable half, which you should know before you put this in an architecture decision record.

The IETF webbotauth working group was chartered in 2025 with the IESG approving it unanimously. It has met three times as a chartered group. As of August 2026 it has not adopted a single draft - the nine active Internet-Drafts in the datatracker are all still individual submissions. A direction poll at IETF 126 came back 22 in favour to 6 against, which is comfortable but is not an adoption call. The authors committed to refining key discovery, key-rotation semantics and the signature structure before seeking formal adoption, after Justin Richer, an editor of RFC 9421 itself flagged that signing a signature value is a known cryptographic risk.

Meanwhile Cloudflare, AWS, Akamai, Vercel and Shopify are all enforcing it in production.

You can hold both of those facts at once and you should, because the design consequences are concrete:

• Pin the draft revision you implemented against, in code and in your ADR. Cloudflare currently implements the Signature-Agent format from draft-…-directory-03 and explicitly rejects the dictionary form from later drafts. That is a versioning hazard, not a bug.

• Treat it as a vendor integration, not a standard. Write the verification path behind an interface. The wire format has a real chance of changing before the RFC number lands.

• Keep reverse DNS alive. It is not legacy cruft during a transition this long; it is the majority path.

• Put it on the risk register with an owner and a review date next to every other dependency you carry on a moving specification.

None of that is an argument against shipping. It is an argument for shipping it the way you would ship any pre-1.0 dependency that four of your vendors have already made load-bearing.

---

Proving it works before you enforce anything

Run the whole thing in shadow mode first. Resolve the tier, log it and apply the old policy. You want a week of data on what fraction of your traffic falls into each tier before a single budget is enforced because the number is never what you guessed.

# 1. Does your signing implementation produce something a real verifier accepts?
#    200 = verified · 401 = well-formed but unknown key · 400 = malformed
curl -sS -o /dev/null -w '%{http_code}\n' \
  -H "Signature-Agent: \"https://agent.example\"" \
  -H "Signature-Input: $SIG_INPUT" \
  -H "Signature: $SIG" \
  https://crawltest.com/cdn-cgi/web-bot-auth

// 2. retry-storm.js — the failure the per-user limiter cannot see.
//    Stay under the ceiling. Never stop. Assert the system notices anyway.
import http from 'k6/http';
import { check } from 'k6';

export const options = { vus: 1, duration: '20m', rps: 1 };

export default function () {
  const res = http.get(`https://api.example.com/items/${__ITER}`);
  check(res, {
    'sheds the loop before 10k requests':
      () => !(__ITER > 10000 && res.status === 200),
    'never returns 200 with an error body':
      (r) => !(r.status === 200 && r.json('error') !== null),
  });
}

Three assertions decide whether this worked and none of them is "signature verifies":

1. Cache hit rate by trust tier. If T2 hit rate is still near zero after a week, your cache policy did not change and neither did your origin load.

2. Origin requests per 1,000 edge requests, by tier. This is the number your CFO cares about whether or not they know it.

3. False-tier rate. How much traffic you know is legitimate landed in T2. If that number is not small, your fallback path is broken and you are about to throttle real users.

---

What actually goes wrong

The directory goes down and you fail closed. A two-second timeout on a JWKS fetch should not demote every request from a verified operator. Serve the last known good directory well past its max-age and refresh asynchronously; a stale key is a far smaller problem than a synchronous fetch on the request path.

Key rotation without overlap. Operators rotate. If you cache aggressively and they publish a new key without keeping the old one live, verification fails for exactly one cache TTL and looks like an attack. Accept every key in the directory, not just the first.

Clock skew eats your expires. A sixty-second window is generous until you have a node drifting. Monitor NTP on verification hosts. This one presents as an intermittent, unreproducible verification failure and burns a full afternoon.

Corporate proxies land in T2. Enterprise customers behind an egress proxy can look a lot like unattributed automation. This is why the default tier throttles rather than blocks and why you watch the false-tier rate before enforcing.

Someone treats a signature as authorisation. It is not. A signature proves the operator is who it claims to be. It says nothing about whether that operator should be reading the endpoint it just requested. Authorisation is still yours to write and conflating them is the security bug that will eventually come out of this pattern.

Transitive trust confusion. An agent reaching you is often not operated by the company that built it; a platform runs automations for many end users. Cloudflare is experimenting with RFC 7239's Forwarded header to carry operator identity through that chain. It is explicitly experimental. Do not build billing on it yet.

---

Where to start

• Week one - ship the verification path in shadow mode and log a trust_tier on every request. Do not change a single limit.

• Week two - look at the distribution and at your cache hit rate broken out by tier and this is where the actual argument for the work gets made in numbers from your own traffic rather than someone else's blog post.

• Week three - tier your cache policy because it is the change with the largest cost impact and the smallest blast radius.

• Week four - move the limiter's key from IP to identity and turn on the retry budget. Enforce differentiated rate budgets last, after the false-tier rate is small enough that you would bet a customer relationship on it.

---

The uncomfortable thing about this whole class of problem is that none of the individual components were badly built. The rate limiter was correct. The cache was correct. The retry logic in the agent was from its own perspective is correct. Each one held an assumption that was true when it was written and quietly stopped being true and the failure only became visible where they compose.

Agent traffic is not a bigger version of human traffic. It is a different population with different statistics, arriving through infrastructure that was fitted to the old one. You cannot tune your way out of that with a smaller number in a config file. You have to give the system a way to know who is calling and then decide, deliberately, what each kind of caller is allowed to cost you.

---

Further reading

• RFC 9421 - HTTP Message Signatures. The underlying standard. Stable since February 2024.

• Cloudflare: Web Bot Auth integration guide. The most complete operator documentation currently in existence, including the unsupported-parameter list.

• draft-meunier-web-bot-auth-architecture. The protocol draft. Check the revision before you implement.

• RFC 8037 - JWK thumbprint for Ed25519. How keyid is computed.

• Cloudflare: using cryptography to verify bot and agent traffic. The design rationale from the people who shipped it first.

• Reference implementations: web-bot-auth (Rust), web-bot-auth (npm), http-signature-directory.