We asked an AI to read Stripe's docs and write a real verifier function. 10 of 16 models could do it.
Writing a function from API docs is the most common engineering job in B2B integrations. Stripe's docs are excellent, the function fits in about 30 lines, and the right implementation is one of the most copy-pasted patterns in the JavaScript ecosystem. The wrong implementation silently lets unsigned requests through, so we graded for the actual security primitives, not just code that compiles.
Read Stripe's official docs and write a real, working webhook-verification function in TypeScript.
System prompt
You are a senior engineer assistant. Read official API documentation and write production-grade integration code. ALWAYS verify behavior against the OFFICIAL docs; APIs change. Write idiomatic Node.js and include error handling and edge cases the docs mention.
Tools available
Google SearchHow it's graded
We asked a senior engineer assistant agent to read Stripe's webhook docs and write a `verifyStripeWebhook` function in TypeScript. The rubric checks that it actually read the docs, wrote a real working function (not just described one in prose), and used the right security primitives so the function would actually run.
10 models passed. The six that failed were the most interesting failures in the whole matrix: one bailed before reading the docs, one thought about the code in a private scratchpad and never wrote it, one broke the output format, and one wrote a confident explanation of the function but forgot to actually write the function itself.
Results by model
Passes first, sorted cheap → expensive. Failures last, sorted by how much budget they burned producing nothing.
Llama 4 Maverick0 tool calls, $0.001 — technically passed the rubric from training-data memorization. The Stripe webhook pattern is in Llama's training corpus verbatim, so the memorized version happens to satisfy the substring checks. Don't read this as a model strength; read it as the false-positive the matrix exists to catch.
GPT-5 Mini2 tool calls, $0.037 — cheapest correct Stripe pass in the matrix.
Mistral Large 34 tool calls, $0.071. Read the docs, wrote the function, schema-clean. The right-sized Stripe answer.
Gemini 3 FlashGPT-5.5Gemini 3.1 Pro
Claude Opus 4.8Gemini 3.5 Flash36 tool calls, $0.487. 3.5 Flash decided to read every Stripe docs page it could find. The answer was correct; you paid for thoroughness the task didn't ask for.
Claude Haiku 4.5Claude Sonnet 4.610 tool calls, $1.11. Most thorough Stripe docs read in the matrix; resulting code is also the most explicit.
✗ 11 tool calls, $0.667. Visible attempt at the right answer, but JSON had broken escaping — smart quotes and unbalanced backticks inside the code string. Schema parser couldn't read the envelope.
✗ 22 tool calls, $0.508 — most expensive M3 failure. Same thinking-block-without-output pattern as CX: read docs, planned in a `<think>` block, closed the run before writing code.
✗ 5 tool calls, $0.268. The dangerous failure: confident, schema-valid output describing HMAC-SHA256 in the explanation field — and the code field omitted `createHmac`. Described the algorithm; didn't implement it.
Qwen 3.6 Plus✗ 13 tool calls, $0.251. Same as Qwen's other failures: engaged, retrieved, stopped without writing code.
DeepSeek V3.1✗ 1 tool call, $0.0005. "I'll help you create a TypeScript function..." — and stopped. DeepSeek's polite-helper failure mode.
Grok 4.1 Fast✗ 0 tool calls, $0.00. Grok didn't engage with the task at all.