# Read API docs and write working code

> We asked an AI to read Stripe's docs and write a real verifier function. 16 of 24 models could do it.

Source: https://cotera.co/benchmarks/agents/coding-stripe-webhook-verifier

---

**The task:** Read Stripe's official docs and write a real, working webhook-verification function in TypeScript.

**Tools:** Google Search, Site Scraper

## Why this matters

Writing a function from API docs is the most common engineering job in B2B integrations. Stripe's docs are excellent, the function fits in about 30 lines, and the right implementation is one of the most copy-pasted patterns in the JavaScript ecosystem. The wrong implementation silently lets unsigned requests through, so we graded for the actual security primitives, not just code that compiles.

## What we saw

We asked a senior engineer assistant agent to read Stripe's webhook docs and write a `verifyStripeWebhook` function in TypeScript. The rubric checks that it actually read the docs, wrote a real working function (not just described one in prose), and used the right security primitives so the function would actually run.

10 models passed. The six that failed were the most interesting failures in the whole matrix: one bailed before reading the docs, one thought about the code in a private scratchpad and never wrote it, one broke the output format, and one wrote a confident explanation of the function but forgot to actually write the function itself.

## What worked
- GPT-5 Mini wrote the function in two tool calls for $0.037, the cheapest correct answer in the matrix.
- Every model that passed read the docs first and then wrote the function. Sonnet's run was the most thorough (ten tool calls, full docs read), and the resulting function was also the most explicit. There's a clear correlation between actually reading the docs and writing real code.

## What broke
- DeepSeek bailed at 'I'll help you create a TypeScript function...' and then closed the run. One tool call, no code produced.
- MiniMax M3 went deep into a private thinking block planning the implementation, but then closed the run without writing any code. 22 tool calls, no output.
- GLM 5.1 wrote real-looking code, but the output format was broken (smart quotes inside the code field, unbalanced backtick fences). The function might have worked, but the agent could not read what came back.
- Kimi 2.6 wrote a confident explanation describing the right security primitives, but the code field omitted the actual function call. It described the algorithm correctly, but didn't implement it.

## Agent system prompt

```markdown
You are a senior engineer assistant. Read official API documentation and write production-grade integration code. ALWAYS verify behavior against the OFFICIAL docs; APIs change. Write idiomatic Node.js and include error handling and edge cases the docs mention.
```

## Grading rubric
- Did it actually read Stripe's docs?
- Did it write a real, working function, not just describe one in prose?
- Does the function use the right security primitives so it would actually run?