What Made Our Paid LLM Reports Fail, and How We Fixed It

By Govind Sharma

The Problem

After the pose pass, FastBowlingLab turns the measurements and annotated key frames into a coaching report and a 4-week training plan. Both are large typed objects rendered on web, mobile and PDF, and the athlete has paid for them. A schema failure meant "the model did not produce a valid object" and a wasted paid generation. Three separate problems caused those failures, and each one looked like model quality until it was measured.

1. The Schema Was Too Big to Compile

The combined report-and-plan schema compiled to a decoding grammar that Anthropic's API rejected as too large, identically on every retry. The tempting workaround was to pass the schema as an ordinary tool, which removes the grammar limit. It also removes the guarantee: the model returned double-encoded JSON, the report nested inside the plan, or the plan alone, and roughly 3 in 4 paid generations failed.

The fix was to split the work into two calls, each well inside the limit. That turned out to be better design anyway: the plan is prompted with the finished report, so it targets the gaps actually found, and the visual "is this really a bowling action" verdict runs between the calls, so a rejected analysis never pays for a plan. The rule I wrote down: if a schema hits the grammar limit, split it. A loud, uniform 400 is better than intermittent shape errors that look like the model being flaky.

2. The SDK Silently Dropped Schema Bounds

On Gemini, the provider SDK's schema converter keeps a fixed list of keywords and drops minimum, maximum, minItems, maxItems and maxLength: 11 bounds on the report schema and 9 on the plan. The model happily satisfied the schema it was actually sent, and our validator rejected the result. Unbounded integers also caused decoder runaway: a sets field whose answer was 3 came back as a twenty-digit number.

The first fix restated the bounds as prose in the field descriptions, and it was reverted the same day. With "week must be at least 1", the model wrote one week. The 12 plans before the change all had 4 weeks and 24 to 28 sessions; the two after it had 1 week and 1 session, and both were delivered and charged. A bound is enforcement; prose is a suggestion.

The real fix is at the wire: a custom fetch on the provider puts the bounds back into the request schema before it leaves the process, and the model honours them. Ask for exactly 4 items and you get 4.

// The SDK strips numeric and array bounds when it converts the schema.
// Put them back on the request body, where the model can enforce them.
export function withSchemaBounds(bounds: BoundsByPath): typeof fetch {
  return async (input, init) => {
    if (init?.body && typeof init.body === "string") {
      const body = JSON.parse(init.body);
      const schema = body.generationConfig?.responseSchema;
      if (schema) restoreBounds(schema, bounds); // minimum, maximum, minItems, maxItems
      init = { ...init, body: JSON.stringify(body) };
    }
    return fetch(input, init);
  };
}

3. Repair, But Never Invent

Bounds on the wire made failures rare, not impossible, so there is a repair step before giving up. It is deliberately narrow:

• Allowed: clamp numbers into range, truncate over-long arrays and strings, drop an optional field that is out of range or runaway.
• Never: pad an array that came back short. A report padded with an invented row is worse than an honest retry.
• The repaired object is validated again, and each repair is logged as a path and an action, never the values.

Errors carry the finish reason, output tokens and schema issues as path-and-code pairs, which separates "truncated at the cap" from "wrong shape under the cap" at a glance. That separation found the last bug: findings and plan had shared a 16k output cap, and measured peaks were about 7k and 26k, so plans were being truncated. The caps are now 16k and 32k.

Designing the Output for the Renderer and the Budget

• One week of seven days repeated for 4 weeks, with a required progression line per item ('weeks 1-2 bodyweight, weeks 3-4 add 5 kg').
• No prose fields that nobody displays; descriptions and how-to text were most of the 26k-token output.
• An explicit unit (reps, balls, seconds, metres) next to every amount, because 'sets: 48' had been used to mean eight overs.
• Plans become routines in the training logger without another model call. Unknown drill names are matched by name, then a trigram shortlist, then a small judge model whose 'same exercise' answer counts only if the movement, equipment and variation it wrote match the candidate.

Guardrails on What the Model May Say

The model sees up to 12 annotated key frames, a compact per-frame signals file and each measurement at the exact number its report row shows, with one rounding rule shared by the overlay, the report, the PDF and the prompt. The system prompt forbids legality rulings (the elbow proxy is not the 15-degree legality test), medical diagnosis, 3D or hip-shoulder separation claims, and inferring any measurement the footage didn't produce. Text the athlete typed reaches the prompt only through a fence that strips control characters and marker-like sequences and caps the length.