My AI agents tested the new AI class on a real actuarial workflow. Here's what they found
Jev matched a thinking model on a real data-interpretation task at 1% error, in seconds instead of minutes, but only after the questions were redesigned; the questions and the code did most of the work.
Yesterday I wrote that TypeSafe’s Jev, a model that makes typed decisions instead of writing prose, could change how enterprise AI gets built. Several asked the right question: does it work? My AI agents and Aria, my virtual actuary, spent a few hours yesterday checking.
The test
I picked two jobs Aria does every day where a model has to judge rather than write. These are standard testing cycles I used for building Aria, extended to the new AI class.
Reading a claims file. A use upload a spreadsheet. Something has to decide what each column means and answer three questions about the file: running totals or movements, all claims or only closed ones, claims-made or occurrence. Get these wrong and the reserve is built on the wrong numbers. 19 files, 227 questions, 14 files written to be awkward on purpose.
Checking a draft for errors. When Aria drafts an answer to an actuarial question, a second model checks it against authored definitions for contradictions, arithmetic slips, reversed logic and dropped qualifications. 82 real drafts plus 163 copies with 1 planted error each.
Pass marks were set before anything ran, half the cases were held back for scoring. Jev ran against Aria’s current path, against Aria’s own model asked the same narrow questions, and against 7 regular LLMs with thinking switched off: Claude Haiku 4.5, GPT-4.1, Mistral Large, Kimi K2, Qwen3, DeepSeek 4.1 and Gemini 3.8 Flash. (thinking has to be switched off for speed in workflow - thinking models take way too long for realtime editing - I use them only at specific request moments or out of cycle). Outside spend: about £3/$4 on the LLMs, $0.09 on Jev.
What worked
On reading files, Jev with well-designed questions got 1% of consequential answers wrong, in 2.9 seconds per file. Aria’s thinking model, asked identical questions, also got 1% wrong. It took 164 seconds. Jev’s probabilities were the best calibrated of any candidate: above 0.8, it was right 98% of the time.
On checking drafts, Jev missed 6.5% of planted errors in under a second, catching every arithmetic slip and reversed implication and 89% of contradictions, and wrongly flagged 7% of correct drafts. Aria’s current checker missed none, in 45 seconds. It also rewrites 40% of drafts that could have passed the same baseline test, in order to make them ‘more correct’. I did not know that until this test, because current LLMs via chat tend to keep telling you to improve things with no %age indicator of how important signal is (you must have experienced this!).
What did not
My first attempt was mediocre: one 22-way question per column, and file-level questions with the rule written into the instruction (“choose complete only if the file shows both open and closed claims”). Jev ignored the rule and was confidently wrong on the awkward files. A whole-draft “does this contradict the definitions?” question gave a weak signal.
3 changes fixed it, each moving work from the model into code. Ask what kind of column this is, then the branch-specific questions in the same call, and let code compose the answer, saying “a person must decide” where the vocabulary can’t. Compute what code can compute; for arithmetic in prose, let code generate candidate equations and Jev pick which one the sentence asserts. And show the model the definition of every term the draft mentions, not only those in the question.
The non-thinking language models were the surprise. On files they were no better than Jev and much more expensive. On drafts they were blind: GPT-4.1 rated faulty and correct drafts identically. Cheap judgement means Jev or a thinking model, not a fast chatbot. This is a major cost learning for all similar expert assistance tasks, I think.
Cost and effectiveness
| candidate | file reading: error, time, $/1,000 | draft checking: missed, time, $/1,000 |
|---|---|---|
| Aria today | 2%, 20 s, ~$1 | 0% missed, 40% false flags, 45 s, ~$1 |
| Jev, best design | 1%, 2.9 s, $0.65 | 6.5%, 0.9 s, $0.23 |
| DeepSeek v4 flash, thinking, decomposed questions | 1%, 164 s, ~$4 | 3.9%, 59 s, ~$1.60 |
| GPT-4.1 | 3%, 3 s, $8.60 | 88%, 1.3 s, $2.40 |
| Claude Haiku 4.5, no thinking | 7%, 3 s, $4.20 | shrugs on 92% |
| Gemini 3.8 Flash, minimal | 2%, 16 s, ~$9.50 | 21%, 2.3 s, ~$1.20 |
Costs are dollars per thousand items at list prices, so everything here is cheap in absolute terms; the differences are relative. Aria today is cheap because its model, DeepSeek v4 flash, charges about a fiftieth of GPT-4.1 or Gemini per token, and a file costs it about 7,000 tokens. Draft-checking figures exclude one planted edit, the deletion of a qualifying phrase, whose materiality is disputed and which every candidate missed.
Be careful with the money. Against DeepSeek’s cheap thinking model, the one Aria already uses, Jev costs about the same per decision, matches it on reading files and trails it slightly on checking drafts. Against Western APIs it is five to fifteen times cheaper and more accurate. The reliable difference is time: seconds against minutes for the same answers, which is what lets Aria check every column in realtime while the actuary watches rather than “come back later”.
How to use it
- Design the decomposition first, then pick the engine. The questions and the code produced the gain; the vendor produced the speed.
- One snap judgment per question, every option defined, an explicit “unresolved” in every set.
- Never put a rule in the prompt. Ask for the observations and apply the rule in code.
- Compute what code can compute. Enumerate candidates and let the model select; never let it calculate.
- Retrieve the definitions the content mentions, not the ones the question mentions.
- Calibrate on your own labels before you automate, then use three bands: act, act with a review flag, escalate.
- Keep a deterministic fallback and a human veto. Jev output is never authority.
Caveats
One vendor, two tasks, a few hundred cases, non-confidential data, intervals of a few points. TypeSafe’s privacy terms do not yet cover client data, and I would not rebuild a regulated process around one lab’s model. But the primitive works, and the discipline it forces, decisions as small typed questions composed in code, made Aria better even where I keep the thinking model.
I work in insurance and write about artificial intelligence, risk and community — Jewish and British. This site collects the writing. More about me.