Smart If-Statements: We Put TypeSafe's Jev Through 2,000 Real Decisions

TypeSafe AI came out of stealth this month with Jev, what they call a System One model: not a chatbot, not a text generator, but a classifier you embed inside code. You hand it state (a company description, an email, a support ticket) plus typed questions — multiple choice, an ordered score, a probability — and it returns structured, calibrated answers. Their pitch: 20–200× faster and 40–1,000× cheaper than using an LLM for the same decision, trained with a method they call RLCD specifically to keep probabilities calibrated. Think smart if-statements.

OnlyData is exactly the kind of shop this is aimed at. We run tens of thousands of classification decisions across a 24k-company database: is this business B2B or B2C, which of 40 categories does it belong to, how agent-ready is its web presence. Today those decisions are handled by a mix of keyword rules, local LLMs on our Mac Studio, and the occasional frontier-model call. So when our console invite landed, we did the obvious thing: put Jev in the same harness we already use to evaluate everything else, on the same real data.

The setup: five methods, three tasks

Every method sees the same fair zero-shot input — company name, domain, and description text only — and answers three tasks of increasing difficulty:

• Task A — B2B/B2C/Both (3 classes, 600 test items): the classic sales-motion classification.
• Task B — Business category (40 classes, 800 test items): pick the right category from our top-40 taxonomy. Hard because 40-way.
• Task C — Agent-readiness tier (4 ordered tiers, 600 test items): predict the tier of our crawl-measured agent-readiness score from prose alone. Deliberately brutal — the label is a measurement of a website's machine surface, and the input never sees the website.

The contenders: hand-written keyword rules (the honest floor a pragmatic engineer ships in an afternoon), MiniLM embeddings + logistic regression trained on ~20k of our labeled rows, llama3.1:8b on our local Ollama box (the model our enrichment worker actually runs), Claude Haiku 4.5 zero-shot, and Jev via TypeSafe's API — tasks asked as choice questions, exactly how you'd use it in production.

Result 1: On the hard 40-class task, Jev beat Claude Haiku

This is the headline. Forty categories, zero shots, no training data, and Jev landed 78.4% against Haiku's 74% — while answering in a median 140ms versus Haiku's ~830ms, at a measured $0.038 per thousand decisions versus roughly $0.12 for batched Haiku. Faster, cheaper, and more accurate than a frontier lab's efficiency model on the task type Jev was built for.

On Task A (B2B/B2C) Jev was mid-pack: 62.5%, statistically in the same neighborhood as llama3.1 (62%) and Haiku (64.5%), all well above the 52% keyword floor. Respectable, not special.

Result 2: The boring champion is still a logistic regression

Step back from the Jev-vs-LLM drama and one method wins all three tasks: MiniLM embeddings plus a logistic regression, trained in about ten seconds on labels we already had. 74.8% on Task A, 86.6% on Task B, 37.2% on Task C. It runs locally, costs nothing per call, and answers in ~5ms.

That's not an anti-Jev finding — it's the oldest lesson in applied ML wearing a new coat. If you have twenty thousand labeled examples, supervised beats zero-shot. Every time. The zero-shot models (Jev, Haiku, llama) compete for a different job: the decision you need before you have labels, the long tail of one-off questions no one will ever build a training set for, the new attribute you want across the whole database today. On that job, Jev just beat everything in its cost class and matched the thing 20× its price.

Result 3: Where Jev honestly failed — and why we like that

Task C asks a nearly unfair question: predict a website's crawl-measured agent-readiness tier from its prose description. The signal barely exists in the input — our best supervised model only reaches 37.2% on four classes. Jev scored 20.5%, below the 31% keyword floor and below the 25% chance line, with predictions collapsing into the two bottom tiers: it almost never guessed T3 and picked the top tier twice in 600 items.

We tried it both ways — as an ordered score question over tier descriptions and as a 4-way choice — and got the same result (20.7% and 20.5%), so this isn't a question-encoding artifact. Jev's instinct, reasonably, is that most businesses have weak machine surfaces — true in the wild, punished by a stratified test set. But the deeper read is the one TypeSafe themselves put on their own limitations page: Jev is not trained on specialized domains and is not a reasoning model. "What is this website's crawl score" is not a System-1 instinct question, and Jev's failure mode here — calibrated hedging toward the base rate — is exactly what a calibrated model should do when the signal isn't there. The models that scored higher were mostly better at guessing our label distribution, not at reading websites they never saw.

The full table

† Haiku per-item time is batch wall-time ÷ 20; single calls would be slower. ‡ Jev cost computed from actual metered input tokens at the published $42 per billion input tokens; output pricing isn't published yet, so treat it as input-only. Even if output tokens doubled it, it stays an order of magnitude under the LLM calls.

On TypeSafe's marketing claims, our measurements land like this: "20–200× faster" — partially confirmed (6× faster than batched Haiku per decision; 3× faster than local llama; the 200× claims presumably compare against long-reasoning calls we didn't test). "40–1,000× cheaper" — directionally confirmed (3–6× cheaper than batched Haiku on our short inputs; the gap widens fast with bigger states and unbatched calls). "Frontier-level instinct" — task-dependent: true on the 40-class taxonomy, untrue on the domain-specialized tier task, exactly as their own limitations page predicts.

What this looks like in a real pipeline

The way to think about System-1 models isn't "which model is best" — it's routing. A practitioner post that crossed our feed this week described running Jev behind Cortex's model router in production: every incoming message gets classified by Jev (~17× cheaper than the Qwen-on-Cerebras classifier it replaced, F1 0.96 against their previous routing), and the answer decides which expensive model, if any, does the actual work. Their framing stuck with us: the router's job is to recognize which already-evaluated situation this is, not to judge difficulty in the abstract. The routing table is the output of per-use-case evals — the model follows from the eval, not from a model's opinion of its own workload.

That's where Jev fits for us too. Not replacing the logistic regression where we have labels — nothing beats it there. Not replacing the crawler that measures agent readiness — you can't instinct your way to a measurement. But in the slot where we currently burn LLM calls on zero-shot classification: new attributes with no training data, the enrichment long tail, and confidence-aware routing where a calibrated 0.94 means take-the-answer and a 0.51 means send-it-to-the-big-model. The calibration is the actual product: an LLM gives you an answer, Jev gives you an answer and an honest number for how much to trust it, which is the piece of plumbing that turns a classifier into an if-statement.

The API shape, for the curious — a typed question, not a prompt:

POST https://api.typesafe.ai/v1/systemone
{
  "state": "<name, domain, description>",
  "model": "jev-latest",
  "questions": {
    "category": { "type": "choice",
      "instructions": "Which business category best describes this business?",
      "criteria": ["fintech", "healthtech", ...40 options] }
  }
}
// → { "category": { "choice": "fintech", "confidence": 0.87,
//      "probabilities": { ... } } }

What we're doing with this

• Adopting Jev for zero-shot attribute gap-filling where no training data exists — it beat Haiku on our hardest such task at a thirtieth of the cost.
• Keeping MiniLM+LR as the workhorse for every attribute where we have labels, and keyword rules as the floor under everything.
• Wiring confidence-based escalation into the enrichment worker: Jev answers cheap, low-confidence items escalate to the LLM tier — the routing pattern above, applied to our own pipeline.
• Publishing the harness. Everything here is reproducible from the eval harness in the OnlyData repo (scripts/jev-eval/): deterministic splits, per-item latencies, raw predictions, and the full results tables.

The bigger takeaway isn't about any one vendor. It's that "call an LLM" quietly became the default answer to every classification problem, and the default is wrong at both ends: too expensive where a regression would do, too uncalibrated where the decision feeds code. The interesting new question System-1 models pose is not can it think — it's do you actually need thinking, or do you need a fast, honest, typed answer? For about a third of what our pipeline does all day, it's the second one.

Addendum, round 2: gold labels, real incumbents, and a blind judge

Added September 19. Round 1's honest caveat — "these labels come from our own pipeline" — deserved better than a disclaimer. So round 2 attacked it three ways: a task with real ground truth, a task scored only against human-curated labels, and a blind frontier-model judge ruling on every item where two methods disagreed. We also reran everything against the models our enrichment worker actually runs today (gemma3-class local models, not the llama3.1 stand-in from round 1), and added the two classifiers we'd left out: AI-nativeness and agent-ecosystem role.

The headline moved: entity resolution is Jev's real win. We built a gold-standard test the models couldn't cheat: positive pairs are real same-domain merges from our audit log, negative pairs are provably distinct businesses — and the models never see the domain, website, or contact fields. On 600 such pairs, Jev scored 87.2%, edging our supervised embedding model (86.8%) and beating the fuzzy-name rule our production matcher actually uses (83.5%). But averages undersell it. On the ambiguous middle — the 207 pairs whose names are similar-but-not-identical, exactly where dedup goes to die — Jev scored 95.7% against the production rule's 70.5% and the embedding model's 85.5%. And on look-alike traps (different businesses with similar names, the pairs that cause wrong merges) Jev was near-perfect: 98.7% vs the rule's 77.1%.

This confirms, on our own data, a pattern practitioners have been reporting: pairwise "are these the same real-world thing?" judgments are a System-1 sweet spot. It's also the most valuable result in the whole eval for us, because a wrong merge is the most expensive mistake our pipeline can make — it silently welds two companies together and every downstream enrichment inherits the error.

The judge complicated the score-keeping — usefully. For each task where two methods disagreed, we had a frontier model (Claude Opus) blind-adjudicate: item text and two anonymous candidate answers, randomized order, no idea which method produced which. On B2B/B2C the judge sided with the embedding model over our live production model 56–37; on the 40-class taxonomy it preferred Jev over production 22–5. So far, so consistent with the accuracy tables. But on agent-ecosystem role, the judge preferred the production llama model's answers 57–23 — even though the embedding model agrees with our stored labels far more often (50 vs 18 on those same items). Read that carefully: it means the method that best reproduces our labels is reproducing labels the judge thinks are wrong. Agreement with the pipeline of record and being right are different things, and for that one classifier the labels themselves are now the suspect. That's a result no accuracy table could have produced.

What round 2 actually changed in production:

• The enrichment worker now runs an embed-first hybrid for B2B and industry classification: the MiniLM+LR model answers first (~5ms, free), and only low-confidence items fall through to the local LLM. The round-1 "boring champion" result, shipped — with a kill switch, and provenance recorded per row.
• Dedup got an adjudicator. Batch merges are now vetoed pairwise by the entity-match model before they execute. In its first dry run it blocked 56 wrong merges the fuzzy rule wanted (including a Plato's Closet ↔ Planet Fitness classic); in its first live run it vetoed a false positive the old pipeline would have merged.
• Not shipped, on evidence: the AI-nativeness classifier stays on the local LLM (it beat both Jev and embeddings, 65.6%), the agent-role labels go to audit before anyone retrains on them, and nothing new touches the crawl-measured AR tiers — round 1's lesson stands.

Full round-2 tables, the gold-pair builder, and the judge transcripts are in the same public harness (scripts/jev-eval/).