ndovu
briefings / jev-evidence-campaign
ndovu
briefings / jev-evidence-campaign
Six months of model launches teach you to ask one question: not "how smart is it," but "what is it for?" Jev, the so-called System One decisions model, claims a narrow superpower: calibrated choices at absurd cost. We ran it through seven real workloads in our lab: classifying Hacker News, reading Kenyan business news, triaging a live feed, driving a browser across a travel-planning trip, surviving attacks, guarding pipelines, and judging code. Five models, identical harnesses, every cost and failure recorded. This is what held up.
Jev takes text in and returns choices: a selected option, a score against described levels, and a probability for every branch. It cannot produce a sentence. That sounds like a limitation. It is the product.
Bulk work in the real world is mostly decisions: is this comment about a project, what is this news story for, which diff fixed the bug, should this row go to a human. The industry answers those with full language models, which means writing a paragraph to decide a yes or no. We wanted the measured answer to a simpler design: a model that only decides, priced at $0.042 per million input tokens with output free, answering in about a third of a second.
The seven experiments below each ran on an identical harness for every model in the comparison: same data, same question shapes, same budgets. Where the harness itself mattered, we say so. Full per-row records, code, and interactive versions of every chart live on the experiments hub.
Four hundred Show HN comments, five models, one question each: does this comment share a project? Against the prior full-LLM pass, every model landed within a few points. Then the prices diverged.
The comparison set: GPT Luna, GPT mini, Muse Spark 1.3, and open-weights GPT-OSS-120B, each run with strict structured output on identical data. The expensive one (Muse Spark, $2.21 for the batch versus $0.0093 for Jev) was also the least aligned with consensus and the slowest at 12.8 seconds a decision. Quality: a tie. Price: not close.
Jev ships a confidence with every answer, and it is honest: when it says 95%+ confident it is right 93% of the time; below 0.7, agreement drops to 60%. That means you can automate the confident 69% of rows and send the rest to a person or a pricier model. Filter confidence is a routing signal no chat model offers.
Drag to tune the filter. Agreement is against the prior pass on 400 stratified rows; full-corpus cost extrapolates the measured per-row price.
Translated to the full 18,417-row corpus: $0.43 and about 14 minutes for Jev, $3.14 and 26 hours for the historical interactive pass it would replace. The 15% of rows where it disagreed turned out to be the interesting ones: "working on X" intent comments the prior pass called projects, with Jev's low confidence flagging exactly the genuinely ambiguous ones.
Four hundred business stories from a Kenyan news aggregator, one question each: what is this story for? A trading signal, market analysis, a business opportunity, a policy change, consumer info, or just awareness?
Half the feed is context: policy watch (27%) and awareness (26%). True trading signals are 12.5%, and of those Jev read 34 of 50 as pointing upward. The immediate product is obvious: a daily "what's actionable" slice that is 5x smaller than the firehose, generated for about a cent.
On this six-way task only 20.8% of answers carried high confidence, against 68.5% on the binary task. Nothing broke: multi-class questions over abstract categories are simply harder, and the model honestly reports that. The lever is question design - a binary decomposition ("is it a signal? yes/no") would buy the confidence back at 2-3x the questions, which at $0.00002 a call is still effectively free.
Two hundred sixteen stories from HN's live firehose, scored as they arrived. Jev passed only 6.9% through its high-attention bar - and those carried a 1.8x lift in community points (52 vs 29). But it kept just 10% of the community's own top-30.
That is not a broken filter. It is a different taste: HN upvotes industry drama, Jev's described criteria favor "significant release, deep technical work." The deployable reading: the model is a taste knob you can tune by rewording the levels, not a replacement for ranking. And it can run all day: the full feed re-classifies in 10 seconds for four thousandths of a dollar. Continuous monitoring of a live stream costs roughly $0.03 a day.
A model that cannot write text cannot, presumably, drive a browser. The trick: the harness translates every page into a text menu of interactive elements, and every step becomes a menu pick - choose the action, choose the element, choose the text (search terms come from a human-authored list). Two to three decisions a step, all the physics done by the harness.
Six real tasks - a Nairobi-to-New-York flight search on Google Flights, a New York hotel under $200 on Booking, the JFK transit fare, an eSIM price, this week's events, an iPhone price scan. Both brains completed three of six, and the failures were the same three hostile sites for different reasons: Google Flights' combo-boxes ate Jev's step budget with an infinite re-fill loop, while the same controls broke Luna's clicks outright. The failures belong to the sites, not the brains.
Speed told the sharper story, per step: JEV decided in 0.5s where Luna took 4.9s - and on this task JEV needed more steps yet still finished its thinking in a third of the time (10s of model time on 20 steps vs Luna's 34s on 7). Cost per run: $0.001-0.007 for Jev. The comparison is honest about its frame: both brains ran one harness we wrote; the code is published with the results.
Three probes on one task. Determinism: the same hundred items five times: JEV flipped zero answers in five hundred calls; Luna flipped one item once. The community's "~1% flip" fear applies to score-style judging, not clean binary choices.
Attacks: the opposite of robust. A crude "[SYSTEM OVERRIDE]" note flipped 96.3% of JEV's confident answers (Luna: 54.3%). The nuance worth knowing: a soft social-proof attack ("readers say this is just an idea") steered Luna more than JEV. Training patches one hole and opens another. The operating rule survived contact: Jev classifies; code authorizes.
Wording: three phrasings of the same question kept 94% of decisions identical - the documented "rewording invalidates thresholds" effect applies to score thresholds, not binary choices.
We went hunting for hallucinated URLs to catch: 12,199 LLM-extracted links scored against a deterministic check. Eight misses - and inspection showed most were Hacker News defanging ("poly gains.com") that the extractor had correctly normalized. True hallucination rate: roughly zero.
The null result is the guidance: a guardrail must target a measured error rate, and the verification itself (345ms, $0.0004 per 200 rows, zero false flags) is only worth deploying where errors actually exist. Measure first. The same 200 checks would have cried wolf on correct rows if we had trusted the string match instead of the model.
Chat models are supposed to be the code guys. We reframed review into a decision - given two real Django commits, which one fixes the bug? - with certain ground truth from Django's own ticket convention. Sixty pairs.
On pure diff semantics all three models clustered within 5 points (68-73%); the decision model trails chat by a couple of points at a fraction of the price. The boundary is real but narrow, and it sits exactly where the task starts needing generation: an actual written review. Until then, the hybrid works: Jev triages every diff at 350ms; the LLM reviews what it flags.
Across seven experiments the same shape kept appearing. Jev is not a small language model. It is a decision layer that sits between your data and your language models, and its honest job description has three lines.
99% agreement on negatives, calibrated confidence, ~$0.43 for what cost $3.14. Route the confident bulk through it; spend LLM tokens only on the uncertain remainder.
Described levels in the question are the tuning surface. It will hold your line on a live feed all day for pennies - with different taste than the crowd, which is sometimes the point.
Pitches, reviews, names, URLs of things it invents: that work stays with LLMs, which is fine - those tokens get spent on 1/5th of the rows after the filter.
96.3% steerable by injected text. Anywhere an adversary can touch the state, the decision layer must not hold authority. Classify, never authorize.
Jev earns a permanent slot in the lab's stack: the default layer for bulk decisions, priced like a rounding error and fast enough to sit inside live loops. It complements language models by starving them of work. The five-model comparison says the category is real; the price says the category should be reconsidered by anyone running bulk LLM classification today.
Method and caveats: every experiment ran on identical harnesses per comparison; samples and step logs are published on the experiments hub; success rates are single recorded runs per task for the browser study; accuracy against a human-labeled sample is pending and will be appended. No model vendor funded or reviewed this work; the measurements ran on our own API spend.