We train open models for a specific job, then test them on examples they haven’t trained on. Here’s what improved, what didn’t, and how we measured it.
Explore the findingsRecorded experiments. Published evidence.
Results apply to each test, not every task.
One recorded correction
The agent asks for account and order IDs to verify the customer’s identity. The customer provides both.
Selected correction from public ABCD role-play data. Context paraphrased; the scores above cover all 500 test cases.
Read the training studyRead each result against its own task and baseline. These experiments do not form a single leaderboard.
79.2% vs 70.6%
Tuned open model vs official hosted TypeSafe JEV 1.13.0
Before task-specific training: 57.8%. Same 500 test conversations.
The open JevK5 4B model with a support adapter surpassed official JEV on recorded next actions. This does not measure whether the whole customer issue was resolved.
50.78% vs 43.55%
Tuned open model vs official hosted TypeSafe JEV 1.13.0
Before chess training: 42.97%. Chess-rules software: 100%.
A separate JevK5 4B chess adapter surpassed official JEV on this restricted task. It does not establish full-game chess strength or an advantage over a rules engine.
H2O: 2.99× lower median latency
59.7 ms vs 178.4 ms for the JevK5 4B adapter on B300
2B used 54% less CUDA memory; 9B showed no clear accuracy gain.
This follow-up compares local open models, not hosted TypeSafe JEV. H2O and 2B accuracy differences from the best 4B adapter remain inconclusive on two reused 500-case cohorts. New checkpoints have not been validated on miner hardware.
6.5% less probability error
Relative Brier reduction vs the base model
Historical delay rates had the best measured score.
The adapter improved risk estimates but did not meet the requirement to beat both baselines. No flight model was deployed.
147 correct for Zils; 147 for Kev
Same accuracy; worse probability estimates
The unchanged published Kev checkpoint is the baseline.
This earlier run did not establish an improvement. Its tasks, models, and scoring differ from the studies above.
Support training and the JEV comparison reuse the same trained model and 500 test cases. They are two comparisons of one experiment, not independent replications.
A model that assigns probabilities to a set of possible answers. Each experiment tests a specific decision, such as which action to take next.
The support, flight, and chess studies (5–7 October 2026) used the published JevK5 4B weights, the base at the time. In each study, that unchanged model is the baseline for measuring what training adds. JevK5 was retired on 17 October 2026; customer adapters now train on d3-mini 4.5B.
A LoRA adapter is a small set of learned weights added to the model. The base weights stay frozen while the adapter learns from examples for that task.
Compare models on the same held-out cases. Measure correctness, probability quality, and simple alternatives. Test data does not select the adapter.
Each study records its own split, selection rule, and evaluation protocol. The support study selects a checkpoint on development data; the flight study calibrates probabilities on a later, separate month; the chess study reserves calibration data but does not use it. A held-out test prevents direct training leakage, but cannot rule out public-data exposure during base-model pretraining.
Completed comparison · 6 October 2026 · 500 ABCD test conversations
Given a customer conversation, its previous actions, and a workflow guide, choose the next action from 30 options. ABCD is a public dataset of support role-play conversations; the recorded action is the test answer.
Our trained model matched that answer 79.2% of the time, compared with 70.6% for TypeSafe JEV 1.13.0. Both received the same inputs, instructions, and choices.
The Zils adapter trained on 1,024 separate conversations and was selected on a different development set. Its weights were frozen before the final test cases were prepared.
We then called TypeSafe’s hosted JEV API on those same test inputs. The Zils scores reuse verified predictions from the completed training study. No test answers were sent to either model, and no model or prompt was tuned during this comparison.
Our trained model got 72 cases right that JEV missed. JEV got 29 right that our trained model missed.
Each mark is one conversation. Grouped by outcome.
43 more correct decisionsOn this test, with the same inputs and choices.
View all 500 predictionsAccuracy gain
+8.6 points
Compared with TypeSafe JEV
Fewer mistakes
29.3%
147 mistakes reduced to 104
More correct actions
43
Across the same 500 conversations
The 95% interval for the accuracy advantage is +4.8 to +12.2 percentage points, from 2,000 paired resamples of test conversations. This measures uncertainty across these conversations, not across different training runs.
JEV was more accurate among its smaller set of answers assigned at least 90% probability. Our model assigned that probability to more cases, but made more mistakes within them. These groups contain different cases and have different coverage.
| Model | Cases / 500 | Mistakes | Accuracy in group |
|---|---|---|---|
| TypeSafe JEV 1.13 | 176 | 8 | 95.5% |
| Trained with Zils | 381 | 44 | 88.5% |
| Model | Accuracy | Macro F1 | Brier error |
|---|---|---|---|
| TypeSafe JEV 1.13 | 70.6% | 0.6108 | 0.403536 |
| Trained with Zils | 79.2% | 0.7583 | 0.337545 |
Model foundation. The trained model in this comparison combined the published JevK5 4B weights, the base at the time (retired 17 October 2026), with our ABCD-specific adapter.
Identical evidence. Each model received the prior conversation, completed actions and results, compact workflow catalog, and the same ordered choices. Current answers, future turns, and labeled workflow metadata were excluded. Each case comes from a distinct conversation.
Preserved API responses. All 500 JEV calls returned answers. 47 probability vectors summed to 0.99; they were normalized for Brier scoring. One returned choice differed from its probability ranking. Accuracy uses the returned choice; using the highest-probability option instead gives 70.8%. No cases were dropped or re-queried.
Usage and timing. JEV reported 1,595,919 input tokens, an estimated $0.06703 at the provider’s listed rate. API response time includes networking; the earlier Zils timings measure local GPU inference. They do not establish a comparable speed advantage.
This measures recorded next-action agreement in public role-play conversations. It does not establish complete conversation success or a general model ranking. One training recipe and seed were used; public-data pretraining overlap cannot be excluded. The prompt was developed for Zils and was not separately optimized for JEV. The research adapter has not been deployed.
Support training protocol · 6 October 2026
After training on 1,024 separate conversations, Zils’ accuracy rose from 57.8% to 79.2% on the untouched test set. The task was to choose the next recorded support action from all 30 tool choices.
One fixed recipe. Three saved checkpoints.
Choose the strongest qualifying checkpoint.
Freeze the adapter, then compare both models on identical cases.
The adapter corrected 128 baseline mistakes and introduced 21 new ones. Total mistakes fell from 211 to 104.
+21.4 percentage points vs before training
Accuracy gain
+21.4 points
On the final test set
Fewer mistakes
51%
Compared with before training
Lower probability error
45.3%
Relative decrease in multiclass Brier score
The 95% interval for the accuracy gain is +16.8 to +25.8 percentage points, from 2,000 paired resamples of test conversations. The trained adapter also beat both simple frequency baselines.
| Model | Accuracy | Macro F1 | Brier error |
|---|---|---|---|
| Before training | 57.8% | 0.4964 | 0.616934 |
| After training | 79.2% | 0.7583 | 0.337545 |
| Previous-action frequency | 44.8% | 0.1930 | 0.707464 |
| Training-action frequency | 19.8% | 0.0110 | 0.926030 |
| Training examples | Accuracy | Brier error |
|---|---|---|
| Before training | 54.0% | 0.656521 |
| 256 | 63.0% | 0.498994 |
| 512 | 74.2% | 0.381126 |
| 1,024 | 77.4% | 0.366084 |
What the model sees. The conversation so far, previously completed tool calls and their results, and a compact catalog of workflow sequences. The current answer, future turns, and labeled workflow metadata are excluded. Each case comes from a distinct conversation in its official split.
One fixed recipe. JevK5 4B with an attention-only rank-16 LoRA adapter, one epoch, learning rate 0.00001, and seed 553. The base weights stayed frozen. Training used groups of 15 or 16 answer choices; every evaluation offered all 30 through JevK5’s native selection process. Published temperatures stayed at 1.22 and 0.93; no confidence recalibration was fitted.
Selection before testing. A checkpoint needed at least 59% development accuracy, lower Brier error, and no macro-F1 regression. Highest accuracy won among qualifying checkpoints. All three qualified; the 1,024-example checkpoint was frozen before the final test.
How improvement is measured. The +21.4-point gain is the difference in accuracy. The 51% figure is the relative reduction in mistakes. The 45.3% probability improvement is the relative reduction in multiclass Brier error, the summed squared difference between predicted probabilities and the recorded answer.
ABCD support workflows · Recorded test examples
Three mistakes the adapter corrected and one it introduced in the 500-conversation test. These examples were selected after evaluation, not sampled to represent overall performance. The excerpts are paraphrased; both models saw the same full conversation, workflow catalog, and all 30 possible actions.
The agent asks for account and order IDs to verify the customer’s identity. The customer provides both.
The agent approves a return and asks for the address needed for a shipping label. The customer provides it.
A customer complains about an out-of-stock jacket. The agent offers to look up their account and collects their name.
The shopping cart still fails after logging out and back in. The agent asks the customer to refresh the page and try adding the item again.
Before training, Zils chose try again, matching the recorded action. After training, it chose provide instructions. This was one of the 21 new mistakes it introduced, alongside 128 corrected mistakes.
Examples selected to explain the decisions. The 79.2% result above includes all 500 test cases. This measures action selection, not tool execution or a complete customer conversation.
Recorded example predictions (JSON)Training improved next-action prediction on ABCD’s public support role-play conversations. It does not measure complete conversation success, flight prediction, or customer readiness. This used an isolated research training runner; the full customer-facing training service was not evaluated.
Confidence still runs high: 44 of 381 predictions made with at least 90% confidence were wrong. One recipe and one training seed were tested. Public-data exposure during base-model pretraining cannot be ruled out. The adapter has not been deployed.
Chess specialization · 7 October 2026 UTC
Choose an immediate checkmate from a supplied list of legal checking moves. A correct answer ends the game in one move; this does not test playing a whole game.
A fresh chess adapter selected a checkmating move in 260 of 512 held-out positions. Accuracy improved by +7.81 percentage points over the model before training and +7.23 over TypeSafe JEV.
Read the chess studyAll legal checking moves were offered. Every move that immediately checkmates received credit.
This measures one-move choices among legal checks. Deterministic chess software solves 100% of these positions; the study tests specialization on a new task. One training seed, one recipe, no deployment.
Completed experiment · 5 October 2026 UTC
Estimate whether a flight will arrive at least 15 minutes late, using scheduled information only. On 1,024 later test flights, the trained adapter reduced probability error by 6.5% versus the base. A simple historical-rate baseline still had the best measured score.
Public US Bureau of Transportation Statistics records, departing JFK, LGA, and EWR. The decision: will a completed flight arrive at least 15 minutes late?
3,072 flights
512 flights
1,024 flights
Only scheduled information entered the model: date, airline, route, times, duration, and distance. Actual delays and arrival times supplied the answers for training and scoring, never the prediction inputs.
Both models had their confidence scores adjusted on separate February data. All three were tested on the same March flights.
The adapter produced better probability estimates and ranked delay risk more accurately. Its risk-ranking score (ROC-AUC) rose from 0.516 to 0.640. The measured probability-error improvement over the base was supported by a 95% interval that stayed above zero.
Historical rates scored slightly better; the difference was uncertain. The experiment required the adapter to beat both references. It did not meet that bar, and no model was deployed from this run.
Most flights were not late. At a 50% probability cutoff, the adapter predicted “not late” for every flight and flagged none of the 201 late arrivals. The improvement was in its risk estimates, not in useful delay alerts at that threshold. An alert threshold needs its own validation.
| Model | Brier error | Accuracy | ROC-AUC |
|---|---|---|---|
| Before training, calibrated | 0.164076 | 80.18% | 0.5165 |
| After training, calibrated | 0.153357 | 80.37% | 0.6401 |
| Historical delay rates | 0.151347 | 80.37% | 0.6503 |
| Before training, raw | 0.170116 | 80.18% | 0.5165 |
| After training, raw | 0.156231 | 80.37% | 0.6401 |
What the percentage means. The 6.5% headline is the relative decrease in binary Brier error: (base error − adapter error) ÷ base error. It is not a percentage-point increase in accuracy. Binary Brier is the average squared difference between the predicted chance of a late arrival and what happened.
A fair comparison. Both models had their confidence adjusted independently using February data only. Raw temperatures were 1.22 for the base and 1.0 for the adapter; fitted temperatures were 0.933033 and 0.870551. Historical rates used the same January sample, grouped by airline, origin, and four-hour departure window, with smoothing toward the overall rate.
Uncertainty. We resampled whole test dates 2,000 times. The 95% interval for adapter Brier improvement versus the base was [+0.006059, +0.015117]. Versus historical rates it was [-0.005960, +0.002194], crossing zero. Positive values favor the adapter.
One fixed training recipe. JevK5 4B, one epoch, rank-16 LoRA, seed 553. The base weights stayed frozen. Training took 27.5 minutes on an RTX 4090. Median forward calls took 49.7 ms for the base and 64.7 ms for the adapter, excluding tokenization, loading, and API overhead. These are recorded timings, not a serving guarantee.
Scope and limits. One recipe, three airports, one season, and 31 test dates. Canceled and diverted flights were excluded, so this does not cover all scheduled flights. There were no live weather, congestion, or inbound-aircraft feeds. Public records may have appeared in base-model pretraining. Historical schedules are not a timestamped record of what was known at booking time. This result does not establish general improvement or customer readiness.
Base tier · status as of 19 October 2026
Customer adapters train on a pinned base model. Since 17 October 2026 that base is d3-mini 4.5B (below); before that it was H2O Lightning 4B v1.2.3. The base tier is the competition that decides when that base changes. Miners submit a full-weight replacement for the pinned base of a text profile. Validators score it on a sealed holdout against the incumbent, and a candidate is promoted only if it passes every step below.
| Step | Rule |
|---|---|
| Sealed score | Validators score the submitted weights on a sealed holdout, against the incumbent. |
| Eligibility | Skill is positive, accuracy is not below the incumbent’s, and Brier improves by at least 0.01 with a paired-bootstrap 95% lower bound above zero. |
| King of the hill | Only kings are paid. A winner becomes king when the owner promotes it. The base-tier share splits equally over the last five distinct kings still registered. |
| No king | An empty track’s share is burned. The vendor base (stock d3-mini today) has no miner, so there is no king yet. |
| Confirmation | The best eligible candidate is re-scored on a second, disjoint sealed draw at least 12 hours later and must stay eligible. |
| Regression suite | Before promotion, a fixed suite of reference adapter jobs is retrained on the incumbent and on the candidate. The candidate must be no worse than 0.005 Brier on every job and better on at least one. |
| Channel | Advances | Who gets it |
|---|---|---|
| latest | Advances on each confirmed winner. | Opt-in, per job. |
| stable | Advances at most quarterly, to a base that held latest through a full window without rollback. | The default for new customer jobs. |
New customer jobs default to stable; latest is opt-in per job. Existing releases keep serving on their base.
Status, 19 October 2026. Open to anyone registered on testnet subnet 579: upload to Hugging Face, then commit on chain. Every validator reads the commitments and scores them. Customer-job admission stays off until miner terms and intake are ready. Mainnet is not live.
Round one’s sealed holdout has 1,458 teacher-authored cases in six families. The incumbent, stock H2O Lightning 4B v1.2.3, scores Brier 0.184 and 87.7% accuracy on it. d3-mini’s confirmation draw did not confirm a winner, but d3-mini became the base by decision on 17 October (see below), so both channels now point at it and stock d3-mini is the incumbent for round two.
Want to compete? Read the miner guide in the Zils docs. For every attempt so far, in order, see the miners page; for how the bases have changed, see bases.
d3-mini · recorded 16–17 October 2026 · one rented B300
Under the base tier’s own rules, stock d3-mini was not promotable. On the night of 16–17 October 2026, one rented B300 ran for 6.75 hours. The first sealed draw cleared, but the confirmation draw did not confirm a winner (Brier 0.1790 for d3-mini against 0.1794 for H2O, an improvement of 0.0004 where the rule needs at least 0.01 with a positive lower bound). The adapter regression suite passed, on one reference job where the rule asks for three or four. Under those rules d3-mini was not promoted; it became the base by decision instead (below). No round-two decision was frozen under the exam.
| Property | Value |
|---|---|
| Profile | d3-mini-4.5b-v1 |
| Vendor model | vllm-sr/d3-mini · 4.54B parameters · Apache-2.0 |
| Backbone | Qwen3.5-4B (the same as H2O and Imajev), plus a 0.33B vision tower |
| Inputs | Text, up to four images, and one video |
| Answers | Yes/no, choice (up to 255 outcomes), and score |
| Pinned by Zils | A byte-identical mirror, zils-ai/d3-mini-4.5b-v1, and a 4,096-token cap with no truncation |
Merged on 16–17 October 2026 in the subnet and platform repositories: the profile, runtime, node qualification, images and video, platform routing, a latest channel that can point at d3-mini, and cross-profile upgrades. Through the training API, an existing task on H2O can be upgraded to a d3-mini adapter by setting "base_change": true in the acceptance policy; the frozen previous release is scored on its own base and compared on the same test data. The dashboard does not offer this yet.
Decision, 17 October 2026. Zils is adopting d3-mini as its base anyway, by the owner’s decision and recorded as an override of the promotion rule rather than a win under it. The grounds are capability: adapters trained on d3-mini beat adapters trained on H2O on the reference job and on ABCD, and d3-mini takes images and video, which H2O does not serve. Since the afternoon of 17 October 2026 both stable and latest name d3-mini, so new jobs train on it (4,096-token inputs, up to four images). Existing H2O jobs and releases keep their frozen profile and move only through base_change upgrade jobs; JevK5 is retired; H2O and Imajev retire once nothing served depends on them.
| Measure | H2O Lightning 4B v1.2.3 | d3-mini | Reading |
|---|---|---|---|
| Stock model, first draw (round-one sealed holdout), validator decision rule | Brier 0.184 · 87.7% accuracy | Brier 0.142 · 89.8% accuracy | Eligible. Reproduced exactly on the node through the validator path, so real for that set |
| Stock model, confirmation draw (second, disjoint, teacher-authored sealed set: 291 calibration and 1,156 test cases, scored 12 hours later) | Brier 0.1794 · 88.5% accuracy | Brier 0.1790 · 88.6% accuracy | No winner confirmed: improvement 0.0004, paired-bootstrap 95% interval −0.022 to +0.023; the rule needs at least 0.01 with a positive lower bound |
| Families won (of six), first draw and confirmation draw | 0 and 3 | 6 and 3 | The first-draw result did not carry over; the second split three and three, narrowly. H2O improved between draws (0.184 to 0.179) while d3-mini fell (0.142 to 0.179), so the second set is not simply harder |
| Adapter regression suite, one reference job (ABCD; the rule asks for three or four): calibrated Brier and accuracy | Brier 0.2801 · 81.6% | Brier 0.2515 · 84.0% | Passed, on one job |
| Upgrading an existing H2O task to a d3-mini adapter with base_change: Brier | 0.1805 (previous H2O release) | 0.1667 (d3-mini candidate) | Improvement 0.0138 against a 0.01 threshold |
| Adapter on the 500-case ABCD test cohort (frozen production recipe, 4,096 training examples, two seeds) | 83.0% | 83.4% | A tie: the interval includes zero |
| Adapter on the historical ABCD cohort | 83.6% | 86.6% | d3-mini ahead; interval +0.6 to +5.4 points |
| Exam prompt length, median input tokens | 528 | 568 | d3-mini prompts are about 40 tokens longer |
A feature probe answered yes/no, score, multi-question, JSON-state, one-, two- and four-image, and short-video requests correctly. It refused a 13,414-token state at the runtime’s 8,192-token limit.
The merged d3-mini code ran end to end on a GPU for the first time (qualification parity 5.3e-7). A d3-mini training job on the ABCD dataset was accepted, at 84.0% test accuracy and Brier 0.244 against 48.0% for stock d3-mini, and its release answered a text call and a one-image call.
Limits. One card, one night. Nothing above 4,096 tokens was tested. The holdouts are teacher-authored, not customer data. Passing the sealed-holdout decision rule makes a candidate eligible; it does not promote it.
Earlier research · Kev 0.8B
This earlier experiment fine-tuned Kev 0.8B and tested it on the public JevBench items. Accuracy tied the unchanged checkpoint, while probability quality worsened. Its model, tasks, and scoring rules differ from the studies above. Checkpoint and dataset identities are preserved in the method below.
Earlier JevBench comparison
Zils candidate accuracy
63.64%
147 of 231 public items correct
Probability error (Brier)
0.463608
Lower is better. Compare within this study.
Confident mistakes
8
Wrong at ≥90% confidence.
Zils corrected 6 reference-model errors and introduced 6. No general improvement is established.
231 released public items: 48 easy, 72 original, 111 hard
| Metric | Zils candidateExperimental checkpoint | Published Kev 0.8BComparison baseline |
|---|---|---|
| AccuracyHigher is better | 63.64% | 63.64% |
| Correct answersSame public items | 147 / 231 | 147 / 231 |
| Brier lossLower is better | 0.463608 | 0.451648 |
| Calibration error (ECE)Lower is better | 0.113615 | 0.044050 |
| Confident mistakesWrong at ≥90% confidence | 8 | 3 |
| Median latencyLocalhost HTTP | 42.08 ms | 52.78 ms |
| p95 latencyLocalhost HTTP | 205.26 ms | 209.36 ms |
| Valid probability outputs | 100.00% | 100.00% |
Brier loss measures probability error; ECE measures the gap between confidence and observed accuracy. Lower is better for both. Timing is one serial pass per model; Kev then Fez, on NVIDIA GeForce RTX 4090, via localhost HTTP. This does not establish a reliable speedup or production SLA.
| Subset | Zils | Kev baseline |
|---|---|---|
| easy | 48 / 48100.00% | 48 / 48100.00% |
| original | 57 / 7279.17% | 58 / 7280.56% |
| hard | 42 / 11137.84% | 41 / 11136.94% |
This earlier experiment fine-tuned a published Kev 0.8B checkpoint built on Qwen3.5-0.8B-Base. The unchanged published checkpoint is the baseline in this comparison.
No official JevBench rank or composite score is available: private and sealed tests were not run. Hosted cost was not measured. Other experiment suites are documented separately and cannot form an improvement line with this comparison.
63475109e543fe5f7fd2698ec27f6ac0b828c67074606409ad71697abca30da3af9331c6c6331099ce2d28a0298620ca540c991f781b6167ceb0f177f235190fdc3995d8ae1e2fc8e81ce38431add509eb8bb39b85aadfd0c7c32079382dde5126eb72d4e0e60d8ace0adfc77a384063442561cdThe experimental Zils checkpoint is not distributed, so the full comparison cannot be reproduced from the checkout alone.
Being correct and knowing how likely you are to be correct are different properties. We report both.
All results on this page are recorded experiments. The study reports define their scope, artifacts, and reproduction limits.