JEVANY / DOCUMENTATION

Evaluation

Model family v2

This section contains the full results for the five current LoRA SFT releases. Transfer also informed model development and serves as a diagnostic comparison.

Transfer is the cross-domain evaluation built from Kev's frozen transfer-v9 suite for cross-domain and robustness evaluation over 1,046 clean, knowable decisions. Its item-level mean spans Emotion, PAWS, QNLI, TweetEval, MMLU, MMLU-Pro, SciQ, and four robustness slices. JevBench is accuracy across all 231 public development items. NLL, Brier, and ECE in the main table are Transfer metrics; every run covers every item.

The 27B release uses step 44,319. Against the earlier step 22,160 checkpoint, it trades one JevBench answer for three additional Transfer answers and improves NLL on both suites:

Transfer breakdown

Columns are dataset or robustness-slice accuracy. Sample counts are respectively 80 / 80 / 80 / 80 / 80 / 200 / 80 / 80 / 96 / 80 / 110.

Buried tests hidden instructions; Comp. tests AND/OR/conditional composition; Policy contrasts authorization/deadline rules; Ctrl. contains 11 knowable policy controls. Values are percentages.

JevBench breakdown

The direct-token 4B model leads the released 4B models on JevBench, while the pointer 4B model is slightly better on Transfer. Kev and Laya use complete local public-checkpoint runs (Kev-27B pinned to 01b8199, Laya to 55cf4c4); Jev uses a complete local Transfer API run and JevBench's published per-tier accuracy.

What we ablated

• Pointer structure: linear, MLP, and zero-initialized residual heads; the residual head won the controlled screen on development NLL and calibration.
• Representation readout: decision-token, query-mean, option-mean, and combined variants. Query-mean led the early screen; the release recipe uses decision-marker / option-close after the full model-family run.
• Loss: cross-entropy, pure InfoNCE, and mixed objectives; CE gave the best Transfer accuracy in the loss sweep, while small contrastive terms mainly improved calibration. The released checkpoints use CE.
• Readout family: at 4B, direct-token improves JevBench (80.95% vs 80.09%), while pointer is slightly stronger on Transfer (78.68% vs 78.20%).

Reproducibility

Transfer reports all 1,046 scored decisions; JevBench reports all 231 public development items. Exact suite hashes and unrounded metrics are in the machine-readable results. Release manifests, checkpoint-native reload reports, and GPU loader/reconstruction parity were checked before publishing.

The README comparison figure ranks models by the equal-weight mean of Transfer and JevBench accuracy, using unrounded values. The ordering depends on these weights: at 4B, direct-token leads with equal suite weights, while pointer leads with equal item weights. Ranks reflect these point estimates. Model icons and their sources are in model-logos/.

Regenerate the figure from the repository root with matplotlib and cairosvg installed:

python scripts/plot_evaluation_overview.py

Machine-readable release results · Historical results · Technical report.

Training compute

The five released checkpoints represent approximately 2,082 H200 GPU-hours of training, with at most 40 GPUs used in parallel within one run. GPU-hours are elapsed training time through the released checkpoint multiplied by the DDP world size; ablations, evaluation, and training after a selected checkpoint are excluded. Gemma uses run/checkpoint timestamps because its earlier checkpoint format did not store cumulative elapsed seconds; the other figures come from checkpoint or terminal trainer telemetry.

Earlier releases and evaluations

The JevBench and Kev comparison evaluates both released JevAny checkpoints and all 14 distinct released Kev checkpoints across 67 public panels. It includes complete result tables, published official Jev references, and the private partitions that remain unavailable. The earlier release, multimodal, and interactive measurements are retained below.

SFT is the default checkpoint. RLCR improved development NLL by 0.005 and did not improve transfer accuracy.

The v0.2 release record reports results on 1,004 development questions and 1,046 transfer questions. NLL (negative log-likelihood) penalizes low probability on the correct answer; lower is better.

RLCR changed transfer accuracy by -0.10 percentage points against SFT, with 3 fixes and 4 regressions. The paired 95% bootstrap interval is [-0.58, 0.39] points. The small development NLL gain did not survive independent calibration. Jev provides a hosted-system comparison on the same decision suite, using different model weights.

Development accuracy and NLL exclude 100 VideoFeedback questions whose labels are all the same highest score. These questions exercise the video path; the video accuracy evaluation uses MVBench below. AI2D and MMMU use native images through the backbone's vision path. The training set also contains native A-OKVQA and ScienceQA images.

Native image and video decisions

We evaluated the released SFT checkpoint with the real media, a neutral blank asset, and media shuffled between questions within each task. We shuffle by unique media group, so questions that share one image or video receive the same replacement. The media sensitivity gate requires full-media accuracy to exceed the stronger control by at least five points, with a positive paired media-group bootstrap interval. Passing the gate indicates media use. The tables also report task accuracy.

From the image panel we dropped invalid choices and every item that matched the training, calibration, or development splits by media or by normalized question and unordered option text. No exact or perceptual media overlap, question-option overlap, or source-ID overlap remains. Its task-macro random and label-position baselines are 26.7% and 31.8%. The video panel passes the same zero-overlap checks. Its overall media gain is significant, though the action-antonym score is low and the video probabilities are badly calibrated.

Full metrics, per-task intervals, dataset revisions, checkpoint hashes, and control provenance are in the image report and the video report. The internal evaluation checkpoint and public SFT release have identical LoRA and pointer-head tensors; the equivalence record accounts for the embedded release temperature. Upstream terms keep us from redistributing the benchmark media.

The examples below use synthetic media we created and released with this repository. Each modality shows two cases from a predeclared set of three; the selection record lists every probability. The two video cases ask the same question about different clips, with motion determining the answer.

Jev-Image: which destination is still reachable	Jev-Image: which bay needs inspection
Jev-Video: the cart finishes west	Jev-Video: the cart finishes north

Jev-Agent

We expose only the legal actions at each step, and JevAny picks one without generating text. FrozenLake lets you recover from a bad step. Sokoban gives you no way to undo a push, so one choice can decide the episode.

Jev-Test

We tested transductive adaptation without ground-truth labels. The parent produced 16 stochastic decisions per input, a strict majority became the pseudo label, and we dropped ties. We locked the protocol before scoring the adapted models against gold labels.

MMLU-Pro NLL improved by 0.012 while accuracy fell a point. MuSR accuracy moved by at most 0.40 points while its NLL rose from 1.122 to 1.552. This self-training recipe is a negative result, and the code stays experimental. The locked protocol and release measurements document the full experiment.