JEVANY / DOCUMENTATION
Get started with JevAny
JevAny is open infra for System 1 decision model training and deployment, covering data preparation, model adaptation and evaluation. Use a released model or train on your own data to route support tickets, select tools, or choose a robot's next action. One API takes the state, question and candidate options, then directly returns a choice and its probabilities.
🎮 Results and Demos
Explore interactive benchmark results
Full benchmark results and evaluation details.
The following 30 examples are archived replays from an earlier compatible JevAny checkpoint. The current default release is JevAny-Qwen3.8-27B. Explore the cases, or run a model locally to try your own inputs and see its choices and probabilities.
Explore all 30 application replays →
⚡ Jev inside LLM agent loops
Key takeaways
- Task level: T0 controlled → T4 open terminal. Task difficulty and delegation are separate.
- Delegation level: D0 LLM-only → D4 bounded subgoal. Increase delegation only when local choices remain easy to verify.
- Delegate when: the LLM can generate 2–4 valid, meaningfully different, reversible choices with immediate feedback.
- Keep with the LLM: planning, open-ended search, exact edits, recovery, high-risk actions, and completion.
- Success test: reward stays the same or improves while LLM calls, tokens, or time fall; otherwise return control to the LLM.
1 · WebShop — choose the exact color and size from LLM-generated menus
Reward 1→1 · LLM calls 9→4 · tokens 38,852→14,256 · time 18.54s→7.83s
2 · FrozenLake — compare four directions at every state
Reward 1→1 · LLM calls 4→1 · tokens 2,338→663
3 · Terminal-Bench · sqlite-db-truncate — select one of three commands
Reward 1→1 · LLM calls 15→8 · time 187.9s→144.7s
| Task | Success | Efficiency |
|---|---|---|
| GPT-5.6-sol · FrozenLake · 10 pairs | 100% → 100% | LLM calls −64.4% · tokens −63.1% · time −37.6% |
| WebShop · LLM-generated menus · 3 pairs | 67% → 100% | LLM calls −21.4% · tokens −14.3% · time −15.0% |
| WebArena · 6 pairs | 50% → 50% | LLM calls +5.6% · tokens +28.2% · time −0.4% |
| Terminal-Bench · 6 pairs | 1/6 → 3/6 | LLM calls −9.0% |
Full results · task/delegation levels · combined technical report
📑 Table of Contents
- 🎮 Results and Demos
- ⚡ 1. Quickstart
- 💻 1.1 Run locally
- 🛠️ 1.2 JevAny Training
- 🚀 1.3 JevAny Deployment
- 🤗 2. Pretrained Models
- 📊 3. Benchmark Results
- 🕹️ 4. Examples & Test Environments
- 🧩 5. Supported Model Families
- 📚 6. Documentation and Contributing
⚡ 1. Quickstart
Use Python 3.12 or newer. Clone the repository and install the lightweight package:
git clone https://github.com/SimpleJev/JevAny.git
cd JevAny
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -e .
Keep this environment active and work from the repository root. Start with a local demo, then train on your own data or use the API.
💻 1.1 Run locally
Choose a model that fits your computer:
| Model | Hardware | Start here |
|---|---|---|
| Qwen 0.8B starter | CPU · 16 GB RAM recommended | Train the small adapter on the bundled tickets |
| JevAny-Qwen 4B | CUDA · ~8 GB for BF16 base weights, plus runtime memory | Load the released model |
| JevAny-Qwen 27B | CUDA · ~54 GB for BF16 base weights, plus runtime memory | Choose the larger checkpoint |
The local model guide covers preparation and loading. Released models download on first use and reuse the local cache. With the model server running, open a second terminal in the same checkout:
source .venv/bin/activate
jevany demo --base-url http://127.0.0.1:8008 --text-only
Open http://127.0.0.1:8090, choose Test and connect, then edit
Try your own decision and press Ask the model. Change the state or options
to see how its decision changes. Games, robotics and replays
are available in the same playground.
🛠️ 1.2 JevAny Training
Train your own System 1 model on the same state and questions you send at
inference, with a label for each question. Start with the bundled synthetic
support tickets, then train on your own labelled data. The starter recipe uses
Qwen3.5-0.8B on CUDA with BF16 and writes runs/my-jev:
python -m pip install -e '.[train]'
jevany data init --out data/starter
jevany data validate data/starter/train.jsonl
jevany train --config recipes/sft.toml --dry-run
jevany train --config recipes/sft.toml
After training, try the checkpoint on the included ticket request:
jevany decide examples/request.json --checkpoint runs/my-jev
Pass --data to train on your own JSONL data, or use
recipes/finetune.toml to adapt the released 27B model.
See the training guide for CPU settings, multimodal data and
standard torchrun launches. For image/video training or fine-tuning the released
27B model, install .[train,multimodal].
After SFT, you can continue with experimental RLCR, which rewards correctness and probability calibration:
jevany train --config recipes/rlcr.toml
🚀 1.3 JevAny Deployment
Install the serving dependencies and start the released Qwen 4B model on a CUDA GPU. See the hardware and loading guide for memory requirements.
python -m pip install -e '.[serve,multimodal]'
jevany serve --checkpoint SimpleJev/JevAny-Qwen3.5-4B-LoRA \
--device cuda --dtype bf16 --port 8008
The default path favors reproducibility. CUDA deployments can opt into BF16
LoRA merging, SDPA and torch.compile; the useful settings differ between 4B
and 27B. See the inference acceleration guide
for commands, H200 measurements and accuracy caveats.
To serve your training output, replace the checkpoint ID with runs/my-jev.
Keep the server running. In a Python session using the same environment, send
a ticket and the departments that can handle it:
from jevany import Choice, JevClient
jev = JevClient("http://127.0.0.1:8008")
result = jev.system_one(
state={"ticket": "I was charged twice. Please help."},
questions={
"department": Choice(
instructions="Which team should handle this?",
criteria={"billing": "Payment problems", "shipping": "Delivery problems"},
),
},
)
answer = result["answers"]["department"]
print("Selected team:", answer["choice"])
print("Probabilities:", answer["probabilities"])
choice is one of the department names; probabilities maps each name to its
probability. Your application can use these fields to route the ticket or ask
for review when the decision is uncertain. Use Noul for yes/no questions,
such as whether a ticket needs urgent review,
and Score for ordered levels, such as low, normal and high priority.
See the API reference for all three question types.
For in-process inference, load a model in Python and use the same interface. For image and video inputs, follow the media setup.
🤗 2. Pretrained Models
For a first local run, choose a model and hardware in Run locally.
| Model | Readout | Intended use |
|---|---|---|
 JevAny-Gemma-4B |
Pointer | Compact Gemma release |
 JevAny-Qwen3.5-4B |
Pointer | Compact, flexible choice count |
| JevAny-Qwen3.5-4B-Direct-Token |
Direct-token | Best released 4B JevBench accuracy |
| JevAny-Qwen3.8-27B |
Pointer | Default; highest released accuracy |
 JevAny-Muse-Glimmer-30B |
Pointer | Muse Glimmer alternative |
These LoRA adapters were trained with SFT on 1,772,725 text records containing 2,180,242 labelled decisions; see training compute and experiments for the setup. Full-parameter SFT and further post-training improvements are planned.
The corresponding base model is loaded separately and its license and access terms apply. Allow roughly twice the base parameter count in bytes for BF16 weights, plus runtime memory. See the hardware and loading guide.
Pointer and direct-token models share the same API. Pointer supports up to 4,096 options within the context limit; direct-token supports up to 255. See readout choices for training and accuracy tradeoffs.
📊 3. Benchmark Results
JevAny-Qwen3.8-27B leads both benchmarks and has the lowest NLL and Brier. Among 4B releases, direct-token leads on JevBench; pointer leads on Transfer.
Explore interactive benchmark results
Kev-4B](https://huggingface.co/jaredpalmer/kev-4b) | 74.19% | 75.32% | 0.858 | 0.380 | 0.125 |
| [ Kev-27B](https://huggingface.co/jaredpalmer/kev-27b) | 82.31% | 85.28% | 0.533 | 0.265 | 0.050 |
| [
Jev 1.13.0](https://docs.typesafe.ai/models) | 85.37% | 86.58% | 0.644 | 0.212 | 0.033 |
| [
Laya](https://huggingface.co/convaiinnovations/laya) | 52.29% | 58.01% | 1.264 | 0.615 | 0.127 |
| **JevAny releases** | | | | | |
| [ JevAny-Gemma-4B](https://huggingface.co/SimpleJev/JevAny-Gemma-4B-LoRA) | 70.84% | 77.49% | 0.706 | 0.369 | 0.056 |
| [ JevAny-Qwen3.5-4B](https://huggingface.co/SimpleJev/JevAny-Qwen3.5-4B-LoRA) | 78.68% | 80.09% | 0.587 | 0.297 | 0.035 |
| [ JevAny-Qwen3.5-4B-Direct-Token](https://huggingface.co/SimpleJev/JevAny-Qwen3.5-4B-Direct-Token-LoRA) | 78.20% | 80.95% | 0.564 | 0.291 | **0.029** |
| [ JevAny-Muse-Glimmer-30B](https://huggingface.co/SimpleJev/JevAny-Muse-Glimmer-30B-LoRA) | 83.46% | 87.45% | 0.464 | 0.229 | 0.032 |
| [ **JevAny-Qwen3.8-27B**](https://huggingface.co/SimpleJev/JevAny-Qwen3.8-27B-LoRA) | **86.04%** | **90.04%** | **0.388** | **0.195** | **0.026** |
NLL, Brier and ECE are measured on Transfer.
Full results and protocols · Machine-readable results · Method and ablation report
🕹️ 4. Examples & Test Environments
The playground includes the three environments below. These GIFs preserve historical model actions and option probabilities; run the current JevAny-Qwen3.8-27B checkpoint with the commands in the playground guide.
🤖 4.1 Robot peg insertion
Use a Franka gripper to grasp, align and insert a peg, checked by PyBullet contact physics.
🔫 4.2 Doom corridor · 3D
Clear the final room by defeating the enemies on the left and right, then move forward. The environment uses ViZDoom and the included Freedoom assets.
⛏️ 4.3 Crafter survival · 2D
Gather wood, craft tools and mine stone while managing health and supplies.
🎮 4.4 Try the playground
With a model running from Run locally, open the playground:
jevany demo --base-url http://127.0.0.1:8008 --text-only
Open http://127.0.0.1:8090, choose Test and connect, and try your own
decision. To let the model control a game, install the optional engines and
restart the playground:
python -m pip install -e '.[demo]'
jevany demo --base-url http://127.0.0.1:8008 --text-only
Choose Run model, then One decision or Run automatically.
Play yourself lets you control the game. Live control sends text state to the
model; robot control uses the .[robotics] extra.
For the bundled recordings, run jevany demo and choose Replay.
Playback works on CPU without model weights.
See the playground guide for platform
requirements and environment APIs, or integrations to
combine JevAny decisions with an LLM planner.
🧩 5. Supported Model Families
Model IDs, supported inputs and setup requirements.
📚 6. Documentation and Contributing
Training · Deployment · API · Data · Evaluation · Agent harness protocol · Contributing
To contribute a model adapter, evaluation or application example, start with the contribution guide. The combined technical report describes model design, multimodal support, the agent-harness study, negative results, and open questions; the original release report remains unchanged.
Code and starter data are Apache-2.0. Some components are adapted from Kev; see NOTICE and ACKNOWLEDGEMENTS.md. Base models and upstream datasets retain their own terms.