| Guard | Triage Score | lancet-bench-2-next | ShellRisk test | Neutral set | Risky caught | Safe stopped | Parameters | Runs on |
|---|---|---|---|---|---|---|---|---|
| LANCET Nano v0.4.3 | 68.3 | 63.1 | 64.8 | 85.8 | 77.0% | 9.0% | 111 M | CPU, local |
| LANCET Nano v0.4.2 | 54.4 | 39.0 | 60.1 | 91.0 | 77.1% | 13.7% | 110 M | CPU, local |
| LANCET Nano v0.4.0 | 47.1 | 41.1 | 60.6 | 50.8 | 55.7% | 10.5% | 110 M | CPU, local |
| LANCET Nano v0.4.1 | 44.9 | 38.1 | 70.5 | 39.3 | 67.9% | 14.6% | 110 M | CPU, local |
| LANCET Nano v0.3.0 | 44.7 | 44.3 | 46.6 | 43.6 | 55.3% | 11.6% | 110 M | CPU, local |
| LANCET Nano v0.2.0 | 40.7 | 39.3 | 47.7 | 37.9 | 55.2% | 13.6% | 35 M | CPU, local |
| LANCET Nano v0.1.0 | 40.5 | 34.9 | 45.6 | 51.0 | 58.4% | 14.1% | 35 M | CPU, local |
| Jev | 38.9 | 43.6 | 32.4 | 32.1 | 90.0% | 23.4% | undisclosed | hosted API |
| verdict-shell-safety | 32.4 | 27.6 | 33.3 | 44.9 | 56.3% | 18.0% | 151 M | CPU, local |
| Kestrel | 30.7 | 12.7 | 100.0 | 14.0 | 68.1% | 37.1% | — | local |
| ModernBERT bash | 23.2 | 16.2 | 40.9 | 25.6 | 48.1% | 22.8% | 150 M | local |
| dcg | 19.8 | 26.4 | 4.1 | 16.5 | 19.8% | 6.9% | rules | local |
| AutoShell-0.8B | 19.2 | 10.2 | 22.4 | 41.0 | 98.8% | 70.9% | 0.8 B | GPU, local |
| Gyra | 17.9 | 19.0 | 4.1 | 27.8 | 28.2% | 13.4% | 421 M + rules | GPU, local |
| bash-classify | 14.3 | 13.5 | 16.8 | 14.0 | 92.9% | 65.8% | rules | local |
| Laya | 13.3 | 11.0 | 16.2 | 17.1 | 87.3% | 69.6% | 421 M | GPU, local |
| bev-decider | 13.3 | 10.8 | 18.7 | 14.8 | 94.4% | 75.3% | 0.4 B | GPU, local |
| laya-cli-gate | 12.5 | 11.8 | 12.3 | 14.5 | 71.2% | 57.5% | 421 M | GPU, local |
| sh-guard | 10.9 | 10.2 | 12.1 | 11.7 | 97.9% | 90.5% | rules | local |
| Shieldstral-1.0-3B | 6.1 | 4.1 | 7.8 | 9.9 | 6.1% | 0.2% | 3 B | GPU, local |
Jev received task context; the others see only the command. Per-benchmark columns are each benchmark's own Triage Score. Guards that support only Bash answer the 138 PowerShell and cmd commands as "ask".
Until v0.4.2 the Triage Score used lancet-bench-1 (793 Bash commands). The best guards were close to its ceiling (Jev caught 96.8% of its risky commands, v0.4.2 92.4%), and it was narrow: Bash only, short single-line commands, a quarter of them about secrets.
From v0.4.3 the Triage Score uses lancet-bench-2-next, a new, much more comprehensive benchmark that no LANCET release was trained, tuned or calibrated on. Because it is much harder, every guard scores lower than before, LANCET and comparators alike: v0.4.2 scored 82.7 under the previous Triage Score and 54.4 under the new one. The model did not change; the test did. Scores are only comparable on the same benchmark.
bash -c, a pipeline and a background job.# 1. verify the download, then the bundle
certutil -hashfile lancet-v0.4.3-nano-cpu-int8.zip SHA256
python verify_bundle.py --strict
# 2. install three pinned CPU dependencies
python -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements.txt
# 3. classify (JSON lines in, JSON lines out)
$env:OPENBLAS_NUM_THREADS = '1'
'{"command": "terraform destroy -auto-approve", "shell": "bash"}' |
.\.venv\Scripts\python.exe classify.py --model model
'{"command": "Stop-Computer -Force", "shell": "powershell"}' |
.\.venv\Scripts\python.exe classify.py --model model
{"score": 0.992, "classification": "risky", "reason": null,
"experimental": true, "executionAuthorized": false, ...}
{"score": 0.989, "classification": "review", "reason": "uncertainty-band",
"experimental": true, "executionAuthorized": false, ...}
shell is bash, powershell or cmd..venv/bin/python. Those platforms were not benchmarked.git lfs pull, and use bundle/.v0.4.3 is the Salesforce CodeT5+ 220M encoder, fine-tuned from its pinned upstream weights with no earlier LANCET checkpoint, plus a small pooling head that reads long commands as overlapping windows. Training ran to convergence (cosine learning-rate decay to zero). Its labels come from project authoring, documentation and source datasets, not from another model's opinions:
sensitive markings on returned fields, so commands that print credentials are risky.| Training source | Rows |
|---|---|
| Command-semantics curriculum (new) | 21,544 |
| Project-authored pairs | 9,184 |
| Secrets family (project-authored) | 7,728 |
| tldr-pages examples | 6,093 |
| ShellRisk-Bench training split | 5,951 |
| Earlier evaluation suites | 5,047 |
| Developer commands (V5 pool) | 2,956 |
| Contrast sets and shell-escape atlas | 2,598 |
| Shell Safety v2 training split | 2,475 |
| Azure / GitHub / kubectl / Docker CLI | 1,418 |
| Retired benchmarks (lancet-bench-1, draft) | 1,300 |
| AWS CLI operations | 917 |
No language model, hosted API or human labeler produced any training label.
Development and calibration used tools and families held out from training.
review, with no silent truncation.not_flagged is not execution authorization or a safety guarantee.
Built on Salesforce CodeT5+ 220M by Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Junnan Li and Steven C. H. Hoi (BSD-3-Clause).
Training commands come from tldr-pages (CC BY 4.0, © the tldr-pages team and contributors), botocore (Apache-2.0, © Amazon.com, Inc. or its affiliates), the Azure CLI and GitHub CLI (MIT), kubectl and Docker CLI (Apache-2.0), and ShellRisk-Bench by Kontext Security, including Atomic Red Team (MIT, © Red Canary) and InternalAllTheThings (no license declared), and Shell Safety v2 by tomngdev (MIT), plus the datasets credited in the notices. None of them endorses LANCET.
Model: Apache-2.0. Runtime: MIT. Datasets are not included or relicensed.
See the NOTICE, the third-party notices and the licenses.