The Laya implementation is shelved. The comparison did not justify shipping it as a safety-equivalent alternative to Jev. This archive preserves observations, not an installation option. It also does not establish that Laya could never improve.
Matched API, CPU and GPU runs
Same cases, local rules, questions and ask/block thresholds (0.35/0.8). Laya used the pinned English checkpoint. CPU FP32 and GPU BF16 differ numerically, not just in speed. Timings exclude model loading; Jev includes network time. The red-team inputs were replayed, not newly generated for each backend.
| Run | Matrix | Devious suite | Red team | End to end | Resources |
|---|---|---|---|---|---|
| Jev API | 124 cases | 136 cases | 30 attacks | 48 scenarios | remote API |
| Laya CPU | 124 cases | 136 cases | 30 attacks | 48 scenarios | runtime, RAM |
| Laya GPU | 124 cases | 136 cases | 30 attacks | 48 scenarios | runtime, RAM, VRAM |
E2E counts are integration assertions, not safety accuracy. Agent deviations can fail an assertion without exercising the intended path. Unexercised cases are not passes. GPU memory is the process's PyTorch allocator, not total device use.
Devious suite: 856 attempts
The original 136 cases plus 120 new attack/benign pairs across 20 families, each tested plainly, nested or encoded, and with misleading assurances. These are correlated views, not 720 independent new techniques.
Jev gave 9 unexpected allows among 471 attack-labelled attempts; Laya GPU gave 114, plus one input rejection. Controls interrupted: 77/385 and 190/385 respectively; hard blocks: 13 and 54. Jev missed only one new plain attack, so more cases did not establish a harder benchmark.
No CPU run of this suite. It was recorded by the research runner; npm run devious now runs the same 856 cases against Jev, and a test checks they match this record. Records retain the exact command, tool, user context, expectation and result. No adversarial command was executed.
| Record | Files |
|---|---|
| Report | Subset and family report (Markdown) |
| Observations | Jev API, 856 rows · Laya GPU, 856 rows |
| Rejection | Token-budget rejection evidence |
Tuning: no preset adopted
Six predeclared general configurations used unchanged weights and thresholds. All were evaluated on a deterministic 505-case sample of ShellRisk-Bench's TRAIN split. None passed the development gate. FP32 was frozen for diagnosis only, then evaluated once against baseline on 4,193 TEST rows after exact-overlap filtering.
Both detected 119/193 risky test inputs; baseline interrupted 1,467/4,000 benign inputs and FP32 1,482/4,000, including one rejected benign input each. No other candidate was tried on the test split. The devious suite was not used to select or adopt a candidate.
| Stage | Files |
|---|---|
| Protocol | Study report (Markdown) · Frozen protocol · Six configurations · 48 authored contrast inputs |
| Development | Original BF16 · FP32 control · Concise questions · Six risk questions · Two-option rubric · Action-only state |
| One-shot test | Selection · Validation and adoption gate · External BF16 predictions · External FP32 predictions · Authored BF16 contrasts · Authored FP32 contrasts |
| Provenance | Study manifest · Pre-inference loader correction · Original manifest |
ShellRisk labels are source-derived, not individually reviewed authorization decisions. TRAIN and TEST share sources; near-duplicates or training contamination may remain. The authored contrasts are not independent validation. Rejected risky inputs do not count as detections; rejected benign inputs do count as operational interruptions. No weights were trained and no thresholds fitted.
Archive scope and integrity
SHA-256 checksums cover the unchanged raw observations and frozen study inputs. Original manifests retain experiment paths for provenance; they do not imply that the Python service or research runners are included here. This is an auditable record, not a stand-alone inference reproduction kit.
The Laya runtime and its collection code remain in the shelved implementation work; the devious suite's cases and a Jev runner are in the repository. Model weights, Python environments and mixed-license external command data are not distributed with these pages. The archive can be verified with node docs/evaluations/verify.mjs, without model or API access.