bench/dev.gmitch215.drift.bench/EvidenceEval

EvidenceEval

The shipped ranker against the one that classes probe and rule evidence by the observed failure, with the dimension prior beside them, on the same scenarios and the same template bootstrap as Analysis.

Functions

failure

fun failure(result: String): List<Symptom>

The failure the red arm showed, from a scenario's data/results text: the last failure line as a normalized signature and each distinct non-zero exit code.

run

fun run(lab: Lab, ids: List<String>, result: (String) -> String, seed: Long): JsonObject

Scores ids of lab (dev ids only); result reads <id>.json from data/results. The weights are hand-set and nothing is fit, so in-sample is the whole result. A subset of the dev ids is labelled partial.