bench/dev.gmitch215.drift.bench

Package-level declarations

Types

What the dev pairs support beyond a point estimate: template-clustered intervals, the baselines the shipped order is compared with, the sample sizes the claims would need, bundle honesty against the declared members and the false-CONFIRMED bounds.

Arm

class Arm(val image: String, val env: Map<String, String>, val memory: String, val cpus: String, val cpuset: String?, val ulimits: Map<String, String>, val network: String, val files: Map<String, String>, val interpreter: String?, val args: List<String>, val gate: Int, val rate: Int, val shell: String? = null, val workspace: Map<String, String> = emptyMap())

ArmRun

class ArmRun(val capsule: String?, val trials: List<TrialOutcome>, val done: Boolean)

Attr

class Attr(val dimension: String, val path: String)

Binomial

object Binomial

Exact binomial tails with a permille success rate, in integers.

Calibrate

object Calibrate

Calibration

class Calibration(val fit: String, val evaluation: String, val chosen: Model)

The result of one calibration run: both files as canonical JSON text.

Cand

class Cand(val path: String, val dimension: Dimension, val probe: Boolean, val rules: List<Severity>, val tier: Tier, val bundled: Boolean, val rankerScore: Long, val probability: Long?, val label: Boolean, val decoy: Boolean, val verifiedRule: Boolean = false, val dimensionProbe: Boolean = false, val observationalProbe: Boolean = false, val mechanisticRule: Boolean = false)

One ranked candidate with the facts the weights act on, and the scenario's label for it. probe is a differing probe that reads the candidate's path, dimensionProbe one that only shares its dimension.

Capsules

object Capsules

Cause

class Cause(val dimension: String, val path: String, val attribute: String, val direction: String)

Count

class Count(val causes: Int, val total: Int)

Candidates in a group and how many of them are the cause.

Diagnosis

class Diagnosis(val ranked: List<String>, val confirmed: String? = null)

A diagnoser's answer: attribute paths best first, and the one it called CONFIRMED, if any.

Dimensions

object Dimensions

Entry

class Entry(val path: String, val label: Boolean, val decoy: Boolean, val tier: Tier?, val bundled: Boolean, val p: Long, val ruled: Boolean = false, val verified: Boolean = false)

A candidate as one model ordered it, with the probability that model gave it.

Evaluate

object Evaluate

EvidenceEval

object EvidenceEval

The shipped ranker against the one that classes probe and rule evidence by the observed failure, with the dimension prior beside them, on the same scenarios and the same template bootstrap as Analysis.

Fields

class Fields(val where: String, val json: JsonObject)

Typed reads over a JsonObject; scalars may be strings (YAML failsafe) or JSON numbers.

Fit

object Fit

FlagScore

class FlagScore(val flag: String, val reported: Int, val confirmed: Int, val onControl: Int, val precision: Rate, val recall: Rate)

reported counts the findings of the group on break moves and onControl on control moves; confirmed are the break-move findings that name the measured failure. Precision is confirmed over every reported finding; recall is the share of break items with a confirmed finding.

Folds

object Folds

Freeze

object Freeze

The frozen state of the benchmark: data/freeze.json holds the engine commit and one value per key of Work.currentFreeze; an unseal needs every value to match the working tree.

Gate

object Gate

The per-trial fault gate. harness.sh and harness.ps1 compute the same integer function (--hash and -Hash print it), so the trials that fail are known before anything runs.

Generator

object Generator

Harness

object Harness

Text of the files the runner hands to a container: the per-arm plan and the docker command.

Hashing

object Hashing

Ingest

object Ingest

Turns the raw output of both arms into a validation verdict and the capsules to keep.

Inputs

class Inputs(val probed: Lab, val original: Lab, val results: (String) -> String? = null)

The inputs of a run: the capsule pairs with probes (Inputs.probed, the fit and the primary evaluation) and the first attribute-only pairs (Inputs.original, for the before numbers). Inputs.results reads <id>.json of data/results; with it the evaluation also scores the ranker conditioned on the observed failure (see EvidenceEval).

Interval

class Interval(val low: Long, val high: Long)

Isotonic

class Isotonic(val steps: List<Pair<Long, Long>>)

Pool-adjacent-violators over (score, label) pairs; a step function in micro-units.

Lab

class Lab(val scenarios: List<Scenario>, val dev: Set<String>, capsule: (String) -> String, val scenarioListSha256: String, open: Set<String> = emptySet(), val excluded: Set<String> = emptySet())

The dev scenarios and their capsules. Test ids are refused here unless the lab was built with Unsealed, so no fit or evaluation path can reach a sealed scenario before the unseal.

Learner

fun interface Learner

Metrics

object Metrics

Fixed-point metrics in millionths, identical on every target.

Model

class Model(val id: String, val note: String, val tiered: Boolean, val learner: Learner, val table: (List<Observed>) -> Table? = null, val mapped: Boolean = false, val normalized: Boolean = false, val fitsWeights: Boolean = false, val none: (List<Observed>) -> Long? = null)

Models

object Models

Observed

class Observed(val scenario: Scenario, val changed: List<String>, val ranked: List<Cand>, val probes: List<String> = emptyList(), val rules: List<String> = emptyList())

What the real pipeline produced for one dev scenario: the changed attributes, the probes that differ and the rules that matched (ids), and the ranked candidates.

Override

class Override(val path: String, val green: String, val red: String)

Pred

class Pred(val o: Observed, val ranked: List<Entry>, val random: Boolean = false)

Pred.random says the order is only a placeholder: hit values are the expectation over orders.

ProbedImport

data class ProbedImport(val imported: List<String>, val skipped: Map<String, List<String>>)

The cases probed wrote and the ones it skipped, each with the arms it lacks.

Problem

enum Problem : Enum<Problem>

Rate

class Rate(val count: Int, val total: Int)

Scenario

class Scenario(val id: String, val template: String, val title: String, val dimension: String, val salt: Long, val runtime: String, val source: String, val signature: String, val signatureExit: Int?, val trials: Int, val green: Arm, val red: Arm, val cause: Cause?, val bundle: List<Attr>, val decoys: List<Attr>, val visible: Boolean, val controllable: Boolean, val overrides: List<Override>, val note: String)

SealedException

class SealedException(id: String) : Exception

SolveBench

object SolveBench

SolveCase

class SolveCase

SolvedCase

class SolvedCase(val id: String, val certificate: String?)

One solved case directory: its id and the text of certificate.json, null when unreadable.

SolveOutcome

enum SolveOutcome : Enum<SolveOutcome>

SolveRanks

class SolveRanks(val ranker: Int?, val prior: Int?)

1-based rank of the cause in an order, null when the order does not hold it.

SolveScore

class SolveScore(val cases: List<SolveCase>, val notSolved: List<String>)

The dev drift solve certificates scored against the injected causes. One class block per scenario class and all, with Wilson intervals; see bench/README.md.

SolveTruth

class SolveTruth(val id: String, val klass: String, val cause: String?, val bundle: List<String> = emptyList())

The label of a solved scenario: its class, the injected cause path (null for a control) and the attributes the scenario declares as travelling with the cause.

Split

class Split(val dev: List<String>, val test: List<String>)

SplitMix64

class SplitMix64(seed: Long)

SplitMix64 on Long arithmetic, so every target draws the same sequence for a seed.

SplitRule

object SplitRule

Table

class Table(val dimensions: Map<Dimension, Long>, val probe: Long, val rules: Map<Severity, Long>)

The ranker's evidence weights as likelihood ratios in micro-units (spectrum is not fit).

Template

class Template(val id: String, val instances: Int, val variants: List<Map<String, String>>, val draw: Map<String, List<String>>, body: JsonObject)

A hand-maintained scenario template. variants are rows of placeholder values (a seeded shuffle picks instances of them), draw placeholders are drawn independently per instance.

TemplateException

class TemplateException(val problem: Problem, val where: String, detail: String) : Exception

TriageBench

object TriageBench

Triage scoring; a separate category that never enters the environment tables.

TriageData

class TriageData(root: Path)

Builds triage/triage.json from triage/items.yml and the bodies under fixtures/triage.

TriageItem

class TriageItem(val id: String, val source: String, val body: String, val facts: Set<String>, val claims: Set<String>, val correction: String?)

One labelled issue body. facts are key=value (or key=value@passing) strings, claims the claim ids the claim check should raise and correction what a maintainer would answer. source is fixture; a corpus item would carry its own body.

TriageScore

class TriageScore(val items: Int, val precision: Rate, val recall: Rate, val brier: Long, val brierLow: Long, val brierHigh: Long, val facts: Int, val claimPrecision: Rate, val claimRecall: Rate)

TrialOutcome

class TrialOutcome(val index: Int, val exit: Int, val flake: Boolean, val signature: Boolean, val tail: String)

Truth

class Truth(val path: String?)

What the scenario's label says: a cause path, or null for a control.

Unsealed

class Unsealed

Proof that the frozen files matched at unseal; only Work creates one.

UnsealException

class UnsealException(message: String) : Exception

UpgradeBench

object UpgradeBench

Upgrade-check scoring over the migration scenarios; a separate category that never enters the environment tables. Dev items read the committed capsules through a Lab; test items need the Unsealed proof.

UpgradeItem

class UpgradeItem(val scenario: String, val template: String, val move: String, val downgrade: Boolean, val input: Capsule, val target: Target, val signature: String, val explained: Boolean)

One move of a migration scenario. A break item takes the passing arm's capsule and moves its cause attribute to the failing arm's value, so the measured red arm is the truth. A control item takes the failing arm's capsule and moves to the passing value: the green arm passed every trial, so nothing breaks. downgrade is true when the target is older than the capsule.

UpgradeResult

class UpgradeResult(val item: UpgradeItem, val report: UpgradeReport)

UpgradeScore

class UpgradeScore(val scenarios: Int, val strata: List<UpgradeStratum>)

UpgradeStratum

class UpgradeStratum(val name: String, val breaks: Int, val controls: Int, val flags: List<FlagScore>)

breaks and controls count the items of the stratum; flags has one row per flag.

Verdict

class Verdict(val valid: Boolean, val reasons: List<String>, val result: JsonObject, val capsules: Map<String, String>)

Work

class Work(root: Path)

File operations behind the three commands; root is the bench/ directory.

Yaml

object Yaml

Every scalar stays a string; Template.listFromJson does the typed reading.

Functions

main

fun main(args: Array<String>)