bench/dev.gmitch215.drift.bench/Analysis

Analysis

What the dev pairs support beyond a point estimate: template-clustered intervals, the baselines the shipped order is compared with, the sample sizes the claims would need, bundle honesty against the declared members and the false-CONFIRMED bounds.

Properties

BASELINE_IDS

val BASELINE_IDS: List<String>

RESAMPLES

const val RESAMPLES: Int

SOLVE_CONFIRMED

const val SOLVE_CONFIRMED: Int = 59

SOLVE_FALSE_CONFIRMED

const val SOLVE_FALSE_CONFIRMED: Int = 0

SOLVE_RUNS

const val SOLVE_RUNS: Int = 180

The Lab repeats on the 45 dev scenarios; the certificates are not in the repository.

SOLVE_SCENARIOS

const val SOLVE_SCENARIOS: Int = 45

SOLVE_STUCK_FIRST_RUN

const val SOLVE_STUCK_FIRST_RUN: Int = 21

Z_ONE_SIDED

const val Z_ONE_SIDED: Long

Z_POWER

const val Z_POWER: Long

Z_TWO_SIDED

const val Z_TWO_SIDED: Long

Functions

bootstrap

fun bootstrap(shipped: List<Pred>, others: List<Pair<Model, List<Pred>>>, seed: Long): JsonObject

Percentile intervals (2.5 to 97.5) over RESAMPLES draws of whole templates with replacement, the same draws for every model so the differences from the shipped order are paired. Scenarios without a cause do not enter.

bundleHonesty

fun bundleHonesty(preds: List<Pred>): JsonObject

The bundle class scored against its declared members: a ranked bundle should hold the cause and the members the template declares, and every other member is a cost.

falseConfirmed

fun falseConfirmed(): JsonObject

halfWidth