bench/dev.gmitch215.drift.bench/Evaluate

Evaluate

object Evaluate(source)

Properties

const val BINS: Int = 5

val PROBE_RATIOS: List<Long>

Functions

fun brier(preds: List<Pred>): Long

fun brierCovered(preds: List<Pred>): Long

Brier over the rankings the product prints a probability for.

fun crossValidated(obs: List<Observed>, model: Model, k: Int = Folds.K): List<Pred>

fun entries(obs: List<Observed>): List<Entry>

Candidates as entries without a probability, for checks that need no model.

fun gates(entries: List<Entry>): JsonObject

honesty plus how often the tiers named the injected cause.

fun honesty(entries: List<Entry>): JsonObject

What known and plausible rest on; needs no label, so it holds for any pair.

fun inSample(obs: List<Observed>, model: Model): List<Pred>

fun metrics(preds: List<Pred>, tiered: Boolean): JsonObject

fun model(obs: List<Observed>, m: Model): JsonObject

fun mrr(preds: List<Pred>): Long

fun probeSweep(obs: List<Observed>): JsonObject

Out-of-fold Brier of the shipped readout as the probe ratio varies, over the rankings that carry a probe and over all of them, next to the uniform 1/n readout on the same rankings.

fun tail(p: List<Long>, k: Int): Long

The chance, in micro-units, of at least k hits when scenario i hits with probability p[i] (micro-units); the exact Poisson-binomial tail.

fun versusRandom(obs: List<Observed>): JsonObject

The shipped order against a random order of the same candidates: hits, the hits a random order is expected to make, and the chance that a random order makes at least as many.