osrforge.evals
Eval scoring: truth-file models, alignment, and the pinned metric families.
This is the pure half of the eval harness — extraction changes are measured,
not vibed: deterministic, CI-tested code that scores a
workdir's stage caches against verified structural ground truth. The
live-network driver (tools/eval/run_eval.py) is repo-only wiring; everything
with behavior worth testing lives here. The scorer reads the stage caches —
never adventure.json — because evals measure extraction, and assembly's
best-effort fallbacks exist to mask extraction gaps in the playable draft,
which is exactly what a measurement must not let them do.
Truth files are structural-only (printed keys, names, and codes — no prose)
and are authored from the printed module under the independence discipline
(tools/eval/AUTHORING.md) — never from pipeline output; see
tools/eval/README.md for the corpus rules and the authoring conventions.
AreaMetrics
Bases: BaseModel
The areas family: recall (the headline metric), the hallucination guard, and dungeon alignment.
The dungeon counts make the survey mode legible in every scoreboard entry
— a measured mode-flip (ten lairs collapsing into one dungeon on a
re-roll of the same module) reads as truth_dungeons=14,
extracted_dungeons=5 instead of requiring a trip to survey.json.
extracted_dungeons
instance-attribute
extracted_dungeons: int
How many dungeons the survey extracted.
matched_dungeons
instance-attribute
matched_dungeons: int
How many truth dungeons aligned to an extracted dungeon.
truth_areas
instance-attribute
truth_areas: int
How many keyed areas the truth asserts, across all its dungeons — aligned or not, so a whole missed dungeon depresses recall.
extracted_areas
instance-attribute
extracted_areas: int
How many keyed areas extraction produced, across all extracted dungeons.
matched
instance-attribute
matched: int
How many truth areas matched an extracted area (matching happens within aligned dungeons).
precision
instance-attribute
precision: float | None
matched / extracted_areas — the hallucination guard; None on an
empty denominator.
ByomEntry
Bases: BaseModel
One published BYOM record: aggregate-only by construction.
Identity metadata (cross-copy: title, publisher, edition, pages), the run
block, the truth-file hash, the non-default knobs, and the metrics —
nothing else. No PDF hash (copy-specific, meaningless cross-customer), no
license claims, no module text. truth_sha256 is the yardstick pin:
watermark-proof because it hashes the owner's YAML, and its job is
distinguishing "the extraction moved" from "the truth moved" between
entries.
ByomScoreboard
Bases: BaseModel
The committed BYOM scoreboard: advisory, aggregate-only, owner-refreshed.
Answers "how does it perform in general," not "may this PR merge" — the
regression rule binds the corpus scoreboard, never this one. Entries
refresh best-effort by whoever owns the module; a stale entry is visible
via its osrforge_version stamp, never blocking.
ConnectionMetrics
Bases: BaseModel
The connections family: F1 over undirected same-level edges in the asserted universe.
extracted_edges
instance-attribute
extracted_edges: int
How many undirected edges extraction produced within the asserted universe.
precision
instance-attribute
precision: float | None
true_positives / extracted_edges; None on an empty denominator.
recall
instance-attribute
recall: float | None
true_positives / truth_edges; None on an empty denominator.
f1
instance-attribute
f1: float | None
The harmonic mean of precision and recall; None when either is.
CorpusManifest
Bases: BaseModel
A corpus member's manifest — the whole redistribution surface.
The corpus ships pointers plus hashes, never PDFs. Identity and integrity
split for watermarked retail PDFs (the same module hashes differently per
customer): cross-copy identity is metadata (title, publisher,
edition, pages), while integrity is sha256 when pinned (every
committed member — the harness refuses a mismatched file before any model
spend) or the local source.sha256 sidecar when not (the
watermarked-retail case; seeded the first time the harness sees the
module's source). license is optional because a private corpus is the
owner's copy with no redistribution surface — the license-verification
procedure applies only where something derived will be committed.
DoorMetrics
Bases: BaseModel
The doors family: presence recall and precision plus kind/locked accuracy, over the asserted door universe.
The universe is undirected edges between matched areas with at least one
endpoint asserting doors — an asserting area's door set is complete, so
an extracted door its truth omits is a false positive, and a door fact
stated on neither directed mention is a miss. The extracted facts flow
through the edge-fact seam merged with the map reading
(merge_level_edges) — the fact
source phase 11 swapped in without touching these semantics.
truth_doors
instance-attribute
truth_doors: int
How many door edges the truth asserts within the universe.
extracted_doors
instance-attribute
extracted_doors: int
How many door edges extraction stated within the universe.
recall
instance-attribute
recall: float | None
true_positives / truth_doors; None on an empty denominator.
precision
instance-attribute
precision: float | None
true_positives / extracted_doors; None on an empty denominator.
kind_matched
instance-attribute
kind_matched: int
How many true positives agree on kind (door / secret_door).
kind_accuracy
instance-attribute
kind_accuracy: float | None
kind_matched / true_positives; None on an empty denominator.
locked_matched
instance-attribute
locked_matched: int
How many true positives agree on the locked condition.
locked_accuracy
instance-attribute
locked_accuracy: float | None
locked_matched / true_positives; None on an empty denominator.
EncounterMetrics
Bases: BaseModel
The encounters family: name recall and precision, count accuracy, resolution accuracy, custom-emission accuracy.
The custom pair scores the truth's custom: true assertions against the
stat-block cache, so emission is its own legible number rather than
diluting SRD-resolution accuracy; non_srd keeps meaning "no SRD
template and no assertion about emission." The precision triple rides the
asserted-empty convention: it counts only over matched areas whose truth
asserts encounters, because only there is an extracted name that matches
nothing provably a hallucination rather than an unasserted fact.
name_matched
instance-attribute
name_matched: int
How many truth encounters matched an extracted name in their area.
name_recall
instance-attribute
name_recall: float | None
name_matched / truth_encounters; None on an empty denominator.
precision_denominator
instance-attribute
precision_denominator: int
How many distinct folded extracted names appear in matched areas whose truth asserts encounters.
precision_matched
instance-attribute
precision_matched: int
How many of those folds appear in their truth area's fold set.
precision
instance-attribute
precision: float | None
precision_matched / precision_denominator; None on an empty
denominator.
count_denominator
instance-attribute
count_denominator: int
How many name-matched truth encounters assert a count.
count_matched
instance-attribute
count_matched: int
How many asserted counts the extraction reproduced.
count_accuracy
instance-attribute
count_accuracy: float | None
count_matched / count_denominator; None on an empty denominator.
resolution_denominator
instance-attribute
resolution_denominator: int
How many name-matched truth encounters assert an SRD template.
resolution_matched
instance-attribute
resolution_matched: int
How many asserted templates resolution reproduced.
resolution_accuracy
instance-attribute
resolution_accuracy: float | None
resolution_matched / resolution_denominator; None on an empty
denominator.
custom_denominator
class-attribute
instance-attribute
custom_denominator: int = 0
How many name-matched truth encounters assert custom emission
(custom: true).
custom_matched
class-attribute
instance-attribute
custom_matched: int = 0
How many custom assertions have a usable cached stat block.
custom_accuracy
class-attribute
instance-attribute
custom_accuracy: float | None = None
custom_matched / custom_denominator; None on an empty denominator.
non_srd
instance-attribute
non_srd: int
Truth encounters asserting no SRD template and nothing about emission.
EntranceMetrics
Bases: BaseModel
The entrance family: does the pipeline's entrance selection pick the printed way in?
Scores geometry's own selection through the shared
select_entrance — a resolvable
map proposal beating the positional heuristic — so the metric and
geometry can never pick differently.
ManifestLicense
ModuleMetrics
ModuleScore
Bases: BaseModel
One module's scoreboard entry: the run that produced it, its yardstick, its knobs, and its metrics.
truth_sha256 hashes the truth.yaml the metrics were scored against —
recorded at score time so a truth edit between scoring and publishing is
detectable, and the published pin always names the yardstick that
actually produced the numbers. settings_overrides echoes the scored
workdir's non-default ConversionSettings knobs as key=value strings
(knob names and page numbers, never module text) — a run measured with,
say, a blanked page is visible in the record instead of being an
invisible special condition.
ModuleTruth
Bases: BaseModel
A corpus module's verified structural ground truth.
dungeons
class-attribute
instance-attribute
dungeons: tuple[TruthDungeon, ...] = Field(min_length=1)
RunInfo
Scoreboard
Bases: BaseModel
A corpus's scoreboard: per-module scores keyed by corpus module id, sorted for byte stability.
TransitionMetrics
Bases: BaseModel
The transitions family: dungeon-scoped vertical links matched on truth endpoint pairs.
Dungeon-scoped because levels are peers: a claim's endpoints resolve
through the pairing claims to truth (level, key) addresses, so a
printed inter-level stair survives the JN2 collapsed-level shape (one
extracted level holding two printed levels) and the cross-level keyed
shape alike. Kind is a resolved attribute of the merged link — matching
is by endpoints alone, so the kind row can genuinely miss.
asserted_dungeons
instance-attribute
asserted_dungeons: int
How many aligned dungeons assert transitions.
truth_transitions
instance-attribute
truth_transitions: int
How many vertical links those dungeons' truth asserts.
extracted_transitions
instance-attribute
extracted_transitions: int
How many deduplicated vertical links extraction claimed in those dungeons.
recall
instance-attribute
recall: float | None
true_positives / truth_transitions; None on an empty denominator.
precision
instance-attribute
precision: float | None
true_positives / extracted_transitions; None on an empty
denominator.
kind_matched
instance-attribute
kind_matched: int
How many true positives agree on kind (stairs / trapdoor /
chute).
kind_accuracy
instance-attribute
kind_accuracy: float | None
kind_matched / true_positives; None on an empty denominator.
TreasureMetrics
Bases: BaseModel
The treasure family: presence agreement and letter accuracy.
presence_denominator
instance-attribute
presence_denominator: int
How many matched areas the truth asserts treasure presence (or absence) for.
presence_matched
instance-attribute
presence_matched: int
How many of those the extraction agreed with.
presence_agreement
instance-attribute
presence_agreement: float | None
presence_matched / presence_denominator; None on an empty
denominator.
letters_denominator
instance-attribute
letters_denominator: int
How many treasure-type letters the truth asserts on matched areas.
letters_matched
instance-attribute
letters_matched: int
How many asserted letters the extraction reproduced.
letter_accuracy
instance-attribute
letter_accuracy: float | None
letters_matched / letters_denominator; None on an empty
denominator.
TruthArea
Bases: BaseModel
One keyed area, identified by its printed key.
encounters, connections, doors, and treasure are assertion-aware:
None (omitted) means the fact was not asserted — the area is out of the
corresponding metric's universe. A present value asserts the complete
fact set: the area's full encounter list (possibly empty — the
asserted-empty convention the encounter-precision metric rides), its full
same-level connected printed-key list (possibly empty), the complete
door-fact set over its asserted connections, or its treasure facts.
Assertion-awareness is what makes time-boxed partial truth honest: a truth
file covering every area key plus a verified sample of areas still yields
exact area recall and honestly-denominated agreement everywhere else.
doors is keyed by neighbor printed key; every key must appear in the
same area's connections (slug-matched), and asserting doors requires
asserting connections — a door fact on an unasserted edge set would
have no universe to score in.
TruthDoor
Bases: BaseModel
A door fact asserted on one of an area's connections.
kind mirrors the extraction contract's door vocabulary (door /
secret_door); locked is asserted alongside it. Stuck state is
deliberately not asserted — the corpus carries almost no printed signal
for it, so a stuck metric would run on an empty denominator.
TruthDungeon
Bases: BaseModel
One printed adventuring site.
transitions and entrance are assertion-aware, dungeon-scoped facts:
omitted asserts nothing; a present transitions asserts the dungeon's
complete vertical-link set (levels are peers, so an edge between them
belongs to the dungeon, not to either level), and a present entrance
asserts which printed area holds the way in.
transitions
class-attribute
instance-attribute
transitions: tuple[TruthTransition, ...] | None = None
TruthEncounter
Bases: BaseModel
One printed encounter: the creature name as the module's key prints it.
template is the osrlib catalog id the name should resolve to, omitted
when the module's monster genuinely has no SRD template (rank variants
with their own stat blocks, module-specific creatures). custom is legal
only with template omitted: it asserts this creature should emit — the
printed page carries a usable stat block (an AC plus an HD line or a
class-level notation, exactly assembly's refusal-ladder predicate).
Omitted-with-custom moves the encounter into the custom metric pair;
omitted-without stays non_srd — no SRD template and no assertion about
emission. count is omitted when the module states none or a variable
one.
TruthEntrance
TruthLevel
Bases: BaseModel
One printed level.
Area keys must be unique per level under canonical_slug (empty slugs
are exempt — they take distinct positional fallbacks): the scorer matches
areas by slug, and a duplicate would silently attribute the second area's
facts to the first.
TruthProvenance
Bases: BaseModel
How the module's truth file came to be trusted.
The record publish requires before a module's numbers reach the
committed BYOM board: unverified truth can be scored locally all day, but
it cannot put numbers on the committed record. instrument is free text
by design — the cross-instrument rule (tools/eval/AUTHORING.md) is a
stated preference, not a gate, because enforcement is impossible and
false assurance is worse than none.
TruthTransition
Bases: BaseModel
One vertical link between two levels of a dungeon, asserted once.
Endpoint order is free — matching is undirected — and kind uses
geometry's own narrowing (trapdoor and chute as themselves,
everything else stairs). The travel sense is deliberately not asserted.
TruthTreasure
Bases: BaseModel
Whether the printed area contains treasure, and its stated letter codes.
present is true when the entry states coins, valuables, or magic items
in the area (carried by its occupants included; rewards promised
elsewhere excluded). letters only when the module states treasure-type
letter codes.
corpus_means
corpus_means(scoreboard: Scoreboard) -> dict[str, float | None]
The corpus mean of each headline metric, over modules where it is defined.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
scoreboard
|
Scoreboard
|
The scoreboard to summarize. |
required |
Returns:
| Type | Description |
|---|---|
dict[str, float | None]
|
Metric name → mean (None when no module defines it). |
enforce_source_integrity
enforce_source_integrity(manifest: CorpusManifest, module_dir: Path, digest: str, described: str) -> bool
Enforce the truth-to-source chain of custody for one observed digest.
The manifest's sha256 pin gates when present (every committed member);
otherwise the local sidecar gates, seeded on first sight — at convert
(from the PDF) or, for a workdir converted outside the harness, at first
score (from run.json's recorded source hash). The chain runs unbroken
from the file the truth was authored against to any published number.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
manifest
|
CorpusManifest
|
The module's manifest. |
required |
module_dir
|
Path
|
The corpus member's directory (where the sidecar lives). |
required |
digest
|
str
|
The observed source sha256 (from the PDF or from |
required |
described
|
str
|
What was hashed, for the refusal message. |
required |
Returns:
| Type | Description |
|---|---|
bool
|
True when this call seeded the sidecar (the harness's first sight of |
bool
|
the module's source); False when the digest matched an existing gate. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the digest matches neither the manifest pin nor the sidecar — the source is not the file the truth was authored against. |
load_byom_scoreboard
load_byom_scoreboard(path: Path) -> ByomScoreboard
Load the committed BYOM scoreboard.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
Path
|
The |
required |
Returns:
| Type | Description |
|---|---|
ByomScoreboard
|
The board; an empty one if the file does not exist yet. |
load_manifest
load_manifest(path: Path) -> CorpusManifest
Load and validate a corpus manifest.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
Path
|
The |
required |
Returns:
| Type | Description |
|---|---|
CorpusManifest
|
The validated manifest. |
load_scoreboard
load_scoreboard(path: Path) -> Scoreboard
Load a corpus's scoreboard.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
Path
|
The |
required |
Returns:
| Type | Description |
|---|---|
Scoreboard
|
The scoreboard; an empty one if the file does not exist yet. |
load_truth
load_truth(path: Path) -> ModuleTruth
Load and validate a corpus truth file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
Path
|
The |
required |
Returns:
| Type | Description |
|---|---|
ModuleTruth
|
The validated truth. Unknown keys are rejected — a typo in a |
ModuleTruth
|
hand-authored truth file must fail loudly, not silently drop a fact. |
publish_module
publish_module(
board: ByomScoreboard,
module_id: str,
manifest: CorpusManifest,
private_board: Scoreboard,
current_truth_sha256: str,
committed_ids: Collection[str],
) -> ByomScoreboard
Copy one private scoreboard entry onto the committed BYOM board.
The deliberate, outward-facing act, separate from scoring. The chain of custody holds because only scored entries are copied, scoring runs under the source-integrity check, and the published yardstick pin is the hash recorded at score time — a truth edited after scoring is refused, not silently paired with stale metrics.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
board
|
ByomScoreboard
|
The current committed BYOM board. |
required |
module_id
|
str
|
The private corpus module id to publish. |
required |
manifest
|
CorpusManifest
|
The module's manifest (identity plus provenance). |
required |
private_board
|
Scoreboard
|
The private corpus's scoreboard. |
required |
current_truth_sha256
|
str
|
The hash of the module's |
required |
committed_ids
|
Collection[str]
|
The committed corpus's module ids (the shared-namespace guard). |
required |
Returns:
| Type | Description |
|---|---|
ByomScoreboard
|
A new board with the module's entry added or replaced. |
Raises:
| Type | Description |
|---|---|
ValueError
|
On any pinned refusal — no scored entry for the id, a truth file that changed since scoring, missing truth provenance, id collision with a committed corpus member, or (on update) a title mismatch with the entry being replaced. |
rescore_modules
rescore_modules(board_path: Path, targets: Sequence[tuple[str, Path, Path]]) -> dict[str, ModuleMetrics]
Re-score existing scoreboard entries' workdirs against current truth — one rebuilt board, one save.
The offline-regeneration mechanism: same runs, re-scored. The stale board
is read as raw JSON — never through the models, whose required fields a
pre-extension entry cannot satisfy — and each rebuilt entry carries its
raw run block verbatim (the run's own date, tokens, cost, model id,
package version: the entry stays a record of the run) plus the carried
settings_overrides, a freshly pinned truth_sha256, and the fresh
metrics from score_workdir. The entire
rebuilt board then validates through the extended models before the
single save, so a save never persists a board the current schema rejects.
Taking the targets together is what makes a schema migration saveable at all: a board whose every entry predates a model extension can only become valid whole, so its entries rebuild in one invocation and the board saves once. A leftover entry the extended models still refuse is a loud refusal naming the stale ids — the whole-board pin working, not an obstacle to route around. A single target stays valid whenever the rest of the board is already current-shaped.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
board_path
|
Path
|
The corpus's |
required |
targets
|
Sequence[tuple[str, Path, Path]]
|
|
required |
Returns:
| Type | Description |
|---|---|
dict[str, ModuleMetrics]
|
Module id → the fresh metrics written into its entry, in target |
dict[str, ModuleMetrics]
|
order. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If no targets are given, an id repeats, the board does not exist, an id holds no entry, or the rebuilt board still contains stale entries (named in the message) — nothing is written. |
ValidationError
|
If the rebuilt board fails the extended models for any other reason — nothing is written. |
save_byom_scoreboard
save_byom_scoreboard(path: Path, board: ByomScoreboard) -> None
Write the BYOM scoreboard in the pinned artifact byte format.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
Path
|
The |
required |
board
|
ByomScoreboard
|
The board to persist. |
required |
save_scoreboard
save_scoreboard(path: Path, scoreboard: Scoreboard) -> None
Write the scoreboard in the pinned artifact byte format.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
Path
|
The |
required |
scoreboard
|
Scoreboard
|
The scoreboard to persist. |
required |
score_workdir
score_workdir(workdir_path: Path, truth: ModuleTruth) -> ModuleMetrics
Score one converted workdir's stage caches against a module's truth.
Reads stages/survey.json (area recall/precision, the entrance
selection), the stages/areas.*.json content caches (encounters,
connections, doors, transitions, treasure), stages/monsters.json
(resolution accuracy), stages/statblocks.json (custom-emission
accuracy — a missing file scores no matches, the honest state of a
workdir converted before the stat-block pass existed, never an error),
and stages/mapread.json (the map readings the edge and entrance
families reconcile with — absent tolerated under the same posture: the
honest state of an older workdir, scored prose-only, never an error).
Deterministic: scoring the same workdir twice yields byte-identical
metrics.
Encounter names match under a minimal morphological fold (_match_fold) —
the truth's singular authoring convention meets extraction's printed
plural on folded forms; a truth encounter's count compares against the
fold-matched encounter group's summed fixed counts, and its resolution
matches only when every fold-matched extracted name resolved to the
asserted template. A custom-asserted encounter matches only when every
fold-matched extracted name carries a usable block in the stat-block
cache — usability being exactly assembly's refusal-ladder predicate,
shared as one helper, so the metric can never score an emission assembly
would refuse; the signal is honest by construction, because the pass only
runs over unresolved names, so a wrongly-SRD-resolved bespoke creature
has no block and scores a miss. An area whose truth lists one name twice
(two separately statted groups printed under one name) scores each entry
against the whole group — the summed count can then match neither entry
and resolution can credit at most one of the two templates; a known
conservative shape, recorded rather than special-cased.
The edge families ride the edge-fact seam (_edge_facts) rerouted
through merge_level_edges: the
prose facts merge with the pairing level's map reading — endpoints
resolved exactly-then-slug, map-only pairs filtered to matched slugs —
and the merged set flows through the unchanged asserted-universe gates,
so map-only edges and map-adopted doors enter exactly where truth
asserts. Transitions and the entrance score dungeon-scoped, after the
level loop, over each aligned dungeon's recorded pairing claims — the
entrance through the shared
select_entrance, so the metric
and geometry can never pick differently.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
workdir_path
|
Path
|
A workdir whose extraction stages have completed. |
required |
truth
|
ModuleTruth
|
The module's ground truth. |
required |
Returns:
| Type | Description |
|---|---|
ModuleMetrics
|
The seven metric families. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If a required stage cache is missing. |
Examples:
settings_overrides
settings_overrides(settings: ConversionSettings) -> tuple[str, ...]
The non-default conversion knobs, as key=value strings in field order.
Values render as YAML-parseable text (blank_page_renders=[21],
render_dpi=300) — the same shape the --set flag was typed with. Knob
names and numbers only, never module text.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
settings
|
ConversionSettings
|
The settings echoed in a scored workdir's |
required |
Returns:
| Type | Description |
|---|---|
tuple[str, ...]
|
One |
sidecar_path
The module's local integrity sidecar: the copy-specific source hash.
Only meaningful for manifests without a sha256 pin (watermarked retail
PDFs hash differently per customer, so a committed pin would be
meaningless); its one job is proving later re-runs score the same file
the truth was authored against. Never committed for repo corpus members —
they all pin.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
module_dir
|
Path
|
The corpus member's directory. |
required |
Returns:
| Type | Description |
|---|---|
Path
|
|
verify_source
verify_source(manifest: CorpusManifest, module_dir: Path, pdf_path: Path) -> bool
Hash a local PDF and enforce the chain of custody, before any model spend.
Truth authored against one printing scores a different printing as noise — the hash, never the URL, is the integrity gate.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
manifest
|
CorpusManifest
|
The module's manifest. |
required |
module_dir
|
Path
|
The corpus member's directory. |
required |
pdf_path
|
Path
|
The locally downloaded PDF. |
required |
Returns:
| Type | Description |
|---|---|
bool
|
True when the call seeded the module's sidecar (first sight). |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the file is not the source the truth was authored against. |