`osrforge.survey`

Stage 1: the survey pass — one structured-output request over the whole module, or chunked page windows.

The survey identifies title, hooks, town info, dungeons, levels, keyed areas with page locations, and every monster name — the index that plans the content passes. Normalization to canonical ids and keys happens here, at the source: contracts/report.py pins the address grammar, and the content stage's cache filenames and per-batch key enums need canonical forms before content runs.

Sources at or under survey_max_pages pages survey in one request whose bytes are identical to what every committed fixture was recorded against — chunking is purely additive. Larger sources split into contiguous page windows of that size, each request carrying a window-naming preamble, and the windows' raw answers merge deterministically (merge_survey_answers) before one normalize_survey pass.

The census is the survey checking itself: a second, reduced request over the same page windows (a distinct short system prompt, a strict projection of the survey schema — sites, levels, and printed key ranges only), merged by its own census-shaped merge and compared deterministically against the normalized index (census_disputes). Disagreements land on the cache; assembly turns them into module-scope survey_disputed flags. A mode-flipped survey thereby flags itself in the run it happens in.

Every system-prompt rule is pinned against an observed extraction failure in recorded runs; a prompt edit strands the recorded fixtures and re-runs the eval sweep — see the re-record rule before changing one.

CENSUS_SCHEMA `module-attribute`

CENSUS_SCHEMA: dict[str, object] = {
    "type": "object",
    "properties": {
        "dungeons": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "name": {"type": "string"},
                    "levels": {
                        "type": "array",
                        "minItems": 1,
                        "items": {
                            "type": "object",
                            "properties": {
                                "number": {"type": "integer"},
                                "first_key": {"type": "string"},
                                "last_key": {"type": "string"},
                            },
                            "required": ["number", "first_key", "last_key"],
                            "additionalProperties": False,
                        },
                    },
                },
                "required": ["name", "levels"],
                "additionalProperties": False,
            },
        }
    },
    "required": ["dungeons"],
    "additionalProperties": False,
}

The census schema: a strict projection of the survey's — dungeons, levels, and printed key ranges only.

No per-area records, no town, no monster names — the census is the survey checking itself, and the ranges are collected to be used: forcing the model to commit to specifics is itself part of the census's grounding.

CENSUS_SYSTEM `module-attribute`

CENSUS_SYSTEM = 'You take a census of a tabletop adventure module\'s keyed adventuring sites. The user message interleaves every page\'s extracted text (each headed by a [page N] marker) with that page\'s image. Answer only what the census schema asks: the dungeons, their levels, and each level\'s first and last printed area keys.\n\nRules:\n- A dungeon is a keyed adventuring site: caves, ruins, lairs, and the like. A dungeon exists only where the module prints a keyed area list for it. The town or home base and its buildings are never dungeons.\n- One dungeon per independently keyed site: separate lairs or sites with their own maps and entrances are separate dungeons, even when they are drawn together on one regional map or share a running area-number sequence; maps connected internally by stairs or shafts are levels of one dungeon.\n- Each level\'s "first_key" and "last_key" are the first and last printed area keys in that level\'s keyed list, exactly as printed (like "1" or "4a"). A level\'s key range comes from its printed key list, never from the map alone.\n'

SURVEY_SCHEMA `module-attribute`

SURVEY_SCHEMA: dict[str, object] = {
    "type": "object",
    "properties": {
        "title": {"type": "string"},
        "description": {"type": "string"},
        "hooks": {"type": "array", "items": {"type": "string"}},
        "town": {
            "type": "object",
            "properties": {
                "name": {"type": "string"},
                "description": {"type": "string"},
                "services": {"type": "array", "items": {"type": "string"}},
            },
            "required": ["name", "description", "services"],
            "additionalProperties": False,
        },
        "dungeons": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "name": {"type": "string"},
                    "levels": {
                        "type": "array",
                        "minItems": 1,
                        "items": {
                            "type": "object",
                            "properties": {
                                "number": {"type": "integer"},
                                "map_pages": {"type": "array", "items": {"type": "integer"}},
                                "areas": {"type": "array", "items": {"$ref": "#/$defs/area"}},
                            },
                            "required": ["number", "map_pages", "areas"],
                            "additionalProperties": False,
                        },
                    },
                },
                "required": ["name", "levels"],
                "additionalProperties": False,
            },
        },
        "monster_names": {"type": "array", "items": {"type": "string"}},
    },
    "required": ["title", "description", "hooks", "town", "dungeons", "monster_names"],
    "additionalProperties": False,
    "$defs": {
        "area": {
            "type": "object",
            "properties": {
                "key": {"type": "string"},
                "name": {"type": "string"},
                "source_pages": {"type": "array", "items": {"type": "integer"}},
                "kind": {"type": "string", "enum": list(AREA_KINDS)},
            },
            "required": ["key", "name", "source_pages", "kind"],
            "additionalProperties": False,
        }
    },
}

The survey JSON schema, refined from the capability probes' proven shape.

Two deliberate changes from the probe-era schema: no model-supplied dungeon id (canonical ids are osr-forge's construct, derived by slugging name), and map_pages added per level (feeds the content stage's direction extraction). The schema is a fraction of the largest probe-proven schema budget.

SURVEY_SYSTEM `module-attribute`

SURVEY_SYSTEM = 'You survey tabletop adventure modules. The user message interleaves every page\'s extracted text (each headed by a [page N] marker) with that page\'s image. Fill the survey schema from those pages.\n\nRules:\n- A dungeon is a keyed adventuring site: caves, ruins, lairs, and the like. A dungeon exists only where the module prints a keyed area list for it — lettered or unkeyed callouts on another site\'s map are that site\'s features, never sites of their own. The town or home base and its buildings are never dungeons — describe them only in "town". Leave "town.name" empty only when the module genuinely leaves the town unnamed.\n- One dungeon per independently keyed site: first enumerate the module\'s independently keyed sites, then emit exactly that many dungeons. Maps connected internally by stairs or shafts are levels of one dungeon; separate lairs or sites with their own maps and entrances are separate dungeons, even when they are drawn together on one regional map or share a running area-number sequence (a module keying "A. Orc Lair", "B. Goblin Lair" describes separate dungeons — one dungeon each, never merged).\n- "description" is the module\'s own pitch: an excerpt of its printed introduction or back-cover text, quoted or tightened from the module\'s own words — never invented. Leave it empty when the module states none.\n- "town.services" lists the named establishments and services the module states the town offers (an inn, a temple, a general store). List only what the module states.\n- "hooks" are the rumors, jobs, and reasons the party goes on the adventure — usually found in the module\'s introduction or background.\n- Each level\'s "map_pages" lists the pages showing that level\'s map; each area\'s "source_pages" lists the pages describing it. Both refer to the [page N] markers in this request, never to page numbers printed on the pages themselves.\n- An area\'s "key" is the module\'s printed key for it (like "5" or "4a"); when an area has no printed key, use its name. "monster_names" collects every monster name that appears anywhere in the module.\n'

build_census_request

build_census_request(parts: Sequence[TextPart | ImagePart]) -> ModelRequest

Build the census request over already-built page parts.

Public and pure like build_survey_request, so the extraction runner's targeted census-recording leg and fixture tests build fingerprint-identical requests without duplicating prompt code.

Parameters:

Name	Type	Description	Default
`parts`	`Sequence[TextPart \| ImagePart]`	The interleaved page parts, from `page_request_parts`.	required

Returns:

Type	Description
`ModelRequest`	The request, tagged `census`.

build_chunked_census_request

build_chunked_census_request(
    parts: Sequence[TextPart | ImagePart], first_page: int, last_page: int, page_count: int
) -> ModelRequest

Build one chunked-census window's request: the census prompt plus a window-naming preamble.

Mirrors build_chunked_survey_request: the single-request path goes through build_census_request untouched, keeping its recorded fixture valid.

Parameters:

Name	Type	Description	Default
`parts`	`Sequence[TextPart \| ImagePart]`	The window's interleaved page parts.	required
`first_page`	`int`	The window's first page (1-based, absolute).	required
`last_page`	`int`	The window's last page (1-based, absolute).	required
`page_count`	`int`	The whole source's page count.	required

Returns:

Type	Description
`ModelRequest`	The request, tagged `census`.

build_chunked_survey_request

build_chunked_survey_request(
    parts: Sequence[TextPart | ImagePart], first_page: int, last_page: int, page_count: int
) -> ModelRequest

Build one chunked-survey window's request: the survey prompt plus a window-naming preamble.

The preamble is appended only here — the single-request path goes through build_survey_request untouched, which is what keeps every committed fixture valid. The schema and tag are the single-request path's; distinct windows still produce distinct request fingerprints because their pages (and this preamble) differ.

Parameters:

Name	Type	Description	Default
`parts`	`Sequence[TextPart \| ImagePart]`	The window's interleaved page parts, from `page_request_parts`.	required
`first_page`	`int`	The window's first page (1-based, absolute).	required
`last_page`	`int`	The window's last page (1-based, absolute).	required
`page_count`	`int`	The whole source's page count.	required

Returns:

Type	Description
`ModelRequest`	The request, tagged `survey`.

build_survey_request

build_survey_request(parts: Sequence[TextPart | ImagePart]) -> ModelRequest

Build the survey request over already-built page parts.

Public and pure so the extraction runner and fixture tests build fingerprint-identical requests without duplicating prompt code.

Parameters:

Name	Type	Description	Default
`parts`	`Sequence[TextPart \| ImagePart]`	The interleaved page parts, from `page_request_parts`.	required

Returns:

Type	Description
`ModelRequest`	The request, tagged `survey`.

canonical_slug

canonical_slug(text: str) -> str

Slug free text into the canonical id/key alphabet.

NFKD-decompose, drop non-ASCII, lowercase, collapse every run outside [a-z0-9] to a single -, strip leading/trailing -. The result matches CANONICAL_SLUG_PATTERN or is empty (callers apply fallbacks).

Parameters:

Name	Type	Description	Default
`text`	`str`	The text to slug, e.g. a printed dungeon name or area key.	required

Returns:

Type	Description
`str`	The canonical slug, possibly empty.

census_disputes

census_disputes(index: SurveyIndex, census: dict[str, Any]) -> tuple[str, ...]

Compare the census answer against the normalized survey index — pure, deterministic.

Both sides compare on canonical_slug of the dungeon name as answered (never the normalized SurveyIndex.id, whose uniquing suffixes and fallbacks a census answer can't reproduce), as multisets so duplicate-named sites don't silently collapse; empty slugs carry no evidence of identity and stay out of the comparison. Three disagreement shapes, pinned: a site in one answer and not the other (a count mismatch on a shared slug is the same shape, with both counts named); level counts differing for a matched site (sites pair occurrence-indexed in listed order); and a matched level — census and survey level numbers equal — whose census key range disagrees with the survey's keyed-area extremes (first and last in the survey's listed area order, compared on printed-key slugs; a survey level with no areas has no extremes and takes no range check).

Parameters:

Name	Type	Description	Default
`index`	`SurveyIndex`	The normalized survey index.	required
`census`	`dict[str, Any]`	The (merged) raw census answer, already validated against `CENSUS_SCHEMA`.	required

Returns:

Type	Description
`str`	The disagreements as stable human-readable entries, survey-side order
`...`	first, census-only sites after, in listed order.

filter_index_to_pages

filter_index_to_pages(index: SurveyIndex, page_numbers: Iterable[int]) -> SurveyIndex

Restrict every page reference in an index to the given pages.

This is the excerpt-recording closure step: the extraction runner's excerpt mode filters the normalized index down to the committed page subset before batch planning, and both the recorder and the replay test source page parts exclusively from the committed pages — so an in-range model reference to an uncommitted page cannot make a batch request unbuildable at replay time.

Parameters:

Name	Type	Description	Default
`index`	`SurveyIndex`	The normalized survey index.	required
`page_numbers`	`Iterable[int]`	The pages that are actually available.	required

Returns:

Type	Description
`SurveyIndex`	A copy with every `source_pages`/`map_pages` intersected with
`SurveyIndex`	`page_numbers`.

merge_census_answers

merge_census_answers(answers: Sequence[dict[str, Any]]) -> dict[str, Any]

Merge chunked-census windows' schema-valid raw answers into one, under the survey merge's discipline.

The census needs its own small merge — merge_survey_answers hard-requires hooks/town/monster-name fields the census schema deliberately lacks — but the join rules are the survey merge's, restated over the census shape: entries from the same window never join each other; across windows, dungeons join occurrence-indexed on canonical_slug(name) (an empty slug never joins) and levels join occurrence-indexed on number within a joined dungeon; on a join, the first occurrence wins every scalar field.

Parameters:

Name	Type	Description	Default
`answers`	`Sequence[dict[str, Any]]`	The windows' raw answers in window order, each already validated against `CENSUS_SCHEMA`. The inputs are not mutated.	required

Returns:

Type	Description
`dict[str, Any]`	One merged raw answer, shaped exactly like a single window's.

merge_survey_answers

merge_survey_answers(answers: Sequence[dict[str, Any]]) -> dict[str, Any]

Merge chunked-survey windows' schema-valid raw answers into one raw answer, deterministically.

The merge happens at the raw level so the merged dict flows through normalize_survey exactly once — merging normalized indexes would fight reserve-then-bump uniquing (a key bumped to 5-2 in one window and not the other could never re-join).

Join rules, pinned: entries from the same window never join with each other — intra-window multiplicity is preserved verbatim (duplicate slugs denote genuinely distinct entities under reserve-then-bump). Across windows, entities join occurrence-indexed: a later window's n-th occurrence of a join key joins the accumulator's n-th occurrence, and occurrences past the accumulator's count append as new entries. Join keys: dungeons on canonical_slug(name), levels on number within a joined dungeon, areas on canonical_slug(key) within a joined level. An empty slug never joins — two empty slugs carry no evidence of identity, so each empty-slug entry stays distinct and takes normalize_survey's positional fallback. On a join, the first occurrence wins every scalar field; source_pages and map_pages union in first-seen order. title, description, and town take the first non-empty occurrence in window order (town as a unit: first entry with a non-empty name, else first with a non-empty description, else empty — services riding with the chosen entry); hooks concatenate deduplicated by exact string; monster_names union in first-seen order.

Parameters:

Name	Type	Description	Default
`answers`	`Sequence[dict[str, Any]]`	The windows' raw answers in window order, each already validated against `SURVEY_SCHEMA`. The inputs are not mutated.	required

Returns:

Type	Description
`dict[str, Any]`	One merged raw answer, shaped exactly like a single window's.

normalize_survey

normalize_survey(raw: dict[str, Any], page_count: int) -> SurveyIndex

Normalize a schema-valid survey answer into the canonical index, deterministically.

Dungeon ids are the slug of the model's name (empty slug falls back to dungeon-<position>, 1-based document order); area keys are the slug of the model's key (empty falls back to area-<position> within the level). Collisions resolve by reserve-then-bump: the first occurrence of a slug keeps it, later duplicates take the lowest free -2, -3, … suffix, and a genuine printed key is never renamed by someone else's bump. Level numbers invalid or non-unique within a dungeon are renumbered 1..n in listed order. Page references outside 1..page_count are dropped; page lists are deduplicated and sorted. source_label preserves the model's original key wherever the canonical form differs; the human-facing name is never touched. A dungeon the model gave zero levels is dropped — unreachable through a conforming provider (the schema requires one).

Parameters:

Name	Type	Description	Default
`raw`	`dict[str, Any]`	The model's answer, already validated against `SURVEY_SCHEMA`.	required
`page_count`	`int`	The source's page count, for page clamping.	required

Returns:

Type	Description
`SurveyIndex`	The canonical survey index.

Raises:

Type	Description
`ExtractionError`	If normalization yields zero dungeons or zero areas — a dead conversion; osrlib requires at least one dungeon.

survey

survey(workdir: Workdir, provider: ModelProvider) -> SurveyIndex

Run stage 1: survey the module and write stages/survey.json.

A source at or under survey_max_pages pages surveys in one request built by build_survey_request — byte-identical to the pre-chunking request, which the committed fixture replay gates prove. A larger source surveys in survey_windows-sized chunks whose raw answers merge through merge_survey_answers before the one normalize_survey pass. Page markers stay absolute in both modes, so downstream stages see the same page-number space either way.

After normalization the census runs — the survey checking itself: the same page windows through the reduced build_census_request builders, merged by merge_census_answers, compared by census_disputes, with the disagreements landing on the cache's census_disputes before the write. Census usage folds into this stage's usage block (no new stage — rerun survey re-runs both). Flag-only in v1: a re-roll policy is deferred until a measured recurrence.

Stale stages/areas.*.json caches and stages/monsters.json are deleted only on success — a re-run survey can change canonical ids and the encounter-name population, orphaning the downstream caches, but a transient provider failure on a re-run leaves the previous consistent cache set intact.

Parameters:

Name	Type	Description	Default
`workdir`	`Workdir`	A workdir whose preprocess stage is `completed`.	required
`provider`	`ModelProvider`	The model provider.	required

Returns:

Type	Description
`SurveyIndex`	The normalized survey index, as written to the cache.

Raises:

Type	Description
`ValueError`	If the preprocess stage is not `completed` (programmer misuse).
`ExtractionError`	If the survey finds no dungeons or areas.
`ProviderError`	On provider transport, auth, or rate-limit exhaustion.
`SchemaValidationError`	If the provider exhausts its schema budget.

survey_windows

survey_windows(page_count: int, window_size: int) -> tuple[tuple[int, int], ...]

Split pages 1..page_count into contiguous, disjoint windows of at most window_size pages.

No overlap, pinned: overlapping windows would double-extract boundary areas and force the merge to arbitrate conflicting duplicates of the same key; a dungeon spanning a boundary is already covered because each window reports the parts it saw and the merge unions them.

Parameters:

Name	Type	Description	Default
`page_count`	`int`	The source's page count.	required
`window_size`	`int`	The window size (`survey_max_pages`).	required

Returns:

Type	Description
`tuple[int, int]`	`(first, last)` page pairs, 1-based and inclusive: `(1, K)`,
`...`	`(K+1, 2K)`, … A source at or under `window_size` yields one window.

osrforge.survey

CENSUS_SCHEMA module-attribute

CENSUS_SYSTEM module-attribute

SURVEY_SCHEMA module-attribute

SURVEY_SYSTEM module-attribute

build_census_request

build_chunked_census_request

build_chunked_survey_request

build_survey_request

canonical_slug

census_disputes

filter_index_to_pages

merge_census_answers

merge_survey_answers

normalize_survey

survey

survey_windows

`osrforge.survey`

CENSUS_SCHEMA `module-attribute`

CENSUS_SYSTEM `module-attribute`

SURVEY_SCHEMA `module-attribute`

SURVEY_SYSTEM `module-attribute`