Skip to content

osrforge.providers.decisions

The decision provider: Clef's role in the tracing stage.

Clef answers a fixed-choice question about one image with a probability for each option. The tracing stage asks it about each edge that touches floor, each cell the ink measurements don't settle, and each label-sized mark, one question per request, with a red box on the crop marking the item. A DecisionRequest is one such question, and a DecisionResponse is its probabilities. The live adapter is ClefProvider.

A recording is one answer filed under its key, the hash of the request's pixels and parameters, so tests replay it without calling the service. RecordedDecisionProvider replays a directory of recordings and RecordingDecisionProvider writes them. Clef has no version pin, so before a sweep the drift_check sends a sample of recorded requests again and reports every answer that moved. A recording of a private corpus member's map may contain the module's picture, so those recordings live in the owner's private corpus directory and never in the repository; only the committed corpus members' and the synthetic maps' recordings are committed.

JPEG_QUALITY module-attribute 

JPEG_QUALITY = 90

The JPEG quality a large crop is re-encoded at. Part of every key.

JPEG_THRESHOLD_BYTES module-attribute 

JPEG_THRESHOLD_BYTES = 100000

A request body over this many bytes is rebuilt with its image as JPEG. Part of every key.

Workers AI rejects bodies far smaller than its documented 13 MiB: the probe saw rejections from about 512 KiB. A 2 by 2 cell crop at 128 pixels per cell is about 32 KB as PNG and 20 KB as JPEG, so edge requests go as PNG and only the larger crops are re-encoded.

ClefModel module-attribute 

ClefModel = Literal['clef', 'clef-flash']

The two models Workers AI hosts: Clef, and its smaller sibling.

DriftKind module-attribute 

DriftKind = Literal['body', 'choice', 'probability']

What moved: the body sent, the top answer, or one option's probability.

ImageFormat module-attribute 

ImageFormat = Literal['png', 'jpeg']

How a request's image was sent.

DecisionProvider

Bases: Protocol

The decision role of the tracing stage, which Clef fills.

decide 

decide(request: DecisionRequest) -> DecisionResponse

Send one question and return its probabilities.

Parameters:

Name Type Description Default
request DecisionRequest

The question.

 required

Returns:

Type Description
DecisionResponse

The answer.

Raises:

Type Description
LedgerStopError

If the request's reservation would pass the ledger's stop.
ProviderError

On transport, auth, or rate-limit exhaustion, or a rejection that stays after its retries.

DecisionRequest dataclass 

DecisionRequest(
    tag: str,
    image: bytes,
    state: str,
    question: str,
    options: tuple[tuple[str, str | None], ...],
    model: ClefModel = "clef",
)

One question about one image.

Attributes:

Name Type Description
tag str

A stable identifier like edge or cell, never free prose. It names recording files and is the question's id in the request, which Clef reads along with the question.
image bytes

The crop, as PNG.
state str

The text the service reads before the question: the map style in words, and what the red box marks.
question str

The question, like "What is drawn on the cell edge inside the red box?".
options tuple[tuple[str, str | None], ...]

The options, each an id and a description (or None for an option whose id says enough), in the order they're given. Clef sorts options by id before it reads them, so the order doesn't change the answer, but it's part of the key.
model ClefModel

Which model answers.

tag instance-attribute 

tag: str

image instance-attribute 

image: bytes

state instance-attribute 

state: str

question instance-attribute 

question: str

options instance-attribute 

options: tuple[tuple[str, str | None], ...]

model class-attribute instance-attribute 

model: ClefModel = 'clef'

key 

key() -> str

Return the recording key: the hash of the request's pixels and parameters.

The key is the sha256 hex of the canonical JSON (sorted keys, compact separators, UTF-8) of the model, the tag, the image's pixel_digest (of the source PNG, whatever format is sent), the state, the question, the options as a list of [id, description] pairs, JPEG_QUALITY, and JPEG_THRESHOLD_BYTES, both read when the key is computed, so a change in how images are sent changes every key. The format a particular request goes as isn't in the key, because it depends on the PNG encoder's output size rather than on the pixels. The recording stores that format.

Returns:

Type Description
str

A 64-character sha256 hex digest.

DecisionResponse

Bases: BaseModel

One answer: a probability for every option.

The provider computes choice, confidence, and margin from the probabilities and ignores the confidence field the service returns, whose formula isn't documented.

probabilities instance-attribute 

probabilities: dict[str, float]

Each option id's probability. They sum to 1 within rounding.

model_id instance-attribute 

model_id: str

The model the service says answered.

input_tokens class-attribute instance-attribute 

input_tokens: int = Field(ge=0)

The input tokens the service charged. Clef has no output tokens.

format instance-attribute 

format: ImageFormat

How the image was sent: PNG, or JPEG when the PNG body was too large.

body_sha256 instance-attribute 

body_sha256: str

The sha256 of the exact request body sent, which the drift check compares with a recording's.

choice property 

choice: str

The option with the highest probability, the first by id on a tie.

confidence property 

confidence: float

The top probability.

margin property 

margin: float

How far the top probability leads the second: the drift sample favours small margins.

Drift dataclass 

Drift(tag: str, key: str, kind: DriftKind, reason: str)

One way a recorded answer no longer holds.

Attributes:

Name Type Description
tag str

The request's tag.
key str

The recording key.
kind DriftKind

What moved.
reason str

The recorded and the new values, in words.

tag instance-attribute 

tag: str

key instance-attribute 

key: str

kind instance-attribute 

kind: DriftKind

reason instance-attribute 

reason: str

RecordedDecisionProvider 

RecordedDecisionProvider(recordings: Path)

Replays recorded answers, so tests and continuous integration never call Clef.

A recording file is JSON with schema_version, key, tag, request, body_bytes, and response. request is {"model", "state", "question", "options", "image": {"pixels": digest, "width", "height"}}: the exact body's content with the image's data replaced by its pixel digest. response is the DecisionResponse as dumped, so it records the format the image went as and the sha256 of the exact body sent.

Bind to a directory of recordings.

Parameters:

Name Type Description Default
recordings Path

The directory of <tag>.<key[:12]>.json files.

 required

recordings instance-attribute 

recordings = recordings

decide 

decide(request: DecisionRequest) -> DecisionResponse

Replay the recording whose key matches the request.

Parameters:

Name Type Description Default
request DecisionRequest

The request to replay.

 required

Returns:

Type Description
DecisionResponse

The recorded answer.

Raises:

Type Description
FixtureMissError

If no recording has the request's key.
ProviderError

If the recording's schema version isn't this package's.

RecordingDecisionProvider 

RecordingDecisionProvider(inner: DecisionProvider, recordings: Path)

Passes each request to a live provider and writes its answer as a recording.

Writing is idempotent by key.

Wrap a live provider.

Parameters:

Name Type Description Default
inner DecisionProvider

The provider that answers.

 required
recordings Path

Where recordings land; created if missing.

 required

inner instance-attribute 

inner = inner

recordings instance-attribute 

recordings = recordings

decide 

decide(request: DecisionRequest) -> DecisionResponse

Decide through the inner provider and record the exchange.

Parameters:

Name Type Description Default
request DecisionRequest

The request.

 required

Returns:

Type Description
DecisionResponse

The inner provider's answer, unchanged.

drift_check 

drift_check(
    recordings: Path, provider: DecisionProvider, requests: Sequence[DecisionRequest], tolerance: float = 0.03
) -> tuple[Drift, ...]

Send recorded requests again and report every answer that moved.

Each request is sent through provider, and its answer is compared with the recording under the request's key. A body drift is reported when the body the provider sent doesn't have the recording's body_sha256, so the request wasn't resent exactly. The probabilities are compared whether or not the body matched: a choice drift when the top answer changed, and a probability drift when any option's probability moved by more than tolerance. One request can yield several drifts. A sweep stops when this returns anything.

Parameters:

Name Type Description Default
recordings Path

The recordings directory.

 required
provider DecisionProvider

The live provider.

 required
requests Sequence[DecisionRequest]

The sample to resend, each with a recording, as drift_sample chooses them.

 required
tolerance float

How far a probability may move.

 0.03

Returns:

Type Description
Drift

The drifts, in the requests' order and then body, choice,
...

probability; empty when nothing moved.

Raises:

Type Description
FixtureMissError

If a request has no recording.

drift_sample 

drift_sample(recordings: Path, requests: Sequence[DecisionRequest], size: int = 50) -> tuple[DecisionRequest, ...]

Choose the requests the drift check resends from those that have recordings.

A recording holds no image, so the caller supplies the requests: the sweep's planned requests, which it builds from the maps. Among those with a recording, half of the sample (rounded down) is the ones whose recorded margins are smallest, where the top answer led the next by the least, because those are the answers a changed model moves first. The rest are taken at even intervals through the remaining recorded requests in key order. When at most size requests have recordings, the sample is all of them.

Parameters:

Name Type Description Default
recordings Path

The recordings directory.

 required
requests Sequence[DecisionRequest]

The requests to choose from.

 required
size int

The sample size.

 50

Returns:

Type Description
tuple[DecisionRequest, ...]

The chosen requests, sorted by key.

encode_request 

encode_request(request: DecisionRequest) -> tuple[bytes, ImageFormat]

Return the body the provider sends: PNG, or JPEG when the PNG body is too large.

Parameters:

Name Type Description Default
request DecisionRequest

The request.

 required

Returns:

Type Description
bytes

The PNG body and "png" when that body is at most
ImageFormat

JPEG_THRESHOLD_BYTES
tuple[bytes, ImageFormat]

long, else the JPEG body and "jpeg".

recording_filename 

recording_filename(tag: str, key: str) -> str

Return the filename a recording is stored under: <tag>.<key[:12]>.json.

Parameters:

Name Type Description Default
tag str

The request's tag.

 required
key str

The recording key.

 required

Returns:

Type Description
str

The filename.

request_body 

request_body(request: DecisionRequest, image_format: ImageFormat) -> bytes

Build the exact body for a request, with its image in the given format.

The body is the canonical JSON (sorted keys, compact separators, UTF-8) of {"model": request.model, "images": [{"content_type": "image/png" or "image/jpeg", "base64": ...}], "state": request.state, "questions": {request.tag: {"type": "choice", "instructions": request.question, "criteria": {option id: description}}}}. The JPEG is the PNG's pixels saved by Pillow at JPEG_QUALITY.

Parameters:

Name Type Description Default
request DecisionRequest

The request.

 required
image_format ImageFormat

How to send the image.

 required

Returns:

Type Description
bytes

The body bytes.