torch_world.evaluation

Planning

goal_reaching

goal_reaching(
    env,
    policies: dict[str, Callable[[Tensor], Tensor]],
    start: Tensor,
    goal: Tensor,
    steps: int,
    record: Callable[[object], Tensor] | None = None,
) -> GoalReaching

Run each policy(observation) -> action for steps from start and score it against goal states.

The environment provides distance(states, goal) and success(states, goal), and optionally object_position(states) (for SPL, in environments where a path length is meaningful) and coverage(states, goal). record(env), if given, returns images (N, 3, H, W) of the batch at every step, for example lambda env: env.render(size=192).

GoalReaching dataclass

GoalReaching(
    paths: dict[str, Tensor],
    distances: dict[str, Tensor],
    success: dict[str, Tensor],
    reached: dict[str, Tensor] = dict(),
    actions: dict[str, Tensor] = dict(),
    seconds: dict[str, float] = dict(),
    positions: dict[str, Tensor] = dict(),
    goals: Tensor | None = None,
    coverage: dict[str, Tensor] = dict(),
    frames: dict[str, Tensor] = dict(),
)

Closed-loop episodes of each policy from the same starts toward the same goals.

Per policy: paths are states (N, steps + 1, state_dim), distances the distance to the goal at every step (N, steps + 1), reached whether each step counts as reaching it (N, steps + 1), success whether the final step does (N,), actions the actions taken (N, steps, action_dim) and seconds the mean wall time of one policy call. positions (N, steps + 1, D) and goals (N, D) locate what the task moves (env.object_position), for SPL, coverage (N, steps + 1) is the goal-region overlap for environments that define it (Push-T), and frames holds uint8 videos (N, steps + 1, 3, H, W) when record was given.

rates

rates() -> dict[str, float]

Final success rate of each policy.

intervals

intervals(confidence: float = 0.95) -> dict[str, Rate]

Final success rate of each policy with its Wilson interval, as plots.bars draws it.

summary

summary() -> dict[str, dict[str, float | Rate]]

The standard measures of goal-reaching episodes, per policy:

• success: final success rate with its 95% Wilson interval;
• reached: fraction of episodes that reach the goal at any step;
• steps: mean number of steps to first reach the goal, over the episodes that do;
• final distance: median distance to the goal at the end;
• progress: mean fraction of the initial distance closed;
• SPL: success weighted by path length of the scored object;
• smoothness: mean change between consecutive actions (lower is smoother);
• ms / step: wall time of one policy call;
• coverage: mean of the best goal-region overlap of each episode, when the environment defines it.

random_policy

random_policy(env) -> Callable[[Tensor], Tensor]

Uniform actions in [-1, 1]^action_dim.

Question answering

vqa

VQAResults dataclass

VQAResults(
    dataset: Dataset,
    predictions: list[str],
    truths: list[str],
    kinds: list[str],
)

Predicted and true answers of every question in dataset, with the kind of each question.

accuracy

accuracy() -> dict[str, float]

Accuracy per question kind, and over all questions.

majority

majority() -> dict[str, float]

Accuracy of always giving the most common true answer of each kind.

scores

scores(name: str = 'model') -> dict[str, dict[str, float]]

Accuracy beside the majority baseline per kind, as plots.bars expects.

cards

cards(
    per_kind: int = 2, seed: int = 0
) -> tuple[Tensor, list[str], list[str], list[str]]

per_kind questions of each kind, a mistake first when there is one, as plots.answers expects.

compare

compare(
    results: dict[str, VQAResults],
    baseline: str = "majority answer",
) -> dict[str, dict[str, float]]

Accuracy per kind of several evaluations, e.g. on train and test, beside the majority baseline of the last one, as plots.bars expects.

vqa

vqa(
    model,
    dataset: Dataset,
    tokenizer,
    candidates: list[str] | None = None,
    batch_size: int = 1024,
    shuffle_images: bool = False,
    generator: Generator | None = None,
) -> VQAResults

Answer every question of dataset by retrieval over candidates (by default dataset.answers).

model follows the VLJEPA interface (embed_text and retrieve) and items carry image, question, answer and kind. With shuffle_images, each question is asked about a random image of the whole dataset: a control that measures how much the answers come from the image rather than from the question.

VQAResults dataclass

VQAResults(
    dataset: Dataset,
    predictions: list[str],
    truths: list[str],
    kinds: list[str],
)

Predicted and true answers of every question in dataset, with the kind of each question.

accuracy

accuracy() -> dict[str, float]

Accuracy per question kind, and over all questions.

majority

majority() -> dict[str, float]

Accuracy of always giving the most common true answer of each kind.

scores

scores(name: str = 'model') -> dict[str, dict[str, float]]

Accuracy beside the majority baseline per kind, as plots.bars expects.

cards

cards(
    per_kind: int = 2, seed: int = 0
) -> tuple[Tensor, list[str], list[str], list[str]]

per_kind questions of each kind, a mistake first when there is one, as plots.answers expects.