Scoring a Mesh-Stage Model Against Images

A mm error against the surfaces a registration produced says how well the network reproduces that registration. WorkflowEvaluateMovement asks the other question: how close are the size and shape of the inferred anatomy to the anatomy that was actually imaged, structure by structure. See Tutorial 11 in Tutorials.

For every gated time point it carries the reference frame’s labelmap into that time point with the network’s own deformation, and compares the result to the labelmap of the frame that was acquired: volume difference, Dice and surface RMSE per lung lobe or per heart chamber.

Per-structure scoring

class physiotwin4d.WorkflowEvaluateMovement(movement_workflow, cohort=None, label_names=None, log_level=20)[source]

Bases: PhysioTwin4DBase

Score inferred motion per anatomical structure against acquired frames.

Parameters:

  • movement_workflow (WorkflowInferMovement) – The displacement decoder whose predictions are scored.

  • cohort (Optional[EvaluateMovementBase]) – The cohort being scored — physiotwin4d.EvaluateMovementLung, physiotwin4d.EvaluateMovementDukeHeart or another physiotwin4d.EvaluateMovementBase. It supplies the structures to score and this cohort’s defaults, and its assemble_ground_truth() produces what process() scores against. The workflow never learns which cohort it holds, which is what keeps it free of anatomy branches.

  • label_names (Optional[dict[int, str]]) – Structures to score, {label_id: name}, for a caller with no cohort. Ids the reference frame does not contain are dropped with a warning; ids a single acquired frame does not contain are skipped for that frame alone, since a structure can leave the field of view.

  • log_level (int | str) – Logging level. Default: logging.INFO.

__init__(movement_workflow, cohort=None, label_names=None, log_level=20)[source]

Initialize the base class with logging configuration.

Parameters:

  • class_name – Name for the class (used in log messages). If None, uses the class name. Default: None

  • log_level (int | str) – Logging level. Can be an integer (logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR, logging.CRITICAL) or a string (‘DEBUG’, ‘INFO’, ‘WARNING’, ‘ERROR’, ‘CRITICAL’). Default: logging.INFO

  • log_to_file – Optional file path to write logs to in addition to console output. Default: None

process(case_id, shape_parameters, fitted_reference_mesh, ground_truth, output_directory, smoothing_sigma_mm=None, evaluation_spacing_mm=None, report_dice=None, report_displacement_data=False, include_predicted_displacements=False, include_true_displacements=False, include_displacement_error=False)[source]

Score every gated time point of one case.

Parameters:

  • case_id (str) – Name of the case being scored, recorded in every output.

  • shape_parameters (Path) – JSON file with the case’s PCA coefficient vector.

  • fitted_reference_mesh (Path) – The case’s fitted reference-frame SSM surface, as produced by physiotwin4d.WorkflowFitStatisticalModelToPatient. The predicted displacements are added to its points, and its extent defines the evaluation grid.

  • ground_truth (MovementGroundTruth) – What this case is scored against — the acquired labelmap per stage, the reference frame’s labelmap, and the per-stage fits the point-by-point error is measured against — as assembled by physiotwin4d.EvaluateMovementBase.assemble_ground_truth().

  • output_directory (Path) – Directory the report, the CSV and the per-stage geometry are written to.

  • smoothing_sigma_mm (Optional[float]) – Gaussian sigma, in millimeters, that turns the network’s surface-shell deformation into a continuous field. None takes the cohort’s own value.

  • evaluation_spacing_mm (Optional[float]) – Isotropic pitch every metric is measured on. It sets both the voxel volume the Dice and volume figures are quantized to and the resolution of the deformation fields, whose memory grows with its cube. None takes the cohort’s own value.

  • report_dice (Optional[bool]) – Report the Dice overlap. Turn it off for a structure whose motion is small against its own size: Dice is an overlap fraction, so a lung lobe scores over 0.96 undeformed and the column says more about the organ’s bulk than about the motion. The volume and surface figures still resolve it. None takes the cohort’s own value.

  • report_displacement_data (bool) – Write displacement_per_point.csv, one row per mesh point per stage carrying that point’s predicted and true displacement and the error between them.

  • include_predicted_displacements (bool) – Carry predicted_displacement_mm as point data on every stage’s predicted surface.

  • include_true_displacements (bool) – Carry true_displacement_mm as point data on every stage’s predicted surface.

  • include_displacement_error (bool) – Carry displacement_error_mm, the distance between the predicted and the true position, as point data on every stage’s predicted surface, and report that error’s RMS, 95th percentile and maximum — per structure on every metric row and in the report, and pooled over every point and stage in the report and the returned dict. It is the one metric here measured point by point, so a displacement predicted in the wrong direction cannot cancel against another. A shape-model point is attributed to the structure its own per-cell SegmentationLabelIds names, or, on a model carrying no such ids, to the structure whose reference-frame surface is nearest — the same nearest-label partition the labelmap metrics are already scored under.

Return type:

dict[str, Any]

Returns:

Dict with rows (every metric row), csv_file, report_file, volume_plot_file, predicted_surfaces, warped_labelmaps, displacement_data_file, displacement_statistics (one entry per stage, empty unless include_displacement_error) and that error pooled over every point and stage as displacement_rms_mm, displacement_95th_mm and displacement_max_mm.

Raises:

ValueError – If ground_truth.labelmaps is empty, any stage it carries a labelmap for has no entry in ground_truth.meshes — checked whether or not an option needing the true displacement was requested — none of the requested labels are in the reference frame, or no label survives scoring because the acquired frames contain none of them.

static dice(truth, predicted, label)[source]

Dice overlap of one label. nan when neither volume contains it.

Return type:

float

static volume_mm3(labels, label, voxel_volume_mm3)[source]

Volume of one label, in cubic millimeters.

Return type:

float

static surface_rmse_mm(truth, predicted)[source]

Symmetric point-to-surface RMSE, in millimeters.

Both directions are pooled before the root-mean-square. A one-sided RMSE misses a prediction that covers the truth everywhere but also bulges somewhere the truth does not reach.

Return type:

float

static pool_displacement_error(stage_errors)[source]

RMS, 95th percentile and maximum over every point of every stage.

The errors themselves are pooled rather than their per-stage summaries: a percentile of percentiles is not the percentile of the whole. Every value is nan when no stage was scored point by point.

Return type:

dict[str, float]

static displacement_error_row(case_id, stage, differences, errors)[source]

One stage’s displacement error, summarized over the mesh points.

Parameters:

  • case_id (str) – Case being scored.

  • stage (float) – Normalized stage.

  • differences (ndarray) – (n_points, 3) predicted minus true position.

  • errors (ndarray) – (n_points,) Euclidean norm of those differences, computed once by the caller rather than again here.

Return type:

dict

Example

from physiotwin4d import (
    WorkflowEvaluateMovement,
    WorkflowInferMovement,
    WorkflowInferPhysicsNeMo,
)

evaluate = WorkflowEvaluateMovement(
    movement_workflow=WorkflowInferMovement(
        WorkflowInferPhysicsNeMo(model_directory=model_dir)
    ),
    label_names={28: "lung_upper_lobe_left", 29: "lung_lower_lobe_left"},
)
result = evaluate.process(
    case_id="Case1Pack",
    shape_parameters=pca_coefficients_file,
    fitted_reference_mesh=fitted_reference_mesh_file,
    reference_labelmap=reference_labelmap,
    ground_truth_labelmaps={0.0: frame_00, 0.1: frame_10},
    output_directory=out_dir,
)
print(result["report_file"], result["csv_file"])

Notes

Why labelmaps rather than the model’s surface. The lung shape model carries its five lobes as per-cell labels, but the heart model is a single structure — the whole heart minus its chamber cavities — so its chambers exist only in the acquired labelmaps. Warping those labelmaps scores every structure the acquisition contains, whether or not the shape model represents it separately.

The evaluation grid. Everything is measured on one isotropic grid built around the reference anatomy, so a case whose gated frames carry different slice pitches is still scored on a single, stated voxel volume. Its pitch sets both that voxel volume and the memory the per-stage deformation fields take, which grows with its cube.

See Also