Tutorials

PhysioTwin4D tutorials

From a CT scan to an animated digital twin

Thirteen numbered stages across 29 Python scripts, 19 of them runnable today: the ten duke_heart variants wait on a dataset that is being released soon. Each one drives the real workflow classes end-to-end on downloadable data, shows what it produced, and ends with the handful of constants to change so it runs on your own scans.

Before You Start

1. Get the scripts. They ship with the source repository, not with the pip package — pip install physiotwin4d gives you the library and the physiotwin4d-* commands but no tutorials/ directory:

git clone https://github.com/Project-MONAI/physiotwin4d.git
cd physiotwin4d

See Quick Start for version-matched clones and the release tarball link.

2. Get the data, running every download from the top level of the clone. The tutorials resolve their inputs against the repository root (<repo>/data/<dataset>), while the CLI writes to data/<dataset> relative to the current working directory:

physiotwin4d-download-data Slicer-Heart-CT --directory data/Slicer-Heart-CT
physiotwin4d-download-data KCL-Heart-Model --directory data/KCL-Heart-Model
physiotwin4d-download-data Chest-CT --directory data/Chest-CT

That covers Heart Tutorials 1, 3, 4 and 6 (Slicer-Heart-CT and KCL-Heart-Model) and Lung Tutorial 7 (Chest-CT), which Tutorial 13 also animates. DirLab-4DCT — used by Lung Tutorials 1, 2, 3, 4, 6, 8, 10, 11 and 12, and by Heart Tutorial 7 — is not auto-downloaded: DIR-Lab distributes each case individually and may require registration.

Tutorials 5 and 9 need no dataset of their own; they consume the outputs of Tutorials 4 and 8. Duke-Heart-4DLabelmaps drives the ten duke_heart variants: a nine-tutorial chain from Tutorial 4 through Tutorial 12, plus the separate, optional Tutorial 2 ICON finetuning variant; the dataset is being released soon, and until then access can be requested from Stephen Aylward (saylward@nvidia.com). See data/DirLab-4DCT/README.md, data/Duke-Heart-4DLabelmaps/README.md, and Download Example Data for every dataset’s size and source.

3. Know where output lands. Every tutorial writes to tutorials/output/<tutorial_name>/ and reuses what it finds there, so a second run is cheap and later tutorials pick up earlier results automatically.

01

Gated 4D CT to Animated USD

Segment, register and assemble a 4D CT series into an animated OpenUSD scene.

Slicer-Heart-CT · DIR-Lab 02

Finetune ICON Registration

Adapt uniGradICON to your own cohort and measure what the finetuning bought you.

DIR-Lab 03

Reconstruct High-Resolution 4D CT

Register every phase to one reference and reconstruct the series at its resolution.

Slicer-Heart-CT · DIR-Lab 04

CT Segmentation to VTK Surfaces

Segment one CT phase and export patient anatomy as VTK PolyData surfaces.

Slicer-Heart-CT · DIR-Lab 05

VTK Surfaces to Animated USD

Convert meshes into a time-sampled USD scene for Omniverse playback.

Tutorial 4 output 06

Create a PCA Shape Model

Turn a population of meshes into a statistical shape model and its modes.

KCL-Heart-Model · DIR-Lab 07

Fit the Shape Model to a Patient

Fit the shape model to one ungated clinical scan, PCA coefficients and all.

Chest-CT · Tutorial 6 output 08

Propagate the Model Through 4D

Fit each case at its reference phase and carry the mesh through every phase.

DIR-Lab · Tutorials 2 and 6 09

Train a PhysicsNeMo Surrogate

Train a MeshGraphNet to predict per-vertex motion from shape and phase.

Tutorial 8 output 10

Predict Motion With the Surrogate

Replace the registration solve with one forward pass, then export to USD.

Tutorials 8 and 9 output 11

Score the Surrogate Against the Images

Volume and surface RMSE per lobe, plus Dice per chamber, on the held-out case.

Tutorials 8, 9 and 10 output 12

The Whole Inference Pipeline in One Script

Go from a gated series to an animated prediction without registering a single phase.

Tutorials 6 and 9 output 13

Breathe and Beat a Static Clinical CT

Animate one ungated breath-hold scan with both rhythms, from two networks at once.

Chest-CT · Tutorials 7 and 9 output

Recommended Run Order

Tutorials are straightforward Python scripts: run one with python tutorials/tutorial_01_heart_gated_ct_to_usd.py, or open it in your editor and read it top to bottom. Numbers 1, 4 and 5 are the fastest way to see the toolkit work end-to-end; 6 through 13 build the statistical-model and AI-surrogate pipeline on top.

1. Tutorial 1 — after downloading Slicer-Heart-CT.

2. Tutorial 2 — after obtaining DIR-Lab. It writes the finetuned ICON weights Tutorial 8 uses when present; optional if stock weights are acceptable.

3. Tutorial 3 — after obtaining its dataset; it registers with Greedy and needs no finetuned weights.

4. Tutorial 4 — after downloading Slicer-Heart-CT.

5. Tutorial 5 — after Tutorial 4, whose surfaces it converts.

6. Tutorial 6 — heart needs KCL-Heart-Model, lung needs DIR-Lab.

7. Tutorial 7 — after Tutorial 6; the lung variant also needs Chest-CT.

8. Tutorial 8 — after Tutorial 6 (lung); Tutorial 2 optional.

9. Tutorial 9 — after Tutorial 8, whose fitted meshes it trains on.

10. Tutorial 10 — after Tutorial 9, whose checkpoint it loads.

11. Tutorial 11 — after Tutorial 9. The lung variant segments every gated frame of the held-out case, so it needs a GPU and the segmentation weights.

12. Tutorial 12 — after Tutorial 6 and Tutorial 9 for its anatomy; it fits the model to the patient itself, so nothing is read from Tutorial 8.

13. Tutorial 13 — after Tutorial 7 (lung) and Tutorial 9 for both anatomies. It also needs Simpleware Medical, which segments the heart it fits.

Tutorial 1: Gated 4D CT to Animated USD

Script

tutorials/tutorial_01_heart_gated_ct_to_usd.py (Slicer-Heart-CT)

tutorials/tutorial_01_lung_gated_ct_to_usd.py (DIR-Lab)

Workflow

WorkflowConvertImageToUSD, driving RegisterImagesGreedy and a SegmentAnatomyBase subclass.

Dataset

Slicer-Heart-CT (auto-download) for the heart, DIR-Lab (manual) for the lung. The phase roughly 70% through the series is the segmentation and registration reference.

Requirements

Greedy registers every phase against the reference on the CPU; a GPU is still needed for segmentation.

Preview

The animated cardiac model, played back in Omniverse.

The same workflow on a DIR-Lab respiratory series.

Inner API usage

workflow = WorkflowConvertImageToUSD(
    time_series_images=time_series_images,
    reference_image=reference_image,
    output_directory=str(output_dir),
    usd_project_name="cardiac_model",
    registration_method=registration_method,
    segmentation_method=segmentation_method,
    save_assets=True,
)
workflow_results = workflow.process()

Run

python tutorials/tutorial_01_heart_gated_ct_to_usd.py
python tutorials/tutorial_01_lung_gated_ct_to_usd.py

Outputs

The animated USD named after usd_project_name, the per-phase registered volumes and labelmaps, and screenshots — all under tutorials/output/tutorial_01_{heart,lung}/.

Adapt to your data

Point data_dir and the file glob near the top of the script at your own series: any set of 3D volumes ITK can read (.mha, .nrrd, .nii.gz) in acquisition order, or a 4D .seq.nrrd split first with physiotwin4d-convert-image-4d-to-3d. Choose the reference phase by changing the index expression, and swap segmentation_method for the one matching your anatomy and contrast — see Segmentation Modules. For command-line use without editing code, run physiotwin4d-convert-image-to-usd (Heart Gated CT Processing).

Tutorial 2: Finetune ICON Registration

Script

tutorials/tutorial_02_lung_finetune_icon.py

tutorials/tutorial_02_lung_distancemap_finetune_icon.py — the lung distance-map variant, which finetunes on distance maps rather than image intensities so the labelmap-to-labelmap stage of Tutorials 7 and 8 has in-distribution weights.

tutorials/tutorial_02_duke_heart_distancemap_finetune_icon.py — the same for the heart, on Duke-Heart-4DLabelmaps. The heart needs its own run because it registers with a much tighter mask than the lungs, so its distance maps saturate over a shorter radius and do not share an intensity distribution with lung ones. The per-organ values live in tutorials/parameters_lung_ct_dirlab.py for the lung variant and tutorials/parameters_duke_heart_labelmaps.py for this one. This is a duke_heart tutorial: Duke-Heart-4DLabelmaps is being released soon (see Before You Start), and until then access can be requested from Stephen Aylward (saylward@nvidia.com) — see data/Duke-Heart-4DLabelmaps/README.md.

Workflow

WorkflowFinetuneICONRegistration, then RegisterImagesGreedy and RegisterImagesGreedyICON to score the result, with SegmentNVSegmentCTMRI supplying the labelmaps.

Dataset

DIR-Lab (manual). Every case except Case1Pack trains; Case1Pack is held out and registered three ways — Greedy alone with its defaults, then Greedy+ICON with the stock uniGradICON weights and with the finetuned ones — so the improvement is measured, not asserted.

Scoring

The fixed image is segmented once, and each registered moving image is segmented again after warping. The table reports the mean, 5th percentile, median, 95th percentile, minimum and maximum of the per-class Dice scores, plus the mislabeled voxel count, with the unregistered moving image as a reference row. Segmenting each warped volume separately costs one GPU segmentation per method and folds segmentation variability into the scores.

Requirements

GPU required. 100 epochs over nine cases: the longest-running tutorial before the AI-surrogate chain. The experiment directory is cleared on every run, so it does not resume.

Preview

The held-out case scored per method — unregistered, Greedy, Greedy+ICON with the stock weights, and with the finetuned weights.

Inner API usage

workflow = WorkflowFinetuneICONRegistration(
    subject_image_files=list(subject_image_files.values()),
    output_dir=weights_dir,
    finetune_name=finetune_name,
    subject_ids=list(subject_image_files.keys()),
    epochs=epochs,
    dice_loss_weight=0.0,
)
weights_path = workflow.process()

Run

python tutorials/tutorial_02_lung_finetune_icon.py
python tutorials/tutorial_02_lung_distancemap_finetune_icon.py

Outputs

The finetuned checkpoint under tutorials/network_weights/icon_dirlab_4dct/, plus registration_summary.csv, the fixed-minus-registered difference images (residual structure is what separates the methods), the fixed and warped labelmaps, and before/after screenshots in tutorials/output/tutorial_02_lung/.

Adapt to your data

Replace the training cohort glob with your own volumes and set epochs to fit your budget — the workflow needs only a list of image files and matching subject ids. Raise dice_loss_weight above 0.0 when you also have labelmaps to supervise with. Load the resulting weights anywhere by passing them to RegisterImagesICON.

Tutorial 3: Reconstruct High-Resolution 4D CT

Script

tutorials/tutorial_03_heart_reconstruct_highres_4d_ct.py

tutorials/tutorial_03_lung_reconstruct_highres_4d_ct.py

Workflow

WorkflowReconstructHighres4DCT with RegisterImagesGreedy.

Dataset

Slicer-Heart-CT for the heart; DIR-Lab for the lung, which reconstructs against its T70 (end-exhale) phase — the same reference Tutorial 8 fits to.

Requirements

CPU is enough. One coarse-to-fine registration per phase, greedy schedule [30, 15, 7, 3].

Preview

The acquired cardiac phases.

The same phases reconstructed at the reference resolution.

Side by side on the lung series.

Inner API usage

registration_method = RegisterImagesGreedy()
registration_method.set_number_of_iterations([30, 15, 7, 3])

workflow = WorkflowReconstructHighres4DCT(
    time_series_images=time_series,
    reference_image=reference_image,
    reference_time_frame=reference_time_frame,
    registration_method=registration_method,
)
workflow.set_modality("ct")
result = workflow.process()

Run

python tutorials/tutorial_03_heart_reconstruct_highres_4d_ct.py
python tutorials/tutorial_03_lung_reconstruct_highres_4d_ct.py

Outputs

reconstructed_frame_<i>.mha plus forward and inverse transforms for every phase, and two screenshots, under tutorials/output/tutorial_03_{heart,lung}/.

Adapt to your data

Set case_glob and data_dir to your series and pick the reference with reference_time_frame. If you have a separate breath-hold or contrast-enhanced volume, pass it as reference_image instead of one of the phases — that is what the workflow is really designed for. Tune number_of_iterations_greedy down for a fast smoke test. The saved .hdf transforms are reusable: TransformTools applies them to meshes and labelmaps.

Tutorial 4: CT Segmentation to VTK Surfaces

Script

tutorials/tutorial_04_heart_ct_to_vtk.py

tutorials/tutorial_04_lung_ct_to_vtk.py

tutorials/tutorial_04_duke_heart_labelmap_to_vtk.py — starts from gated labelmaps rather than CT, and also extracts tetrahedral meshes. Needs Duke-Heart-4DLabelmaps (see Before You Start).

Workflow

WorkflowConvertImageToVTK with SegmentChestTotalSegmentatorWithContrast (heart) or SegmentChestTotalSegmentator (lung).

Dataset

One frame of Slicer-Heart-CT or DIR-Lab — a single static volume is enough.

Requirements

GPU recommended for segmentation; no registration, so this is the quickest way to confirm your environment and model weights work.

Preview

Cardiac anatomy surfaces exported from one CT phase.

The same workflow on a DIR-Lab respiratory case.

The duke_heart variant, which starts from a gated labelmap rather than a CT and also writes tetrahedral meshes.

Inner API usage

workflow = WorkflowConvertImageToVTK(
    segmentation_method=segmentation_method,
)
result = workflow.process(
    input_image=ct_image,
    surface_reduction_rate=HEART_CT_KCL.surface_reduction_rate,
    extract_label_surfaces=save_label_surfaces,
)

Run

python tutorials/tutorial_04_heart_ct_to_vtk.py
python tutorials/tutorial_04_lung_ct_to_vtk.py

Outputs

patient_surfaces.vtp (all anatomy in one mesh, with a per-cell SegmentationLabelIds array so each cell still names the structure it came from), per-group and per-label .vtp files, patient_labelmap.mha and two screenshots, under tutorials/output/tutorial_04_{heart,lung}/.

Adapt to your data

Change the input volume path, then choose the segmenter matching your scan: contrast versus non-contrast CT, or SegmentNVSegmentCTMRI for CT and MRI. Raise surface_reduction_rate in the tutorial’s parameter module toward 1.0 for lighter meshes. Every segmenter declares its own labels through AnatomyTaxonomy, so downstream grouping and USD materials follow automatically — see Segmentation Modules.

Tutorial 5: VTK Surfaces to Animated USD

Script

tutorials/tutorial_05_heart_vtk_to_usd.py

tutorials/tutorial_05_duke_heart_vtk_to_usd.py — the 4D counterpart, animating Tutorial 4 (duke heart)’s per-phase surfaces. Needs Duke-Heart-4DLabelmaps (see Before You Start).

Workflow

WorkflowConvertVTKToUSD.

Dataset

Tutorial 4’s per-structure patient_*.vtp surfaces — no image data, no download.

Requirements

CPU only, seconds to run. The cheapest tutorial in the set.

Preview

The exported USD scene, split by anatomy and painted with OmniSurface materials.

Inner API usage

workflow = WorkflowConvertVTKToUSD(
    input_meshes=meshes,
    usd_project_name=project_name,
    output_directory=output_dir,
    appearance="anatomy",
    static_merge=True,
    separate_by_connectivity=True,
)
results = workflow.process()

Each input surface keeps the structure name that WorkflowConvertImageToVTK wrote into its field_data['SegmentationLabelNames']. That name becomes the USD prim name and, with anatomy_type left unset, selects the prim’s material — so the left chambers, right chambers, myocardium and great vessels each get their own look rather than one shared heart material.

Run

python tutorials/tutorial_05_heart_vtk_to_usd.py

Outputs

The USD scene and a rendered screenshot under tutorials/output/tutorial_05_heart/.

Adapt to your data

input_meshes takes any list of PyVista meshes — pass one per time point, in order, for an animated scene instead of a static one (drop static_merge), and set frames_per_second to control playback. appearance="anatomy" binds per-organ materials through USDAnatomyTools; set anatomy_type to force one palette onto every object, or object_names to name the prims yourself. For file-in, file-out conversion without Python, see VTK to USD Conversion.

Tutorial 6: Create a PCA Shape Model

Script

tutorials/tutorial_06_heart_create_statistical_model.py

tutorials/tutorial_06_lung_create_statistical_model.py

tutorials/tutorial_06_duke_heart_create_statistical_model.py — builds the cardiac model the duke_heart surrogate chain trains against. Needs Duke-Heart-4DLabelmaps (see Before You Start).

Workflow

WorkflowCreateStatisticalModel; the lung variant first builds an unbiased atlas with WorkflowCreateMeanSurface.

Dataset

KCL-Heart-Model (auto-download) for the heart. The lung variant starts from raw DIR-Lab volumes, segmenting each case’s T70 phase itself.

Requirements

The heart variant is CPU-only and quick. The lung variant is the slowest of Tutorials 1-7: one GPU segmentation per case, then a deformable registration per case per atlas iteration. Every intermediate is cached, so a re-run costs almost nothing.

Preview

Heart model: the mean shape at ±2σ along its leading modes.

The same decomposition for the lung population.

Inner API usage

mean_workflow = WorkflowCreateMeanSurface(surfaces=sample_surfaces)
mean_workflow.set_number_of_iterations(mean_surface_iterations)
reference_surface = mean_workflow.process()["mean_surface"]

workflow = WorkflowCreateStatisticalModel(
    sample_meshes=sample_surfaces,
    reference_mesh=reference_surface,
    number_of_pca_components=number_of_pca_components,
)
result = workflow.process()

Run

python tutorials/tutorial_06_heart_create_statistical_model.py
python tutorials/tutorial_06_lung_create_statistical_model.py

Outputs

pca_model.json, pca_mean_surface.vtp, the ±2σ mode surfaces and their renders, under tutorials/output/tutorial_06_{heart,lung}/. The lung variant also leaves its per-case segmentations there, which Tutorial 8 reuses.

Adapt to your data

The workflow wants a population of meshes plus one reference; point sample_meshes at your own cohort and let WorkflowCreateMeanSurface build the reference when no natural template exists. number_of_pca_components trades fidelity against cohort size — you need more subjects than modes. The saved pca_model.json is the portable artifact: Tutorials 7 and 8 and Create Statistical Model all speak it.

Tutorial 7: Fit the Shape Model to a Patient

Script

tutorials/tutorial_07_heart_fit_statistical_model_to_patient.py

tutorials/tutorial_07_lung_fit_statistical_model_to_patient.py

tutorials/tutorial_07_duke_heart_fit_statistical_model_to_patient.py — fits the Tutorial 6 (duke heart) model. Needs Duke-Heart-4DLabelmaps (see Before You Start).

Workflow

WorkflowFitStatisticalModelToPatient.

Dataset

Tutorial 6’s model plus one patient scan. The lung variant fits to Chest-CT — an ungated, single-acquisition chest CT, the kind a patient-specific model is normally fitted to. See data/Chest-CT/README.md for the data source and required citation.

Requirements

One segmentation pass plus a PCA-constrained fit; GPU recommended for the segmentation, and no registration over time.

Preview

The heart model fitted to a non-contrast scan.

The lung model fitted to the ungated Chest-CT volume.

Inner API usage

workflow = WorkflowFitStatisticalModelToPatient(
    template_model=pca_mean,
    patient_models=[patient_surface],
    patient_image=patient_image,
    patient_labelmap=patient_labelmap,
)
workflow.set_use_pca_registration(
    use_pca_registration=True,
    pca_model=pca_model,
)
result = workflow.process()

Run

python tutorials/tutorial_07_heart_fit_statistical_model_to_patient.py
python tutorials/tutorial_07_lung_fit_statistical_model_to_patient.py

Outputs

The registered template surface, the fitted mesh, and — the piece the rest of the pipeline needs — *_registered_coefficients.json, the patient’s position in shape space. Under tutorials/output/tutorial_07_{heart,lung}/.

Adapt to your data

Set the patient image path and keep the segmenter consistent with the one that built the model. labelmap_interior_object_ids (heart) tells the fit which labels are interior structures — those ids are TotalSegmentator’s chamber labels, so change them if you change segmenter. Turn set_use_pca_registration off to fall back to an unconstrained template-to-patient fit when you have no model. The CLI equivalent is Heart Model to Patient Registration.

Tutorial 8: Propagate the Shape Model Through 4D

Script

tutorials/tutorial_08_lung_fit_model_to_4d_patients.py

tutorials/tutorial_08_duke_heart_fit_model_to_4d_patients.py — the same fit-then-propagate pass over cardiac phases, using RegisterModelsDistanceMaps in place of the image registration. Needs Duke-Heart-4DLabelmaps (see Before You Start).

Workflow

WorkflowFitStatisticalModelToPatient at the reference phase, then WorkflowReconstructHighres4DCT to carry the fitted surface through every other phase.

Dataset

DIR-Lab, plus Tutorial 6 (lung)’s model. Tutorial 2’s finetuned distance-map ICON weights are used by the model fit when present; without them the tutorial warns and fits with the stock uniGradICON weights.

Requirements

GPU required, and the heaviest registration workload in the set: one segmentation and one fit per case, plus one registration per phase per case.

Preview

The fitted shape-model surface propagated across the phases of a DIR-Lab case.

The duke_heart variant, coloured by deformation magnitude across the cardiac phases.

Inner API usage

fit_workflow = WorkflowFitStatisticalModelToPatient(
    template_model=pca_mean_surface,
    patient_models=[lung_surface],
    patient_image=reference_image,
    patient_labelmap=lung_labelmap,
)
fit_workflow.set_use_pca_registration(True, pca_model=pca_model)

reg_workflow = WorkflowReconstructHighres4DCT(
    time_series_images=time_series,
    reference_image=reference_image,
    reference_time_frame=phase_ids.index(reference_phase),
    register_reference_time_frame_to_reference_image=False,
    registration_method=registration_method,
)

Run

python tutorials/tutorial_08_lung_fit_model_to_4d_patients.py

Outputs

Per case, under tutorials/output/tutorial_08_lung/<case>/: the fitted reference surface, its PCA coefficients, and one warped surface plus forward/inverse transform per phase. Those per-phase surfaces are exactly the training set Tutorial 9 consumes.

Adapt to your data

Point data_dir at a directory of per-case 4D series and set reference_phase to the phase your model was built at; the case and phase file patterns are two globs near the top of the script. Everything is cached per case, so adding a subject re-runs only that subject.

Tutorial 9: Train a PhysicsNeMo Surrogate

Script

tutorials/tutorial_09_lung_train_physicsnemo_mgn.py

tutorials/tutorial_09_duke_heart_train_physicsnemo_mgn.py — trains the cardiac network Tutorial 13 uses for its heartbeat. Needs Duke-Heart-4DLabelmaps (see Before You Start).

Workflow

WorkflowTrainPhysicsNeMo driving TrainPhysicsNeMoMGN, then WorkflowInferPhysicsNeMo and WorkflowInferMovement to score the held-out case. A fully connected TrainPhysicsNeMoMLP method is a drop-in replacement; no separate tutorial ships for it.

Dataset

Tutorial 8’s per-phase surfaces and Tutorial 6 (lung)’s mean surface. The tutorial writes one JSON manifest per case, plus the per-vertex displacement targets those manifests point at.

Requirements

GPU, plus the optional extra:

pip install "physiotwin4d[physicsnemo]"
pip install torch-geometric

Python >= 3.11. 1500 epochs by default.

Preview

The held-out lung case, predicted at every stage by the trained network.

The same surface coloured by per-vertex error against the registration that produced the training data.

Deformation magnitude, which is what the error above should be read against — the largest errors sit where the motion is largest.

The duke_heart variant over a cardiac cycle, with its own RMSE and deformation-magnitude captures in tutorial_09_duke_heart_rmse.gif and tutorial_09_duke_heart_deformation_magnitude.gif.

Inner API usage

training_method = TrainPhysicsNeMoMGN()
training_method.set_epochs(epochs)
training_method.set_processor_size(processor_size)

train_workflow = WorkflowTrainPhysicsNeMo(
    train_manifests=train_manifests,
    val_manifests=val_manifests,
    pca_mean_mesh=ssm_mean_surface_file,
    output_directory=output_dir,
    training_method=training_method,
)
train_result = train_workflow.process()

Run

python tutorials/tutorial_09_lung_train_physicsnemo_mgn.py

Outputs

mgn_stage_model.pt, its metadata and loss/RMSE logs, in the shared weights directory Tutorial 10 reads (tutorials/network_weights/physicsnemo_mgn_lung_motion/, a fresh sibling of it when resuming). The per-case manifests and the held-out evaluation under eval_mgn/ stay in tutorials/output/tutorial_09_lung_mgn/.

Adapt to your data

The contract is the manifest, not the tutorial. Each JSON names a reference mesh, a PCA coefficient file, a target_array name and one entry per phase; the workflow reads that array verbatim, so the target can be displacement — as here — or any per-point quantity of any width. Produce manifests in that shape from your own pipeline and nothing else changes. See PhysicsNeMo AI Surrogates for the schema, and Train a PhysicsNeMo Surrogate for the command-line path.

Tutorial 10: Predict Motion With the Surrogate

Script

tutorials/tutorial_10_lung_infer_physicsnemo_mgn.py

tutorials/tutorial_10_duke_heart_infer_physicsnemo_mgn.py — the same prediction over a cardiac cycle. Needs Duke-Heart-4DLabelmaps (see Before You Start).

Workflow

WorkflowInferPhysicsNeMo for the raw prediction, WorkflowInferMovement to turn it back into geometry, and WorkflowConvertVTKToUSD to export it.

Dataset

Tutorial 8’s fitted surfaces for one case, and Tutorial 9’s checkpoint.

Requirements

The [physicsnemo] extra; otherwise trivial — one forward pass per stage replaces the per-phase registration solve that produced the training data.

Preview

The exported USD scene, played back over the respiratory cycle — every frame a forward pass rather than a registration solve.

The duke_heart variant over a cardiac cycle.

Inner API usage

infer_workflow = WorkflowInferPhysicsNeMo(
    model_directory=model_dir,
    epoch=epoch,
)
infer_result = WorkflowInferMovement(infer_workflow).process_time_series(
    shape_parameters=pca_file,
    stages=stages,
    output_directory=output_dir,
    reference_mesh=reference_file,
    ground_truth=phase_files,
    reference_image=itk.imread(str(reference_ct_file)),
    warp_interpolation="linear",
    warp_background_value=-1000.0,
    usd_project_name=f"{case_id}_mgn_motion",
    anatomy_type="lung",
)

Run

python tutorials/tutorial_10_lung_infer_physicsnemo_mgn.py

Outputs

One predicted surface and one warped CT per stage, one animated USD across all of them, and statistics_per_stage.csv with the mm error against each acquired phase, under tutorials/output/tutorial_10_lung_mgn/<case>/.

Adapt to your data

Change case_id to predict a different subject, or pass stages that were never acquired — which is the point of the surrogate. Omit reference_mesh to displace the mesh reconstructed from the PCA coefficients alone, needing no per-subject geometry at all. Omit reference_image to write meshes without warping anything. Use WorkflowInferPhysicsNeMo on its own to get the raw target array when your model predicts something other than displacement.

Tutorial 11: Score the Surrogate Against the Images

Script

tutorials/tutorial_11_lung_evaluate_physicsnemo.py

tutorials/tutorial_11_duke_heart_evaluate_physicsnemo.py

Workflow

WorkflowEvaluateMovement, driving WorkflowInferMovement and, for the lung variant, SegmentNVSegmentCTMRI.

Dataset

The gated sequence itself — DIR-Lab for the lung, Duke-Heart-4DLabelmaps for the heart — plus Tutorial 8’s fitted surface and Tutorial 9’s checkpoint for the held-out case.

Requirements

The [physicsnemo] extra. The lung variant also segments every gated frame on first run, so it needs a GPU and the segmentation weights; the labelmaps are cached, and a re-run skips them.

Preview

volume_vs_stage.png for the held-out lung case: acquired volume solid, predicted dashed, one pair per lobe across every gated stage.

The same run summarised per lobe. No Dice column — see the note below.

The duke_heart variant, which does report Dice per chamber, alongside its own tutorial_11_duke_heart_volumes.png.

Inner API usage

evaluate = WorkflowEvaluateMovement(
    movement_workflow=WorkflowInferMovement(infer_workflow),
    label_names=lobe_names,
)
result = evaluate.process(
    case_id=case_id,
    shape_parameters=pca_file,
    reference_mesh=reference_mesh_file,
    reference_labelmap=itk.imread(str(reference_labelmap_file)),
    ground_truth_labelmaps=ground_truth_labelmaps,
    output_directory=output_dir,
    evaluation_spacing_mm=2.0,
    include_dice=False,
)

Run

python tutorials/tutorial_11_lung_evaluate_physicsnemo.py

Outputs

evaluation_report.md, evaluation_metrics.csv and volume_vs_stage.png under tutorials/output/tutorial_11_lung/<case>/, carrying volume difference and surface RMSE per lobe at every gated stage; the duke variant adds Dice per chamber. The plot traces each structure’s acquired and predicted volume across the stages. Report and CSV both record the hold-out case name, its shape parameters, and the network weights path with its dates, so a number can be traced back to the run that produced it.

The lung variant passes include_dice=False. Dice is an overlap fraction, so a lobe that moves a few millimeters against its own bulk scores over 0.96 however well or badly the motion is predicted; the column would describe the lobe rather than the model. Chambers change shape enough over a heartbeat for it to discriminate, so the duke variant keeps it.

Adapt to your data

Change LOBE_LABEL_IDS (or HEART_LABEL_IDS) to score a different set of structures — any label your segmenter writes and your reference frame contains. Raise evaluation_spacing_mm if the deformation fields do not fit in memory; lower it to resolve a thin wall, at the cost of its cube.

Tutorial 12: The Whole Inference Pipeline in One Script

Script

tutorials/tutorial_12_lung_end_to_end_inference.py

tutorials/tutorial_12_duke_heart_end_to_end_inference.py

Workflow

WorkflowConvertImageToVTK (lung) or ContourTools (heart), WorkflowFitStatisticalModelToPatient, then process_time_series().

Dataset

The gated sequence alone — DIR-Lab for the lung, Duke-Heart-4DLabelmaps for the heart — plus the Tutorial 6 shape model and the Tutorial 9 checkpoint. Unlike Tutorial 10, nothing is read from Tutorial 8: this script fits the model to the patient itself, so the chain from image to animation runs in one place.

Requirements

The [physicsnemo] extra. The output directory is emptied at the start of every run, so nothing is reused and the reported runtimes are the whole pipeline’s. Neither variant registers a phase — that is what the network replaces, and it is why this runs in minutes where Tutorial 8 runs in hours.

Preview

The whole chain on one DIR-Lab case: segment, fit, infer, animate — no phase registered anywhere in it.

The duke_heart variant, starting from gated labelmaps instead of CT.

Inner API usage

# The fit puts the model in this patient: coefficients condition the
# network, and the fitted surface is what its displacements move.
fit = WorkflowFitStatisticalModelToPatient(
    template_model=pca_mean_surface,
    patient_models=[lung_surface],
    patient_image=reference_image,
    patient_labelmap=lung_labelmap,
)
fit.set_use_pca_registration(
    use_pca_registration=True,
    pca_model=pca_model,
    number_of_pca_components=6,
    use_surface=False,
)
fit_result = fit.process()

infer_result = WorkflowInferMovement(infer_workflow).process_time_series(
    shape_parameters=pca_coefficients_file,
    stages=stages,
    output_directory=output_dir,
    reference_mesh=reference_mesh_file,
    reference_image=reference_image,
    usd_project_name=f"{case_id}_mgn_motion",
    anatomy_type="lung",
)

Run

python tutorials/tutorial_12_lung_end_to_end_inference.py

Outputs

Under tutorials/output/tutorial_12_lung/<case>/ (or tutorial_12_duke_heart): the patient’s fitted <case>_ssm_surface.vtp and <case>_ssm_pca_coefficients.json, one predicted *_pred.vtp surface and one *_warped.mha volume per stage, <case>_mgn_motion.usd animating the whole cycle, and <case>_runtimes.csv timing each step of the run.

Adapt to your data

Point the script at any case of the same cohort by changing case_id; the stages come from the filenames, so a sequence with a different number of phases needs no other change. To predict stages the acquisition never sampled, pass your own stages list — the network is continuous in stage, and nothing downstream requires a matching image.

Tutorial 13: Breathe and Beat a Static Clinical CT

Script

tutorials/tutorial_13_heart_and_lung_motion.py

Workflow

WorkflowInferMovement over both Tutorial 9 networks, WorkflowFitStatisticalModelToPatient for the heart fit, and ConvertVTKToUSD with USDAnatomyTools for the animation.

Dataset

data/Chest-CT/Chest-CT.mha, one ungated breath-hold scan, plus Tutorial 7 (lung)’s fit of it and both Tutorial 9 checkpoints. No 4D acquisition is involved: every deformation comes from a network, none from a registration. See data/Chest-CT/README.md for the data source and required citation.

Requirements

The [physicsnemo] extra, and Simpleware Medical for the heart segmentation. Both segmentations and the heart fit are cached, so a re-run goes straight to inference. Budget disk: 100 combined frames, each with its own warped CT and labelmap, come to roughly 43 GB.

Preview

heart_and_lung_motion.usd: one ungated breath-hold scan, breathing and beating at once, with every deformation coming from a network and none from a registration.

The same per-frame deformation applied to the CT itself — the voxels move with the surfaces, so the scan breathes and beats along with them.

Inner API usage

infer = WorkflowInferMovement(
    WorkflowInferPhysicsNeMo(model_directory=lung_model_dir)
)
# "forward" moves mesh vertices; "inverse" is what resampling an image
# into the stage's frame needs.
field = infer.create_deformation_field(
    shape_parameters=lung_coefficients_file,
    stage=0.0,
    reference_image=patient_image,
    reference_mesh=lung_reference_mesh_file,
    direction="forward",
)
transform = TransformTools().smooth_deformation_field_transform(
    field["deformation_field"], 15.0, field["weight_image"]
)

Run

python tutorials/tutorial_13_heart_and_lung_motion.py

Outputs

Under tutorials/output/tutorial_13_heart_and_lung/: one 4D USD per rhythm (breathing_lungs.usd, beating_heart.usd), 100 combined frames as VTP plus heart_and_lung_motion.usd split by anatomy and painted with organ materials, and the CT and labelmap warped by the same per-frame deformation.

Adapt to your data

Point patient_image_file at your own chest CT and rerun Tutorial 7 (lung) on it to get the lung fit; the heart fit happens inside this script. Change cardiac_cycles_per_phase to re-time the heartbeat against the breath, and the two *_sigma_mm values to change how far each rhythm’s surface motion is carried into the surrounding tissue.

Where to Go Next

• Viewing USD Files — installing an Omniverse Kit application and opening the scenes these tutorials produce.

• Bring Your Own Data - DICOM, Images & VTK to USD — running the workflows on your own DICOM, NRRD or VTK data, including directory layout and conversion.

• API Reference — every workflow, segmenter, registrar and utility class.

• Architecture — how the workflow layer fits together and where to extend it.

• Testing — tests/test_tutorials.py runs these scripts end-to-end behind the --run-tutorials flag.