NV-Segment-CTMR

SegmentNVSegmentCTMRI runs NVIDIA’s NV-Segment-CTMR model (a VISTA3D derivative finetuned on 30K+ CT and MRI scans) and groups its 345-class labelmap into the anatomy masks used by PhysioTwin4D workflows.

Warning

The NV-Segment-CTMR weights are released under the NVIDIA OneWay Non-Commercial License (academic research use only); the surrounding bundle code is Apache 2.0. Use SegmentChestTotalSegmentator or NV-Segment-CT if you need a commercially licensed model.

Class Reference

class physiotwin4d.SegmentNVSegmentCTMRI(log_level=20)[source]

Bases: SegmentAnatomyBase

CT and MRI segmentation using NVIDIA’s NV-Segment-CTMR model.

NV-Segment-CTMR is a VISTA3D-architecture network finetuned on more than 30,000 CT and MRI scans. It covers 345 classes across three modalities (CT_BODY, MRI_BODY, MRI_BRAIN) and, unlike VISTA3D, supports only automatic (label-prompt) segmentation — there is no point-click interactive branch.

Model weights (~872 MB) are downloaded from hf_repo_id on the first call to segmentation_method() and cached by huggingface_hub thereafter.

Labelmap ids are the model’s published class indices, used verbatim. Because those run to 345, SegmentAnatomyBase.labelmap_dtype is np.uint16 rather than the np.uint8 used by the other segmenters.

Anatomy groups (heart, major_vessels, lung, bone, soft_tissue, brain_parcellation) are populated into SegmentAnatomyBase.taxonomy. The first five reuse the names the TotalSegmentator backend uses, so downstream consumers see the same group keys; brain_parcellation is new and renders with the grey-matter entry registered for it in physiotwin4d.usd_anatomy_tools.DEFAULT_RENDER_PARAMS, plus organ-level overrides for the tissues that differ (white matter, CSF-filled ventricles, brainstem, cerebellum, pallidum).

Licensing:

The NV-Segment-CTMR weights are released under the NVIDIA OneWay Non-Commercial License (academic research use only); the surrounding bundle code is Apache 2.0. This is more restrictive than the rest of this repository. NV-Segment-CT (CT only, 132 classes) is the commercially licensed alternative. license_warning is logged at WARNING on the first call to segmentation_method().

target_spacing

1.5mm, matching the model bundle’s internal resampling, so the image is interpolated once rather than twice.

Type:

float

modality

One of modalities; selects the model’s predefined “segment everything” class list. Defaults to "CT_BODY".

Type:

str

modalities

Modalities the model accepts in place of an explicit label_prompt.

Type:

tuple[str, …]

model_cache_dir

Download destination passed to huggingface_hub. None uses the default Hugging Face cache.

Type:

Optional[str]

hf_repo_id

Hugging Face repository holding the bundle and weights.

Type:

str

hf_revision

Pinned commit of hf_repo_id to download.

Type:

str

hf_allow_patterns

Files pulled from hf_repo_id.

Type:

tuple[str, …]

license_warning

Banner logged at WARNING on first use.

Type:

str

The anatomy labels populated by this class are accessed through the inherited SegmentAnatomyBase.taxonomy (taxonomy.labels_in_group("heart") etc.).

Note

SegmentAnatomyBase.fast_mode is ignored: this model has a single network and no reduced-accuracy variant.

Example

>>> segmenter = SegmentNVSegmentCTMRI()
>>> result = segmenter.segment(ct_image)
>>> labelmap = result['labelmap']
>>> heart_labelmap = result['heart']
__init__(log_level=20)[source]

Initialize the NV-Segment-CTMR-based segmentation.

Populates SegmentAnatomyBase.taxonomy with the model’s class indices, then calls SegmentAnatomyBase._finalize_other_group() over the model’s full [1, 346) class index space so unclaimed ids end up in the other group. Constructing the class downloads nothing; weights are fetched lazily by segmentation_method().

Parameters:

log_level (int | str) – Logging level (default: logging.INFO)

set_modality(modality)[source]

Set the modality whose predefined class list the model segments.

Parameters:

modality (str) – One of modalities. CT_BODY segments the 117-class CT set, MRI_BODY the 50-class body MR set, and MRI_BRAIN the 132-class LUMIR brain parcellation.

Raises:

ValueError – If modality is not one of modalities.

Return type:

None

Note

MRI_BRAIN expects a T1 volume that has already been skull-stripped and affinely aligned to the LUMIR template. This class does not perform that preprocessing.

Example

>>> segmenter.set_modality("MRI_BODY")
segmentation_method(preprocessed_image)[source]

Run NV-Segment-CTMR on the preprocessed image and return the result.

The model’s Hugging Face pipeline reads and writes NIfTI files, so the image is written to a temporary file and the prediction read back with ITK. That round trip also handles the coordinate-system conversion between ITK (LPS) and the bundle’s internal RAS orientation.

The bundle inverts its own preprocessing before saving, so the prediction is returned on the same grid as preprocessed_image.

Parameters:

preprocessed_image (itk.image) – The preprocessed CT or MR image with isotropic spacing

Returns:

The segmentation labelmap with NV-Segment-CTMR class

indices, as uint16.

Return type:

itk.image

Raises:

RuntimeError – If the model pipeline produced no output volume.

Note

Requires a CUDA GPU (device cuda:0).

Example

>>> labelmap = segmenter.segmentation_method(preprocessed_ct)

Basic Usage

import itk

from physiotwin4d import SegmentNVSegmentCTMRI

image = itk.imread("chest_ct.nrrd")
segmenter = SegmentNVSegmentCTMRI()

masks = segmenter.segment(image)

heart = masks["heart"]
lungs = masks["lung"]
labelmap = masks["labelmap"]

itk.imwrite(labelmap, "labelmap.nrrd", compression=True)

For MR studies, select the matching modality before calling segment():

segmenter = SegmentNVSegmentCTMRI()
segmenter.set_modality("MRI_BODY")   # or "CT_BODY", "MRI_BRAIN"

MRI_BRAIN expects a T1 volume that has already been skull-stripped and affinely aligned to the LUMIR template; this class does not perform that preprocessing.

Returned Keys

For this segmenter, segment() returns a dictionary with the following keys:

  • labelmap

  • heart

  • major_vessels

  • lung

  • bone

  • soft_tissue

  • brain_parcellation

  • other

Label Ids

Unlike the other segmenters, the labelmap is uint16: label ids are the model’s own published class indices (see configs/label_dict.json in https://github.com/NVIDIA-Medtech/NV-Segment-CTMR), which run to 345. For example, 6 is the aorta and 115 the heart. The full group→id mapping is available through the segmenter’s taxonomy attribute (segmenter.taxonomy.labels_in_group("heart"), segmenter.taxonomy.all_labels()).

Rendering

brain_parcellation is a group name this segmenter introduces. Its group-level entry in physiotwin4d.usd_anatomy_tools.DEFAULT_RENDER_PARAMS is a grey-matter look, which is the right default because most of its labels are cortical gyri or deep grey nuclei (caudate, putamen, thalamus, amygdala, hippocampus).

The brain tissues whose gross appearance genuinely differs from cortex carry organ-level overrides, which win over the group entry on a substring match (longest key first):

  • white_matter - glossy creamy off-white myelin; also claims the cerebellar white matter.

  • 3rd_ventricle, 4th_ventricle, lateral_ventricle, inf_lat_vent - a shared clear-fluid CSF look. Four keys because a bare ventricle key would lose to the heart’s ventricle_left / ventricle_right overrides.

  • brain_stem - pale, fiber-tract dominated.

  • cerebell - darker, browner, more matte cerebellar cortex; matches both cerebellum_exterior_* and cerebellar_vermal_lobules_*.

  • pallidum - myelin-rich, paler than the neighboring putamen and caudate.

  • basal_forebrain - grey matter; present only to outrank the whole-organ brain override.

Note that white_matter_hyperintensity is in the soft_tissue group, not brain_parcellation; its hyperintensity override keeps the lesion dull and matte instead of inheriting the glossy white-matter look.

Operational Notes

The first call to segment() downloads ~872 MB of model weights from https://huggingface.co/nvidia/NV-Segment-CTMR into the Hugging Face cache (override the destination with the model_cache_dir attribute). Inference requires a CUDA GPU.

See Also