SemanticSegmentor

tiatoolbox . models . engine . semantic_segmentor . SemanticSegmentor

class SemanticSegmentor(batch_size=8, num_loader_workers=0, num_postproc_workers=0, model=None, pretrained_model=None, pretrained_weights=None, dataset_class=<class 'tiatoolbox.models.engine.semantic_segmentor.WSIStreamDataset'>, *, verbose=True, auto_generate_mask=False)

Pixel-wise segmentation predictor.

The tiatoolbox model should produce the following results on the BCSS dataset using fcn_resnet50_unet-bcss.

Semantic segmentation performance on the BCSS dataset

	Tumour	Stroma	Inflammatory	Necrosis	Other	All
Amgad et al.	0.851	0.800	0.712	0.723	0.666	0.750
TIAToolbox	0.885	0.825	0.761	0.765	0.581	0.763

Note, if model is supplied in the arguments, it will ignore the pretrained_model and pretrained_weights arguments.

Parameters:

• model (nn.Module) – Use externally defined PyTorch model for prediction with weights already loaded. Default is None. If provided, pretrained_model argument is ignored.

• pretrained_model (str) – Name of the existing models support by tiatoolbox for processing the data. For a full list of pretrained models, refer to the docs. By default, the corresponding pretrained weights will also be downloaded. However, you can override with your own set of weights via the pretrained_weights argument. Argument is case-insensitive.

• pretrained_weights (str) – Path to the weight of the corresponding pretrained_model.

• batch_size (int) – Number of images fed into the model each time.

• num_loader_workers (int) – Number of workers to load the data. Take note that they will also perform preprocessing.

• num_postproc_workers (int) – This value is there to maintain input compatibility with tiatoolbox.models.classification and is not used.

• verbose (bool) – Whether to output logging information.

• dataset_class (obj) – Dataset class to be used instead of default.

• auto_generate_mask (bool) – To automatically generate tile/WSI tissue mask if is not provided.

process_prediction_per_batch

A flag to denote whether post-processing for inference output is applied after each batch or after finishing an entire tile or WSI.

Type:

bool

Examples

>>> # Sample output of a network
>>> wsis = ['A/wsi.svs', 'B/wsi.svs']
>>> predictor = SemanticSegmentor(model='fcn-tissue_mask')
>>> output = predictor.predict(wsis, mode='wsi')
>>> list(output.keys())
[('A/wsi.svs', 'output/0.raw') , ('B/wsi.svs', 'output/1.raw')]
>>> # if a network have 2 output heads, each head output of 'A/wsi.svs'
>>> # will be respectively stored in 'output/0.raw.0', 'output/0.raw.1'

Initialize SemanticSegmentor.

Methods

filter_coordinates	Indicates which coordinate is valid basing on the mask.
get_coordinates	Calculate patch tiling coordinates.
get_reader	Define how to get reader for mask and source image.
merge_prediction	Merge patch-level predictions to form a 2-dimensional prediction map.
predict	Make a prediction for a list of input data.

static filter_coordinates(mask_reader, bounds, resolution=None, units=None)

Indicates which coordinate is valid basing on the mask.

To use your own approaches, either subclass to overwrite or directly assign your own function to this name. In either cases, the function must obey the API defined here.

Parameters:

• mask_reader (VirtualReader) – A virtual pyramidal reader of the mask related to the WSI from which we want to extract the patches.

• bounds (ndarray and np.int32) – Coordinates to be checked via the func. They must be in the same resolution as requested resolution and units. The shape of coordinates is (N, K) where N is the number of coordinate sets and K is either 2 for centroids or 4 for bounding boxes. When using the default func=None, K should be 4, as we expect the coordinates to be bounding boxes in [start_x, start_y, end_x, end_y] format.

• resolution (Resolution) – Resolution of the requested patch.

• units (Units) – Units of the requested patch.

Returns:

List of flags to indicate which coordinate is valid.

Return type:

numpy.ndarray

Examples

>>> # API of function expected to overwrite `filter_coordinates`
>>> def func(reader, bounds, resolution, units):
...   # as example, only select first bound
...   return np.array([1, 0])
>>> coords = [[0, 0, 256, 256], [128, 128, 384, 384]]
>>> segmentor = SemanticSegmentor(model='unet')
>>> segmentor.filter_coordinates = func

static get_coordinates(image_shape, ioconfig)

Calculate patch tiling coordinates.

By default, internally, it will call the PatchExtractor.get_coordinates. To use your own approach, either subclass to overwrite or directly assign your own function to this name. In either cases, the function must obey the API defined here.

Parameters:

• image_shape (tuple(int), numpy.ndarray) – This argument specifies the shape of mother image (the image we want to extract patches from) at requested resolution and units and it is expected to be in (width, height) format.

• ioconfig (IOSegmentorConfig) – Object that contains information about input and output placement of patches. Check IOSegmentorConfig for details about available attributes.

Returns:

List of patch inputs and outputs

• list - patch_inputs:

  A list of corrdinates in [start_x, start_y, end_x, end_y] format indicating the read location of the patch in the mother image.

• list - patch_outputs:

  A list of corrdinates in [start_x, start_y, end_x, end_y] format indicating to write location of the patch in the mother image.

Return type:

tuple

Examples

>>> # API of function expected to overwrite `get_coordinates`
>>> def func(image_shape, ioconfig):
...   patch_inputs = np.array([[0, 0, 256, 256]])
...   patch_outputs = np.array([[0, 0, 256, 256]])
...   return patch_inputs, patch_outputs
>>> segmentor = SemanticSegmentor(model='unet')
>>> segmentor.get_coordinates = func

static get_reader(img_path, mask_path, mode, *, auto_get_mask)

Define how to get reader for mask and source image.

Parameters:

• img_path (str | Path)

• mask_path (str | Path)

• mode (str)

• auto_get_mask (bool)

Return type:

tuple[WSIReader, WSIReader]

static merge_prediction(canvas_shape, predictions, locations, save_path=None, cache_count_path=None)

Merge patch-level predictions to form a 2-dimensional prediction map.

When accumulating the raw prediction onto a same canvas (via calling the function multiple times), save_path and cache_count_path must be the same. If either of these two do not exist, the function will create new files. However, if save_path is None, the function will perform the accumulation using CPU-RAM as storage.

Parameters:

• canvas_shape (numpy.ndarray) – HW of the supposed assembled image.

• predictions (list) – List of np.ndarray, each item is a patch prediction, assuming to be of shape HWC.

• locations (list) – List of np.ndarray, each item is the location of the patch at the same index within predictions. The location is in the to be assembled canvas and of the form (top_left_x, top_left_y, bottom_right_x, bottom_right_x).

• save_path (str) – Location to save the assembled image.

• cache_count_path (str) – Location to store the canvas for counting how many times each pixel get overlapped when assembling.

Returns:

An image contains merged data.

Return type:

numpy.ndarray

Examples: >>> SemanticSegmentor.merge_prediction( … canvas_shape=[4, 4], … predictions=[ … np.full((2, 2), 1), … np.full((2, 2), 2)], … locations=[ … [0, 0, 2, 2], … [2, 2, 4, 4]], … save_path=None, … ) … array([[1, 1, 0, 0], … [1, 1, 0, 0], … [0, 0, 2, 2], … [0, 0, 2, 2]])

predict(imgs, masks=None, mode='tile', ioconfig=None, patch_input_shape=None, patch_output_shape=None, stride_shape=None, resolution=1.0, units='baseline', save_dir=None, *, on_gpu=True, crash_on_exception=False)

Make a prediction for a list of input data.

By default, if the input model at the object instantiation time is a pretrained model in the toolbox as well as patch_input_shape, patch_output_shape, stride_shape, resolution, units and ioconfig are None. The method will use the ioconfig retrieved together with the pretrained model. Otherwise, either patch_input_shape, patch_output_shape, stride_shape, resolution, units or ioconfig must be set else a Value Error will be raised.

Parameters:

• imgs (list, ndarray) – List of inputs to process. When using “patch” mode, the input must be either a list of images, a list of image file paths or a numpy array of an image list. When using “tile” or “wsi” mode, the input must be a list of file paths.

• masks (list) – List of masks. Only utilised when processing image tiles and whole-slide images. Patches are only processed if they are within a masked area. If not provided, then a tissue mask will be automatically generated for whole-slide images or the entire image is processed for image tiles.

• mode (str) – Type of input to process. Choose from either tile or wsi.

• ioconfig (IOSegmentorConfig) – Object defines information about input and output placement of patches. When provided, patch_input_shape, patch_output_shape, stride_shape, resolution, and units arguments are ignored. Otherwise, those arguments will be internally converted to a IOSegmentorConfig object.

• on_gpu (bool) – Whether to run the model on the GPU.

• patch_input_shape (tuple) – Size of patches input to the model. The values are at requested read resolution and must be positive.

• patch_output_shape (tuple) – Size of patches output by the model. The values are at the requested read resolution and must be positive.

• stride_shape (tuple) – Stride using during tile and WSI processing. The values are at requested read resolution and must be positive. If not provided, stride_shape=patch_input_shape is used.

• resolution (float) – Resolution used for reading the image.

• units (Units) – Units of resolution used for reading the image. Choose from either “level”, “power” or “mpp”.

• save_dir (str or pathlib.Path) – Output directory when processing multiple tiles and whole-slide images. By default, it is folder output where the running script is invoked.

• crash_on_exception (bool) – If True, the running loop will crash if there is any error during processing a WSI. Otherwise, the loop will move on to the next wsi for processing.

• self (SemanticSegmentor)

Returns:

A list of tuple(input_path, save_path) where input_path is the path of the input wsi while save_path corresponds to the output predictions.

Return type:

list

Examples

>>> # Sample output of a network
>>> wsis = ['A/wsi.svs', 'B/wsi.svs']
>>> predictor = SemanticSegmentor(model='fcn-tissue_mask')
>>> output = predictor.predict(wsis, mode='wsi')
>>> list(output.keys())
[('A/wsi.svs', 'output/0.raw') , ('B/wsi.svs', 'output/1.raw')]
>>> # if a network have 2 output heads, each head output of 'A/wsi.svs'
>>> # will be respectively stored in 'output/0.raw.0', 'output/0.raw.1'