WSIStreamDataset

class WSIStreamDataset(ioconfig, wsi_paths, mp_shared_space, preproc=None, mode='wsi')[source]

Reading a wsi in parallel mode with persistent workers.

To speed up the inference process for multiple WSIs. The torch.utils.data.Dataloader is set to run in persistent mode. Normally, this will prevent workers from altering their initial states (such as provided input etc.). To sidestep this, we use a shared parallel workspace context manager to send data and signal from the main thread, thus allowing each worker to load a new wsi as well as corresponding patch information.

Parameters:

  • mp_shared_space (Namespace) – A shared multiprocessing space, must be from torch.multiprocessing.

  • ioconfig (IOSegmentorConfig) – An object which contains I/O placement for patches.

  • wsi_paths (list) – List of paths pointing to a WSI or tiles.

  • preproc (Callable) – Pre-processing function to be applied to a patch.

  • mode (str) – Either “wsi” or “tile” to indicate the format of images in wsi_paths.

Examples

>>> ioconfig = IOSegmentorConfig(
...     input_resolutions=[{"units": "baseline", "resolution": 1.0}],
...     output_resolutions=[{"units": "baseline", "resolution": 1.0}],
...     patch_input_shape=[2048, 2048],
...     patch_output_shape=[1024, 1024],
...     stride_shape=[512, 512],
... )
>>> mp_manager = torch_mp.Manager()
>>> mp_shared_space = mp_manager.Namespace()
>>> mp_shared_space.signal = 1  # adding variable to the shared space
>>> wsi_paths = ['A.svs', 'B.svs']
>>> wsi_dataset = WSIStreamDataset(ioconfig, wsi_paths, mp_shared_space)

Initialize WSIStreamDataset.

Methods

collate_fn

Prototype to handle reading exception.
static collate_fn(batch)[source]

Prototype to handle reading exception.

This will exclude any sample with None from the batch. As such, wrapping __getitem__ with try-catch and return None upon exceptions will prevent crashing the entire program. But as a side effect, the batch may not have the size as defined.

Parameters:

batch (list | ndarray)

Return type:

Tensor