torch#

PyTorch is the primary backend for metrics that require neural networks.

While these metrics can take in custom models, DataEval provides utility classes to create a seamless integration between custom models and DataEval’s metrics.

Functions#

dataeval.utils.torch.read_dataset(dataset: Dataset[Any]) → list[list[Any]]#

Extract information from a dataset at each index into individual lists of each information position

Parameters:

dataset (torch.utils.data.Dataset) – Input dataset

Returns:

All objects in individual lists based on return position from dataset

Return type:

List[List[Any]]

Warning

No type checking is done between lists or data inside lists

See also

torch.utils.data.Dataset

Examples

>>> import numpy as np
>>> data = np.ones((10, 1, 3, 3))
>>> labels = np.ones((10,))
>>> class ICDataset:
...     def __init__(self, data, labels):
...         self.data = data
...         self.labels = labels
...
...     def __getitem__(self, idx):
...         return self.data[idx], self.labels[idx]

>>> ds = ICDataset(data, labels)

>>> result = read_dataset(ds)
>>> len(result)  # images and labels
2
>>> np.asarray(result[0]).shape  # images
(10, 1, 3, 3)
>>> np.asarray(result[1]).shape  # labels
(10,)

Datasets#

class dataeval.utils.torch.datasets.CIFAR10(root: str | Path, train: bool = True, transform: Callable | None = None, target_transform: Callable | None = None, download: bool = False)#

CIFAR10 Dataset.

Parameters:

  • root (str or pathlib.Path) – Root directory of dataset where directory cifar-10-batches-py exists or will be saved to if download is set to True.

  • train (bool, optional) – If True, creates dataset from training set, otherwise creates from test set.

  • transform (callable, optional) – A function/transform that takes in a PIL image and returns a transformed version. E.g, transforms.RandomCrop

  • target_transform (callable, optional) – A function/transform that takes in the target and transforms it.

  • download (bool, optional) – If true, downloads the dataset from the internet and puts it in root directory. If dataset is already downloaded, it is not downloaded again.

class dataeval.utils.torch.datasets.MNIST(root: str | Path, train: bool = True, download: bool = False, size: int = -1, unit_interval: bool = False, dtype: type | None = None, channels: Literal['channels_first', 'channels_last'] | None = None, flatten: bool = False, normalize: tuple[float, float] | None = None, corruption: Literal['identity', 'shot_noise', 'impulse_noise', 'glass_blur', 'motion_blur', 'shear', 'scale', 'rotate', 'brightness', 'translate', 'stripe', 'fog', 'spatter', 'dotted_line', 'zigzag', 'canny_edges'] | None = None, classes: TClassMap | None = None, balance: bool = True, randomize: bool = True, slice_back: bool = False, verbose: bool = True)#

MNIST Dataset and Corruptions.

Parameters:

  • root – str | pathlib.Path Root directory of dataset where the mnist_c/ folder exists.

  • train – bool, default True If True, creates dataset from train_images.npy and train_labels.npy.

  • download – bool, default False If True, downloads the dataset from the internet and puts it in root directory. If dataset is already downloaded, it is not downloaded again.

  • size – int, default -1 Limit the dataset size, must be a value greater than 0.

  • unit_interval – bool, default False Shift the data values to the unit interval [0-1].

  • dtype – type | None, default None Change the NumPy dtype - data is loaded as np.uint8

  • channels – Literal[‘channels_first’ | ‘channels_last’] | None, default None Location of channel axis if desired, default has no channels (N, 28, 28)

  • flatten – bool, default False Flatten data into single dimension (N, 784) - cannot use both channels and flatten, channels takes priority over flatten.

  • normalize – tuple[mean, std] | None, default None Normalize images acorrding to provided mean and standard deviation

  • corruption – Literal[‘identity’ | ‘shot_noise’ | ‘impulse_noise’ | ‘glass_blur’ | ‘motion_blur’ | ‘shear’ | ‘scale’ | ‘rotate’ | ‘brightness’ | ‘translate’ | ‘stripe’ | ‘fog’ | ‘spatter’ | ‘dotted_line’ | ‘zigzag’ | ‘canny_edges’] | None, default None The desired corruption style or None.

  • classes – Literal[“zero”, “one”, “two”, “three”, “four”, “five”, “six”, “seven”, “eight”, “nine”] | int | list[int] | list[Literal[“zero”, “one”, “two”, “three”, “four”, “five”, “six”, “seven”, “eight”, “nine”]] | None, default None Option to select specific classes from dataset.

  • balance – bool, default True If True, returns equal number of samples for each class.

  • randomize – bool, default True If True, shuffles the data prior to selection - uses a set seed for reproducibility.

  • slice_back – bool, default False If True and size has a value greater than 0, then grabs selection starting at the last image.

  • verbose – bool, default True If True, outputs print statements.

class dataeval.utils.torch.datasets.VOCDetection(root: str | Path, year: str = '2012', image_set: str = 'train', download: bool = False, transform: Callable | None = None, target_transform: Callable | None = None, transforms: Callable | None = None)#

Pascal VOC Detection Dataset.

Parameters:

  • root (str or pathlib.Path) – Root directory of the VOC Dataset.

  • year (string, optional) – The dataset year, supports years "2007" to "2012".

  • image_set (string, optional) – Select the image_set to use, "train", "trainval" or "val". If year=="2007", can also be "test".

  • download (bool, optional) – If true, downloads the dataset from the internet and puts it in root directory. If dataset is already downloaded, it is not downloaded again. (default: alphabetic indexing of VOC’s 20 classes).

  • transform (callable, optional) – A function/transform that takes in a PIL image and returns a transformed version. E.g, transforms.RandomCrop

  • target_transform (callable, required) – A function/transform that takes in the target and transforms it.

  • transforms (callable, optional) – A function/transform that takes input sample and its target as entry and returns a transformed version.

Models#

class dataeval.utils.torch.models.AriaAutoencoder(channels: int = 3)#

An autoencoder model with a separate encoder and decoder.

Parameters:

channels (int, default 3) – Number of input channels

class dataeval.utils.torch.models.Decoder(channels: int)#

A simple decoder to be used in an autoencoder model.

This is the decoder used by the AriaAutoencoder model.

Parameters:

channels (int) – Number of output channels

class dataeval.utils.torch.models.Encoder(channels: int = 3)#

A simple encoder to be used in an autoencoder model.

This is the encoder used by the AriaAutoencoder model.

Parameters:

channels (int, default 3) – Number of input channels

Trainer#

class dataeval.utils.torch.trainer.AETrainer(model: Module, device: str | device = 'auto', batch_size: int = 8)#

A class to train and evaluate an autoencoder<Autoencoder>` model.

Parameters:

  • model (nn.Module) – The model to be trained.

  • device (str or torch.device, default "auto") – The hardware device to use for training. If “auto”, the device will be set to “cuda” if available, otherwise “cpu”.

  • batch_size (int, default 8) – The number of images to process in a batch.

encode(dataset: Dataset[Any]) → Tensor#

Create image embeddings for the dataset using the model’s encoder.

If the model has an encode method, it will be used; otherwise, model.forward will be used.

Parameters:

dataset (Dataset) – The dataset to encode. Torch Dataset containing images in the first return position.

Returns:

Data encoded by the model

Return type:

torch.Tensor

Note

This function should be run after the model has been trained and evaluated.

eval(dataset: Dataset[Any]) → float#

Basic image reconstruction evaluation function for autoencoder models

Uses torch.nn.MSELoss as default loss function.

Parameters:

dataset (Dataset) – The dataset to evaluate on. Torch Dataset containing images in the first return position.

Returns:

Total reconstruction loss over the entire dataset

Return type:

float

Note

To replace this function with a custom function, do:

AETrainer.eval = custom_function

train(dataset: Dataset[Any], epochs: int = 25) → list[float]#

Basic image reconstruction training function for Autoencoder models

Uses torch.optim.Adam and torch.nn.MSELoss as default hyperparameters

Parameters:

  • dataset (Dataset) – The dataset to train on. Torch Dataset containing images in the first return position.

  • epochs (int, default 25) – Number of full training loops

Returns:

A list of average loss values for each epoch.

Return type:

List[float]

Note

To replace this function with a custom function, do:

AETrainer.train = custom_function