Drift Detection Tutorial Using Multiple Drift Detectors¶

Problem Statement¶

When evaluating and monitoring data after model deployment, it is important to test incoming data for potential drift which may affect model performance.

When to use¶

The dataeval.detectors drift detection classes should be used when you would like to measure new data for operational drift.

What you will need¶

1. A set of image embeddings for each dataset (usually obtained with an AutoEncoder)

2. A Python environment with the following packages installed:

   • dataeval or dataeval[all]

Setting up¶

Let’s import the required libraries needed to set up a minimal working example

from functools import partial

import numpy as np
import torch

from dataeval.detectors.drift import (
    DriftCVM,
    DriftKS,
    DriftMMD,
    preprocess_drift,
)
from dataeval.utils.dataset.datasets import MNIST
from dataeval.utils.torch.models import Autoencoder

device = "cuda" if torch.cuda.is_available() else "cpu"

Loading in data¶

Let’s start by loading in torchvision’s mnist dataset, then we will examine it

# Load in the training mnist dataset and use the first 4000
train_ds = MNIST(root="./data/", train=True, download=True, size=4000, dtype=np.float32, channels="channels_first")

# Split out the images and labels
images, labels = train_ds.data, train_ds.targets

Files already downloaded and verified

print("Number of samples: ", len(images))
print("Image shape:", images[0].shape)

Number of samples:  4000
Image shape: (1, 28, 28)

Test reference against control¶

Let’s check for drift between the first 2000 images and the second 2000 images from this sample.

data_reference = images[0:2000]
data_control = images[2000:]

In order to reduce the dimensionality of the data, we can set a simple Autoencoder to the preprocess_fn. While this is optional for the MNIST data set, it is highly recommended for datasets that have higher dimensionality.

For the purposes of the tutorial, we will use 3 forms of drift detectors: Maximum Mean Discrepancy (MMD), Cramér-von Mises (CVM), and Kolmogorov-Smirnov (KS).

# define encoder
encoder_net = Autoencoder(1).encoder.to(device)

# define preprocessing function
preprocess_fn = partial(preprocess_drift, model=encoder_net, batch_size=64, device=device)

# initialise drift detectors
detectors = [detector(data_reference, preprocess_fn=preprocess_fn) for detector in [DriftMMD, DriftCVM, DriftKS]]

We estimate that the test for drift is false for all detectors as both the reference and test data set is from the same MNIST training dataset.

results = {type(detector).__name__: detector.predict(data_control).is_drift for detector in detectors}
print(results)

{'DriftMMD': False, 'DriftCVM': False, 'DriftKS': False}

Loading in corrupted data¶

Now let’s load in a corrupted MNIST dataset.

corruption = MNIST(
    root="./data",
    train=True,
    download=False,
    size=2000,
    dtype=np.float32,
    channels="channels_first",
    corruption="translate",
)
corrupted_images = corruption.data

Files already downloaded and verified

print("Number of corrupted samples: ", len(corrupted_images))
print("Corrupted image shape:", corrupted_images[0].shape)

Number of corrupted samples:  2000
Corrupted image shape: (1, 28, 28)

Check for drift against corrupted data¶

Test for drift between the corrupted dataset and the original reference set using all 3 detectors.

corrupted = {type(detector).__name__: detector.predict(corrupted_images).is_drift for detector in detectors}
print(corrupted)

{'DriftMMD': True, 'DriftCVM': True, 'DriftKS': True}

We conclude that the translated MNIST images are significantly different from the original images according to all 3 measures of drift.