Out-of-Distribution (OOD) Detection Tutorial#

Problem Statement#

For most computer vision tasks like image classification and object detection, out-of-distribution (OOD) detection can provide insight into operational drift, or training problems. A way to identify these is through autoencoding reconstruction error.

To help with this, DataEval has an OOD detector that allows a user to identify these images.

When to use#

The OOD_AE class and similar should be used when you would like to find individual images in a dataset which are the most different from the others in the provided set.

What you will need#

A training image dataset with the approximate percentage of known OOD images.
A test image dataset to evaluate for OOD images.
A python environment with the following packages installed:
- dataeval[tensorflow] or dataeval[all]

Setting up#

Let’s import the required libraries needed to set up a minimal working example

import numpy as np

from dataeval.detectors.ood import OOD_AE, OOD_VAEGMM
from dataeval.utils.tensorflow.models import AE, VAEGMM, create_model
from dataeval.utils.torch.datasets import MNIST

WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1730287673.920437    1326 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1730287673.926248    1326 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered

Load the data#

We will use the tensorflow mnist dataset for this tutorial

# Load in the training mnist dataset and use the first 4000
train_ds = MNIST(root="./data/", train=True, download=True, size=4000, unit_interval=True, channels="channels_first")

# Split out the images and labels
images, labels = train_ds.data, train_ds.targets
input_shape = images[0].shape

Files already downloaded and verified

Initialize the model#

Now, lets look at how to use DataEval’s OOD detection methods.
We will focus on a simple autoencoder network from our Alibi Detect provider

detectors = [
    OOD_AE(create_model(AE, input_shape)),
    OOD_VAEGMM(create_model(VAEGMM, input_shape)),
]

W0000 00:00:1730287678.205340    1326 gpu_device.cc:2344] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...

Train the model#

Next we will train a model on the dataset. For better results, the epochs can be increased. We set the threshold to detect the most extreme 1% of training data as out-of-distribution.

for detector in detectors:
    print(f"Training {detector.__class__.__name__}...")
    detector.fit(images, threshold_perc=99, epochs=20, verbose=False)

Training OOD_AE...

Training OOD_VAEGMM...

Test for OOD#

We have trained our detector on a dataset of digits.
What happens when we give it corrupted images of digits (which we expect to be “OOD”)?

corruption = MNIST(
    root="./data",
    train=True,
    download=False,
    size=2000,
    unit_interval=True,
    channels="channels_first",
    corruption="translate",
)
corrupted_images = corruption.data

Files already downloaded and verified

Now we evaluate the two datasets using the trained model.

[(type(detector).__name__, np.mean(detector.predict(images).is_ood)) for detector in detectors]

[('OOD_AE', 0.01), ('OOD_VAEGMM', 0.01225)]

[(type(detector).__name__, np.mean(detector.predict(corrupted_images).is_ood)) for detector in detectors]

[('OOD_AE', 0.958), ('OOD_VAEGMM', 0.1115)]

Results#

We can see that the Autoencoder based OOD detector was able to identify most of the translated images as outliers, while the AEGMM was resilient to the perturbation.

Depending on your needs, certain outlier detectors will work better under specific conditions.