Bayes Error Rate Estimation Tutorial¶

Problem Statement¶

For classification machine learning tasks, there is an inherent difficulty associated with signal to noise ratio in the images. One way of quantifying this difficulty is the Bayes Error Rate, or irreducable error.

DataEval has introduced a method of calculating this error rate that uses image embeddings.

When to use¶

The BER metric should be used when you would like to measure the feasibility of a machine learning task. For example, if you have an operational accuracy requirement of 80%, and would like to know if this is feasibly achievable given the imagery.

What you will need¶

A set of image embeddings and their corresponding class labels. This requires training an autoencoder to compress the images.
A python environment with the following packages installed:
- dataeval[all]

Setting up¶

Let’s import the required libraries needed to set up a minimal working example

import numpy as np

from dataeval.metrics.estimators import ber
from dataeval.utils.dataset.datasets import MNIST

Loading in data¶

Let’s start by loading in torchvision’s mnist dataset, then we will examine it

# Load in both the training and testing mnist dataset
train_ds = MNIST(root="./data/", train=True, download=True, flatten=True)
test_ds = MNIST(root="./data/", train=False, download=True, channels="channels_last")

# Split out the images and labels for each set
images, labels = train_ds.data, train_ds.targets
test_images, test_labels = test_ds.data, test_ds.targets

Files already downloaded and verified

Files already downloaded and verified

print("Number of training samples: ", len(images))
print("Image shape:", images[0].shape)
print("Label counts: ", np.unique(labels, return_counts=True))

Number of training samples:  54210
Image shape: (784,)
Label counts:  (array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=uint8), array([5421, 5421, 5421, 5421, 5421, 5421, 5421, 5421, 5421, 5421]))

To highlight the effects of modifying the dataset on its Bayes Error Rate, we will only include a subset of 6,000 images and their labels for digits 1, 4, and 9

images_split = {}
labels_split = {}

# Keep only 1, 4, and 9
for label in (1, 4, 9):
    subset_indices = np.where(labels == label)
    images_split[label] = images[subset_indices][:2000]
    labels_split[label] = labels[subset_indices][:2000]

images_subset = np.concatenate(list(images_split.values()))
labels_subset = np.concatenate(list(labels_split.values()))
print(images_subset.shape)
print(np.unique(labels_subset, return_counts=True))

(6000, 784)
(array([1, 4, 9], dtype=uint8), array([2000, 2000, 2000]))

We have taken a subset of the data that is only the digits 1, 4, and 9. The BER estimate requires 1 dimension, that’s why we have flattened images. This is ok since MNIST images are small, in practice we would need to do some dimension reduction (autoencoder) here.

We now have 6,000 flattened images of size 784. Next we can move on to evaluation of the dataset.

Evaluation¶

Suppose we would like to build a classifier that differentiates between the handwritten digits 1, 4, and 9 with predetermined accuracy requirement of 99%.

We will use BER to check the feasibility of the task. As the images are small, we can simple use the flattened raw pixel intensities to calculate BER (no embedding necessary). Note: This will not be the case in general.

# Evaluate the BER metric for the MNIST data with digits 1, 4, 9.
# One minus the value of this metric gives our estimate of the upper bound on accuracy.
base_result = ber(images_subset, labels_subset, method="MST")

print("The bayes error rate estimation:", base_result.ber)

The bayes error rate estimation: 0.022833333333333334

The estimate of the maximum achievable accuracy is one minus the BER estimate.

print("The maximum achievable accuracy:", 1 - base_result.ber)

The maximum achievable accuracy: 0.9771666666666666

Results¶

The maximum achievable accuracy on a dataset of 1, 4, and 9 is about 97.7%. This does not meet our requirement of 99% accuracy!

Modify dataset classification¶

To address insufficient accuracy, lets modify the dataset to classify an image as “1” or “Not a 1”. By combining classes, we can hopefully achieve the desired level of attainable accuracy.

# Creates a binary mask where current label == 1 that can be used as the new labels
labels_merged = labels_subset == 1
print("New label counts:", np.unique(labels_merged, return_counts=True))

New label counts: (array([False,  True]), array([4000, 2000]))

# Evaluate the BER metric for the MNIST data with updated labels
new_result = ber(images_subset, labels_merged, method="MST")

print("The bayes error rate estimation:", new_result)

The bayes error rate estimation: BEROutput: {'ber': np.float64(0.005333333333333333), 'ber_lower': np.float64(0.002673815958446346)}

The estimate of the maximum achievable accuracy is one minus the BER estimate.

print("The maximum achievable accuracy:", 1 - new_result.ber)

The maximum achievable accuracy: 0.9946666666666667

Results¶

The maximum achievable accuracy on a dataset of 1 and not 1 (4, 9) is about 99.5%. This does meet our accuracy requirement.

By using BER to check for feasibility early on, we were able to reformulate the problem such that it is feasible under our specifications