Class Parity Label Analysis Tutorial¶

Problem Statement¶

For machine learning tasks, a discrepancy in label frequencies between train and test datasets can result in poor model performance.

To help with this, DataEval has a tool that compares the label distributions of two datasets.

When to use¶

The Parity class and similar should be used when you would like to determine if two datasets have statistically independent labels.

What you will need¶

1. A labeled training image dataset

2. A labeled test image dataset to evaluate the label distribution of

Setting up¶

Let’s import the required libraries needed to set up a minimal working example

from dataeval.metrics.bias import label_parity
from dataeval.utils.dataset.datasets import MNIST

Load the data¶

We will use the MNIST dataset from torchvision for this tutorial on class label statistics

train_ds = MNIST("./data", train=True, download=True, size=2000)
test_ds = MNIST("./data", train=False, download=True, size=500)

# Take a subset of 2000 training images and 500 test images
train_labels = train_ds.targets
test_labels = test_ds.targets

Files already downloaded and verified

Files already downloaded and verified

Evaluate label statistical independence¶

Now, let’s look at how to use DataEval’s label statistics analyzer. Start by initializing a Parity object. Compute the chi-squared value of hypothesis that test_ds has the same class distribution as train_ds by specifying the two datasets to be compared, as well as the number of unique classes (for MNIST, there are 10 unique classes). It also returns the p-value of the test.

results = label_parity(train_labels, test_labels)
print(f"The chi-squared value for the two label distributions is {results.score}, with p-value {results.p_value}")

The chi-squared value for the two label distributions is 0.0, with p-value 1.0