Class Parity Label Analysis Tutorial

Problem Statement

For machine learning tasks, a discrepancy in label frequencies between train and test datasets can result in poor model performance.

To help with this, DataEval has a tool that compares the label distributions of two datasets.

When to use

The Parity class and similar should be used when you would like to determine if two datasets have statistically independent labels.

What you will need

1. A labeled training image dataset

2. A labeled test image dataset to evaluate the label distribution of

Setting up

Let’s import the required libraries needed to set up a minimal working example

from dataeval.metrics.bias import label_parity
from dataeval.utils.torch.datasets import MNIST

WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1730234217.944678    6747 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1730234217.955283    6747 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered

Load the data

We will use the MNIST dataset from torchvision for this tutorial on class label statistics

train_ds = MNIST("./data", train=True, download=True, size=2000)
test_ds = MNIST("./data", train=False, download=True, size=500)

# Take a subset of 2000 training images and 500 test images
train_labels = train_ds.targets
test_labels = test_ds.targets

Files already downloaded and verified

Files already downloaded and verified

Evaluate label statistical independence

Now, let’s look at how to use DataEval’s label statistics analyzer. Start by initializing a Parity object. Compute the chi-squared value of hypothesis that test_ds has the same class distribution as train_ds by specifying the two datasets to be compared, as well as the number of unique classes (for MNIST, there are 10 unique classes). It also returns the p-value of the test.

results = label_parity(train_labels, test_labels)
print(f"The chi-squared value for the two label distributions is {results.score}, with p-value {results.p_value}")

The chi-squared value for the two label distributions is 0.0, with p-value 1.0