{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bayes Error Rate Estimation Tutorial\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## _Problem Statement_\n",
    "\n",
    "For classification machine learning tasks, there is an _inherent difficulty_ associated with signal to noise ratio in the images. One way of quantifying this difficulty is the Bayes Error Rate, or irreducable error.\n",
    "\n",
    "DataEval has introduced a method of calculating this error rate that uses image embeddings.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### _When to use_\n",
    "\n",
    "The `BER` metric should be used when you would like to measure the feasibility of a machine learning task. For example, if you have an operational accuracy requirement of 80%, and would like to know if this is feasibly achievable given the imagery.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### _What you will need_\n",
    "\n",
    "1. A set of image embeddings and their corresponding class labels. This requires training an autoencoder to compress the images.\n",
    "2. A python environment with the following packages installed:\n",
    "   - `dataeval[all]`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### _Setting up_\n",
    "\n",
    "Let's import the required libraries needed to set up a minimal working example\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "remove_cell"
    ]
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    import google.colab  # noqa: F401\n",
    "\n",
    "    # specify the version of DataEval (==X.XX.X) for versions other than the latest\n",
    "    %pip install -q dataeval[all]\n",
    "except Exception:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "from dataeval.metrics.estimators import ber\n",
    "from dataeval.utils.torch.datasets import MNIST"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading in data\n",
    "\n",
    "Let's start by loading in torchvision's mnist dataset,\n",
    "then we will examine it\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load in both the training and testing mnist dataset\n",
    "train_ds = MNIST(root=\"./data/\", train=True, download=True, flatten=True)\n",
    "test_ds = MNIST(root=\"./data/\", train=False, download=True, channels=\"channels_last\")\n",
    "\n",
    "# Split out the images and labels for each set\n",
    "images, labels = train_ds.data, train_ds.targets\n",
    "test_images, test_labels = test_ds.data, test_ds.targets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Number of training samples: \", len(images))\n",
    "print(\"Image shape:\", images[0].shape)\n",
    "print(\"Label counts: \", np.unique(labels, return_counts=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To highlight the effects of modifying the dataset on its Bayes Error Rate,\n",
    "we will only include a subset of 6,000 images and their labels for digits 1, 4, and 9\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "images_split = {}\n",
    "labels_split = {}\n",
    "\n",
    "# Keep only 1, 4, and 9\n",
    "for label in (1, 4, 9):\n",
    "    subset_indices = np.where(labels == label)\n",
    "    images_split[label] = images[subset_indices][:2000]\n",
    "    labels_split[label] = labels[subset_indices][:2000]\n",
    "\n",
    "images_subset = np.concatenate(list(images_split.values()))\n",
    "labels_subset = np.concatenate(list(labels_split.values()))\n",
    "print(images_subset.shape)\n",
    "print(np.unique(labels_subset, return_counts=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have taken a subset of the data that is only the digits 1, 4, and 9.\n",
    "The BER estimate requires 1 dimension, that's why we have flattened images. This is ok since MNIST images are small, in practice we would need to do some dimension reduction (autoencoder) here.\n",
    "\n",
    "We now have 6,000 flattened images of size 784. Next we can move on to evaluation of the dataset.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "\n",
    "Suppose we would like to build a classifier that differentiates between the handwritten digits 1, 4, and 9 with predetermined accuracy requirement of 99%.\n",
    "\n",
    "We will use BER to check the feasibility of the task.\n",
    "As the images are small, we can simple use the flattened raw pixel intensities to calculate BER (no embedding necessary).\n",
    "_Note_: This will not be the case in general.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Evaluate the BER metric for the MNIST data with digits 1, 4, 9.\n",
    "# One minus the value of this metric gives our estimate of the upper bound on accuracy.\n",
    "base_result = ber(images_subset, labels_subset, method=\"MST\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"The bayes error rate estimation:\", base_result.ber)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "remove_cell"
    ]
   },
   "outputs": [],
   "source": [
    "### TEST ASSERTION CELL ###\n",
    "assert 0.976 < 1 - base_result.ber < 0.978"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The estimate of the maximum achievable accuracy is one minus the BER estimate.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"The maximum achievable accuracy:\", 1 - base_result.ber)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results\n",
    "\n",
    "The maximum achievable accuracy on a dataset of 1, 4, and 9 is about 97.7%.\n",
    "This _does not_ meet our requirement of 99% accuracy!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modify dataset classification\n",
    "\n",
    "To address insufficient accuracy, lets modify the dataset to classify an image as \"1\" or \"Not a 1\".\n",
    "By combining classes, we can hopefully achieve the desired level of attainable accuracy.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creates a binary mask where current label == 1 that can be used as the new labels\n",
    "labels_merged = labels_subset == 1\n",
    "print(\"New label counts:\", np.unique(labels_merged, return_counts=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Evaluate the BER metric for the MNIST data with updated labels\n",
    "new_result = ber(images_subset, labels_merged, method=\"MST\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"The bayes error rate estimation:\", new_result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "remove_cell"
    ]
   },
   "outputs": [],
   "source": [
    "### TEST ASSERTION CELL ###\n",
    "assert 0.994 < 1 - new_result.ber < 0.996"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The estimate of the maximum achievable accuracy is one minus the BER estimate.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"The maximum achievable accuracy:\", 1 - new_result.ber)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results\n",
    "\n",
    "The maximum achievable accuracy on a dataset of 1 and not 1 (4, 9) is about 99.5%.\n",
    "This _does_ meet our accuracy requirement.\n",
    "\n",
    "By using BER to check for feasibility early on, we were able to reformulate the problem such that it is feasible under our specifications\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv-3.12",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.7"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}