{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dataset Deduplication Tutorial\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problem Statement\n",
    "\n",
    "Exploratory data analysis (EDA) can be overwhelming. There are so many things to check.\n",
    "Duplicates in your dataset, bad/corrupted images in the set, blurred or bright/dark images, the list goes on.\n",
    "\n",
    "DataEval created a Duplicates class to assist you with your EDA so you can start training your models on high quality data.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### _When to use_\n",
    "\n",
    "The Duplicates class should be used if you need to check for duplicates in your dataset.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### _What you will need_\n",
    "\n",
    "1. A dataset to analyze\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## _Getting Started_\n",
    "\n",
    "Let's import the required libraries needed to set up a minimal working example\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "remove_cell"
    ]
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    import google.colab  # noqa: F401\n",
    "\n",
    "    # specify the version of DataEval (==X.XX.X) for versions other than the latest\n",
    "    %pip install -q dataeval\n",
    "except Exception:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "from dataeval.detectors.linters import Duplicates\n",
    "from dataeval.utils.torch.datasets import MNIST"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading in the data\n",
    "\n",
    "Let's start by loading in torchvision's MNIST dataset, then we will examine it\n",
    "\n",
    "The MNIST dataset contains 70,000 images - 60,000 in the train set and 10,000 in the test set.\n",
    "For the purposes of this demonstration, we are just going to use the test set.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load in the mnist dataset\n",
    "testing_dataset = MNIST(root=\"./data/\", train=False, download=True, unit_interval=True)\n",
    "test_data = testing_dataset.data\n",
    "labels = testing_dataset.targets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because the MNIST dataset does not contain any exact duplicates we are going to adjust the dataset to include some.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating some duplicates\n",
    "print(\"Exact duplicates\")\n",
    "duplicates = {}\n",
    "for i in [1, 2, 5, 9]:\n",
    "    matching_indices = np.where(labels == i)[0]\n",
    "    test_data[matching_indices[78]] = test_data[matching_indices[23]]\n",
    "    print(f\"\\t{i} - ({matching_indices[23]}, {matching_indices[78]})\")\n",
    "    duplicates[i] = (matching_indices[23], matching_indices[78], matching_indices[2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Number of samples: \", len(test_data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finding the Duplicates\n",
    "\n",
    "Now we are asking our Duplicates class to find the needle in the haystack.\n",
    "There are only 4 exact duplicates and then there are 3 shifted exact duplicates.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the Duplicates class\n",
    "duplicator = Duplicates()\n",
    "\n",
    "# Evaluate the data\n",
    "results = duplicator.evaluate(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results can be returned as a dictionary with exact and near as the keys. So we will extract those to view the results.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for category, images in results.dict().items():\n",
    "    print(f\"{category} - {len(images)}\")\n",
    "    print(f\"\\t{images}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we recall from above, our exact duplicates were:\n",
    "\n",
    "(231, 781), (232, 782), (235, 785), (239, 789)\n",
    "\n",
    "Which exactly matches what the Duplicates class was able to find.\n",
    "\n",
    "It also found several sets of images that are very closely related to each other, and since we are using hand written digits we would expect it to find some images that were nearly identical.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "remove_cell"
    ]
   },
   "outputs": [],
   "source": [
    "### TEST ASSERTION CELL ###\n",
    "assert len(results.exact) == 4\n",
    "assert [231, 781] in results.exact\n",
    "assert [232, 782] in results.exact\n",
    "assert [235, 785] in results.exact\n",
    "assert [239, 789] in results.exact"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv-3.11",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}