Identify duplicates

Duplicates add no new information and can distort model training by over-emphasizing the features that appear in them. Duplicates finds exact and near duplicate images so you can remove them before you train. It also looks past pixels: when a dataset carries object detection targets, it checks whether two items share one annotation (a synthetic copy) or share pixels but disagree on annotation (one collect labeled twice).

Use it when you need to find duplicate images in a dataset, as one of the first steps of exploratory data analysis.

You need dataeval, maite-datasets and a dataset to analyze.

from dataclasses import dataclass

import numpy as np
from IPython.display import display
from maite_datasets.image_classification import MilitaryVehicles

from dataeval import Metadata
from dataeval.config import set_max_processes
from dataeval.data import Indices, View
from dataeval.protocols import DatasetMetadata, DatumMetadata
from dataeval.quality import Duplicates

set_max_processes(4)

Load the data

Load the MilitaryVehicles train split, 7,823 photographs of 24 types of ground vehicle, and read the class of each image.

train_ds = MilitaryVehicles("./data", image_set="train", download=True, lazy=True)
index2label = train_ds.metadata["index2label"]

# Get the labels
labels = Metadata(train_ds).class_labels

Find the duplicates

Ask the Duplicates class to find the duplicates. You do not need to plant any: the dataset holds its own. See Clean a dataset for how to inspect the groups before you remove anything.

# Initialize the Duplicates class to begin to identify duplicate images. A hash radius of 0 counts two images as
# duplicates only when their hashes are identical.
identifyDuplicates = Duplicates(hash_radius=0)

# Evaluate the data
results = identifyDuplicates.evaluate(train_ds)

The results can be viewed as a DataFrame with exact and near groups enumerated.

display(results)
shape: (105, 7)
┌──────────┬───────┬──────────┬──────────────┬────────────┬────────────┬───────────────────────┐
│ group_id ┆ level ┆ dup_type ┆ item_indices ┆ methods    ┆ differs_on ┆ annotation_divergence │
│ ---      ┆ ---   ┆ ---      ┆ ---          ┆ ---        ┆ ---        ┆ ---                   │
│ i64      ┆ str   ┆ str      ┆ list[i64]    ┆ list[str]  ┆ list[str]  ┆ f64                   │
╞══════════╪═══════╪══════════╪══════════════╪════════════╪════════════╪═══════════════════════╡
│ 0        ┆ item  ┆ exact    ┆ [478, 6118]  ┆ ["xxhash"] ┆ null       ┆ null                  │
│ 1        ┆ item  ┆ exact    ┆ [487, 7811]  ┆ ["xxhash"] ┆ null       ┆ null                  │
│ 2        ┆ item  ┆ exact    ┆ [508, 3725]  ┆ ["xxhash"] ┆ null       ┆ null                  │
│ 3        ┆ item  ┆ exact    ┆ [520, 7613]  ┆ ["xxhash"] ┆ null       ┆ null                  │
│ 4        ┆ item  ┆ exact    ┆ [544, 7615]  ┆ ["xxhash"] ┆ null       ┆ null                  │
│ …        ┆ …     ┆ …        ┆ …            ┆ …          ┆ …          ┆ …                     │
│ 100      ┆ item  ┆ near     ┆ [6566, 6903] ┆ ["phash"]  ┆ null       ┆ null                  │
│ 101      ┆ item  ┆ near     ┆ [7294, 7660] ┆ ["dhash"]  ┆ null       ┆ null                  │
│ 102      ┆ item  ┆ near     ┆ [7416, 7720] ┆ ["dhash",  ┆ null       ┆ null                  │
│          ┆       ┆          ┆              ┆ "phash"]   ┆            ┆                       │
│ 103      ┆ item  ┆ near     ┆ [7538, 7628] ┆ ["phash"]  ┆ null       ┆ null                  │
│ 104      ┆ item  ┆ near     ┆ [7574, 7757] ┆ ["dhash",  ┆ null       ┆ null                  │
│          ┆       ┆          ┆              ┆ "phash"]   ┆            ┆                       │
└──────────┴───────┴──────────┴──────────────┴────────────┴────────────┴───────────────────────┘

Each exact group is one photograph that appears more than once with identical pixels, and each near group holds photographs whose perceptual hashes match. Check what labels the images in each exact group carry.

different = sum(len({labels[i] for i in group}) > 1 for group in results.exact)
first_group = results.exact[0]
print(f"{len(results.exact)} exact groups and {len(results.near)} near groups in {len(train_ds):,} images")
print(f"Exact groups whose images carry different labels: {different} of {len(results.exact)}")
print(f"The first exact group, images {first_group}, is filed under {[index2label[labels[i]] for i in first_group]}")
67 exact groups and 38 near groups in 7,823 images
Exact groups whose images carry different labels: 67 of 67
The first exact group, images [478, 6118], is filed under ['BM-30', 'T-64']

Every exact group holds identical pixels filed under different vehicles, so the same photograph teaches the model two answers. Duplicates finds the copies. It cannot tell which label is right, and that decision is yours.

Remove the duplicates

deduplicate() turns the result into a plan of what to remove. Its arguments state the policy: which kinds of duplicate to collapse, and which member of each set to keep. By default it collapses exact duplicates and keeps the first member of each set. Near duplicates are a judgment about how similar is too similar, so you add them with dup_types=["exact", "near"] once you have looked at them.

Pass the plan to Indices with exclude=True. Build the view on train_ds, the dataset the result was computed on, because the plan names positions in that dataset.

plan = results.deduplicate()
deduplicated_ds = View(train_ds, Indices(plan, exclude=True))
print(plan)
print(f"{len(train_ds)} items before, {len(deduplicated_ds)} after")
RemovalPlan(items=72)
7823 items before, 7751 after

The plan removes 72 images from 67 groups because five groups hold three copies. Here the choice of survivor decides the label. Compare keep="first", the default, with keep="last" on the first group.

kept = {
    keep: set(View(train_ds, Indices(results.deduplicate(keep=keep), exclude=True)).resolve_indices())
    for keep in ("first", "last")
}
for keep, survivors in kept.items():
    print(f'keep="{keep}" keeps', [index2label[labels[i]] for i in first_group if i in survivors])
keep="first" keeps ['BM-30']
keep="last" keeps ['T-64']

The two policies keep different labels for the same photograph, and neither knows which is correct. Pass exclude_groups the group_id of any group you want to leave alone and settle by hand, or correct the labels first and then apply the plan.

Find annotation duplicates

MilitaryVehicles has no object detection targets, so everything above compared pixels only. When a dataset does carry targets (boxes and labels), Duplicates also digests the annotation itself, which is on by default and needs no argument. Two accessors narrow the result to the annotation-driven relations:

  • divergent(): items whose annotation was checked and found to disagree, meaning one collect labeled two different ways.

  • augmented(): items sharing one annotation over different pixels, the signature of a synthetic copy.

Real detection datasets rarely hold a clean pair of each kind, so this guide builds a tiny object detection dataset with four items to see both. Items 0 and 1 have identical pixels and two different boxes, a divergent pair. Items 2 and 3 have the same box over two independently sampled images, an augmented pair.

Hide code cell source

@dataclass
class BoxTarget:
    """A minimal object detection target: boxes, labels, and scores."""

    boxes: np.ndarray
    labels: np.ndarray
    scores: np.ndarray


class TinyObjectDetectionDataset:
    """Four items as two duplicate pairs: one relabeled, one augmented."""

    def __init__(self, images: list[np.ndarray], boxes: list[list[list[float]]], labels: list[list[int]]) -> None:
        self._images = images
        self._boxes = boxes
        self._labels = labels
        self.metadata: DatasetMetadata = DatasetMetadata(id="tiny-od", index2label={0: "object"})

    def __len__(self) -> int:
        return len(self._images)

    def __getitem__(self, index: int) -> tuple[np.ndarray, BoxTarget, DatumMetadata]:
        item_labels = np.asarray(self._labels[index], dtype=np.intp)
        target = BoxTarget(
            boxes=np.asarray(self._boxes[index], dtype=np.float32),
            labels=item_labels,
            scores=np.ones(len(item_labels), dtype=np.float32),
        )
        return self._images[index], target, DatumMetadata(id=index)


rng = np.random.default_rng(0)
same_image = rng.random((3, 16, 16))

annotation_images = [same_image, same_image.copy(), rng.random((3, 16, 16)), rng.random((3, 16, 16))]
annotation_boxes = [[[1.0, 1.0, 6.0, 6.0]], [[9.0, 9.0, 14.0, 14.0]], [[0.0, 0.0, 5.0, 5.0]], [[0.0, 0.0, 5.0, 5.0]]]
annotation_labels = [[0], [0], [0], [0]]

annotation_ds = TinyObjectDetectionDataset(annotation_images, annotation_boxes, annotation_labels)
annotation_results = Duplicates(hash_radius=0).evaluate(annotation_ds)

divergent() finds the pair whose pixels matched but whose annotation did not. That is a label-quality issue, not a redundancy one.

display(annotation_results.divergent())
shape: (1, 7)
┌──────────┬───────┬──────────┬──────────────┬────────────┬────────────────┬───────────────────────┐
│ group_id ┆ level ┆ dup_type ┆ item_indices ┆ methods    ┆ differs_on     ┆ annotation_divergence │
│ ---      ┆ ---   ┆ ---      ┆ ---          ┆ ---        ┆ ---            ┆ ---                   │
│ i64      ┆ str   ┆ str      ┆ list[i64]    ┆ list[str]  ┆ list[str]      ┆ f64                   │
╞══════════╪═══════╪══════════╪══════════════╪════════════╪════════════════╪═══════════════════════╡
│ 0        ┆ item  ┆ exact    ┆ [0, 1]       ┆ ["xxhash"] ┆ ["annotation"] ┆ 1.0                   │
└──────────┴───────┴──────────┴──────────────┴────────────┴────────────────┴───────────────────────┘

augmented() finds the pair whose annotation matched but whose pixels did not. That is a synthetic copy’s signature.

display(annotation_results.augmented())
shape: (1, 7)
┌──────────┬───────┬────────────┬──────────────┬────────────────┬─────────────┬────────────────────┐
│ group_id ┆ level ┆ dup_type   ┆ item_indices ┆ methods        ┆ differs_on  ┆ annotation_diverge │
│ ---      ┆ ---   ┆ ---        ┆ ---          ┆ ---            ┆ ---         ┆ nce                │
│ i64      ┆ str   ┆ str        ┆ list[i64]    ┆ list[str]      ┆ list[str]   ┆ ---                │
│          ┆       ┆            ┆              ┆                ┆             ┆ f64                │
╞══════════╪═══════╪════════════╪══════════════╪════════════════╪═════════════╪════════════════════╡
│ 1        ┆ item  ┆ annotation ┆ [2, 3]       ┆ ["annotation"] ┆ ["content"] ┆ null               │
└──────────┴───────┴────────────┴──────────────┴────────────────┴─────────────┴────────────────────┘

Next steps