Identify duplicates¶
Duplicates add no new information and can distort model training by over-emphasizing the features that appear in them.
Duplicates finds exact and near duplicate images so you can remove them before you train. It also looks past
pixels: when a dataset carries object detection targets, it checks whether two items share one annotation (a synthetic
copy) or share pixels but disagree on annotation (one collect labeled twice).
Use it when you need to find duplicate images in a dataset, as one of the first steps of exploratory data analysis.
You need dataeval, maite-datasets and a dataset to analyze.
from dataclasses import dataclass
import numpy as np
from IPython.display import display
from maite_datasets.image_classification import MilitaryVehicles
from dataeval import Metadata
from dataeval.config import set_max_processes
from dataeval.data import Indices, View
from dataeval.protocols import DatasetMetadata, DatumMetadata
from dataeval.quality import Duplicates
set_max_processes(4)
Load the data¶
Load the MilitaryVehicles train split, 7,823 photographs of 24 types of ground vehicle, and read the class of each image.
train_ds = MilitaryVehicles("./data", image_set="train", download=True, lazy=True)
index2label = train_ds.metadata["index2label"]
# Get the labels
labels = Metadata(train_ds).class_labels
Find the duplicates¶
Ask the Duplicates class to find the duplicates. You do not need to plant any: the dataset holds its own. See
Clean a dataset for how to inspect the groups before you remove anything.
# Initialize the Duplicates class to begin to identify duplicate images. A hash radius of 0 counts two images as
# duplicates only when their hashes are identical.
identifyDuplicates = Duplicates(hash_radius=0)
# Evaluate the data
results = identifyDuplicates.evaluate(train_ds)
The results can be viewed as a DataFrame with exact and near groups enumerated.
display(results)
shape: (105, 7)
┌──────────┬───────┬──────────┬──────────────┬────────────┬────────────┬───────────────────────┐
│ group_id ┆ level ┆ dup_type ┆ item_indices ┆ methods ┆ differs_on ┆ annotation_divergence │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ str ┆ list[i64] ┆ list[str] ┆ list[str] ┆ f64 │
╞══════════╪═══════╪══════════╪══════════════╪════════════╪════════════╪═══════════════════════╡
│ 0 ┆ item ┆ exact ┆ [478, 6118] ┆ ["xxhash"] ┆ null ┆ null │
│ 1 ┆ item ┆ exact ┆ [487, 7811] ┆ ["xxhash"] ┆ null ┆ null │
│ 2 ┆ item ┆ exact ┆ [508, 3725] ┆ ["xxhash"] ┆ null ┆ null │
│ 3 ┆ item ┆ exact ┆ [520, 7613] ┆ ["xxhash"] ┆ null ┆ null │
│ 4 ┆ item ┆ exact ┆ [544, 7615] ┆ ["xxhash"] ┆ null ┆ null │
│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │
│ 100 ┆ item ┆ near ┆ [6566, 6903] ┆ ["phash"] ┆ null ┆ null │
│ 101 ┆ item ┆ near ┆ [7294, 7660] ┆ ["dhash"] ┆ null ┆ null │
│ 102 ┆ item ┆ near ┆ [7416, 7720] ┆ ["dhash", ┆ null ┆ null │
│ ┆ ┆ ┆ ┆ "phash"] ┆ ┆ │
│ 103 ┆ item ┆ near ┆ [7538, 7628] ┆ ["phash"] ┆ null ┆ null │
│ 104 ┆ item ┆ near ┆ [7574, 7757] ┆ ["dhash", ┆ null ┆ null │
│ ┆ ┆ ┆ ┆ "phash"] ┆ ┆ │
└──────────┴───────┴──────────┴──────────────┴────────────┴────────────┴───────────────────────┘
Each exact group is one photograph that appears more than once with identical pixels, and each near group holds photographs whose perceptual hashes match. Check what labels the images in each exact group carry.
different = sum(len({labels[i] for i in group}) > 1 for group in results.exact)
first_group = results.exact[0]
print(f"{len(results.exact)} exact groups and {len(results.near)} near groups in {len(train_ds):,} images")
print(f"Exact groups whose images carry different labels: {different} of {len(results.exact)}")
print(f"The first exact group, images {first_group}, is filed under {[index2label[labels[i]] for i in first_group]}")
67 exact groups and 38 near groups in 7,823 images
Exact groups whose images carry different labels: 67 of 67
The first exact group, images [478, 6118], is filed under ['BM-30', 'T-64']
Every exact group holds identical pixels filed under different vehicles, so the same photograph teaches the model two
answers. Duplicates finds the copies. It cannot tell which label is right, and that decision is yours.
Remove the duplicates¶
deduplicate() turns the result into a plan of what to remove. Its arguments state the policy: which kinds of
duplicate to collapse, and which member of each set to keep. By default it collapses exact duplicates and keeps
the first member of each set. Near duplicates are a judgment about how similar is too similar, so you add them with
dup_types=["exact", "near"] once you have looked at them.
Pass the plan to Indices with exclude=True. Build the view on train_ds, the dataset the result was
computed on, because the plan names positions in that dataset.
plan = results.deduplicate()
deduplicated_ds = View(train_ds, Indices(plan, exclude=True))
print(plan)
print(f"{len(train_ds)} items before, {len(deduplicated_ds)} after")
RemovalPlan(items=72)
7823 items before, 7751 after
The plan removes 72 images from 67 groups because five groups hold three copies. Here the choice of survivor decides
the label. Compare keep="first", the default, with keep="last" on the first group.
kept = {
keep: set(View(train_ds, Indices(results.deduplicate(keep=keep), exclude=True)).resolve_indices())
for keep in ("first", "last")
}
for keep, survivors in kept.items():
print(f'keep="{keep}" keeps', [index2label[labels[i]] for i in first_group if i in survivors])
keep="first" keeps ['BM-30']
keep="last" keeps ['T-64']
The two policies keep different labels for the same photograph, and neither knows which is correct. Pass
exclude_groups the group_id of any group you want to leave alone and settle by hand, or correct the labels first
and then apply the plan.
Find annotation duplicates¶
MilitaryVehicles has no object detection targets, so everything above compared pixels only. When a dataset does carry
targets (boxes and labels), Duplicates also digests the annotation itself, which is on by default and needs no
argument. Two accessors narrow the result to the annotation-driven relations:
divergent(): items whose annotation was checked and found to disagree, meaning one collect labeled two different ways.augmented(): items sharing one annotation over different pixels, the signature of a synthetic copy.
Real detection datasets rarely hold a clean pair of each kind, so this guide builds a tiny object detection dataset with four items to see both. Items 0 and 1 have identical pixels and two different boxes, a divergent pair. Items 2 and 3 have the same box over two independently sampled images, an augmented pair.
annotation_results = Duplicates(hash_radius=0).evaluate(annotation_ds)
divergent() finds the pair whose pixels matched but whose annotation did not. That is a label-quality issue, not a
redundancy one.
display(annotation_results.divergent())
shape: (1, 7)
┌──────────┬───────┬──────────┬──────────────┬────────────┬────────────────┬───────────────────────┐
│ group_id ┆ level ┆ dup_type ┆ item_indices ┆ methods ┆ differs_on ┆ annotation_divergence │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ str ┆ list[i64] ┆ list[str] ┆ list[str] ┆ f64 │
╞══════════╪═══════╪══════════╪══════════════╪════════════╪════════════════╪═══════════════════════╡
│ 0 ┆ item ┆ exact ┆ [0, 1] ┆ ["xxhash"] ┆ ["annotation"] ┆ 1.0 │
└──────────┴───────┴──────────┴──────────────┴────────────┴────────────────┴───────────────────────┘
augmented() finds the pair whose annotation matched but whose pixels did not. That is a synthetic copy’s signature.
display(annotation_results.augmented())
shape: (1, 7)
┌──────────┬───────┬────────────┬──────────────┬────────────────┬─────────────┬────────────────────┐
│ group_id ┆ level ┆ dup_type ┆ item_indices ┆ methods ┆ differs_on ┆ annotation_diverge │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ nce │
│ i64 ┆ str ┆ str ┆ list[i64] ┆ list[str] ┆ list[str] ┆ --- │
│ ┆ ┆ ┆ ┆ ┆ ┆ f64 │
╞══════════╪═══════╪════════════╪══════════════╪════════════════╪═════════════╪════════════════════╡
│ 1 ┆ item ┆ annotation ┆ [2, 3] ┆ ["annotation"] ┆ ["content"] ┆ null │
└──────────┴───────┴────────────┴──────────────┴────────────────┴─────────────┴────────────────────┘
Next steps¶
Data integrity — Analyze image-level and target-level statistics to identify data quality issues.
Identify duplicate video — Find duplicate footage, train/test leakage, and redundant frames in video and MOT datasets.
Detect common augmentations as duplicates — Find near-identical images created through synthetic transformations and augmentations.