dataeval.quality.Duplicates¶

class dataeval.quality.Duplicates(flags=None, cluster_threshold=None, merge_near_duplicates=None, extractor=None, cluster_algorithm=None, n_clusters=None, config=None)¶

Finds duplicate images using hashing and/or embedding-based clustering.

Supports multiple complementary detection methods:

Hash-based exact (xxhash): Detects exact duplicates (identical pixel values) using xxhash.
Hash-based near (phash): DCT-based perceptual hashing for compression/resize detection.
Hash-based near (dhash): Gradient hash for brightness-invariant detection.
Multidirectional hashing (phash_d4, dhash_d4): Rotation/flip-invariant variants that detect duplicates regardless of orientation.
Cluster-based: Uses neural network embeddings to find semantic duplicates.

The multiple perceptual hash methods (phash, dhash) are complementary and can catch different types of image modifications. Using all hashes provides more robust near-duplicate detection without requiring a trained model.

Three convenience flags are provided for common use cases:

ImageStats.HASH_DUPLICATES_BASIC: Standard duplicate detection (xxhash + phash + dhash)
ImageStats.HASH_DUPLICATES_D4: Rotation/flip-invariant detection (xxhash + phash_d4 + dhash_d4)
ImageStats.HASH: All hash statistics (enables rotation/flip awareness)

Parameters:¶

flags : ImageStats, default ImageStats.HASH_DUPLICATES_BASIC¶: Statistics to compute for hash-based duplicate detection. Set to ImageStats.NONE to disable hash-based detection.
extractor : FeatureExtractor, optional¶: Feature extractor for cluster-based duplicate detection. Must be provided together with cluster_threshold to enable clustering. When provided alone without cluster_threshold, clustering is skipped.
cluster_threshold : float, optional¶: Threshold for cluster-based near duplicate detection. Must be provided together with extractor to enable clustering. When None or when extractor is None, cluster-based detection is skipped entirely. Lower values are stricter.
cluster_algorithm : {"kmeans", "hdbscan"}, default "hdbscan"¶: Clustering algorithm for cluster-based detection.
n_clusters : int, optional¶: Expected number of clusters. For HDBSCAN, this is a hint that adjusts min_cluster_size. For KMeans, this is the exact number of clusters.
merge_near_duplicates : bool, default True¶: If True, overlapping near duplicate groups from different detection methods are merged into unified groups. Each group tracks which methods detected it, providing confidence information. If False, groups from each method are kept separate.
config : Duplicates.Config or None, default None¶: Optional configuration object with default parameters. Parameters specified directly in __init__ will override config defaults.

stats¶

Hash statistics computed during the last evaluate() call.

Type:¶: CalculationResult

flags¶

Statistics to compute for duplicate detection.

Type:¶: ImageStats

extractor¶

Feature extractor for cluster-based detection.

Type:¶: FeatureExtractor | None

cluster_threshold¶

Threshold for cluster-based near duplicate detection.

Type:¶: float | None

cluster_algorithm¶

Clustering algorithm to use.

Type:¶: Literal[“kmeans”, “hdbscan”]

n_clusters¶

Expected number of clusters.

Type:¶: int | None

merge_near_duplicates¶

Whether to merge overlapping near duplicate groups.

Type:¶: bool

Examples

Basic hash-based detection (default):

>>> detector = Duplicates()
>>> result = detector.evaluate(images)

Fast exact-only detection for large datasets:

>>> fast_detector = Duplicates(flags=ImageStats.HASH_XXHASH)
>>> result = fast_detector.evaluate(images)

Combined hash and cluster-based detection:

>>> from dataeval.extractors import FlattenExtractor

>>> detector = Duplicates(extractor=FlattenExtractor(), cluster_threshold=1.0)
>>> result = detector.evaluate(train_ds)

Using configuration:

>>> config = Duplicates.Config(
...     extractor=FlattenExtractor(),
...     cluster_algorithm="kmeans",
...     merge_near_duplicates=False,
... )
>>> detector = Duplicates(config=config)

evaluate(data, *, per_image=True, per_target=True)¶

Find duplicates by computing hashes and/or analyzing embeddings.

Performs duplicate detection using hash statistics and/or cluster-based analysis depending on configuration.

Parameters:¶

data : Dataset[ArrayLike] or Dataset[tuple[ArrayLike, Any, Any]]¶: Dataset of images in array format.
per_image : bool, default True¶: Whether to compute hashes for full items (images/videos).
per_target : bool, default True¶: Whether to compute hashes for individual targets/detections.

Returns:¶

Duplicate detection results with separate item and target groups.

items.exact: Exact duplicates (hash-based and/or cluster-based)
items.near: Near duplicate groups with detection method metadata. Each group has indices and methods (e.g., {“phash”, “rhash”}).
targets: Target-level duplicates (hash-based only)

Return type:¶

DuplicatesOutput

Raises:¶

ValueError – If flags is NONE and no extractor is provided.

Examples

Hash-based duplicates with merged near duplicates (default):

>>> detector = Duplicates()
>>> result = detector.evaluate(images)
>>> print(result.items.exact)
[[3, 20], [7, 11, 18, 25], [16, 37]]
>>> for group in result.items.near:
...     print(f"Index count: {len(group.indices)}, Methods: {sorted(group.methods)}")
Index count: 50, Methods: ['dhash', 'phash']

Fast exact-only detection:

>>> detector = Duplicates(flags=ImageStats.HASH_XXHASH)
>>> result = detector.evaluate(images)

Combined hash and cluster-based detection:

>>> from dataeval.extractors import FlattenExtractor
>>>
>>> detector = Duplicates(extractor=FlattenExtractor(), cluster_threshold=1.0)
>>> result = detector.evaluate(train_ds)

from_clusters(cluster_result)¶

Find duplicates using cluster-based detection from minimum spanning tree.

Analyzes the minimum spanning tree and cluster assignments to identify near duplicates based on distance relationships within clusters.

Parameters:¶

cluster_result : ClusterResult¶: Clustering results from the cluster() function.

Returns:¶

Duplicate detection results with item-level duplicate groups. Cluster-based detection operates on items only (no target separation).

Return type:¶

DuplicatesOutput

Notes

This method identifies duplicates in embedding space. All cluster-based duplicates are returned as near duplicates because embeddings are approximate representations - identical embeddings don’t guarantee pixel-identical images.

Classes¶

`Config`	Configuration for Duplicates detector.