Data integrity

Training data quality is a major determinant of machine learning (ML) model performance and generalization. Redundant samples, corrupted samples, statistical and semantic anomalies, and label noise can reduce training efficiency, introduce unintended statistical associations, and compromise the validity of evaluation results. Data integrity is the degree to which a dataset is free from these four kinds of noise.

In test and evaluation (T&E) contexts, data integrity failures can have operational consequences beyond a poor benchmark score. A model trained on sensor-corrupted imagery may learn to associate artifacts with specific targets. A dataset inflated with near-duplicates from a single collection event may pass validation and still perform poorly when deployed on a different sensor platform or in a different environment. Problems found before training are typically less costly to correct than problems found after deployment (Polyzotis et al., 2017).

What data integrity is

Data integrity is a property of a dataset assessed along several dimensions, not a single metric. A dataset has high integrity when its samples are loadable, non-redundant, within the expected statistical range, and relevant to the problem domain. A dataset has low integrity when a significant proportion of its samples are duplicated, corrupted, or anomalous, even if those samples are technically valid files.

DataEval approaches data integrity through four complementary tools: Duplicates for redundancy detection, Outliers for anomaly identification, label_errors() for detecting mislabeled samples, and label_stats() for auditing label distribution structure. Both Duplicates and Outliers can operate on raw image statistics; Outliers can additionally operate on embeddings from a pre-trained feature extractor, and Duplicates can use embedding-based clustering as a third detection mode alongside its hash methods.

Taxonomy of data noise

Redundancy and duplication

A duplicate is more than “a sample identical or near-identical to another”. That description only covers content, and content is one of three projections a duplicate can be found in. A duplicate is a collision in some projection of the datum: reduce a sample through its content, through its annotation, or through its coded metadata factors, and two samples whose reduction agrees are duplicates in that projection, whether or not they agree in any other.

Projection

What it reduces to

Content

phash / dhash / xxhash over decoded frames

Annotation

a digest over boxes, labels, and track ids

Metadata factors

the coded Metadata factor row

Content is the common case: for images and video frames, the pixels. Near-duplicates here include images with minor lighting shifts, JPEG re-compression artifacts, slight crops of the same underlying scene, or rotated and flipped versions of the same image.

Annotation duplicates share a box/label/track-id digest even when their content differs or was never compared. Factor duplicates share a named Metadata factor row (a capture timestamp, GPS fix, or source filename) even before either image is decoded.

The pattern of agreement across projections indicates the kind of redundancy. Agreement on content with disagreement on annotation is consistent with one collection event carrying two conflicting label passes. Agreement on annotation with disagreement on content is consistent with a synthetically augmented copy. Agreement on both is consistent with a re-ingest under a new name. Duplicates reports which projections it checked and which of those agreed on every group. See Duplicates and near-duplicates for how to read that against a specific result.

Redundancy affects training and evaluation in two ways. When a sample appears multiple times, the model receives repeated gradient updates from information it has already processed. First, redundant samples inflate the effective size of the training set without adding new signal, which makes training less compute-efficient. Second, and more important for T&E, duplicate samples shared between training and validation sets make the validation metrics non-representative. If 30% of the validation images are near-duplicates of training images, the reported accuracy does not estimate out-of-distribution generalization.

Birodkar et al. (2019) demonstrated that large benchmark datasets contain substantial rates of semantic redundancy, and that removing it does not degrade, and in some cases improves, model performance.

Deduplication does not replace data augmentation. The two address different problems at different points in the pipeline. Deduplication is a pre-training step applied to the raw dataset: removing near-duplicates before splitting reduces the risk of accidental train/validation leakage, so held-out metrics are less likely to overstate generalization. Runtime augmentation, applied to the clean, split dataset during training, introduces controlled variation that the model has not already memorized, which can improve generalization. The two practices are complementary.

Statistical outliers and semantic anomalies

Anomalous samples differ in kind, and the kind determines how a flagged sample is handled.

Statistical outliers are samples that fall at the extreme edges of a measurable distribution, such as an extremely overexposed image, a frame with near-zero entropy, or a dimension ratio far outside the norm for the dataset. These samples may or may not be semantically valid. A statistically extreme image might be a rare but operationally important edge case, or it might be a broken sensor frame. The statistical test does not distinguish the two cases; it flags the sample for inspection.

Semantic anomalies are samples that are statistically unremarkable but belong to the wrong problem domain entirely, such as an image of a domestic animal in an industrial inspection dataset, or a clear daytime frame in a dataset intended for low-light detection. These pass pixel-level quality checks but can reduce model performance by introducing irrelevant class structure. Embedding-based outlier detection can find semantic anomalies that statistical linting does not, because such samples can occupy different regions of the embedding space even when their pixel statistics are normal.

An outlier, like a duplicate, can be found in any projection of the datum: its content, its annotation (box geometry, labels, tracks), or its metadata factors. The reference population a value is judged against determines the kind of finding:

Kind

Judged against

Example

Marginal

every row at its level

a track with 11 gaps where tracks have 0–1

Stratified

rows sharing a categorical context (class, sequence, or categorical factor)

a class that is always a small box near the corner, drawn once large and centered

Sequential

the row’s neighbors within its sequence

a timestamp that jumps three hours for one frame; a GPS fix that moves 40 km and back

Joint

a multivariate or continuous-on-continuous relationship

in one class, distant boxes are small and near ones large; a large box at the horizon

A marginal check does not find the other three kinds. A timestamp is roughly uniform over a dataset, so a wrong one is unremarkable until it is compared with its neighbors. A large, centered box is unremarkable if other classes are also large and centered. Many annotation and metadata defects are conditional: a value that is improbable only given the rest of the datum.

Corruption and sensor artifacts

Physical sensors can introduce systematic errors that statistical linting can detect. Common examples in computer vision include motion blur from high-speed collection platforms, lens flare from direct sun exposure, dead pixel rows from sensor damage, and blocking artifacts from aggressive JPEG compression in transmission pipelines.

The integrity concern extends beyond individual corrupted samples. A model trained on a dataset with a consistent rate of blur artifacts may train successfully but on a distribution that differs from deployment. If blur is consistently associated with a particular collection scenario, the model can learn blur as a feature, which is a form of sensor-conditioned bias. The bias may appear only when the deployment sensor differs from the collection sensor. Where collection hardware is upgraded or substituted over multiple phases, this risk is more likely to arise.

Label noise and mislabeling

Image-level and pixel-level integrity checks catch problems with the data itself. A fourth category of integrity failure is harder to detect: label noise, meaning samples that are correctly captured but incorrectly annotated.

Label noise is common in large-scale annotation pipelines. Annotators make mistakes, instructions are interpreted inconsistently, class boundaries are ambiguous at the margins, and adversarial or rushed labeling produces systematic errors in particular categories. In T&E datasets where labels may be generated from automated pipelines or inherited from third parties, label quality is often assumed rather than verified (Sculley et al., 2015).

Mislabeled samples degrade both training and evaluation. During training, a mislabeled sample contributes gradient signal toward an incorrect decision boundary, and clusters of mislabeled samples near class boundaries can shift the learned boundary. In evaluation data, mislabeled samples bias the assessment of model performance: a correct model prediction on a mislabeled sample counts as an error, and a wrong prediction counts as correct.

label_errors() detects potential mislabeling by examining embedding geometry. If the embedding separates the classes, a correctly labeled sample is closer to other samples of its own class than to samples of any other class. The metric is the intra/extra class distance ratio: the mean distance to the \(k\) nearest neighbors within the same class, divided by the mean distance to the \(k\) nearest neighbors in any other class. A ratio ≥ 1.0 means the sample is at least as close to another class as to its own. This is consistent with a labeling problem. It is also consistent with an ambiguous sample or an embedding that does not separate the classes.

In a tracking dataset, label noise can also sit in the track ids. An identity switch is an id that moves to another object partway through a track: two ids exchanged where two objects cross, or a box that drifts onto another track’s object until a corrected keyframe. track_switches() looks where two tracks of one class meet. It compares the embeddings of each track before and after a frame where their boxes meet, and asks whether the two tracks fit better crossed than straight. A flagged meeting marks a place to look, not a certain error: the ids may have swapped, the boxes may sit off their objects, or one object may pass behind an occluder as the other comes out. An empty result does not mean the ids are right. A switch that lasts a few frames, a switch with no second track nearby, and a jump to a distant object are not looked at; track_steps() shows the last as a large step.

Label statistics (label_stats()) address a different but related concern: structural problems in the label distribution that concern completeness and consistency and not correctness. The function counts label_counts_per_class, image_counts_per_class, label_counts_per_image, and, for object detection, empty_image_indices (images with no annotations). An empty image can be an annotation error: an image that was included in the dataset but never labeled, or one where annotators missed all objects. Training on such an image teaches the model to predict no objects, which is incorrect when objects are present. An empty image can also be a legitimate negative example, so whether it is an error depends on the task.

Theory

Information density and gradient efficiency

The information density of a training set is the ratio of novel signal to total sample count. Redundant samples lower it.

Consider a training set of \(N\) samples, of which a fraction \(r\) are near-duplicates of existing samples. The effective number of unique training examples is \(N(1 - r)\). This is the approximate number that remains after deduplication under the chosen similarity criterion and is not a statistical effective sample size: dependencies can remain among samples that are unique under that criterion. The duplicate samples still contribute gradients during training, but those gradients largely repeat information already contributed by the originals. At minimum this wastes compute. For near-identical samples it can also bias the gradient toward the over-represented region of the input space, which re-weights the loss without the practitioner being aware of it.

Duplicate detection: hashing and clustering

Duplicates computes the content projection through three complementary approaches, each suited to a different kind of redundancy. Two further projections, annotation and metadata factors, are covered afterward.

Exact duplicate detection uses xxHash (the 64-bit xxh3 variant), a fast non-cryptographic hash of the raw image bytes. Images with identical bytes have identical hashes, and no threshold is required. The converse holds with high probability and not with certainty. Two different byte strings can collide. If the hashes of \(n\) images behave as uniform 64-bit values, the probability of at least one collision is approximately \(n^2 / 2^{65}\), which is about \(3 \times 10^{-8}\) for one million images. The hash input is the array’s raw bytes, so arrays of different shape or dtype that hold the same bytes have the same hash. DataEval groups images by hash value and does not compare the pixels of the grouped images.

Near-duplicate detection uses perceptual hashing, where two images that are visually similar but not pixel-identical, due to compression, minor cropping, or brightness adjustment, produce similar hash values. DataEval supports two perceptual hash algorithms:

  • pHash (perceptual hash): the image is resized to a square \(N \times N\) grid, a discrete cosine transform (DCT) is applied, and the lowest-frequency components are encoded as a bit array relative to the median coefficient value. The result is a compact hex string that is robust to minor photometric perturbations. Zauner (2010) provides the foundational treatment.

  • dHash (difference hash): horizontal adjacent-pixel differences are computed on a downsampled image and encoded as a bit string. This approach is particularly robust to brightness and contrast shifts.

Both algorithms are available in D4 variants (phash_d4, dhash_d4) that apply the hash across all eight orientations of the dihedral group (four rotations × two reflections) and take the minimum, producing a hash invariant to 90°/180°/270° rotations and horizontal/vertical flips. This matters for datasets where images may be re-oriented during ingestion.

Two images are considered near-duplicates when the Hamming distance between their hash values falls below a configurable threshold. Groups detected by multiple methods carry higher confidence. When both basic and D4 hashes are computed, the orientation column in the duplicates DataFrame is automatically set to "same" (detected by basic hashes) or "rotated" (detected only by D4 hashes), letting practitioners distinguish the two cases. Video relations follow the same rule: a whole-video match, a shared segment or a shared track found only under a D4 hash is "rotated", so a video re-uploaded a quarter turn on is reported as related to the original. Redundancy within one video is read from the basic hashes only, because consecutive frames of a video are not rotations of each other.

Cluster-based detection is an optional third mode that operates in embedding space rather than pixel space. When a feature extractor and a cluster_sensitivity are provided, images are projected into embedding space, clustered, and pairs whose embeddings fall within the threshold distance are treated as near-duplicates. Because embeddings are approximate representations, cluster-based matches are always reported as near rather than exact duplicates, even when their embedding distance is zero. This mode catches semantic duplicates, which are distinct photographs of the same object or scene that are not similar at the pixel level but occupy the same region of embedding space.

Annotation detection is on by default whenever a dataset carries targets (object detection boxes and labels). Each item’s annotation (its boxes, labels, and, for tracking data, track ids) is reduced to a digest; items whose digest agrees are grouped as an annotation duplicate, whether or not their content does. Because it runs alongside content detection rather than instead of it, the same pair of items can be checked on both projections at once: a pair agreeing on content but not on annotation surfaces as an exact content match whose annotation disagreed (consistent with one collection event carrying two conflicting label passes); a pair agreeing on annotation but not on content surfaces as an annotation duplicate whose content disagreed (consistent with a synthetically augmented copy). No feature extractor or threshold is needed: it is an exact, transitive comparison, like xxHash for pixels.

Factor detection is the only projection that is opt-in. Passing duplicate_factors to Duplicates.evaluate() names one or more Metadata factors whose exact agreement makes two items duplicates, such as a capture timestamp, a GPS fix, or a source filename.

The factor names form a conjunction. Two items are duplicates only when every named factor agrees. Agreeing on some factors while differing on another does not produce a duplicate. Because matching applies to the entire row, adding a factor can only split groups. Use additional factors to narrow an identifier that is not unique on its own (for example, duplicate_factors=["capture_date", "camera_id"] requires agreement on both). Items with unstated factors (null, or NaN in numeric columns) are excluded from grouping.

Factor detection is disabled unless factors are explicitly named. There is no default set or selection heuristic. The signal is valid only when the named factors identify an individual item rather than a shared condition. For example, items sharing weather=rain reflect shared conditions that belong in Balance and Coverage. Grouping on low-cardinality factors produces large groups that obscure actual duplicates.

Outlier detection: image statistics and embeddings

Outliers identifies anomalous samples through two independent paths that can be used separately or together.

Image statistics-based detection computes pixel, visual, and dimension statistics for each image using image_stats() with ImageStats flags, then applies a statistical threshold test to each metric distribution to flag samples at the extremes. Three tests are available:

The z-score method measures how many standard deviations a sample’s value \(x_i\) lies from the distribution mean \(\mu\):

\[z_i = \frac{|x_i - \mu|}{\sigma}\]

Samples exceeding the threshold (default: 3.0) are flagged. This method works well for roughly normal distributions but is sensitive to the influence of existing extreme values on the mean and standard deviation.

The modified z-score method substitutes the median \(\tilde{x}\) and median absolute deviation (MAD) for the mean and standard deviation, making it robust to that influence:

\[\tilde{z}_i = \frac{0.6745 \cdot |x_i - \tilde{x}|}{\text{MAD}}\]

The constant 0.6745 is the 75th percentile of the standard normal distribution, chosen so that the modified z-score is on the same scale as the standard z-score for normally distributed data. The default threshold is 3.5.

The interquartile range (IQR) method flags samples whose distance from the nearest quartile boundary exceeds a multiple of the IQR:

\[d_i = \max(Q_1 - x_i,\ x_i - Q_3) > \text{threshold} \times (Q_3 - Q_1)\]

The default threshold of 1.5 corresponds to the standard Tukey fence. This method is the least sensitive to extreme values of the three tests and does not assume a normal distribution.

Each statistical test operates independently on each metric. A sample is flagged if it exceeds the threshold on any single metric. The output records both which metric triggered the flag and the measured value, so practitioners can distinguish a brightness outlier from a dimension outlier and prioritize accordingly.

Cluster-based detection operates in embedding space. When a feature extractor is provided, images are projected into embedding space and clustered. For each sample, the distance to its nearest cluster center is computed. Samples exceeding the cluster_threshold (default: 2.5) are flagged with a cluster_distance metric value: the distance itself, reported beside that cluster’s own population_mean and population_std, so the number of standard deviations it sits out by is (metric_value - population_mean) / population_std. This path catches semantic anomalies: samples that look statistically normal at the pixel level but do not belong to any established class or scene type in the dataset.

Both detection paths can run simultaneously and their results are merged into a single output DataFrame. Review a sample flagged by both paths first.

Outlier detection over annotation and metadata factors

Outliers also reads a Metadata. It then judges ordered metadata factors against the rows at the level they were measured at, categorical factors by their rarity within a stratum, combinations of factors, and values that depart from the rows next to them along a video or track. The extension raises three questions the image path never faced: which columns can be thresholded, how a categorical value can be an outlier, and how small a population can support a finding. Outliers in metadata factors describes the method behind each and the evidence behind its defaults.

Image statistics in integrity checks

image_stats() computes the statistics that the integrity evaluators read. Image statistics maps the families and says what each can support. Statistics of named groups of bands are columns rather than rows, so they reach Metadata and are visible to Balance and Diversity like any other factor.

Object detection datasets can also be measured on the pixels that are not annotated. per_background=True unions every box in an image into a mask, excludes those pixels, and reduces the requested statistics over what is left, meaning the scene an image was captured in, rather than the things annotated inside it:

stats = image_stats(
    dataset,
    stats=ImageStats.PIXEL | ImageStats.VISUAL,
    per_background=True,
    normalize_pixel_values=False,
)
# One value per row, as every stats array is. With the default per_target=True the
# rows are images *and* boxes, and a box has no background, so its entry is null.
# `source_index` says which is which; `add_factors` uses it to place them.
stats["stats"]["background_brightness"]

Background values are returned under background_-prefixed names on the same rows as the whole-image ones, because they describe the same thing a whole-image statistic describes, the image, and so belong at the unit level (see Metadata levels). The background statistics have three properties:

  • background_fraction is always returned. It is the share of the image left unmasked, and a background statistic measured over a few percent of an image is noise that looks like a measurement. Read it before the other background statistics. Where an image’s boxes cover it entirely, every other background statistic for that image is NaN.

  • A masked region has no meaningful hash and no geometry of its own, so the hash and dimension statistics are computed for the image and its boxes and not for the background. background_blur_edge_width pools the edges outside the boxes and a margin around them, so it is not measured over exactly the background_fraction of the image. See Blur statistics.

  • Boxes are rounded outwards and unioned, so the background excludes slightly more than the annotations strictly cover. This is deliberate: a retained pixel is background with high confidence, at the cost of discarding a boundary ring of genuine background along with the object.

The default configuration for Outliers uses DIMENSION | PIXEL | VISUAL. It does not compute hashes, which belong to Duplicates.

Label error detection: embedding geometry

label_errors() operates on embeddings and class labels. For each sample, it computes:

\[\text{score}_i = \frac{\bar{d}_{\text{intra},i}}{\bar{d}_{\text{extra},i}}\]

where \(\bar{d}_{\text{intra},i}\) is the mean distance from sample \(i\) to its \(k\) nearest neighbors within the same class, and \(\bar{d}_{\text{extra},i}\) is the mean distance to its \(k\) nearest neighbors in any other class (\(k\) defaults to 50, capped at min_class_size - 1).

The output contains three fields:

errors: a dictionary mapping sample index to (original_label, [suggested_labels]) for all samples with score ≥ 1.0. Suggested labels are derived from rank-weighted voting over the \(k\) nearest out-of-class neighbors: closer neighbors receive higher weight. The suggestion logic applies two thresholds: if the top candidate’s vote share is below min_confidence (default 0.4), no suggestion is returned; if the margin between the top two candidates is smaller than ambiguity_threshold (default 0.2), both are returned as a tie.

error_rank: all sample indices sorted by descending score (a triage list for human review, starting with the highest scores).

scores: the raw distance ratio for every sample, useful for setting custom thresholds or examining the distribution of scores.

A higher score indicates a larger gap between the same-class and other-class neighbor distances, so error_rank orders samples for review from the largest gap down. Reviewers can stop at a score cutoff they choose. The score is a ratio of distances and is not a probability that the label is wrong.

Interpretation

Each tool reports evidence about samples. None of them reports that a sample is invalid.

  • Duplicates reports groups of samples that agree in a projection. An exact content match means identical raw bytes, apart from hash collisions. A near-duplicate match means the perceptual hashes lie within the Hamming threshold, or the embeddings within the cluster distance. It does not establish that two samples show the same scene. The projections that agree indicate the kind of redundancy, as described in Redundancy and duplication.

  • Outliers reports samples whose value lies beyond a threshold fitted to the population it was judged against. A flag establishes that a value is extreme relative to that population. It does not establish that the sample is invalid, because a rare but valid edge case is flagged in the same way. For metadata factors, a population below the floor is not judged (see Small populations), and the absence of a flag there is not evidence that its rows are normal.

  • label_errors() reports a score, the ratio of the mean same-class to the mean other-class neighbor distance. A score of at least 1.0 is consistent with a labeling problem, an ambiguous sample, or an embedding that does not separate the classes. The score is not a probability.

  • label_stats() reports counts. It describes the label distribution and does not assess whether labels are correct.

To decide what to do after running an assessment, see Data integrity findings in Acting on results.

Limitations

Statistical linting and perceptual hashing detect the integrity failures they were designed for, and both have blind spots.

Hash-based detection will not catch semantic duplicates, meaning two different photographs of the same object taken from different angles or lighting conditions that are different images at the pixel level. Cluster-based duplicate detection in embedding space addresses this, but requires a feature extractor whose embedding space is meaningful for the target domain.

Image statistics-based outlier detection is only sensitive to pixel-level and dimensional anomalies. A semantically incorrect image with normal brightness, contrast, and dimensions will not be flagged by statistical tests. Cluster-based outlier detection addresses this case, again contingent on embedding quality.

Neither approach detects label errors, meaning images that are correctly exposed, non-duplicated, and statistically normal but assigned the wrong class label. Label quality is assessed separately with label_stats() and label_errors().

The statistical outlier tests flag samples relative to the distribution of the dataset being evaluated. Results are therefore dataset-dependent: adding or removing samples changes what counts as an outlier. When comparing results across dataset versions, re-run the full analysis rather than assuming prior flags remain valid.

Stratified and categorical outlier tests assume the rows of a stratum are exchangeable, which consecutive video frames are not, and they return no verdict on strata too small to support one. Relationships between two ordered quantities (box size against distance from the camera, image brightness against capture hour) are joint and are not found by testing one column at a time, within a stratum or not. The neighbor check (neighbors=) is the test that reads consecutive frames in order: it judges a frame against the frames next to it, and reads a run of more than five departing frames as a segment. Outliers in metadata factors describes these tests.

When to use it

Assess data integrity at three points in the data lifecycle.

Before a training run: assess a new or merged dataset before committing compute to training. Problems found at this point cost a data review. Problems found after a failed training run cost a training run and a data review.

After merging datasets: near-duplicate rates can rise when datasets from different collection events or public sources are combined without deduplication, because the sources can overlap.

When evaluation metrics look suspicious: anomalously high validation accuracy, especially accuracy that degrades sharply in operational testing, can indicate validation set contamination by training-set duplicates. In that case, run deduplication across the train/validation split boundary.

See this in practice

How-to guides

Tutorials

References

  1. Birodkar, V., Mobahi, H., & Bengio, S. (2019). Semantic redundancy in image classification datasets. arXiv preprint arXiv:1901.11409. paper

  2. Polyzotis, N., Roy, S., Whang, S. E., & Zinkevich, M. (2017). Data management challenges in production machine learning. In Proceedings of the 2017 ACM SIGMOD International Conference on Management of Data (pp. 1723–1726). paper

  3. Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhuri, V., Young, M., Crespo, J.-F., & Dennison, D. (2015). Hidden technical debt in machine learning systems. In Advances in Neural Information Processing Systems (Vol. 28). paper

  4. Zauner, C. (2010). Implementation and benchmarking of perceptual image hash functions. Bachelor’s thesis, Upper Austria University of Applied Sciences. thesis