Migrating from v1.1 to v1.2

v1.2 removes features deprecated in v1.1. It also updates SourceIndex into an address that names one row at one level. For the capabilities v1.2 adds, see New in v1.2.

v1.1 reorganized metadata addressing, dataset transformations, and image statistics scaling. Each change retained a deprecated interface that emitted a warning pointing to its replacement. v1.2 removes those deprecated interfaces.

To identify code that requires updates, run your v1.1 pipeline with warnings promoted to errors:

python -W error::DeprecationWarning your_script.py

This flags SourceIndex.target, which is newly deprecated in v1.2 and emits a warning. It does not flag the removals listed below, because those raise exceptions directly. Check for those removals by name. ClassFilter, compute_ratios, compute_stats, and two uses of label_parity (passing num_classes, or giving neither groups nor assume_independent=True) are also newly deprecated, but they warn with a FutureWarning, so add -W error::FutureWarning to find them.

At a glance

Area

You are affected if you

Status

Python 3.10

run DataEval on Python 3.10

removed

Metadata levels

read Metadata.image_data/target_data/raw_data, get_image_factors/get_target_factors/has_targets/filter_by_factor_type, set target_factors_only, or pass level="combined"

removed

View and operations

import from dataeval.selection, import a dataset operation from dataeval.utils.data, or reference Select

removed

Filter replaces ClassFilter

use ClassFilter, or want to filter a dataset by metadata factors

deprecated / additive

Per-channel rows

pass per_channel= (with any value) or channels=True to compute_stats, or read SourceIndex.channel

removed

Preprocessing helpers

use BitDepth/get_bitdepth/clip_and_pad

removed

Full-image box for a missing box

call to_bounding_box(None, shape), or pass None box entries to image_stats or boxes_to_mask, on non-square or 2-D images

numbers moved

SourceIndex is an address

read SourceIndex.target, or build or unpack a SourceIndex positionally

renamed

Evaluators report at a level

read Outliers/Duplicates results from statistics that address a level

additive

MetadataLike

implement the protocol with is_discrete

removed

discrete_features

call mutual_info without discrete_features=, or pass it to mutual_info_classwise

required / removed

Extractors

use ClassifierUncertaintyExtractor

removed

Embeddings come from images

wrap an extractor that returns other than one row per image in Embeddings, build Embeddings from a Metadata, or call Coverage.evaluate with a Metadata and no embeddings=

now raises / numbers moved

A class axis of your own

run bias, coverage, or splitting against a custom grouping rather than dataset labels

additive

The score column

read score from the metadata dataframe, or relabel a dataset whose targets score every class

changed

Duplicates checks annotation and metadata factors

run Duplicates on a dataset with targets, or group duplicates by a metadata factor

additive

Removing what an evaluator flagged

remove duplicates or outliers from a dataset

additive

Evaluators take precomputed inputs

hold metadata, labels, or embeddings already, or pass Prioritize class labels that do not match its embeddings

additive / now raises

Constructor arguments are validated

pass a wrong type, an unknown choice, or an out-of-range value to an evaluator’s constructor

now raises

Drift and OOD configs are validated

build a drift or OOD Config positionally, or call dataclasses.replace/asdict on one

changed

ChunkedDrift needs three reference chunks

fit a chunked() detector whose chunking splits the reference into fewer than 3 chunks, with a threshold other than ConstantThreshold

now raises

DriftWasserstein decides on the mean ratio

run DriftWasserstein, or read its drifted, distance, feature_ratios or score() ratios

numbers moved

Outliers reads pixel statistics in stored units

call Outliers.evaluate with a ConstantThreshold on a PIXEL statistic, or on images stored at different bit depths

numbers moved

Outliers reports the raw cluster distance

read the metric_value of cluster_distance rows from Outliers run with an extractor

numbers moved

Relative box geometry in the dimension statistics

compute ImageStats.DIMENSION, or run Outliers with its default flags and per_target=True

additive

compute_ratios is deprecated

call compute_ratios()

deprecated

compute_stats is now image_stats

call compute_stats, or leave normalize_pixel_values unset

renamed

Track statistics declare their labels and add a gap fraction

attach track_stats() to a Metadata

additive

video_stats describes each video

want one row of statistics per video, such as frame rate, redundancy or camera motion

additive (experimental)

image_stats refuses images and boxes of different lengths

pass boxes= to image_stats or compute_stats with a different number of entries than there are images

now raises

AddressedResult

annotate or subclass the result of image_stats, or write code that combines addressed results

additive

Duplicates hashes the native pixel values

run Duplicates, or read phash or dhash from image_stats, on integer images whose maximum is at most 1

numbers moved

Grayscale scales integer images deeper than 8 bits

hash, or measure ego motion on, integer images with values above 255

numbers moved

Re-detecting honors the flags

call with_threshold, classwise or itemwise on the result of the default Outliers.from_stats(stats) call, or of from_stats with per_image=False

numbers moved

Outliers needs 20 rows per population

run Outliers with a statistical threshold on a level, or a classwise() class, of fewer than 20 rows

numbers moved

aggregate_by_class counts a box under its own class

call aggregate_by_class on Outliers results computed with per_target=True, or on images that hold more than one box of a class

numbers moved

Outliers thresholds metadata factors

pass a Metadata to Outliers.evaluate, pass levels=, or use .items/.targets and the other level views on any Outliers result

additive

Outliers names metrics by level

read metric_name, the aggregate_by_metric rows, or the aggregate_by_item or aggregate_by_class columns of Outliers.evaluate on a detection dataset with per_target=True

renamed

Balance scores factor pairs over images

read the factors table of Balance on a detection dataset

numbers moved

Parity reports cells and statuses

read ParityOutput.insufficient_data or is_significant, read the insufficient_data key of core parity, or compare Parity p-values or flags across versions

removed / numbers moved (experimental)

Parity needs an independent sampling unit

run Parity on a custom metadata container without item_indices, or call core parity without groups; run Parity or Balance over a DetectionCrops view or a merge that includes one dataset twice; or load metadata saved from a crop view

now raises (experimental) / numbers moved

label_parity tests homogeneity

call label_parity, or pass it num_classes

numbers moved / deprecated

Overwriting a factor keeps the old column

call Metadata.add_factors with overwrite=True, or read FactorInfo

changed / additive

Partial keyed source_index

attach values to some of a level’s frames, tracks or detections with Metadata.add_factors(source_index=...)

additive

The datum id is item_id

read an id factor from a Metadata, or an id row from Balance or Diversity

changed

Factors Metadata cannot read

hold datum metadata whose key mixes numbers, text or booleans, holds None, or is named timestamp, frame_rate, video_duration, latitude or longitude; or pass one of those names to add_factors

changed / numbers moved / now raises

ber_knn lower bounds move

read ber_knn’s lower_bound with an even k, with more than two classes, or where it exceeded upper_bound

numbers moved

Feasibility judges targets with evidence

want to know whether a classification accuracy, AP or per-class recall target is reachable on your data, with the evidence behind its determination

additive (experimental)

Relabel.fold_scores

relabel a classification dataset and need a model’s class scores folded through the same map, such as for Feasibility’s predictions=

additive (experimental)

split_dataset(split_on=) on detection and frame datasets

pass split_on to split_dataset for a detection dataset, including a SequenceFrames view, or name an integer factor with many distinct values

numbers moved

split_dataset refuses a split it cannot fill

call split_dataset with stratify=True

now raises / additive

Outliers judges a tracking dataset in video terms

call Outliers.evaluate on a tracking dataset, or on a SequenceFrames over one

additive

DriftKnownDomains

need to tell which of several known reference domains (a sensor, a site, a season) new data belongs to, or whether it belongs to none

additive (experimental)

Videos without tracking targets or metadata

run Duplicates, Outliers, SequenceFrames or SourceLocator on videos with no annotations, or on a dataset with no metadata attribute

additive

If none of these items apply, upgrading requires no code changes.


Python 3.10 is no longer supported

DataEval v1.2 requires Python 3.11 or later, and supports 3.11 through 3.14. v1.1 also ran on 3.10. On Python 3.10, pip resolves dataeval to the newest v1.1 release, so upgrade the interpreter before upgrading DataEval.


Metadata levels

v1.1 deprecated legacy image and target attributes with warnings. Those attributes are now removed. Use the following replacements:

Removed in v1.2

Use

md.image_data

md.rows_at(md.item_level)

md.target_data

md.rows_at(md.label_level)

md.raw_data

md.rows_at(md.view).select(md.factor_names).to_numpy()

md.target_factors_only

md.at(level) to choose rows, md.inherited to choose whether ancestor factors count

md.get_image_factors(i)

md.rows_at("unit").filter(pl.col("item_index") == i)

md.get_target_factors(i, t)

md.rows_at(md.label_level).filter((pl.col("item_index") == i) & (pl.col("target_index") == t))

md.has_targets()

md.multi_target

md.filter_by_factor_type(t)

md.filter_by_factor(lambda _, fi: fi.factor_type == t)

Two factor behaviors are also removed. The argument level="combined" was the v1.1 name for an array ordered by (item, target). It is no longer accepted. In addition, Metadata.add_factors() no longer infers a level from factor length. Every add_factors call must pass level= or source_index=. Omitting both raises ValueError.


View and operations

The dataeval.selection module is removed. The dataeval.utils.data module no longer forwards dataset operations. Use View and its operations in dataeval.data:

Removed in v1.2

Use

dataeval.selection.Select

View

Select(ds, selections=...)

View(ds, operations=...)

.selection_groups

.operation_groups

Selection / Selector subclass

subclass Operation, implement apply(view)

dataeval.selection.*

dataeval.data.*

dataeval.utils.data.split_dataset / unzip_dataset / TrainValSplit / DatasetSplits

dataeval.data.*

The dataeval.utils.data module remains available for utility functions. validate_dataset(), DatasetKind, and requires_maite_dataset were never deprecated and are still imported from that module.


Filter replaces ClassFilter

Filter keeps a dataset’s items, and the detections and tracks inside them, by class and by metadata factor. ClassFilter is deprecated and will be removed in v2.0. It warns with DeprecatedWarning, a FutureWarning, so -W error::DeprecationWarning does not flag it; run with -W error::FutureWarning to find it. A filter that keeps nothing now warns with a UserWarning, which ClassFilter never did.

v1.1

v1.2

ClassFilter([0, 2])

Filter(classes=[0, 2])

ClassFilter([0, 2], filter_detections=False)

Filter(classes=[0, 2], drop="items")

ClassFilter([])

remove the operation; Filter refuses an empty classes

Class labels must be integers, numpy integers included. ClassFilter used to accept integral floats such as 0.0, and now raises TypeError for them, as Filter does.

New with Filter:

  • Factor predicates, as SQL or a polars expression: Filter("time_of_day = 'night' AND altitude > 100"). A string predicate is what a configuration file can hold.

  • Cleaning without dropping images: Filter("occlusion < 0.5", drop="detections") keeps every image and removes the detections that do not match.

  • Multi-object tracking: class and factor filtering on tracking datasets, removing detections or whole tracks; ClassFilter raised TypeError there.


The per-channel row path is removed

In v1.1, compute_stats(per_channel=True) returned one row per channel, addressed by a third field on SourceIndex. That field is removed. The per_channel argument, the channels=True alias, and the channel_index column in Outliers are also removed.

To extract channel statistics, specify band names to produce columns:

compute_stats(images, per_channel=True)  # v1.1, deprecated
image_stats(images, channels={"r": 0, "g": 1, "b": 2})  # v1.2

Band groups produce factor columns rather than separate rows. Channels do not correspond to a metadata level, so per-channel rows could not be stored in Metadata. Consequently, evaluators such as balance, diversity, and parity could not process them. Factor columns such as nir_mean are stored in metadata and evaluated directly.

per_channel=False is also removed. Because the parameter itself is removed, passing any value raises TypeError. v1.1 warned only when per_channel was truthy. Remove per_channel wherever it appears, including calls that passed the default False:

compute_stats(images, stats=ImageStats.PIXEL, per_channel=False)  # v1.1
image_stats(images, stats=ImageStats.PIXEL)  # v1.2

Outliers no longer emits channel_index. That column was populated only by per-channel rows. Band groups are thresholded independently per column. Previously, per-channel rows used a single pooled threshold across every band of every image.

compute_ratios() matches on item index alone. Previously, it paired each bounding box row with the image row sharing (item, channel). Channel indices are removed, so rows are matched by item index. The validation check requiring matching per_channel settings between inputs is also removed.


Preprocessing helpers

The deprecated names in utils.preprocessing are removed:

Removed in v1.2

Use

BitDepth

ValueRange

get_bitdepth()

get_value_range()

clip_and_pad(img, box)

crop_with_fill() with fill=np.nan, take the first tuple value

clip_and_pad was deprecated in v1.1 and is now removed.


Full-image box for a missing box

to_bounding_box() called with None and an image shape returns a box covering the whole image. In v1.1 it swapped the two sides, returning (0, 0, height, width) instead of (0, 0, width, height). On a square image the two are identical. On a non-square image the box was clipped to the wrong region, so a None box entry in image_stats() measured a window that did not match the image, and boxes_to_mask() left part of the image unmasked. On a 2-D (height, width) image it returned an empty box and masked nothing. Both now cover the full image.

Statistics stored from None box entries on non-square images, and any background mask built from them, were measured over the wrong region. Square images and images with a real box for every target are unaffected.


SourceIndex is an address

SourceIndex retains three fields. The third field is now level, which specifies the metadata level addressed by the second field (key). A single tuple can now name a row at any level in a dataset’s level graph, rather than only an item and its bounding boxes.

SourceIndex(item, target, channel)  # v1.1
SourceIndex(item, key, level)  # v1.2

target is renamed to key. Each level identifies rows by a specific column. key holds that column’s value, such as target_index for an instance, unit_index for a video frame, or track_id for a track. Accessing target or passing target= to the constructor still works but emits a DeprecationWarning. The target attribute and parameter will be removed in v1.3.0.

si.target  # v1.2: works, warns
si.key  # v1.2

SourceIndex(item=3, target=7)  # v1.2: works, warns
SourceIndex(item=3, key=7)  # v1.2

Positional construction and unpacking still take three values, but the third value is now a level name. Unpacking source_index[0] binds (item, key, level). Positional construction validates the level name. For example, SourceIndex(0, 1, 2) raises ValueError (“level=2 is not a level”) because 2 is not a valid level name.

item, target, channel = source_index[0]  # v1.1
item, key, level = source_index[0]  # v1.2

SourceIndex(0, 1, 2)  # v1.2: ValueError, "level=2 is not a level"

Available levels depend on the dataset. An address naming a level absent from your data is caught when the address is used, rather than when it is built.

The string representation retains up to three parts. str(si) produces "3", "3/7", or "3/12/unit". SourceIndex.from_string accepts those formats, as well as "3/-/sequence" for a level stated without a key. The third part must be one of sequence, unit, track, or instance. Any other value is rejected. In v1.1, that segment was read as a channel number.

Leave the level unstated unless addressing unit or track. An unstated level resolves to the item level when key is omitted, or to the label level when key is provided. SourceIndex(3) addresses an item. SourceIndex(3, 7) addresses a label. Specify a level explicitly only when addressing intermediate levels such as unit or track.

Address equality and hashing depend on exact representation. SourceIndex(3, 7) and SourceIndex(3, 7, "instance") are not equal and have different hash values. They act as distinct keys in dictionary results returned by Outliers and Duplicates. image_stats() emits the minimal form without an explicit level.

Addressing arbitrary metadata levels

Metadata.add_factors() accepts addresses at any metadata level. Per-frame or per-track measurements can be attached as individual values instead of entire keyed columns:

md.add_factors({"speed": [...]}, source_index=[SourceIndex(0, 5, "track"), ...])

This matches the column-oriented contract of add_factors(level=..., key=...). The address consists of (item_index, key) and the level name. See Naming a row: keys or addresses.

An address identifies a row, but does not specify hierarchical containment. For example, a source index does not indicate which frame contains a detection. Consequently, Metadata.from_factors() given addresses builds only the two-level structure. To preserve full hierarchy, construct Metadata from the dataset first, then add factors.

Unchanged functionality

The channels= mapping, value_range, normalize_pixel_values, per_image, per_target, and per_background parameters remain unchanged. Computed statistic values are also unchanged. Calls that omitted per_channel in v1.1 return the same results in v1.2.


Evaluators report at the level they detected at

Outliers and Duplicates read the level directly from each address rather than inferring it from missing fields. Both evaluators return the input addresses in their results.

Addressing changes nothing for image datasets. For image datasets, image_stats() emits addresses without an explicit level, so no level column is added and the same rows are reported. The changes below apply only when a statistic source index specifies an explicit level. Other v1.2 changes still apply to image datasets, among them the context columns that every Outliers result now adds, direction, bound, percentile, population_mean and population_std, and the Duplicates columns described in Duplicates checks annotation and metadata factors.

Both evaluators add a level column. Outliers results add a level column, and Duplicates results add an address_levels column. These columns distinguish rows that share key indices, such as video frames and tracks. When all findings occur at either the item level or the label level, as in image datasets, these columns are omitted. In Duplicates, the existing level column is retained with values "item" and "target". That column indicates the detection role of a group, while address_levels indicates the metadata level of each member.

Findings normalize addresses to canonical form. Evaluators return canonical addresses rather than echoing input formats verbatim. For example, SourceIndex(3, 7, "instance") normalizes to SourceIndex(3, 7). SourceIndex(3, None, "sequence") normalizes to item index 3. An item has one root row, and a key indicates a label row. Normalization ensures that runs over equivalent addresses produce identical results.

Evaluators partition statistics by metadata level. Outliers fits a separate threshold distribution for each metadata level. Per-frame statistics are evaluated only against other per-frame statistics, rather than across per-sequence statistics. Duplicates partitions candidates by level before hashing. This prevents comparisons across different metadata levels.

per_image and per_target filter item and label levels regardless of address format. Equivalent addresses are filtered identically. SourceIndex(3, 7) and SourceIndex(3, 7, "instance") are filtered identically, as are SourceIndex(3) and SourceIndex(3, None, "sequence"). Intermediate levels, such as video frames or tracks, are unaffected by these flags and are always included. Result accessors for intermediate levels use SourceIndex keys, because integer item indices cannot identify those rows.

Evaluator output addresses can be passed directly to Metadata.add_factors():

outliers = Outliers().from_stats(frame_stats)
md.add_factors(
    {"is_outlier": [address in outliers.outliers for address in addresses]},
    source_index=addresses,
)

MetadataLike

The fallback for is_discrete in MetadataLike is removed. In v1.1, defining is_discrete emitted a warning. In v1.2, the fourth protocol member is is_binned. Containers that define only is_discrete no longer satisfy the protocol.

MetadataLike remains available as an alias for CodedMetadataLike. Metadata is unchanged. This change affects only custom classes that implement the protocol directly.


discrete_features

The discrete_features parameter was updated across mutual information functions.

  • In mutual_info(), discrete_features is now required. Omitting it raises TypeError. In v1.1, omitting it triggered auto-detection and emitted a warning. Pass discrete_features=[not b for b in md.is_binned].

  • In mutual_info_classwise(), the discrete_features parameter is removed. Passing it raises TypeError. The parameter previously had no effect and was discarded downstream.

Balance is unaffected. It always passed the value explicitly.


Extractors

ClassifierUncertaintyExtractor is removed. Wrap a TorchExtractor or any FeatureExtractor in UncertaintyExtractor for per-instance uncertainty. For per-class uncertainty, wrap the extractor in ClasswiseUncertaintyExtractor.


Embeddings come from images, one row per item

Embeddings stores one row for each dataset item, and now checks that it gets one.

An extractor must return one row per image. In v1.1, when an extractor returned more rows than its batch held images, as a detection model that returns a row per detection does, Embeddings dropped the extra rows without a warning, so the rows it kept could belong to other images. It now raises ShapeMismatchError, a ValueError, when a batch returns a different number of rows or a single value, and TypeError when the extractor returns a mapping such as a dict. Evaluators that build an Embeddings from your extractor inherit the check, though Prioritize.evaluate re-raises it as a TypeError, with the ShapeMismatchError as its cause. Call a per-detection extractor directly, and give its result, or the extractor itself, to a drift detector.

A Metadata is not a dataset of images. Indexing a Metadata returns factor rows, so in v1.1 Embeddings(metadata, extractor=...) embedded those rows, and Coverage given a Metadata and an extractor but no embeddings= measured the coverage of the factors rather than of the images. Embeddings now raises MaiteShapeError, a TypeError and a ValueError, for a Metadata. Coverage.evaluate embeds the dataset the metadata describes, Metadata.dataset, so its results change, and raises ValueError for a Metadata read back with load, which has no dataset to embed.

Embeddings(metadata, extractor=extractor, batch_size=32)  # v1.1: embedded factor rows; v1.2: MaiteShapeError
Embeddings(metadata.dataset, extractor=extractor, batch_size=32)  # v1.2

Metadata.dataset is None for a Metadata read back with load; pass the dataset it was built from instead.


The score column is one confidence per row

In MAITE, a target can provide scores per bounding box ((N,)) or per class for each box ((N, CLASSES)). A classification target is a score vector. In v1.1, metadata stored target scores directly in the dataframe. The score column was Array(Float32, n_classes) for multi-class targets and Float32 for single-score targets. Because column width depended on class count, datasets with different vocabularies could not be concatenated into a single dataframe. In those cases, merge_datasets() raised an error from np.concatenate.

The score column is now always Float32. It stores the confidence of a row for its assigned class, extracted from whichever layout the target provides. Rows without scores contain null. To access per-class probabilities, read them from the target rather than md.dataframe["score"].to_numpy(). SourceLocator retrieves individual detections. ScoresExtractor extracts per-class prediction arrays from model outputs.

Relabel aligns score arrays when transforming labels. Per-class arrays are indexed by their original vocabulary. In v1.1, relabeled detections retained source-indexed score columns alongside relabeled targets. Consequently, scores mapped to incorrect classes or exceeded array bounds. Detection scores are now reduced to a single confidence value for the source label before vocabulary transformation. Classification vectors are mapped into the target vocabulary, summing probabilities when classes are merged. A Mapping[int, str] target vocabulary must use non-negative integer indices. Non-contiguous class indices are supported and no longer raise IndexError.

If code accesses dataset[i][1].scores from a relabeled detection dataset and requires per-class arrays during migration, pass Relabel(..., reduce_detection_scores=False). This maps scores into the target vocabulary. In v1.1, the array retained the source vocabulary indexing and width. Verify any code that indexes the resulting array. Classes missing from the source array receive a score of 0.0 rather than being treated as absent. Merged classes combine their scores by summation.

Setting reduce_detection_scores=False does not change the metadata score column width. The column retains one value per row under both settings. However, it changes the numeric values in that column because metadata reads from the transformed target. When classes are merged, the column reflects the summed class score. Where scores cannot be read, the column contains 0.0 instead of null. The reduce_detection_scores parameter emits a warning and will be removed in v1.3. Omitting the parameter uses the default reduction behavior.


A class axis of your own

Metadata.classed_by() configures one or more factors as the class axis. Unmodified metadata instances continue to report default class labels.

by_weather = metadata.classed_by("weather")
Balance().evaluate(by_weather)  # conditions on weather
Coverage().evaluate(by_weather, embeddings=emb)  # per-weather coverage
split_dataset(by_weather, stratify=True)  # folds balanced over weather

This configuration introduces three behaviors:

Original dataset labels become a factor. The pivoted metadata gains a factor named class containing the original labels. Results retain the relationship between the custom axis and the original dataset labels.

All class-label consumers use the pivoted axis. Metadata.class_labels and Metadata.index2label update together. Evaluators without an explicit label= parameter evaluate against the pivoted axis. Representation is an exception because it validates label names against an ontology. It raises an error when passed pivoted metadata.

Results record the active class axis. Class-conditional outputs add a class_axis field holding a ClassAxis. The meta().state mapping adds class_axis, class_axis_source, and class_axis_level entries:

result = Balance().evaluate(by_weather)
result.class_axis.source  # "derived" (custom axis rather than dataset labels)
result.class_axis.rows_per_group_entity  # 3.0 if a per-image axis was read from detections

For standard evaluations, class_axis_source records "ground_truth". In previous releases, this field recorded the string "NoneType". When setting label= on bias evaluators, class_axis_source is recorded identically.

The class axis configuration persists across at, where, having, and agg operations. Like view, include, and exclude, it is omitted during serialization with Metadata.save().

See Analyze bias against a custom class axis for a worked example covering composite axes and metadata level selection.


Duplicates checks annotation and metadata factors

Duplicates now evaluates annotations and optional metadata factors alongside pixel checks. Datasets without targets or duplicate_factors produce the same pixel-only results as v1.1.

Datasets with targets include annotation duplicate rows. When a dataset yields (image, target, metadata) and targets contain boxes or labels, Duplicates.evaluate inspects annotations by default. Two additional result patterns can appear:

  • Groups whose methods list contains "annotation". These represent items sharing identical annotations regardless of pixel similarity.

  • Existing pixel groups (dup_type="exact" or "near") where differs_on contains "annotation". These represent matching images with differing annotations.

These additions do not replace previous findings. They provide new rows or additional fields on existing rows.

Results add two columns: differs_on and annotation_divergence. differs_on (list[str] | None) uses a three-state representation. A null value indicates the axis was not evaluated. An empty list [] indicates the axis was evaluated and values agreed. A populated list names the attributes that disagreed. annotation_divergence (float | None) records divergence scores. See Reading differs_on and the divergence columns for the complete three-state contract and pair row mean_iou behavior.

Metadata factor grouping requires explicit configuration. Passing metadata= and duplicate_factors=[...] to evaluate() groups items whose specified factor values match exactly. These groups are reported with dup_type="factors". Omitting duplicate_factors disables factor grouping. No factors are evaluated by default.

Every named factor must agree. duplicate_factors functions as a conjunction. Items are grouped only when all named factors match; matching some and differing on one produces no group. Adding factors narrows matches. Items with unstated factors (null or NaN) are excluded from grouping. factor_cardinality counts stated values only.

evaluate() accepts a Metadata instance directly. When evaluating metadata alone, no pixels or annotations are decoded, so duplicate_factors is required. The run reports at level="item" only.

# Unchanged from v1.1: image datasets without targets produce identical results
Duplicates().evaluate(images)

# New in v1.2: object detection datasets include annotation duplicate rows
Duplicates().evaluate(od_dataset)

# New in v1.2: optional grouping by specified metadata factors
Duplicates().evaluate(od_dataset, metadata=Metadata(od_dataset), duplicate_factors=["capture_id"])

# New in v1.2: all specified factors must agree
Duplicates().evaluate(od_dataset, metadata=Metadata(od_dataset), duplicate_factors=["capture_id", "camera_id"])

# New in v1.2: evaluate metadata directly without decoding images
Duplicates().evaluate(Metadata(od_dataset), duplicate_factors=["capture_id"])

Existing group types are preserved. The evaluator continues to report exact, near, and video group types (segment, aligned, and redundant) as in previous releases.

Removing what an evaluator flagged

DuplicatesOutput.deduplicate() and OutliersOutput.prune() return a RemovalPlan: the SourceIndex addresses of the rows to remove. Pass it to Indices with exclude=True.

plan = Duplicates().evaluate(dataset).deduplicate() | Outliers().evaluate(dataset).prune()
cleaned = View(dataset, Indices(plan, exclude=True))

Indices accepts addresses as well as ints. With exclude=True, it removes a detection named by its target_index and, on a tracking dataset, a whole track named by its track_id. Every image and frame stays, even when all of its detections are removed.

Build the view on the dataset the evaluators read. A plan names positions in that dataset, and they are renumbered once anything is removed.

Evaluators take precomputed inputs

Each evaluator family now has a method per kind of input you may already hold, beside evaluate, which still accepts everything it did. evaluate converts its argument and calls the method, so both give the same result.

Evaluator

Method

Takes

Balance, Diversity, Parity

from_metadata(metadata)

a Metadata, or any metadata container

Representation

from_labels(labels, index2label=None)

any container with class_labels

Coverage, Prioritize

from_embeddings(embeddings, labels=None)

an embeddings array, and optionally one label per embedding

DriftUnivariate, DriftMMD, DriftKNeighbors, DriftDomainClassifier

from_embeddings(reference, data)

reference and test embeddings: fit(reference).predict(data) in one call

DriftWasserstein

from_embeddings(reference, validation, data)

the validation set fit requires, between them

ChunkedDrift

from_embeddings(*embeddings)

the arrays its detector’s from_embeddings takes

OODKNeighbors, OODDomainClassifier

from_embeddings(reference, data)

reference and test embeddings, and predict’s keywords

Build the metadata or embeddings once and share them:

metadata = Metadata(dataset)
balance = Balance().from_metadata(metadata)
parity = Parity().from_metadata(metadata)

The drift and OOD detectors keep fit and predict for fitting once and testing several datasets. The reconstruction detectors have no from_embeddings: their model reads whatever array it was trained on, usually images.

Coverage.from_embeddings without labels runs every item as one class, 0. Coverage and Prioritize refuse labels that do not hold one entry per embedding, such as a detection dataset’s labels, which hold one per target. For Prioritize.evaluate this is new: v1.1 accepted mismatched class_labels without complaint.

Constructor arguments are validated

In v1.1 an evaluator validated its Config only when you built the Config yourself. Arguments passed to the constructor were stored unchecked, so Diversity(method="nope") or Balance(num_neighbors="abc") succeeded and failed later, or not at all. v1.2 validates constructor arguments the same way, and raises a pydantic ValidationError, which is a ValueError, naming the field.

Numeric settings are also checked against the range they are meaningful in: counts above zero, p-value thresholds in (0, 1), shares and scores in [0, 1], and percentiles in [0, 100]. The ranges appear in each Config’s JSON Schema.

Diversity(method="nope")  # v1.1: accepted; v1.2: ValidationError naming `method`
Balance(config=Balance.Config(label="weather"), num_neighbors=7)  # unchanged

Drift and OOD configs are validated

The Config of every drift and OOD detector is now a pydantic model, as every other evaluator’s is. A wrong type or an unknown choice fails when the Config or the detector is built, naming the field, rather than later in fit or predict. The error is a pydantic ValidationError, which is a ValueError.

Build a Config with keywords, and use the pydantic methods in place of the dataclasses helpers:

v1.1

v1.2

DriftUnivariate.Config("ks", 0.05)

DriftUnivariate.Config(method="ks", p_val=0.05)

dataclasses.replace(config, p_val=0.01)

config.model_copy(update={"p_val": 0.01})

dataclasses.asdict(config)

config.model_dump()

dataclasses.fields(config)

type(config).model_fields

ChunkedDrift needs three reference chunks

Every threshold except ConstantThreshold derives a chunked detector’s bounds from the spread of the reference chunks’ baseline scores. Two chunks scored against each other measure one pair twice: a symmetric statistic such as MMD² or KS gives both the same value, so v1.1 either passed every chunk or flagged every chunk as drifted. ChunkedDrift now raises ValueError on fit when its chunking splits the reference into fewer than 3 chunks and its threshold is not a ConstantThreshold.

DriftUnivariate().chunked(chunk_count=2).fit(reference)  # v1.1: degenerate bounds; v1.2: ValueError
DriftUnivariate().chunked(chunk_count=3).fit(reference)  # v1.2

Use a chunk_count of 3 or more, or a chunk_size well below the reference size: on 100 reference samples, chunk_size=50 raises and chunk_size=40 does not. To keep two chunks, pass a ConstantThreshold.

DriftWasserstein decides on the mean ratio

In v1.1 DriftWasserstein declared drift when any single feature’s distance ratio exceeded ratio_threshold. Each ratio divides two noisy distances, so the rule raised false alarms on clean data: with 500 training, 100 validation and 100 test samples from one distribution, 99% of batches with 16 features and every batch with 64. The ratio also depended on the sample sizes. With 2,000 training, 500 validation and 50 test samples, the median mean ratio of clean data was 3.2, and with 500 training, 100 validation and 400 test samples it was 0.7, which hides drift.

v1.2 decides on the mean ratio over the features, and divides each ratio by sqrt((1/n_ref + 1/n_op) / (1/n_fit + 1/n_val)), which corrects for the sizes of the reference at predict time, the operational data, and the training and validation data at fit. The default ratio_threshold stays 1.4.

  • drifted is distance > threshold, where distance (and details["ratio"]) is the mean ratio. It no longer equals details["feature_drift"].any(). The per-feature flags remain, to show where to look.

  • feature_ratios, distance and the ratios that score() returns are corrected for the sample sizes, so they equal distances / baseline_distances only when the sizes match. A feature with a baseline of 0 keeps a ratio of 1.0, or inf when its operational distance is not 0.

  • In chunked mode each chunk’s ratio is corrected for the chunk size. Only the default ConstantThreshold is calibrated there: the baseline chunks come from the training reference, and a threshold derived from their spread, such as "zscore", alarms on 6% to 100% of clean chunks in a simulation, more as the chunks grow relative to the reference.

result = DriftWasserstein().fit(train, validation).predict(test)
result.drifted  # v1.1: any(feature_ratios > 1.4), uncorrected; v1.2: mean(feature_ratios) > 1.4, corrected

At the default threshold, about 0% to 3% of clean batches now raise a false alarm from 16 independent continuous features up, 5% to 19% when the features are correlated, and roughly 15% to 35% with 1 to 4 features, where the KS test of DriftUnivariate is the calibrated choice. The mean does not see shifts confined to a few of many features: a shift in 1 of 512 features was not detected, and a 2 standard deviation shift in 2 of 64 features was detected in 66% to 92% of trials. Drop constant and discrete features before you use the detector, and lower ratio_threshold only after you measure the false alarm rate on held-out batches.

Outliers reads pixel statistics in stored units

Outliers.evaluate now measures PIXEL statistics in the units the data is stored in, as image_stats() does by default. In v1.1 it normalized pixel values to [0, 1] first, so evaluate and from_stats on statistics from compute_stats could flag different images from the same data.

Every threshold except ConstantThreshold scales with the data, so for images stored at one bit depth no outlier flag moves. Two cases change:

  • A ConstantThreshold on a PIXEL statistic is now read in stored units. Rescale its bounds, for example from 0.8 to 204 for 8-bit data.

  • Images stored at different bit depths are no longer put on one scale. To compare them the v1.1 way, compute the statistics normalized and pass them to from_stats:

stats = image_stats(images, stats=ImageStats.PIXEL, normalize_pixel_values=True)
Outliers().from_stats(stats)

VISUAL statistics are unaffected: they always read the display range.


Outliers reports the raw cluster distance

When Outliers is given an extractor, it flags items far from their cluster and reports them with metric_name cluster_distance. In v1.1, metric_value on those rows was the number of standard deviations the item lay from its cluster’s mean distance. It is now the distance itself, and the row carries its cluster’s population_mean and population_std, so the v1.1 value is one expression away. Which items are flagged is unchanged.

cluster = result.data().filter(pl.col("metric_name") == "cluster_distance")
cluster.select((pl.col("metric_value") - pl.col("population_mean")) / pl.col("population_std"))  # the v1.1 value

Relative box geometry in the dimension statistics

ImageStats.DIMENSION now includes seven rel_* statistics: box offset, width, height and area as fractions of the image, and distance to the image center and nearest edge as fractions of the half-diagonal and of that edge’s dimension. They carry exactly the definitions compute_ratios() applies to the absolute statistics, and are grouped as ImageStats.DIMENSION_RELATIVE. Existing statistics are unchanged.

Because Outliers computes DIMENSION by default, a run with per_target=True can report outliers on the rel_* statistics too. An item’s own row is the whole frame, so its relative values are constant and never flagged.


compute_ratios is deprecated

compute_ratios() is deprecated and will be removed in v1.3. It warns with DeprecatedWarning, a FutureWarning. Both of its uses have a replacement that gives the same values:

  • Box geometry relative to its image is a statistic now. Compute ImageStats.DIMENSION_RELATIVE, or the single rel_* flag you need, with per_target=True.

  • For any other statistic, add the per-image and per-target statistics to a Metadata. Each box row at the instance level carries its image’s value, so the ratio is one division.

v1.1

v1.2

compute_ratios(stats)["stats"]["size"]

image_stats(..., stats=ImageStats.DIMENSION_REL_SIZE, per_target=True)["stats"]["rel_size"]

compute_ratios(stats)["stats"]["brightness"]

md.add_factors(stats), then md.rows_at("instance").select(pl.col("instance_brightness") / pl.col("unit_brightness"))

The rel_* statistics use the definitions compute_ratios applied to offset_x/offset_y, width, height, size, distance_center and distance_edge. A custom override_map becomes a polars expression over the same instance_ and unit_ columns.


compute_stats is now image_stats

compute_stats is renamed to image_stats(), which says what it measures now that video has its own statistics. The parameters and the result are unchanged. The old name still works until v2.0 and emits a DeprecatedWarning.

stats = compute_stats(images, stats=ImageStats.PIXEL)  # v1.1
stats = image_stats(images, stats=ImageStats.PIXEL)  # v1.2

Leaving normalize_pixel_values unset no longer emits a FutureWarning. Its default stays False, so results do not change.


Track statistics declare their labels and add a gap fraction

track_stats() adds gap_fraction, the share of the frames a track spans in which it is absent. Its result also declares labels categorical: Metadata.add_factors records the track labels as a vocabulary, as factor_levels= would, so no consumer treats a class id as a quantity. A vocabulary you declared yourself is kept.


video_stats describes each video

video_stats() computes statistics that describe a video as a whole, one row per video, in one walk of each video’s frames. VideoStats selects them, in the groups DESCRIPTIVE (frame count, duration, frame rate), REDUNDANCY (the share of frames that repeat the one before, and the longest such run), EGO (camera motion summaries) and PERSISTENCE (per-video maps of what is fixed to the camera and of local blur). EGO and PERSISTENCE need OpenCV from the dataeval[opencv] extra. The default is VideoStats.NO_EGO, everything that does not need OpenCV.

The PERSISTENCE flags (PERSISTENCE_EGO, PERSISTENCE_BLUR, PERSISTENCE_BLUR_RAW) and their maps ego_map, blur_map and blur_map_raw are experimental ⚠️: selecting any of them, directly, through PERSISTENCE or through ALL, raises ExperimentalWarning once per process. The map shape, the column names and the blur_measure, blur_sigma and persistence_samples keywords may change in a later release, and so may SharpnessMeasure, the protocol a blur_measure follows. The default VideoStats.NO_EGO selection does not warn.

from dataeval.core import video_stats

metadata.add_factors(video_stats(dataset))  # one row per video, at the sequence level

To check measured values against ones the dataset already reports, such as video_duration, pass return_collisions=True to Metadata.add_factors(). It returns the videos where the two disagree, within an optional collision_tolerance.

redundancy_radius is the Hamming distance within which a frame counts as redundant, and shares its default with Duplicates. ego_gap and ego_proc_short_side pass through to ego_stats(), whose per-frame measurements the EGO summaries reduce through the same roll-up Metadata.add_factors uses. ego_stats() now measures videos in parallel under max_processes, and accepts a list of streams. Set max_processes=1 for a dataset whose videos share one open decoder. To look inside one video instead, measure its frames with image_stats() over SequenceFrames. See Volatility and drift of a per-frame statistic. SequenceFrames can walk a video more than once, so each dataset[i] must return a stream that can be walked from its first frame, independently of any earlier stream for the same index.


image_stats refuses images and boxes of different lengths

When boxes= held more entries than there were images, image_stats silently dropped the extra boxes. When it held fewer, the remaining images were measured as if unannotated. Either way the annotation set changed without a warning, and a box list that had slipped out of step with its images went unnoticed. image_stats(), and the deprecated compute_stats alias, now raise ValueError naming the index where one of them ran out:

image_stats(images, boxes=boxes)  # v1.1: 4 images, 3 box lists, the fourth image measured with no boxes
image_stats(images, boxes=boxes)  # v1.2: ValueError: images and boxes differ in length: boxes ran out at index 3

Give every image an entry in boxes. An entry of None still marks an unannotated image. One-pass iterables such as generators raise as soon as the mismatch is found, so no partial result is returned.

A literal None in place of an image under boxes= was skipped silently in v1.1. It now fails inside NumPy (ufunc 'isnan' not supported), as it already did without boxes=.


AddressedResult

AddressedResult is the new base for a statistics result whose values carry an address, a source_index that names the row of each value. StatsResult now extends it with its image bookkeeping (object_count, invalid_box_count and image_count), so image_stats and compute_ratios return what they did before. video_stats() returns an AddressedResult directly, since a video has no image bookkeeping. Code that only reads stats and source_index needs no change, and combine_stats_results now accepts any addressed result.


Duplicates hashes the native pixel values

Perceptual hashes (phash, dhash and their _d4 variants) read an image in the dtype it is stored in. Previously image_stats() converted every image to float64 first, and a float image with a maximum of at most 1 is read as holding values in [0, 1] and stretched to full contrast. Integer images are now hashed at their stored values instead. xxhash still digests the float64 view, so its values are unchanged.

Duplicates groups change only where integer images have a maximum of at most 1. Floating-point images, and integer images with any larger value, hash exactly as before. The case to watch is a binary mask stored as 0 and 1.

  • A 3-channel (RGB) mask, or one with 5 or more channels, now reads as black, because its Rec. 601 luma truncates to 0. It hashes flat, so distinct masks group as near duplicates.

  • A single-channel, 2-channel or 4-channel mask still hashes by its content, though to different values than before.

Scale masks to 0–255 before running Duplicates. That restores the earlier hashes. If you stored hashes of such images, recompute them.


Grayscale scales integer images deeper than 8 bits

to_canonical_grayscale() clipped every value above 255, so 12- and 16-bit integer images, such as 14-bit infrared in uint16, mostly saturated to a flat white frame. Their perceptual hashes then matched each other, and ego motion read them as a still camera. Such an image is now scaled from the encoding get_value_range() decodes for it:

  • a maximum up to 4095 scales from 12 bits;

  • a maximum up to 65535 scales from 16 bits.

Only integer images with a value above 255 are affected. Their phash and dhash values change, and so do Duplicates groups, ego_stats, video_stats and BoVW features that read them. 8-bit images, signed images and float images convert exactly as before. If you stored hashes of deeper integer images, recompute them.


Re-detecting from statistics honors per_image and per_target

Outliers.from_stats defaults to per_target=False, while image_stats defaults to per_target=True, so the default call Outliers().from_stats(image_stats(dataset)) reports item rows only. Its with_threshold, classwise and itemwise re-detected from every row of the statistics, so they returned target rows that the first result did not report. Re-detection now covers the rows the flags selected, so a result and its re-detections describe the same rows. A result built with per_image=False likewise no longer gains item rows when you re-detect. Rows addressed at a level between an item and its labels, such as a frame or a track, are still always included.

To keep target rows in the result and in its re-detections, pass per_target=True to from_stats.


Outliers needs 20 rows per population

Outliers fits a statistical threshold to each population: the rows of one metric at one level, or within one class under classwise(). It now fits one only to a population with at least 20 rows that hold a value. A smaller level, or a smaller classwise() class, yields no findings. Before, AdaptiveThreshold fitted from 3 rows. A ConstantThreshold needs no data, so it still judges every population.

Below 20 rows, the default threshold flags clean data often enough that most findings are noise. On clean, normally distributed values it flags this share of rows:

Rows in the population

Rows flagged

5

3.4 %

10

0.17 %

20

0.05 %

To see what was skipped, read OutliersOutput.populations. It lists every population with its size, whether it was judged, and why not:

result = Outliers().evaluate(metadata)
result.populations.filter(~pl.col("judged"))

If a population you need is skipped, collect more rows for it, or judge it with a ConstantThreshold.

classwise(...).with_threshold(...) now keeps the per-class populations. Before, with_threshold re-detected against whole levels, so the classes were lost.

Results of classwise() and per_class=True now carry a population column that names the class each row was judged in, such as class_label=cat.


aggregate_by_class counts a box under its own class

On a result computed with per_target=True, aggregate_by_class joined each flagged box to the labels of its whole image, so a box was counted once under every class in the image. It now joins a box to its own label. On any result, a flagged image row now counts once under each distinct class among its labels. Before, it counted once per label, so an image with boxes [car, car, bus] counted twice under car; it now counts once.

Expect lower per-class totals for images that hold boxes of more than one class, and for images that hold more than one box of a class.


Outliers thresholds metadata factors

Outliers.evaluate accepts a Metadata as data and thresholds its ordered factors – numeric or temporal by their raw type – each against the rows at the level it was measured at, so a track is compared with tracks and a frame with frames. factors=[...] names which. Only raw values are read, so re-binning a metadata never changes the result. Categorical factors, including integers with a declared vocabulary, are not thresholded.

levels= names the levels to judge, with the same names and rules as Duplicates.evaluate(): "item" and "target" for image and detection data, and "sequence", "unit", "track" and "instance" for tracking data. Passing levels together with per_image or per_target raises. Without either, a tracking metadata judges its sequence and frame levels, as Duplicates does, so judge track factors with levels="track". per_target=True adds every detection and track.

OutliersOutput gains .items, .targets, .sequences, .frames, .tracks and .detections, which narrow a result to one level and keep re-detecting at that level. On a result that holds frames or tracks, aggregate_by_metric adds a level column and counts each flagged frame, track, detection or item once, and aggregate_by_class raises, since a frame or a track has no class. Narrow to .targets first.


Outliers names image and box metrics by level

Outliers.evaluate on a detection dataset with per_target=True now judges the dataset through the Metadata built for it, and names each metric by the level it was measured at, as Metadata.add_factors() names the same statistics: unit_brightness on image rows and instance_brightness on box rows. In v1.1 both were brightness. The names carry through to the result’s aggregations: the rows of aggregate_by_metric and the metric columns of aggregate_by_item and aggregate_by_class.

result = Outliers().evaluate(od_dataset, per_target=True)
result.data().filter(pl.col("metric_name") == "brightness")  # v1.1
result.data().filter(pl.col("metric_name") == "unit_brightness")  # v1.2, image rows

Results that hold one level keep the plain names: per_target=False, per_image=False, a dataset without targets, and Outliers.from_stats.


Balance scores factor pairs over images, not detections

On a detection dataset, Balance reads its factors on detection rows, so a per-image factor repeats once per detection. The factors table took the chance correction for each pair of factors against those rows, which let two unrelated per-image factors read as correlated in proportion to the detections per image. The correction is now taken against the entities each pair varies over: the images for per-image factors, and the videos for a per-sequence factor read on frames.

Pairs of factors defined above the level being read score lower, and independent ones score near zero. Pairs at the level being read are unchanged, and so are the balance and classwise tables. Core mutual_info() and mutual_info_classwise() take an optional effective_n, one entity count per column, and compute what they did before without it.


Parity reports cells and statuses

Parity ⚠️ and parity() ⚠️ are experimental, so these changes move inside the experimental contract. Parity now tests each factor over independent images or sequences and reports every (factor, level, class) cell. See Parity for the method.

ParityOutput.insufficient_data is removed. It listed, per factor and level, the cells with small counts. A combination that never occurs has no row in any table, so it never appeared there, and the mapping could not show what the dataset lacks. The new ParityOutput.cells frame has one row per factor, observed level and observed class, observed zeros included, and marks the cells too thin to rely on:

result.insufficient_data["illum_lux"]  # v1.1
result.cells.filter(pl.col("is_sparse") & (pl.col("factor_name") == "illum_lux"))  # v1.2

Each cell carries observed, expected, deficit (expected minus observed), lift (observed over expected), an exact two-sided p_value and its p_value_adjusted, is_deviant, is_sparse, a status (underrepresented, overrepresented, too_thin, no_finding or untestable) and is_actionable.

The factors frame changes:

v1.1

v1.2

is_significant

is_associated, which reads the Benjamini–Hochberg adjusted p-value

p_value from the G-test (log-likelihood ratio)

p_value from Pearson’s chi-squared test, or the Rao–Scott F when rows are grouped, plus p_value_adjusted

has_insufficient_data: an observed cell count below 5

has_insufficient_data: Cochran’s criterion on the expected counts, divided by the design effect, or fewer than 4 design degrees of freedom

a constant factor scores 0 with p = 1

NaN results and status="untestable", never a verdict of independence

status: associated, weak, inconclusive, not_detected or untestable

design_effect, effective_groups, df_num and df_den

Every flag now reads the adjusted p-value, so a factor significant at a raw p of 0.04 among several factors can stop being associated. Detections are grouped by image and frames by sequence, so p-values on object detection and tracking datasets are larger than in v1.1, whose row-level p-values on such data were invalid.

New configuration: lift_threshold=1.25, at least 1, sets how far a significant cell’s lift must be from 1 for the cell to be actionable, and 1 disables the band. assume_independent=False is described in the next section.

New output fields: test ("pearson" or "rao_scott_f"), group_source, group_level, n_rows and n_groups. meta().state adds lift_threshold, assume_independent, group_source and test.

Core parity. ParityResult removes insufficient_data and adds chi_squared, df_num, df_den, p_values_adjusted, design_effects, effective_groups, insufficient, testable, cells (a ParityCells of flat per-cell arrays), test, n_rows and n_groups. Its third argument is groups, one independent group id per row. Pre-release builds of 1.2 had an effective_n argument in that position; groups replaces it.


Parity needs an independent sampling unit

Parity must know which rows are independent of each other. It reads that from the input’s structure: a Metadata groups by its item_indices at any view, so detections group by image and frames by sequence (group_source="level"), a container with many-to-one item_indices groups by them ("item"), and one row per item needs no grouping ("rows"). A Metadata built from a dataset always records its structure, so these calls are unchanged except for their numbers.

Over a derived view, the unit is the root source item rather than the derived item. Parity over a DetectionCrops view (shipped in v1.1.4) or a frame view now clusters rows by the source image or video (group_source="source"), where it would otherwise have read every crop or frame as an independent sample. A merge_datasets() of crop or frame views keeps each view’s grouping, and a View that repeats items groups the copies too. So does a merge that includes one dataset more than once, such as merge_datasets(ds, View(ds, augment)), which v1.1 read as independent items. Counts and p-values over crops and such merges change accordingly. Metadata saved from a crop view by an earlier version records no root items and is refused on load: rebuild it from its dataset and save it again. So is metadata saved from a frame view before this change, though frame views are new in v1.2. A plain dataset whose own datum metadata has a source_id key is refused too when an earlier version saved it, and is rebuilt the same way.

Balance counts the same unit when it corrects for chance. Over a crop or frame view, a factor that never changes within a source image or video, or a label that doesn’t, now counts the source’s images or videos as its effective sample size rather than its crops or frames, so Balance’s scores over those views change. The same holds for the copies a repeating View or merge holds of one dataset item.

When nothing records the unit, Parity now raises ValueError instead of treating every row as independent. This reaches only custom MetadataLike containers that carry no item_indices. Parity does not apply the protocol’s default of one label per item, because a container that silently lacks item_indices is the case the refusal guards against. Give the container item_indices (label → item), or declare the rows independent:

Parity().evaluate(custom_metadata)  # v1.1: rows read as independent; v1.2: ValueError
Parity(assume_independent=True).evaluate(custom_metadata)  # v1.2, when each row is its own item

Core parity() takes the grouping directly, and needs groups or assume_independent=True:

parity(factor_data, class_labels)  # v1.1: row-level test; v1.2: ValueError
parity(factor_data, class_labels, assume_independent=True)  # v1.2, the v1.1 row-level test
parity(factor_data, class_labels, item_ids)  # v1.2, rows grouped by item

Passing both groups and assume_independent=True raises as well. group_source="declared" records that the caller made the declaration, and the p-values hold only if it is true.


label_parity tests homogeneity

label_parity() is stable, and this is a correctness fix. In v1.1 it ran a goodness-of-fit test that treated the expected labels as the exact population. With both samples drawn from one distribution, that test rejected at α = 0.05 in 27% of draws at 1,000 against 1,000 labels, 87% at 200 against 2,000, and 30% at 100 against 100. A homogeneity test on the same draws rejected 4.4–6.5%. It was also asymmetric: swapping the two arguments moved one p-value from 1.7 × 10⁻⁷ to 0.39. And it broke at the edges: num_classes above the classes present gave NaN, a class absent from the expected labels gave an infinite statistic and p = 0, and samples with different largest labels raised unless num_classes was passed.

v1.2 runs Pearson’s chi-squared test of homogeneity on the 2 × K table of class counts, identical to scipy.stats.chi2_contingency(table, correction=False). It is symmetric in its two arguments and returns no NaN or infinity for those inputs. Every statistic and p-value changes. The result adds score (bias-corrected Cramér’s V between dataset and class), insufficient, df_num, df_den and test.

Detections need groups. Read as independent labels, two detection datasets with the same class distribution but different image counts and detections per image were declared different 84–91% of the time at α = 0.05 in simulation; grouped by image, 4.4–4.7%. New keyword arguments expected_groups and observed_groups take one group id per label, the item index for detections or the sequence id for tracks. Pass both or neither.

Two deprecations, both warning with DeprecatedWarning, a FutureWarning:

  • Calling without groups and without assume_independent=True still runs as if every label were independent, but warns. From 2.0 it raises. assume_independent=False without groups raises now.

  • num_classes is ignored, since the classes are the ones that occur in either sample. Passing it warns, and it is removed in 2.0.

label_parity(expected, observed, num_classes=10)  # v1.1
label_parity(expected, observed, assume_independent=True)  # v1.2, one label per image
label_parity(expected, observed, expected_groups=expected_items, observed_groups=observed_items)  # v1.2, detections

See Label parity for the population, the vocabulary requirement and how to read a result.


Overwriting a factor keeps the old column

Metadata.add_factors() with overwrite=True no longer discards the column it replaces. The incoming values take the name. When the write would replace values, by changing a row’s value, making it missing, or moving the factor to another level, the column as it was moves to <name>_original, or <name>_original_2 and so on when that is taken. A write that only fills empty rows, or writes back the values already there, moves nothing aside, so re-running an overwrite adds no columns. A keyed overwrite still replaces only the rows its keys name, and keeps the whole prior column when it replaces a value.

The moved column is kept but not analyzed. rows_at and dataframe read its values, and factor_info describes it, but it is left out of factor_names, factor_data and every evaluator, whatever exclude holds. To analyze it, name it in include, for example md.include = ["brightness", "brightness_original"]. Because factor_info describes it, the keys of factor_info can include a name factor_names does not, so pair the two by name rather than by position. A column stored under <name>_added by overwrite=False is analyzed as before.

One moved column is not kept for good. A roll-up output keeps its roll-up’s record, so when a source of that roll-up later changes, the stale moved column is dropped and the roll-up is recomputed under the name it asks for, renamed if the overwrite took that name.

FactorInfo gains source_name, the name a factor had in its source before DataEval renamed the column. A dataset key level, stored as metadata_level, records level, and both brightness_added and brightness_original record brightness. Saved metadata keeps these names, and a file saved by an earlier version reads each column as its own source name.


Partial keyed source_index

Metadata.add_factors() accepts addresses that carry a key and name only some rows of their level. Previously, a source index had to name every row at each level it addressed, and naming fewer raised ShapeMismatchError. A key names one row, so findings measured on some of a video’s frames now attach to the full dataset:

md.add_factors({"blur": [0.2, 1.0]}, source_index=[SourceIndex(0, 2, "unit"), SourceIndex(1, 0, "unit")])

Rows no address names are null. A keyed source_index write folds into a factor of the same name exactly as a key= write does: a later write that names only rows still null fills them in rather than being stored as <name>_added, and naming a row that already holds a value is a collision, which overwrite decides. Addresses without a key, such as SourceIndex(3), still have to name every row at their level.


Metadata carries the datum id as item_id

The id that MAITE requires on every datum was a factor in v1.1: it appeared in factor_names, was binned, and gave Balance and Diversity an id row. An identifier names a row rather than measuring the data, so it is now the reserved column item_id, present on every row of the dataframe, and never a factor.

md.dataframe["id"]  # v1.1
md.dataframe["item_id"]  # v1.2

Expect bias results without an id row.


Metadata sets aside factors it cannot read

A key that mixes kinds of value is set aside. In v1.1, a datum metadata key holding both numbers and text, such as an altitude recorded as 100.0 and as "high", was read as text and analyzed as a categorical factor, and booleans mixed with numbers were read as 0 and 1. Neither is read as a factor now: Metadata.dropped_factors lists it with the reason mixed_types, and Metadata.unusable reports the values of each kind. Map the values onto one type at the source, or declare the mapping with Metadata.repair():

from dataeval.types import Remap

md.dropped_factors  # {"altitude": ["mixed_types"]}
md = md.repair([Remap("altitude", {"high": 300.0})])

Known factor names are checked. The names in dataeval.types.KNOWN_FACTORS (timestamp, frame_rate, video_duration, latitude and longitude) each have one meaning, a unit, a range and the levels they are defined at. In v1.1 they were ordinary factors. A column of the dataset’s own metadata that breaks its definition, such as a latitude of 100, text in a timestamp, or a frame_rate per image rather than per video, is now set aside with the reason invalid_known_factor, and Metadata.add_factors() raises ValueError for one. Fix the values, or rename a factor that means something else. See Known factor names.

None is a missing value. A key an item declares with the value None held the category "None" in v1.1. It is now null, and missing values are binned in a group of their own. Select those rows with pl.col(name).is_null() rather than pl.col(name) == "None". For a text key the groups are the same. A number key that held a None was read as text in v1.1: one category per value, or dropped when nearly every value differed. It is now read as numbers, and binned when it holds many values, so its bias results change.


ber_knn lower bounds move

ber_knn could report a lower bound above the true Bayes error. Its plurality vote broke ties toward the smallest label, which made an even k err far more than the constants it divided by assume: with a constant posterior of 0.9 (Bayes error 0.1), k=2 reported a lower bound of 0.125. Three changes keep the bound below the Bayes error:

  • Ties go to the tied label whose member is nearest, so an even k votes like k - 1 and takes its constant.

  • Devroye’s constant applies only for odd k >= 5, where he states it; elsewhere the tighter FeeBee constant applies.

  • With more than two classes, the bound comes from the 1-NN error, for any k, since Cover & Hart’s inequality is about the 1-NN error.

The lower bound is also clamped to the upper bound, which is an error the k-NN classifier achieves. upper_bound changes only where ties occur. ber_knn also accepts Python lists for class_labels, as documented, and a new groups= argument.


Feasibility judges targets with evidence

Feasibility is a new experimental evaluator in dataeval.performance. For each target, a metric and a threshold, it returns one of four determinations (demonstrated_feasible, infeasible_under_recorded_assumptions, not_ruled_out, unresolved) and the evidence behind it: envelopes, upper bounds on what any model in a stated class can reach, and witnesses, lower bounds on what a frozen model reaches. A witness that beats an envelope suspends the conclusion: the determination is unresolved with the limitation conflict. Every bound is a finite-sample bound, and all the bounds of one call hold together with probability at least 1 - alpha.

from dataeval.performance import Feasibility, TargetSpec

target = TargetSpec(metric="accuracy", threshold=0.95)
result = Feasibility().evaluate(dataset, target=[target], embeddings=embeddings)
result.determinations  # one row per target
result.bounds  # the bounds each row cites

This release covers image and video classification, with accuracy and AP targets and, for image classification, a per-class recall target at a stated false-positive limit:

tank = TargetSpec(task="IC", metric="recall", threshold=0.9, class_name="Tank", max_false_positive_rate=0.05)
result = Feasibility().evaluate(dataset, target=[tank], embeddings=embeddings, predictions=decisions)

A recall target counts a witness only when its own false-positive upper bound is within the limit, and its envelope bounds the recall any model in its scope reaches at that limit. It is refused on grouped data and with false_positive_scope="each_class". predictions= also accepts a 1-D integer array of class decisions, model= takes a MAITE model in its place, and when the requirement is stated on a vocabulary coarsened by Relabel, Relabel.fold_scores folds the model’s class scores through the same map. Bounds on Bernoulli trials use exact Clopper–Pearson endpoints on ungrouped data, and print(result) lays out one line per requirement.

Note

The first v1.2.0 release shipped this evaluator with a different surface: targets were a mapping of string keys, allowances a dict, the call took provenance, sampling and look, the constructor took panel_size, n_cells and builtin_witnesses, and the result held determinations with free-text columns, plus evidence. The typed surface above replaces it: determinations keeps its name but holds Enum and number columns only, and bounds replaces evidence. A removed argument raises TypeError, and SamplingPlan, Evidence, Endpoint and aggregate_by_evidence() are gone. TargetSpec now takes metric and threshold first and task after them (it defaults to "IC"), so a positional TargetSpec("IC", "accuracy", 0.9) raises ValueError: pass keywords. It is experimental and may change again.

Object detection and tracking datasets raise NotImplementedError. Feasibility does not use ber_knn or ber_mst, whose bounds are asymptotic, so the ber_knn change above does not move its results. TargetSpec and Allowances are exported from dataeval.performance alongside it, and the bounds it composes are new experimental functions in dataeval.core: mean_endpoint(), binomial_endpoint(), ratio_endpoint(), units_to_cross(), zero_one_consensus(), cost_consensus(), consensus_max(), cost_consensus_max(), kink_costs(), disjoint_panels(), fit_partition(), local_panel_bound(), ap_upper_envelope(), ap_lower_staircase() and witness_thresholds().

See Performance limits and Feasibility evidence.


split_dataset honors split_on on detection and frame datasets

v1.1 ignored split_on on a detection dataset with more detections than images. It logged “split_on is not supported for object detection datasets; ignoring.” and split by image. A sparser one took the classification path, which split its detection rows as though they were images. v1.2 keeps every image of a group in one partition, including an image without detections, however few detections the dataset holds. Over a SequenceFrames view, split_on=["sequence"] keeps every frame of a video in one partition.

from dataeval.data import SequenceFrames, split_dataset

splits = split_dataset(SequenceFrames(dataset), split_on=["sequence"], test_frac=0.2, val_frac=0.2)

The fractions now count groups, not images. Each group is split as one unit, so over five videos test_frac=0.2 aims at one whole video, however many frames it holds, and a split of a few long videos can land far from the fractions you asked for. Stratification counts groups too: a group holds a class if any of its images does, and stratify=True needs each class in at least as many groups as there are partitions, counting train, validation and test.

A group is one combination of the named factors’ values as recorded, not of the bins Metadata cuts for the bias evaluators. A frame view therefore gives one group per video however many videos it holds, and the fractions, the stratification count and the error messages count videos. Two kinds of factor keep their bins: a factor holding floats, a measurement whose every value would be a group of its own, and a factor whose cut you declared with continuous_factor_bins or a recorded encoding. A metadata container other than Metadata is grouped by its factor_data codes, as before.

Grouping by values bins nothing, so a split on such factors alone no longer emits the “binned automatically” warning. Balance, Diversity and Parity still read the binned factor, and the warning now comes from them.

The same rule reaches image classification splits. An integer factor with more distinct values than the level budget allows, more than 20 on a small dataset, was grouped by its automatic bins in v1.1, which merged whole groups. v1.2 groups it by each value:

split_dataset(Metadata.from_factors({"site": sites}, class_labels=labels), split_on=["site"], num_folds=5)  # 30 sites
# v1.1: fewer groups than sites, each a run of consecutive sites
# v1.2: 30 groups, one per site

Where v1.1 logged and carried on, split_on on a detection dataset now raises a ValueError:

  • when it names a factor the metadata does not carry

  • when it names a factor that varies within one image, such as one added at the instance level

  • when there are too few groups to give each partition one, or, with stratify=True, a class is in fewer groups than partitions

  • when the split would leave a partition without a group

  • when the dataset is a metadata container other than Metadata that holds more labels than images, since it has no row for an image without detections

Apart from the refusals in the next section and the grouping of an integer factor past the level budget, image classification splits are unchanged, and so is a detection split without split_on.


split_dataset refuses a split it cannot fill

stratify=True needs the rarest class in at least as many items as there are partitions. v1.1 counted a single split (num_folds=1) with a test holdout as two partitions where it makes three: train, validation and test. A class too rare to reach all three passed the check. On a detection or frame dataset the split then came back with an empty train partition and no error. On an image classification dataset it failed inside scikit-learn on n_splits=1. v1.2 counts the validation partition, and a detection split that still leaves a partition empty raises as well. Both raise a ValueError that gives the counts:

# One class in 2 of 100 frames
splits = split_dataset(frames, stratify=True, test_frac=0.2, val_frac=0.2)
# v1.1: train 0, validation 50, test 50
# v1.2: ValueError: Unable to stratify: the rarest class has only 2 images, fewer than the 3 partitions requested.

These two checks leave unchanged a split that filled every partition in v1.1. For one that now raises, ask for fewer partitions or less extreme fractions, or pass stratify=False.

stratify=True also raises on a filtered detection Metadata that keeps an item without detections. v1.1 numbered the kept items as though each held a detection, so the classes it stratified on slid onto the wrong items: five single-class videos could all read as holding both classes. Without stratify, the split places every item correctly, with or without split_on.

kept = Metadata(frames).where(pl.col("frame") >= 3, level="unit")  # some kept frames hold no detections
splits = split_dataset(kept, stratify=True, test_frac=0.2, val_frac=0.2)
# v1.1: folds stratified on other frames' classes
# v1.2: ValueError: 5 of the 85 items this filtered Metadata keeps hold no detections, ... Pass stratify=False, ...

A grouped split that leaves fewer groups after the test split than its folds need, num_folds of them or two for a single split, raises a ValueError that counts groups, in place of a scikit-learn error that reported n_splits=1 or counted samples:

split_dataset(metadata, split_on=["site"], test_frac=0.5, val_frac=0.2)  # 3 sites
# v1.1: ValueError: k-fold cross-validation requires at least one train/test split ... got n_splits=1.
# v1.2: ValueError: The test split leaves 1 of the 3 groups for 1 [train, val] fold(s), which need at least 2. ...

A stratified detection split, with or without split_on, now logs a warning on the dataeval.data logger when a class misses a partition it could have reached, as a classification split already did. The message names the partition and the class, such as “fold 2 val lacks class 1”.


Outliers judges a tracking dataset in video terms

Outliers().evaluate(tracking_dataset) failed with a numpy error before v1.2. It now measures every frame with image_stats(), places the statistics on the dataset’s Metadata with SequenceFrames.to_source(), and judges the result, as Duplicates already does. A SequenceFrames over a tracking dataset gets the same treatment, and its selector decides which frames are measured.

from dataeval.quality import Outliers

result = Outliers().evaluate(tracking_dataset)  # the sequence and unit levels
result = Outliers().evaluate(tracking_dataset, per_target=True)  # also track and instance
  • Levels. sequence (a video) and unit (a frame) are judged by default. per_target=True adds track and instance (a detection). A video’s value of a statistic is the median over its frames, and a track’s is the median over its detections.

  • Addresses. Rows are addressed in the source dataset, not by position in the frame view: item_index is the video, and target_index is the frame number at unit, the track id at track and null at sequence. A row’s SourceIndex resolves with SourceLocator. result.stats keeps the frame view’s own positions.

  • Names. Metric names follow Metadata.add_factors(): width and height collide with columns the tracking metadata already holds, so they read width_added and height_added, or unit_width and so on with per_target=True.

  • Baselines are pooled over the dataset. To judge a night video against night videos, evaluate the subset on its own; the Outliers.evaluate() docstring gives the recipe. The sequence level is judged only with 20 or more videos.

  • Refused with a ValueError: extractor= (cluster mode), factors, combinations, given, kinds and neighbors, several datasets in one call, a tracking dataset mixed with image datasets, a metadata= of another dataset, and a frame view that keeps no frames.

  • Cost. Each video’s stream is read more than once per call.


DriftKnownDomains names the known domain new data is closest to

DriftKnownDomains is new in v1.2 and experimental ⚠️: it raises ExperimentalWarning and may change or be removed without a deprecation cycle. Each known domain is a named reference dataset, or the samples of one reference dataset whose metadata factor falls in a group of values. A classifier learns to tell the domains apart, and a sample is novel to the extent that it is not clearly any one of them. It reports the closest known domain and whether more of the test data is novel than of held-out reference data, and it can be saved to an ONNX file and restored to monitor without the reference data. Unlike DriftDomainClassifier, which compares the test data with the reference as a whole, it needs at least three domains. Import it from dataeval.shift.


Videos without tracking targets or metadata

Duplicates, Outliers, SequenceFrames and SourceLocator take a dataset of videos that carries no tracking targets: each item is a video stream alone, or a (stream, target, metadata) tuple whose target is not a tracking target, such as None. Each video is measured as video with no detections, and findings are reported by video and frame, as on a tracking dataset. A dataset needs no metadata attribute, with tracking targets or without. ImageDataset and VideoDataset name the inputs the evaluators accept.

from dataeval.quality import Duplicates, Outliers

# `videos[i]` is a stream of frames that carry `pixels`, `time_s`, `pts` and `frame_index`
duplicates = Duplicates().evaluate(videos)
outliers = Outliers().evaluate(videos)  # the sequence and unit levels
  • Levels. There are no tracks, so Outliers raises for levels="track", and with per_target=True it finds no tracks or detections to judge.

  • Frame count. A video’s length is len(stream) where the stream has one, and otherwise one walk of the video, made once and kept.

  • Metadata. The evaluators build their own; Metadata(videos) on such a dataset is not supported.


Checklist

  1. Replace deprecated Metadata members using the table above.

  2. Add level= or source_index= to every add_factors call. Split any level="combined" array into separate factors per level.

  3. Update imports from dataeval.selection and dataeval.utils.data to dataeval.data. Replace Select with View.

  4. Replace ClassFilter(...) with Filter(classes=...), and filter_detections=False with drop="items". Remove ClassFilter([]).

  5. Remove per_channel= arguments, including per_channel=False. Replace per_channel=True with a channels={...} dictionary.

  6. Replace BitDepth, get_bitdepth(), and clip_and_pad() with their v1.2 equivalents.

  7. Rename SourceIndex.target attribute access and target= construction arguments to key. Update positional construction and unpacking of SourceIndex so the third value represents level.

  8. Implement is_binned on custom implementations of MetadataLike.

  9. Pass discrete_features= to mutual_info. Remove discrete_features= from calls to mutual_info_classwise.

  10. Replace ClassifierUncertaintyExtractor with UncertaintyExtractor or ClasswiseUncertaintyExtractor.

  11. Update code reading score to expect a single float per row. Check for missing scores using is_null() instead of array shape. Note that Relabel(..., reduce_detection_scores=False) affects target layout rather than column width, and alters the values extracted into metadata.

  12. Rescale any ConstantThreshold on a PIXEL statistic passed to Outliers to stored units. For images at mixed bit depths, compute statistics with normalize_pixel_values=True and call Outliers.from_stats.

  13. Pass per_target=True to Outliers.from_stats, including in the default from_stats(stats) call, if you re-detect with with_threshold, classwise or itemwise and expect target rows back. Pass per_image=True if you expect item rows back.

  14. Check result.populations for Outliers populations under 20 rows, which are no longer judged by a statistical threshold. Collect more rows for them, or judge them with a ConstantThreshold.

  15. Re-read any per-class outlier counts you recorded from aggregate_by_class on per_target=True results, or on images that hold more than one box of a class.

  16. Replace compute_ratios with the rel_* statistics for box geometry, and with a division of instance_ by unit_ columns on a Metadata for any other statistic.

  17. Replace ParityOutput.insufficient_data with cells.filter(pl.col("is_sparse")), and is_significant with is_associated or status. Re-read any Parity flags you recorded: they now use adjusted p-values and, on detection and tracking data, grouped tests.

  18. Give custom metadata containers item_indices before running Parity, or pass assume_independent=True when each row is its own item. Pass groups or assume_independent=True to core parity.

  19. Pass expected_groups and observed_groups to label_parity for detection or tracking labels, or assume_independent=True for one label per image. Remove num_classes.

  20. Expect a <name>_original column after an add_factors(..., overwrite=True) that replaces values. It is kept for reading but not analyzed; name it in include where the replaced values should be analyzed.

  21. Recompute any stored phash or dhash of integer images with values above 255, which now hash their scaled content rather than a saturated frame.

  22. Re-baseline any stored ber_knn lower bounds computed with an even k or more than two classes.

  23. Re-read any stored box statistics for None box entries on non-square or 2-D images, and any boxes_to_mask result built from one. They covered the image with its width and height swapped, or nothing at all on a 2-D image.

  24. Make boxes the same length as the images passed to image_stats or compute_stats, passing None for an unannotated image.

  25. Re-split any split made with split_on on a detection dataset, which v1.1 ignored and split by image, and expect partition sizes that follow the groups rather than the fractions. Re-split one made with split_on an integer factor past the level budget, which v1.1 grouped by its bins, and drop any factor_levels declared on sequence only to keep one group per video.

  26. Run DataEval on Python 3.11 or later.

  27. Make every extractor wrapped in Embeddings return one row per image. Pass metadata.dataset, not a Metadata, to Embeddings, and re-read any Coverage result computed from a Metadata without embeddings=.

  28. Pass Prioritize one class label per embedding. For a detection dataset, wrap it in DetectionCrops first.

  29. Correct any evaluator constructor argument that now fails validation: a wrong type, an unknown choice, or a value outside its range.

  30. Split a chunked drift detector’s reference into at least 3 chunks, or give it a ConstantThreshold.

  31. Scale masks stored as 0 and 1 to 0 to 255 before running Duplicates, and recompute any stored phash or dhash of integer images whose maximum is at most 1.

  32. Match unit_ and instance_ metric names on Outliers.evaluate results, and their aggregations, from detection datasets with per_target=True.

  33. Re-read any Balance factors results and is_correlated flags recorded on detection datasets.

  34. Read the datum id from the item_id column, and expect no id row in Balance or Diversity results.

  35. Check dropped_factors for mixed_types and invalid_known_factor, and repair or rename those factors. Select missing values with is_null() rather than the string "None", and re-read bias results for number keys that hold missing values.

  36. Compute (metric_value - population_mean) / population_std wherever you read or threshold an Outliers cluster_distance as a number of standard deviations.

  37. Expect split_dataset(..., stratify=True) to raise where the rarest class cannot reach train, validation and test, or where a detection split leaves a partition empty. Ask for fewer partitions or less extreme fractions, or pass stratify=False.

  38. Pass stratify=False to split_dataset for a filtered detection Metadata that keeps items without detections, or filter so that every kept item holds a detection.