Migrating from v1.1 to v1.2¶
v1.2 removes features deprecated in v1.1. It also updates SourceIndex into an address that names one row at
one level. For the capabilities v1.2 adds, see New in v1.2.
v1.1 reorganized metadata addressing, dataset transformations, and image statistics scaling. Each change retained a deprecated interface that emitted a warning pointing to its replacement. v1.2 removes those deprecated interfaces.
To identify code that requires updates, run your v1.1 pipeline with warnings promoted to errors:
python -W error::DeprecationWarning your_script.py
This flags SourceIndex.target, which is newly deprecated in v1.2 and emits a warning. It does not flag the removals
listed below, because those raise exceptions directly. Check for those removals by name. ClassFilter,
compute_ratios, compute_stats, and two uses of label_parity (passing num_classes, or giving neither groups nor
assume_independent=True) are also newly deprecated, but they warn with a FutureWarning, so add
-W error::FutureWarning to find them.
At a glance¶
Area |
You are affected if you |
Status |
|---|---|---|
run DataEval on Python 3.10 |
removed |
|
read |
removed |
|
import from |
removed |
|
use |
deprecated / additive |
|
pass |
removed |
|
use |
removed |
|
call |
numbers moved |
|
read |
renamed |
|
read |
additive |
|
implement the protocol with |
removed |
|
call |
required / removed |
|
use |
removed |
|
wrap an extractor that returns other than one row per image in |
now raises / numbers moved |
|
run bias, coverage, or splitting against a custom grouping rather than dataset labels |
additive |
|
read |
changed |
|
run |
additive |
|
remove duplicates or outliers from a dataset |
additive |
|
hold metadata, labels, or embeddings already, or pass |
additive / now raises |
|
pass a wrong type, an unknown choice, or an out-of-range value to an evaluator’s constructor |
now raises |
|
build a drift or OOD |
changed |
|
fit a |
now raises |
|
run |
numbers moved |
|
call |
numbers moved |
|
read the |
numbers moved |
|
compute |
additive |
|
call |
deprecated |
|
call |
renamed |
|
Track statistics declare their labels and add a gap fraction |
attach |
additive |
want one row of statistics per video, such as frame rate, redundancy or camera motion |
additive (experimental) |
|
pass |
now raises |
|
annotate or subclass the result of |
additive |
|
run |
numbers moved |
|
hash, or measure ego motion on, integer images with values above 255 |
numbers moved |
|
call |
numbers moved |
|
run |
numbers moved |
|
call |
numbers moved |
|
pass a |
additive |
|
read |
renamed |
|
read the |
numbers moved |
|
read |
removed / numbers moved (experimental) |
|
run |
now raises (experimental) / numbers moved |
|
call |
numbers moved / deprecated |
|
call |
changed / additive |
|
attach values to some of a level’s frames, tracks or detections with |
additive |
|
read an |
changed |
|
hold datum metadata whose key mixes numbers, text or booleans, holds |
changed / numbers moved / now raises |
|
read |
numbers moved |
|
want to know whether a classification accuracy, AP or per-class recall target is reachable on your data, with the evidence behind its determination |
additive (experimental) |
|
relabel a classification dataset and need a model’s class scores folded through the same map, such as for Feasibility’s |
additive (experimental) |
|
pass |
numbers moved |
|
call |
now raises / additive |
|
call |
additive |
|
need to tell which of several known reference domains (a sensor, a site, a season) new data belongs to, or whether it belongs to none |
additive (experimental) |
|
run |
additive |
If none of these items apply, upgrading requires no code changes.
Python 3.10 is no longer supported¶
DataEval v1.2 requires Python 3.11 or later, and supports 3.11 through 3.14. v1.1 also ran on 3.10. On Python 3.10,
pip resolves dataeval to the newest v1.1 release, so upgrade the interpreter before upgrading DataEval.
Metadata levels¶
v1.1 deprecated legacy image and target attributes with warnings. Those attributes are now removed. Use the following replacements:
Removed in v1.2 |
Use |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Two factor behaviors are also removed. The argument level="combined" was the v1.1 name for an array ordered by (item, target). It is no longer accepted. In addition, Metadata.add_factors() no longer infers a level from factor
length. Every add_factors call must pass level= or source_index=. Omitting both raises ValueError.
View and operations¶
The dataeval.selection module is removed. The dataeval.utils.data module no longer forwards dataset operations. Use
View and its operations in dataeval.data:
Removed in v1.2 |
Use |
|---|---|
|
|
|
|
|
|
|
subclass |
|
|
|
|
The dataeval.utils.data module remains available for utility functions. validate_dataset(), DatasetKind, and
requires_maite_dataset were never deprecated and are still imported from that module.
Filter replaces ClassFilter¶
Filter keeps a dataset’s items, and the detections and tracks inside them, by class and by metadata factor.
ClassFilter is deprecated and will be removed in v2.0. It warns with DeprecatedWarning, a FutureWarning, so
-W error::DeprecationWarning does not flag it; run with -W error::FutureWarning to find it. A filter that keeps
nothing now warns with a UserWarning, which ClassFilter never did.
v1.1 |
v1.2 |
|---|---|
|
|
|
|
|
remove the operation; |
Class labels must be integers, numpy integers included. ClassFilter used to accept integral floats such as 0.0, and
now raises TypeError for them, as Filter does.
New with Filter:
Factor predicates, as SQL or a polars expression:
Filter("time_of_day = 'night' AND altitude > 100"). A string predicate is what a configuration file can hold.Cleaning without dropping images:
Filter("occlusion < 0.5", drop="detections")keeps every image and removes the detections that do not match.Multi-object tracking: class and factor filtering on tracking datasets, removing detections or whole tracks;
ClassFilterraisedTypeErrorthere.
The per-channel row path is removed¶
In v1.1, compute_stats(per_channel=True) returned one row per channel, addressed by a third field on
SourceIndex. That field is removed. The per_channel argument, the channels=True alias, and the
channel_index column in Outliers are also removed.
To extract channel statistics, specify band names to produce columns:
compute_stats(images, per_channel=True) # v1.1, deprecated
image_stats(images, channels={"r": 0, "g": 1, "b": 2}) # v1.2
Band groups produce factor columns rather than separate rows. Channels do not correspond to a metadata level, so
per-channel rows could not be stored in Metadata. Consequently, evaluators such as balance, diversity, and
parity could not process them. Factor columns such as nir_mean are stored in metadata and evaluated directly.
per_channel=False is also removed. Because the parameter itself is removed, passing any value raises TypeError.
v1.1 warned only when per_channel was truthy. Remove per_channel wherever it appears, including calls that passed
the default False:
compute_stats(images, stats=ImageStats.PIXEL, per_channel=False) # v1.1
image_stats(images, stats=ImageStats.PIXEL) # v1.2
Outliers no longer emits channel_index. That column was populated only by per-channel rows. Band groups
are thresholded independently per column. Previously, per-channel rows used a single pooled threshold across every band
of every image.
compute_ratios() matches on item index alone. Previously, it paired each bounding box row with the image row
sharing (item, channel). Channel indices are removed, so rows are matched by item index. The validation check
requiring matching per_channel settings between inputs is also removed.
Preprocessing helpers¶
The deprecated names in utils.preprocessing are removed:
Removed in v1.2 |
Use |
|---|---|
|
|
|
|
|
|
clip_and_pad was deprecated in v1.1 and is now removed.
Full-image box for a missing box¶
to_bounding_box() called with None and an image shape returns a box covering the whole image. In v1.1 it
swapped the two sides, returning (0, 0, height, width) instead of (0, 0, width, height). On a square image the two
are identical. On a non-square image the box was clipped to the wrong region, so a None box entry in
image_stats() measured a window that did not match the image, and boxes_to_mask() left part of the image
unmasked. On a 2-D (height, width) image it returned an empty box and masked nothing. Both now cover the full image.
Statistics stored from None box entries on non-square images, and any background mask built from them, were measured
over the wrong region. Square images and images with a real box for every target are unaffected.
SourceIndex is an address¶
SourceIndex retains three fields. The third field is now level, which specifies the metadata level addressed
by the second field (key). A single tuple can now name a row at any level in a dataset’s level graph, rather than only
an item and its bounding boxes.
SourceIndex(item, target, channel) # v1.1
SourceIndex(item, key, level) # v1.2
target is renamed to key. Each level identifies rows by a specific column. key holds that column’s value, such
as target_index for an instance, unit_index for a video frame, or track_id for a track. Accessing target or
passing target= to the constructor still works but emits a DeprecationWarning. The target attribute and parameter
will be removed in v1.3.0.
si.target # v1.2: works, warns
si.key # v1.2
SourceIndex(item=3, target=7) # v1.2: works, warns
SourceIndex(item=3, key=7) # v1.2
Positional construction and unpacking still take three values, but the third value is now a level name. Unpacking
source_index[0] binds (item, key, level). Positional construction validates the level name. For example,
SourceIndex(0, 1, 2) raises ValueError (“level=2 is not a level”) because 2 is not a valid level name.
item, target, channel = source_index[0] # v1.1
item, key, level = source_index[0] # v1.2
SourceIndex(0, 1, 2) # v1.2: ValueError, "level=2 is not a level"
Available levels depend on the dataset. An address naming a level absent from your data is caught when the address is used, rather than when it is built.
The string representation retains up to three parts. str(si) produces "3", "3/7", or "3/12/unit".
SourceIndex.from_string accepts those formats, as well as "3/-/sequence" for a level stated without a key. The third
part must be one of sequence, unit, track, or instance. Any other value is rejected. In v1.1, that segment was
read as a channel number.
Leave the level unstated unless addressing unit or track. An unstated level resolves to the item level when key
is omitted, or to the label level when key is provided. SourceIndex(3) addresses an item. SourceIndex(3, 7)
addresses a label. Specify a level explicitly only when addressing intermediate levels such as unit or track.
Address equality and hashing depend on exact representation. SourceIndex(3, 7) and SourceIndex(3, 7, "instance") are
not equal and have different hash values. They act as distinct keys in dictionary results returned by Outliers
and Duplicates. image_stats() emits the minimal form without an explicit level.
Addressing arbitrary metadata levels¶
Metadata.add_factors() accepts addresses at any metadata level. Per-frame or per-track measurements can be
attached as individual values instead of entire keyed columns:
md.add_factors({"speed": [...]}, source_index=[SourceIndex(0, 5, "track"), ...])
This matches the column-oriented contract of add_factors(level=..., key=...). The address consists of (item_index, key) and the level name. See Naming a row: keys or addresses.
An address identifies a row, but does not specify hierarchical containment. For example, a source index does not
indicate which frame contains a detection. Consequently, Metadata.from_factors() given addresses builds only the
two-level structure. To preserve full hierarchy, construct Metadata from the dataset first, then add factors.
Unchanged functionality¶
The channels= mapping, value_range, normalize_pixel_values, per_image, per_target, and per_background
parameters remain unchanged. Computed statistic values are also unchanged. Calls that omitted per_channel in v1.1
return the same results in v1.2.
Evaluators report at the level they detected at¶
Outliers and Duplicates read the level directly from each address rather than inferring it from
missing fields. Both evaluators return the input addresses in their results.
Addressing changes nothing for image datasets. For image datasets, image_stats() emits addresses without
an explicit level, so no level column is added and the same rows are reported. The changes below apply only when a
statistic source index specifies an explicit level. Other v1.2 changes still apply to image datasets, among them the
context columns that every Outliers result now adds, direction, bound, percentile, population_mean and
population_std, and the Duplicates columns described in
Duplicates checks annotation and metadata factors.
Both evaluators add a level column. Outliers results add a level column, and Duplicates results add an
address_levels column. These columns distinguish rows that share key indices, such as video frames and tracks. When
all findings occur at either the item level or the label level, as in image datasets, these columns are omitted. In
Duplicates, the existing level column is retained with values "item" and "target". That column indicates the
detection role of a group, while address_levels indicates the metadata level of each member.
Findings normalize addresses to canonical form. Evaluators return canonical addresses rather than echoing input
formats verbatim. For example, SourceIndex(3, 7, "instance") normalizes to SourceIndex(3, 7). SourceIndex(3, None, "sequence") normalizes to item index 3. An item has one root row, and a key indicates a label row. Normalization
ensures that runs over equivalent addresses produce identical results.
Evaluators partition statistics by metadata level. Outliers fits a separate threshold distribution for each
metadata level. Per-frame statistics are evaluated only against other per-frame statistics, rather than across
per-sequence statistics. Duplicates partitions candidates by level before hashing. This prevents comparisons across
different metadata levels.
per_image and per_target filter item and label levels regardless of address format. Equivalent addresses are
filtered identically. SourceIndex(3, 7) and SourceIndex(3, 7, "instance") are filtered identically, as are
SourceIndex(3) and SourceIndex(3, None, "sequence"). Intermediate levels, such as video frames or tracks, are
unaffected by these flags and are always included. Result accessors for intermediate levels use SourceIndex
keys, because integer item indices cannot identify those rows.
Evaluator output addresses can be passed directly to Metadata.add_factors():
outliers = Outliers().from_stats(frame_stats)
md.add_factors(
{"is_outlier": [address in outliers.outliers for address in addresses]},
source_index=addresses,
)
MetadataLike¶
The fallback for is_discrete in MetadataLike is removed. In v1.1, defining is_discrete emitted a warning.
In v1.2, the fourth protocol member is is_binned. Containers that define only is_discrete no longer satisfy the
protocol.
MetadataLike remains available as an alias for CodedMetadataLike. Metadata is unchanged.
This change affects only custom classes that implement the protocol directly.
discrete_features¶
The discrete_features parameter was updated across mutual information functions.
In
mutual_info(),discrete_featuresis now required. Omitting it raisesTypeError. In v1.1, omitting it triggered auto-detection and emitted a warning. Passdiscrete_features=[not b for b in md.is_binned].In
mutual_info_classwise(), thediscrete_featuresparameter is removed. Passing it raisesTypeError. The parameter previously had no effect and was discarded downstream.
Balance is unaffected. It always passed the value explicitly.
Extractors¶
ClassifierUncertaintyExtractor is removed. Wrap a TorchExtractor or any FeatureExtractor in
UncertaintyExtractor for per-instance uncertainty. For per-class uncertainty, wrap the extractor in
ClasswiseUncertaintyExtractor.
Embeddings come from images, one row per item¶
Embeddings stores one row for each dataset item, and now checks that it gets one.
An extractor must return one row per image. In v1.1, when an extractor returned more rows than its batch held
images, as a detection model that returns a row per detection does, Embeddings dropped the extra rows without a
warning, so the rows it kept could belong to other images. It now raises ShapeMismatchError, a ValueError, when a
batch returns a different number of rows or a single value, and TypeError when the extractor returns a mapping such
as a dict. Evaluators that build an Embeddings from your extractor inherit the check, though Prioritize.evaluate
re-raises it as a TypeError, with the ShapeMismatchError as its cause. Call a per-detection extractor directly,
and give its result, or the extractor itself, to a drift detector.
A Metadata is not a dataset of images. Indexing a Metadata returns factor rows, so in v1.1
Embeddings(metadata, extractor=...) embedded those rows, and Coverage given a Metadata and an extractor
but no embeddings= measured the coverage of the factors rather than of the images. Embeddings now raises
MaiteShapeError, a TypeError and a ValueError, for a Metadata. Coverage.evaluate embeds the dataset the
metadata describes, Metadata.dataset, so its results change, and raises ValueError for a Metadata read
back with load, which has no dataset to embed.
Embeddings(metadata, extractor=extractor, batch_size=32) # v1.1: embedded factor rows; v1.2: MaiteShapeError
Embeddings(metadata.dataset, extractor=extractor, batch_size=32) # v1.2
Metadata.dataset is None for a Metadata read back with load; pass the dataset it was built from instead.
The score column is one confidence per row¶
In MAITE, a target can provide scores per bounding box ((N,)) or per class for each box ((N, CLASSES)). A
classification target is a score vector. In v1.1, metadata stored target scores directly in the dataframe. The score
column was Array(Float32, n_classes) for multi-class targets and Float32 for single-score targets. Because column
width depended on class count, datasets with different vocabularies could not be concatenated into a single dataframe.
In those cases, merge_datasets() raised an error from np.concatenate.
The score column is now always Float32. It stores the confidence of a row for its assigned class, extracted from
whichever layout the target provides. Rows without scores contain null. To access per-class probabilities, read them
from the target rather than md.dataframe["score"].to_numpy(). SourceLocator retrieves individual detections.
ScoresExtractor extracts per-class prediction arrays from model outputs.
Relabel aligns score arrays when transforming labels. Per-class arrays are indexed by their original
vocabulary. In v1.1, relabeled detections retained source-indexed score columns alongside relabeled targets.
Consequently, scores mapped to incorrect classes or exceeded array bounds. Detection scores are now reduced to a single
confidence value for the source label before vocabulary transformation. Classification vectors are mapped into the
target vocabulary, summing probabilities when classes are merged. A Mapping[int, str] target vocabulary must use
non-negative integer indices. Non-contiguous class indices are supported and no longer raise IndexError.
If code accesses dataset[i][1].scores from a relabeled detection dataset and requires per-class arrays during
migration, pass Relabel(..., reduce_detection_scores=False). This maps scores into the target vocabulary. In v1.1, the
array retained the source vocabulary indexing and width. Verify any code that indexes the resulting array. Classes
missing from the source array receive a score of 0.0 rather than being treated as absent. Merged classes combine their
scores by summation.
Setting reduce_detection_scores=False does not change the metadata score column width. The column retains one value
per row under both settings. However, it changes the numeric values in that column because metadata reads from the
transformed target. When classes are merged, the column reflects the summed class score. Where scores cannot be read,
the column contains 0.0 instead of null. The reduce_detection_scores parameter emits a warning and will be removed
in v1.3. Omitting the parameter uses the default reduction behavior.
A class axis of your own¶
Metadata.classed_by() configures one or more factors as the class axis. Unmodified metadata instances continue to
report default class labels.
by_weather = metadata.classed_by("weather")
Balance().evaluate(by_weather) # conditions on weather
Coverage().evaluate(by_weather, embeddings=emb) # per-weather coverage
split_dataset(by_weather, stratify=True) # folds balanced over weather
This configuration introduces three behaviors:
Original dataset labels become a factor. The pivoted metadata gains a factor named class containing the original
labels. Results retain the relationship between the custom axis and the original dataset labels.
All class-label consumers use the pivoted axis. Metadata.class_labels and Metadata.index2label
update together. Evaluators without an explicit label= parameter evaluate against the pivoted axis.
Representation is an exception because it validates label names against an ontology. It raises an error when
passed pivoted metadata.
Results record the active class axis. Class-conditional outputs add a class_axis field holding a
ClassAxis. The meta().state mapping adds class_axis, class_axis_source, and class_axis_level entries:
result = Balance().evaluate(by_weather)
result.class_axis.source # "derived" (custom axis rather than dataset labels)
result.class_axis.rows_per_group_entity # 3.0 if a per-image axis was read from detections
For standard evaluations, class_axis_source records "ground_truth". In previous releases, this field recorded the
string "NoneType". When setting label= on bias evaluators, class_axis_source is recorded identically.
The class axis configuration persists across at, where, having, and agg operations. Like view, include, and
exclude, it is omitted during serialization with Metadata.save().
See Analyze bias against a custom class axis for a worked example covering composite axes and metadata level selection.
Duplicates checks annotation and metadata factors¶
Duplicates now evaluates annotations and optional metadata factors alongside pixel checks. Datasets without
targets or duplicate_factors produce the same pixel-only results as v1.1.
Datasets with targets include annotation duplicate rows. When a dataset yields (image, target, metadata) and
targets contain boxes or labels, Duplicates.evaluate inspects annotations by default. Two additional result
patterns can appear:
Groups whose
methodslist contains"annotation". These represent items sharing identical annotations regardless of pixel similarity.Existing pixel groups (
dup_type="exact"or"near") wherediffers_oncontains"annotation". These represent matching images with differing annotations.
These additions do not replace previous findings. They provide new rows or additional fields on existing rows.
Results add two columns: differs_on and annotation_divergence. differs_on (list[str] | None) uses a
three-state representation. A null value indicates the axis was not evaluated. An empty list [] indicates the axis
was evaluated and values agreed. A populated list names the attributes that disagreed. annotation_divergence (float | None) records divergence scores. See Reading differs_on and the divergence
columns for the complete three-state contract and pair
row mean_iou behavior.
Metadata factor grouping requires explicit configuration. Passing metadata= and duplicate_factors=[...] to
evaluate() groups items whose specified factor values match exactly. These groups are reported with
dup_type="factors". Omitting duplicate_factors disables factor grouping. No factors are evaluated by default.
Every named factor must agree. duplicate_factors functions as a conjunction. Items are grouped only when all
named factors match; matching some and differing on one produces no group. Adding factors narrows matches. Items
with unstated factors (null or NaN) are excluded from grouping. factor_cardinality counts stated values only.
evaluate() accepts a Metadata instance directly. When evaluating metadata alone, no pixels or annotations
are decoded, so duplicate_factors is required. The run reports at level="item" only.
# Unchanged from v1.1: image datasets without targets produce identical results
Duplicates().evaluate(images)
# New in v1.2: object detection datasets include annotation duplicate rows
Duplicates().evaluate(od_dataset)
# New in v1.2: optional grouping by specified metadata factors
Duplicates().evaluate(od_dataset, metadata=Metadata(od_dataset), duplicate_factors=["capture_id"])
# New in v1.2: all specified factors must agree
Duplicates().evaluate(od_dataset, metadata=Metadata(od_dataset), duplicate_factors=["capture_id", "camera_id"])
# New in v1.2: evaluate metadata directly without decoding images
Duplicates().evaluate(Metadata(od_dataset), duplicate_factors=["capture_id"])
Existing group types are preserved. The evaluator continues to report exact, near, and video group types
(segment, aligned, and redundant) as in previous releases.
Removing what an evaluator flagged¶
DuplicatesOutput.deduplicate() and OutliersOutput.prune() return a RemovalPlan: the
SourceIndex addresses of the rows to remove. Pass it to Indices with exclude=True.
plan = Duplicates().evaluate(dataset).deduplicate() | Outliers().evaluate(dataset).prune()
cleaned = View(dataset, Indices(plan, exclude=True))
Indices accepts addresses as well as ints. With exclude=True, it removes a detection named by its target_index
and, on a tracking dataset, a whole track named by its track_id. Every image and frame stays, even when all of its
detections are removed.
Build the view on the dataset the evaluators read. A plan names positions in that dataset, and they are renumbered once anything is removed.
Evaluators take precomputed inputs¶
Each evaluator family now has a method per kind of input you may already hold, beside evaluate, which still accepts
everything it did. evaluate converts its argument and calls the method, so both give the same result.
Evaluator |
Method |
Takes |
|---|---|---|
|
|
a |
|
|
any container with |
|
|
an embeddings array, and optionally one label per embedding |
|
|
reference and test embeddings: |
|
|
the validation set |
|
|
the arrays its detector’s |
|
|
reference and test embeddings, and |
Build the metadata or embeddings once and share them:
metadata = Metadata(dataset)
balance = Balance().from_metadata(metadata)
parity = Parity().from_metadata(metadata)
The drift and OOD detectors keep fit and predict for fitting once and testing several datasets. The reconstruction
detectors have no from_embeddings: their model reads whatever array it was trained on, usually images.
Coverage.from_embeddings without labels runs every item as one class, 0. Coverage and Prioritize refuse
labels that do not hold one entry per embedding, such as a detection dataset’s labels, which hold one per target. For
Prioritize.evaluate this is new: v1.1 accepted mismatched class_labels without complaint.
Constructor arguments are validated¶
In v1.1 an evaluator validated its Config only when you built the Config yourself. Arguments passed to the
constructor were stored unchecked, so Diversity(method="nope") or Balance(num_neighbors="abc") succeeded and
failed later, or not at all. v1.2 validates constructor arguments the same way, and raises a pydantic
ValidationError, which is a ValueError, naming the field.
Numeric settings are also checked against the range they are meaningful in: counts above zero, p-value thresholds
in (0, 1), shares and scores in [0, 1], and percentiles in [0, 100]. The ranges appear in each Config’s JSON Schema.
Diversity(method="nope") # v1.1: accepted; v1.2: ValidationError naming `method`
Balance(config=Balance.Config(label="weather"), num_neighbors=7) # unchanged
Drift and OOD configs are validated¶
The Config of every drift and OOD detector is now a pydantic model, as every other evaluator’s is. A wrong type or
an unknown choice fails when the Config or the detector is built, naming the field, rather than later in fit or
predict. The error is a pydantic ValidationError, which is a ValueError.
Build a Config with keywords, and use the pydantic methods in place of the dataclasses helpers:
v1.1 |
v1.2 |
|---|---|
|
|
|
|
|
|
|
|
ChunkedDrift needs three reference chunks¶
Every threshold except ConstantThreshold derives a chunked detector’s bounds from
the spread of the reference chunks’ baseline scores. Two chunks scored against each other measure one pair twice: a
symmetric statistic such as MMD² or KS gives both the same value, so v1.1 either passed every chunk or flagged every
chunk as drifted. ChunkedDrift now raises ValueError on fit when its chunking splits the reference into
fewer than 3 chunks and its threshold is not a ConstantThreshold.
DriftUnivariate().chunked(chunk_count=2).fit(reference) # v1.1: degenerate bounds; v1.2: ValueError
DriftUnivariate().chunked(chunk_count=3).fit(reference) # v1.2
Use a chunk_count of 3 or more, or a chunk_size well below the reference size: on 100 reference samples,
chunk_size=50 raises and chunk_size=40 does not. To keep two chunks, pass a ConstantThreshold.
DriftWasserstein decides on the mean ratio¶
In v1.1 DriftWasserstein declared drift when any single feature’s distance ratio exceeded ratio_threshold.
Each ratio divides two noisy distances, so the rule raised false alarms on clean data: with 500 training, 100
validation and 100 test samples from one distribution, 99% of batches with 16 features and every batch with 64. The
ratio also depended on the sample sizes. With 2,000 training, 500 validation and 50 test samples, the median mean ratio
of clean data was 3.2, and with 500 training, 100 validation and 400 test samples it was 0.7, which hides drift.
v1.2 decides on the mean ratio over the features, and divides each ratio by
sqrt((1/n_ref + 1/n_op) / (1/n_fit + 1/n_val)), which corrects for the sizes of the reference at predict time, the
operational data, and the training and validation data at fit. The default ratio_threshold stays 1.4.
driftedisdistance > threshold, wheredistance(anddetails["ratio"]) is the mean ratio. It no longer equalsdetails["feature_drift"].any(). The per-feature flags remain, to show where to look.feature_ratios,distanceand the ratios thatscore()returns are corrected for the sample sizes, so they equaldistances / baseline_distancesonly when the sizes match. A feature with a baseline of 0 keeps a ratio of 1.0, orinfwhen its operational distance is not 0.In chunked mode each chunk’s ratio is corrected for the chunk size. Only the default
ConstantThresholdis calibrated there: the baseline chunks come from the training reference, and a threshold derived from their spread, such as"zscore", alarms on 6% to 100% of clean chunks in a simulation, more as the chunks grow relative to the reference.
result = DriftWasserstein().fit(train, validation).predict(test)
result.drifted # v1.1: any(feature_ratios > 1.4), uncorrected; v1.2: mean(feature_ratios) > 1.4, corrected
At the default threshold, about 0% to 3% of clean batches now raise a false alarm from 16 independent continuous
features up, 5% to 19% when the features are correlated, and roughly 15% to 35% with 1 to 4 features, where the KS test
of DriftUnivariate is the calibrated choice. The mean does not see shifts confined to a few of many features:
a shift in 1 of 512 features was not detected, and a 2 standard deviation shift in 2 of 64 features was detected in 66%
to 92% of trials. Drop constant and discrete features before you use the detector, and lower ratio_threshold only
after you measure the false alarm rate on held-out batches.
Outliers reads pixel statistics in stored units¶
Outliers.evaluate now measures PIXEL statistics in the units the data is stored in, as image_stats() does
by default. In v1.1 it normalized pixel values to [0, 1] first, so evaluate and from_stats on statistics from
compute_stats could flag different images from the same data.
Every threshold except ConstantThreshold scales with the data, so for images stored at one bit depth no
outlier flag moves. Two cases change:
A
ConstantThresholdon aPIXELstatistic is now read in stored units. Rescale its bounds, for example from0.8to204for 8-bit data.Images stored at different bit depths are no longer put on one scale. To compare them the v1.1 way, compute the statistics normalized and pass them to
from_stats:
stats = image_stats(images, stats=ImageStats.PIXEL, normalize_pixel_values=True)
Outliers().from_stats(stats)
VISUAL statistics are unaffected: they always read the display range.
Outliers reports the raw cluster distance¶
When Outliers is given an extractor, it flags items far from their cluster and reports them with
metric_name cluster_distance. In v1.1, metric_value on those rows was the number of standard deviations the item
lay from its cluster’s mean distance. It is now the distance itself, and the row carries its cluster’s
population_mean and population_std, so the v1.1 value is one expression away. Which items are flagged is
unchanged.
cluster = result.data().filter(pl.col("metric_name") == "cluster_distance")
cluster.select((pl.col("metric_value") - pl.col("population_mean")) / pl.col("population_std")) # the v1.1 value
Relative box geometry in the dimension statistics¶
ImageStats.DIMENSION now includes seven rel_* statistics: box offset, width, height and area as fractions of the
image, and distance to the image center and nearest edge as fractions of the half-diagonal and of that edge’s
dimension. They carry exactly the definitions compute_ratios() applies to the absolute statistics, and are
grouped as ImageStats.DIMENSION_RELATIVE. Existing statistics are unchanged.
Because Outliers computes DIMENSION by default, a run with per_target=True can report outliers on the rel_*
statistics too. An item’s own row is the whole frame, so its relative values are constant and never flagged.
compute_ratios is deprecated¶
compute_ratios() is deprecated and will be removed in v1.3. It warns with DeprecatedWarning, a
FutureWarning. Both of its uses have a replacement that gives the same values:
Box geometry relative to its image is a statistic now. Compute
ImageStats.DIMENSION_RELATIVE, or the singlerel_*flag you need, withper_target=True.For any other statistic, add the per-image and per-target statistics to a
Metadata. Each box row at theinstancelevel carries its image’s value, so the ratio is one division.
v1.1 |
v1.2 |
|---|---|
|
|
|
|
The rel_* statistics use the definitions compute_ratios applied to offset_x/offset_y, width, height,
size, distance_center and distance_edge. A custom override_map becomes a polars expression over the same
instance_ and unit_ columns.
compute_stats is now image_stats¶
compute_stats is renamed to image_stats(), which says what it measures now that video has its own statistics.
The parameters and the result are unchanged. The old name still works until v2.0 and emits a DeprecatedWarning.
stats = compute_stats(images, stats=ImageStats.PIXEL) # v1.1
stats = image_stats(images, stats=ImageStats.PIXEL) # v1.2
Leaving normalize_pixel_values unset no longer emits a FutureWarning. Its default stays False, so results do not
change.
Track statistics declare their labels and add a gap fraction¶
track_stats() adds gap_fraction, the share of the frames a track spans in which it is absent. Its result also
declares labels categorical: Metadata.add_factors records the track labels as a vocabulary, as
factor_levels= would, so no consumer treats a class id as a quantity. A vocabulary you declared yourself is kept.
video_stats describes each video¶
video_stats() computes statistics that describe a video as a whole, one row per video, in one walk of each
video’s frames. VideoStats selects them, in the groups DESCRIPTIVE (frame count, duration, frame rate),
REDUNDANCY (the share of frames that repeat the one before, and the longest such run), EGO (camera motion
summaries) and PERSISTENCE (per-video maps of what is fixed to the camera and of local blur). EGO and PERSISTENCE
need OpenCV from the dataeval[opencv] extra. The default is VideoStats.NO_EGO, everything that does not need
OpenCV.
The PERSISTENCE flags (PERSISTENCE_EGO, PERSISTENCE_BLUR, PERSISTENCE_BLUR_RAW) and their maps ego_map,
blur_map and blur_map_raw are experimental ⚠️: selecting any of them, directly, through PERSISTENCE or through
ALL, raises ExperimentalWarning once per process. The map shape, the column names and
the blur_measure, blur_sigma and persistence_samples keywords may change in a later release, and so may
SharpnessMeasure, the protocol a blur_measure follows. The default
VideoStats.NO_EGO selection does not warn.
from dataeval.core import video_stats
metadata.add_factors(video_stats(dataset)) # one row per video, at the sequence level
To check measured values against ones the dataset already reports, such as video_duration, pass
return_collisions=True to Metadata.add_factors(). It returns the videos where the two disagree, within an
optional collision_tolerance.
redundancy_radius is the Hamming distance within which a frame counts as redundant, and shares its default with
Duplicates. ego_gap and ego_proc_short_side pass through to ego_stats(), whose per-frame
measurements the EGO summaries reduce through the same roll-up Metadata.add_factors uses.
ego_stats() now measures videos in parallel under max_processes, and accepts a list of streams. Set
max_processes=1 for a dataset whose videos share one open decoder. To look inside one video instead, measure its
frames with image_stats() over SequenceFrames. See
Volatility and drift of a per-frame statistic.
SequenceFrames can walk a video more than once, so each dataset[i] must return a stream that can be walked from its
first frame, independently of any earlier stream for the same index.
image_stats refuses images and boxes of different lengths¶
When boxes= held more entries than there were images, image_stats silently dropped the extra boxes. When it held
fewer, the remaining images were measured as if unannotated. Either way the annotation set changed without a warning,
and a box list that had slipped out of step with its images went unnoticed. image_stats(), and the deprecated
compute_stats alias, now raise ValueError naming the index where one of them ran out:
image_stats(images, boxes=boxes) # v1.1: 4 images, 3 box lists, the fourth image measured with no boxes
image_stats(images, boxes=boxes) # v1.2: ValueError: images and boxes differ in length: boxes ran out at index 3
Give every image an entry in boxes. An entry of None still marks an unannotated image. One-pass iterables such as
generators raise as soon as the mismatch is found, so no partial result is returned.
A literal None in place of an image under boxes= was skipped silently in v1.1. It now fails inside NumPy
(ufunc 'isnan' not supported), as it already did without boxes=.
AddressedResult¶
AddressedResult is the new base for a statistics result whose values carry an address, a source_index that
names the row of each value. StatsResult now extends it with its image bookkeeping (object_count,
invalid_box_count and image_count), so image_stats and compute_ratios return what they did before.
video_stats() returns an AddressedResult directly, since a video has no image bookkeeping. Code that only
reads stats and source_index needs no change, and combine_stats_results now accepts any addressed result.
Duplicates hashes the native pixel values¶
Perceptual hashes (phash, dhash and their _d4 variants) read an image in the dtype it is stored in. Previously
image_stats() converted every image to float64 first, and a float image with a maximum of at most 1 is read as
holding values in [0, 1] and stretched to full contrast. Integer images are now hashed at their stored values instead.
xxhash still digests the float64 view, so its values are unchanged.
Duplicates groups change only where integer images have a maximum of at most 1. Floating-point images, and
integer images with any larger value, hash exactly as before. The case to watch is a binary mask stored as 0 and 1.
A 3-channel (RGB) mask, or one with 5 or more channels, now reads as black, because its Rec. 601 luma truncates to 0. It hashes flat, so distinct masks group as near duplicates.
A single-channel, 2-channel or 4-channel mask still hashes by its content, though to different values than before.
Scale masks to 0–255 before running Duplicates. That restores the earlier hashes. If you stored hashes of
such images, recompute them.
Grayscale scales integer images deeper than 8 bits¶
to_canonical_grayscale() clipped every value above 255, so 12- and 16-bit integer images, such as 14-bit infrared
in uint16, mostly saturated to a flat white frame. Their perceptual hashes then matched each other, and ego motion
read them as a still camera. Such an image is now scaled from the encoding get_value_range() decodes for it:
a maximum up to 4095 scales from 12 bits;
a maximum up to 65535 scales from 16 bits.
Only integer images with a value above 255 are affected. Their phash and dhash values change, and so do
Duplicates groups, ego_stats, video_stats and BoVW features that read them. 8-bit images, signed images
and float images convert exactly as before. If you stored hashes of deeper integer images, recompute them.
Re-detecting from statistics honors per_image and per_target¶
Outliers.from_stats defaults to per_target=False, while image_stats defaults to per_target=True, so the
default call Outliers().from_stats(image_stats(dataset)) reports item rows only. Its with_threshold, classwise
and itemwise re-detected from every row of the statistics, so they returned target rows that the first result did not
report. Re-detection now covers the rows the flags selected, so a result and its re-detections describe the same rows.
A result built with per_image=False likewise no longer gains item rows when you re-detect. Rows addressed at a level
between an item and its labels, such as a frame or a track, are still always included.
To keep target rows in the result and in its re-detections, pass per_target=True to from_stats.
Outliers needs 20 rows per population¶
Outliers fits a statistical threshold to each population: the rows of one metric at one level, or within one class
under classwise(). It now fits one only to a population with at least 20 rows that hold a value. A smaller level, or a
smaller classwise() class, yields no findings. Before, AdaptiveThreshold fitted from 3 rows. A
ConstantThreshold needs no data, so it still judges every population.
Below 20 rows, the default threshold flags clean data often enough that most findings are noise. On clean, normally distributed values it flags this share of rows:
Rows in the population |
Rows flagged |
|---|---|
5 |
3.4 % |
10 |
0.17 % |
20 |
0.05 % |
To see what was skipped, read OutliersOutput.populations. It lists every population with its size, whether it
was judged, and why not:
result = Outliers().evaluate(metadata)
result.populations.filter(~pl.col("judged"))
If a population you need is skipped, collect more rows for it, or judge it with a ConstantThreshold.
classwise(...).with_threshold(...) now keeps the per-class populations. Before, with_threshold re-detected against
whole levels, so the classes were lost.
Results of classwise() and per_class=True now carry a population column that names the class each row was judged
in, such as class_label=cat.
aggregate_by_class counts a box under its own class¶
On a result computed with per_target=True, aggregate_by_class joined each flagged box to the labels of its whole
image, so a box was counted once under every class in the image. It now joins a box to its own label. On any result, a
flagged image row now counts once under each distinct class among its labels. Before, it counted once per label, so an
image with boxes [car, car, bus] counted twice under car; it now counts once.
Expect lower per-class totals for images that hold boxes of more than one class, and for images that hold more than one box of a class.
Outliers thresholds metadata factors¶
Outliers.evaluate accepts a Metadata as data and thresholds its ordered factors – numeric or temporal by
their raw type – each against the rows at the level it was measured at, so a track is compared with tracks and a frame
with frames. factors=[...] names which. Only raw values are read, so re-binning a metadata never changes the result.
Categorical factors, including integers with a declared vocabulary, are not thresholded.
levels= names the levels to judge, with the same names and rules as Duplicates.evaluate(): "item" and
"target" for image and detection data, and "sequence", "unit", "track" and "instance" for tracking data.
Passing levels together with per_image or per_target raises. Without either, a tracking metadata judges its
sequence and frame levels, as Duplicates does, so judge track factors with levels="track". per_target=True adds
every detection and track.
OutliersOutput gains .items, .targets, .sequences, .frames, .tracks and .detections, which narrow
a result to one level and keep re-detecting at that level. On a result that holds frames or tracks,
aggregate_by_metric adds a level column and counts each flagged frame, track, detection or item once, and
aggregate_by_class raises, since a frame or a track has no class. Narrow to .targets first.
Outliers names image and box metrics by level¶
Outliers.evaluate on a detection dataset with per_target=True now judges the dataset through the
Metadata built for it, and names each metric by the level it was measured at, as
Metadata.add_factors() names the same statistics: unit_brightness on image rows and instance_brightness on
box rows. In v1.1 both were brightness. The names carry through to the result’s aggregations: the rows of
aggregate_by_metric and the metric columns of aggregate_by_item and aggregate_by_class.
result = Outliers().evaluate(od_dataset, per_target=True)
result.data().filter(pl.col("metric_name") == "brightness") # v1.1
result.data().filter(pl.col("metric_name") == "unit_brightness") # v1.2, image rows
Results that hold one level keep the plain names: per_target=False, per_image=False, a dataset without targets,
and Outliers.from_stats.
Balance scores factor pairs over images, not detections¶
On a detection dataset, Balance reads its factors on detection rows, so a per-image factor repeats once per
detection. The factors table took the chance correction for each pair of factors against those rows, which let two
unrelated per-image factors read as correlated in proportion to the detections per image. The correction is now taken
against the entities each pair varies over: the images for per-image factors, and the videos for a per-sequence factor
read on frames.
Pairs of factors defined above the level being read score lower, and independent ones score near zero. Pairs at the
level being read are unchanged, and so are the balance and classwise tables. Core mutual_info() and
mutual_info_classwise() take an optional effective_n, one entity count per column, and compute what they did
before without it.
Parity reports cells and statuses¶
Parity ⚠️ and parity() ⚠️ are experimental, so these changes move inside the experimental contract.
Parity now tests each factor over independent images or sequences and reports every (factor, level, class) cell. See
Parity for the method.
ParityOutput.insufficient_data is removed. It listed, per factor and level, the cells with small counts. A
combination that never occurs has no row in any table, so it never appeared there, and the mapping could not show what
the dataset lacks. The new ParityOutput.cells frame has one row per factor, observed level and observed class,
observed zeros included, and marks the cells too thin to rely on:
result.insufficient_data["illum_lux"] # v1.1
result.cells.filter(pl.col("is_sparse") & (pl.col("factor_name") == "illum_lux")) # v1.2
Each cell carries observed, expected, deficit (expected minus observed), lift (observed over expected), an exact
two-sided p_value and its p_value_adjusted, is_deviant, is_sparse, a status (underrepresented,
overrepresented, too_thin, no_finding or untestable) and is_actionable.
The factors frame changes:
v1.1 |
v1.2 |
|---|---|
|
|
|
|
|
|
a constant factor scores 0 with p = 1 |
NaN results and |
|
|
|
Every flag now reads the adjusted p-value, so a factor significant at a raw p of 0.04 among several factors can stop
being associated. Detections are grouped by image and frames by sequence, so p-values on object detection and
tracking datasets are larger than in v1.1, whose row-level p-values on such data were invalid.
New configuration: lift_threshold=1.25, at least 1, sets how far a significant cell’s lift must be from 1 for the
cell to be actionable, and 1 disables the band. assume_independent=False is described in the next section.
New output fields: test ("pearson" or "rao_scott_f"), group_source, group_level, n_rows and n_groups.
meta().state adds lift_threshold, assume_independent, group_source and test.
Core parity. ParityResult removes insufficient_data and adds chi_squared, df_num, df_den,
p_values_adjusted, design_effects, effective_groups, insufficient, testable, cells (a ParityCells of flat
per-cell arrays), test, n_rows and n_groups. Its third argument is groups, one independent group id per row.
Pre-release builds of 1.2 had an effective_n argument in that position; groups replaces it.
Parity needs an independent sampling unit¶
Parity must know which rows are independent of each other. It reads that from the input’s structure: a
Metadata groups by its item_indices at any view, so detections group by image and frames by sequence
(group_source="level"), a container with many-to-one item_indices groups by them ("item"), and one row per item
needs no grouping ("rows"). A Metadata built from a dataset always records its structure, so these calls
are unchanged except for their numbers.
Over a derived view, the unit is the root source item rather than the derived item. Parity over a
DetectionCrops view (shipped in v1.1.4) or a frame view now clusters rows by the source image or video
(group_source="source"), where it would otherwise have read every crop or frame as an independent sample. A
merge_datasets() of crop or frame views keeps each view’s grouping, and a View that repeats items
groups the copies too. So does a merge that includes one dataset more than once, such as
merge_datasets(ds, View(ds, augment)), which v1.1 read as independent items. Counts and p-values over crops and
such merges change accordingly. Metadata saved from a crop view by an earlier version records no root items and
is refused on load: rebuild it from its dataset and save it again. So is metadata saved from a frame view before this
change, though frame views are new in v1.2. A plain dataset whose own datum metadata has a source_id key is refused
too when an earlier version saved it, and is rebuilt the same way.
Balance counts the same unit when it corrects for chance. Over a crop or frame view, a factor that never
changes within a source image or video, or a label that doesn’t, now counts the source’s images or videos as its
effective sample size rather than its crops or frames, so Balance’s scores over those views change. The same holds
for the copies a repeating View or merge holds of one dataset item.
When nothing records the unit, Parity now raises ValueError instead of treating every row as independent. This
reaches only custom MetadataLike containers that carry no item_indices. Parity does not apply the protocol’s
default of one label per item, because a container that silently lacks item_indices is the case the refusal guards
against. Give the container item_indices (label → item), or declare the rows independent:
Parity().evaluate(custom_metadata) # v1.1: rows read as independent; v1.2: ValueError
Parity(assume_independent=True).evaluate(custom_metadata) # v1.2, when each row is its own item
Core parity() takes the grouping directly, and needs groups or assume_independent=True:
parity(factor_data, class_labels) # v1.1: row-level test; v1.2: ValueError
parity(factor_data, class_labels, assume_independent=True) # v1.2, the v1.1 row-level test
parity(factor_data, class_labels, item_ids) # v1.2, rows grouped by item
Passing both groups and assume_independent=True raises as well. group_source="declared" records that the caller
made the declaration, and the p-values hold only if it is true.
label_parity tests homogeneity¶
label_parity() is stable, and this is a correctness fix. In v1.1 it ran a goodness-of-fit test that treated the
expected labels as the exact population. With both samples drawn from one distribution, that test rejected at
α = 0.05 in 27% of draws at 1,000 against 1,000 labels, 87% at 200 against 2,000, and 30% at 100 against 100. A
homogeneity test on the same draws rejected 4.4–6.5%. It was also asymmetric: swapping the two arguments moved one
p-value from 1.7 × 10⁻⁷ to 0.39. And it broke at the edges: num_classes above the classes present gave NaN, a class
absent from the expected labels gave an infinite statistic and p = 0, and samples with different largest labels raised
unless num_classes was passed.
v1.2 runs Pearson’s chi-squared test of homogeneity on the 2 × K table of class counts, identical to
scipy.stats.chi2_contingency(table, correction=False). It is symmetric in its two arguments and returns no NaN or
infinity for those inputs. Every statistic and p-value changes. The result adds score (bias-corrected Cramér’s V
between dataset and class), insufficient, df_num, df_den and test.
Detections need groups. Read as independent labels, two detection datasets with the same class distribution but
different image counts and detections per image were declared different 84–91% of the time at α = 0.05 in simulation;
grouped by image, 4.4–4.7%. New keyword arguments expected_groups and observed_groups take one group id per label,
the item index for detections or the sequence id for tracks. Pass both or neither.
Two deprecations, both warning with DeprecatedWarning, a FutureWarning:
Calling without groups and without
assume_independent=Truestill runs as if every label were independent, but warns. From 2.0 it raises.assume_independent=Falsewithout groups raises now.num_classesis ignored, since the classes are the ones that occur in either sample. Passing it warns, and it is removed in 2.0.
label_parity(expected, observed, num_classes=10) # v1.1
label_parity(expected, observed, assume_independent=True) # v1.2, one label per image
label_parity(expected, observed, expected_groups=expected_items, observed_groups=observed_items) # v1.2, detections
See Label parity for the population, the vocabulary requirement and how to read a result.
Overwriting a factor keeps the old column¶
Metadata.add_factors() with overwrite=True no longer discards the column it replaces. The incoming values take
the name. When the write would replace values, by changing a row’s value, making it missing, or moving the factor to
another level, the column as it was moves to <name>_original, or <name>_original_2 and so on when that is taken.
A write that only fills empty rows, or writes back the values already there, moves nothing aside, so re-running an
overwrite adds no columns. A keyed overwrite still replaces only the rows its keys name, and keeps the whole prior
column when it replaces a value.
The moved column is kept but not analyzed. rows_at and dataframe read its values, and factor_info describes it,
but it is left out of factor_names, factor_data and every evaluator, whatever exclude holds. To analyze it, name
it in include, for example md.include = ["brightness", "brightness_original"]. Because factor_info describes it,
the keys of factor_info can include a name factor_names does not, so pair the two by name rather than by position. A
column stored under <name>_added by overwrite=False is analyzed as before.
One moved column is not kept for good. A roll-up output keeps its roll-up’s record, so when a source of that roll-up later changes, the stale moved column is dropped and the roll-up is recomputed under the name it asks for, renamed if the overwrite took that name.
FactorInfo gains source_name, the name a factor had in its source before DataEval renamed the column. A
dataset key level, stored as metadata_level, records level, and both brightness_added and brightness_original
record brightness. Saved metadata keeps these names, and a file saved by an earlier version reads each column as its
own source name.
Partial keyed source_index¶
Metadata.add_factors() accepts addresses that carry a key and name only some rows of their level. Previously, a
source index had to name every row at each level it addressed, and naming fewer raised ShapeMismatchError. A key names
one row, so findings measured on some of a video’s frames now attach to the full dataset:
md.add_factors({"blur": [0.2, 1.0]}, source_index=[SourceIndex(0, 2, "unit"), SourceIndex(1, 0, "unit")])
Rows no address names are null. A keyed source_index write folds into a factor of the same name exactly as a key=
write does: a later write that names only rows still null fills them in rather than being stored as <name>_added, and
naming a row that already holds a value is a collision, which overwrite decides. Addresses without a key, such as
SourceIndex(3), still have to name every row at their level.
Metadata carries the datum id as item_id¶
The id that MAITE requires on every datum was a factor in v1.1: it appeared in factor_names, was binned, and gave
Balance and Diversity an id row. An identifier names a row rather than measuring the data, so it
is now the reserved column item_id, present on every row of the dataframe, and never a factor.
md.dataframe["id"] # v1.1
md.dataframe["item_id"] # v1.2
Expect bias results without an id row.
Metadata sets aside factors it cannot read¶
A key that mixes kinds of value is set aside. In v1.1, a datum metadata key holding both numbers and text, such as
an altitude recorded as 100.0 and as "high", was read as text and analyzed as a categorical factor, and booleans
mixed with numbers were read as 0 and 1. Neither is read as a factor now: Metadata.dropped_factors lists it
with the reason mixed_types, and Metadata.unusable reports the values of each kind. Map the values onto
one type at the source, or declare the mapping with Metadata.repair():
from dataeval.types import Remap
md.dropped_factors # {"altitude": ["mixed_types"]}
md = md.repair([Remap("altitude", {"high": 300.0})])
Known factor names are checked. The names in dataeval.types.KNOWN_FACTORS (timestamp, frame_rate,
video_duration, latitude and longitude) each have one meaning, a unit, a range and the levels they are defined
at. In v1.1 they were ordinary factors. A column of the dataset’s own metadata that breaks its definition, such as a
latitude of 100, text in a timestamp, or a frame_rate per image rather than per video, is now set aside with the
reason invalid_known_factor, and Metadata.add_factors() raises ValueError for one. Fix the values, or
rename a factor that means something else. See Known factor names.
None is a missing value. A key an item declares with the value None held the category "None" in v1.1. It is
now null, and missing values are binned in a group of their own. Select those rows with pl.col(name).is_null()
rather than pl.col(name) == "None". For a text key the groups are the same. A number key that held a None was
read as text in v1.1: one category per value, or dropped when nearly every value differed. It is now read as numbers,
and binned when it holds many values, so its bias results change.
ber_knn lower bounds move¶
ber_knn could report a lower bound above the true Bayes error. Its plurality vote broke ties toward the smallest
label, which made an even k err far more than the constants it divided by assume: with a constant posterior of
0.9 (Bayes error 0.1), k=2 reported a lower bound of 0.125. Three changes keep the bound below the Bayes error:
Ties go to the tied label whose member is nearest, so an even
kvotes likek - 1and takes its constant.Devroye’s constant applies only for odd
k >= 5, where he states it; elsewhere the tighter FeeBee constant applies.With more than two classes, the bound comes from the 1-NN error, for any
k, since Cover & Hart’s inequality is about the 1-NN error.
The lower bound is also clamped to the upper bound, which is an error the k-NN classifier achieves. upper_bound
changes only where ties occur. ber_knn also accepts Python lists for class_labels, as documented, and a new
groups= argument.
Feasibility judges targets with evidence¶
Feasibility is a new experimental evaluator in dataeval.performance. For each target, a metric and a
threshold, it returns one of four determinations (demonstrated_feasible, infeasible_under_recorded_assumptions,
not_ruled_out, unresolved) and the evidence behind it: envelopes, upper bounds on what any model in a stated class
can reach, and witnesses, lower bounds on what a frozen model reaches. A witness that beats an envelope suspends the
conclusion: the determination is unresolved with the limitation conflict. Every bound is a finite-sample bound, and
all the bounds of one call hold together with probability at least 1 - alpha.
from dataeval.performance import Feasibility, TargetSpec
target = TargetSpec(metric="accuracy", threshold=0.95)
result = Feasibility().evaluate(dataset, target=[target], embeddings=embeddings)
result.determinations # one row per target
result.bounds # the bounds each row cites
This release covers image and video classification, with accuracy and AP targets and, for image classification, a per-class recall target at a stated false-positive limit:
tank = TargetSpec(task="IC", metric="recall", threshold=0.9, class_name="Tank", max_false_positive_rate=0.05)
result = Feasibility().evaluate(dataset, target=[tank], embeddings=embeddings, predictions=decisions)
A recall target counts a witness only when its own false-positive upper bound is within the limit, and its envelope
bounds the recall any model in its scope reaches at that limit. It is refused on grouped data and with
false_positive_scope="each_class". predictions= also accepts a 1-D integer array of class decisions, model= takes
a MAITE model in its place, and when the requirement is stated on a vocabulary coarsened by Relabel,
Relabel.fold_scores folds the model’s class scores through the same map. Bounds on Bernoulli trials use exact
Clopper–Pearson endpoints on ungrouped data, and print(result) lays out one line per requirement.
Note
The first v1.2.0 release shipped this evaluator with a different surface: targets were a mapping of string keys,
allowances a dict, the call took provenance, sampling and look, the constructor took panel_size, n_cells and
builtin_witnesses, and the result held determinations with free-text columns, plus evidence. The typed surface
above replaces it: determinations keeps its name but holds Enum and number columns only, and bounds replaces
evidence. A removed argument raises TypeError, and SamplingPlan, Evidence, Endpoint and
aggregate_by_evidence() are gone.
TargetSpec now takes metric and threshold first and task after them (it defaults to "IC"), so a
positional TargetSpec("IC", "accuracy", 0.9) raises ValueError: pass keywords. It is experimental and may change
again.
Object detection and tracking datasets raise NotImplementedError. Feasibility does not use ber_knn or ber_mst,
whose bounds are asymptotic, so the ber_knn change above does not move its results. TargetSpec and
Allowances are exported from dataeval.performance alongside it,
and the bounds it composes are new experimental functions in dataeval.core: mean_endpoint(),
binomial_endpoint(), ratio_endpoint(), units_to_cross(), zero_one_consensus(),
cost_consensus(), consensus_max(), cost_consensus_max(), kink_costs(),
disjoint_panels(), fit_partition(), local_panel_bound(), ap_upper_envelope(),
ap_lower_staircase() and witness_thresholds().
See Performance limits and Feasibility evidence.
split_dataset honors split_on on detection and frame datasets¶
v1.1 ignored split_on on a detection dataset with more detections than images. It logged “split_on is not supported
for object detection datasets; ignoring.” and split by image. A sparser one took the classification path, which split
its detection rows as though they were images. v1.2 keeps every image of a group in one partition, including an image
without detections, however few detections the dataset holds. Over a SequenceFrames view,
split_on=["sequence"] keeps every frame of a video in one partition.
from dataeval.data import SequenceFrames, split_dataset
splits = split_dataset(SequenceFrames(dataset), split_on=["sequence"], test_frac=0.2, val_frac=0.2)
The fractions now count groups, not images. Each group is split as one unit, so over five videos test_frac=0.2 aims
at one whole video, however many frames it holds, and a split of a few long videos can land far from the fractions you
asked for. Stratification counts groups too: a group holds a class if any of its images does, and stratify=True
needs each class in at least as many groups as there are partitions, counting train, validation and test.
A group is one combination of the named factors’ values as recorded, not of the bins Metadata cuts for the
bias evaluators. A frame view therefore gives one group per video however many videos it holds, and the fractions, the
stratification count and the error messages count videos. Two kinds of factor keep their bins: a factor holding floats,
a measurement whose every value would be a group of its own, and a factor whose cut you declared with
continuous_factor_bins or a recorded encoding. A metadata container other than Metadata is grouped by its
factor_data codes, as before.
Grouping by values bins nothing, so a split on such factors alone no longer emits the “binned automatically” warning.
Balance, Diversity and Parity still read the binned factor, and the warning now comes from them.
The same rule reaches image classification splits. An integer factor with more distinct values than the level budget allows, more than 20 on a small dataset, was grouped by its automatic bins in v1.1, which merged whole groups. v1.2 groups it by each value:
split_dataset(Metadata.from_factors({"site": sites}, class_labels=labels), split_on=["site"], num_folds=5) # 30 sites
# v1.1: fewer groups than sites, each a run of consecutive sites
# v1.2: 30 groups, one per site
Where v1.1 logged and carried on, split_on on a detection dataset now raises a ValueError:
when it names a factor the metadata does not carry
when it names a factor that varies within one image, such as one added at the
instancelevelwhen there are too few groups to give each partition one, or, with
stratify=True, a class is in fewer groups than partitionswhen the split would leave a partition without a group
when the dataset is a metadata container other than
Metadatathat holds more labels than images, since it has no row for an image without detections
Apart from the refusals in the next section and the grouping of an
integer factor past the level budget, image classification splits are unchanged, and so is a detection split without
split_on.
split_dataset refuses a split it cannot fill¶
stratify=True needs the rarest class in at least as many items as there are partitions. v1.1 counted a single split
(num_folds=1) with a test holdout as two partitions where it makes three: train, validation and test. A class too rare
to reach all three passed the check. On a detection or frame dataset the split then came back with an empty train
partition and no error. On an image classification dataset it failed inside scikit-learn on n_splits=1. v1.2 counts
the validation partition, and a detection split that still leaves a partition empty raises as well. Both raise a
ValueError that gives the counts:
# One class in 2 of 100 frames
splits = split_dataset(frames, stratify=True, test_frac=0.2, val_frac=0.2)
# v1.1: train 0, validation 50, test 50
# v1.2: ValueError: Unable to stratify: the rarest class has only 2 images, fewer than the 3 partitions requested.
These two checks leave unchanged a split that filled every partition in v1.1. For one that now raises, ask for fewer
partitions or less extreme fractions, or pass stratify=False.
stratify=True also raises on a filtered detection Metadata that keeps an item without detections. v1.1
numbered the kept items as though each held a detection, so the classes it stratified on slid onto the wrong items:
five single-class videos could all read as holding both classes. Without stratify, the split places every item
correctly, with or without split_on.
kept = Metadata(frames).where(pl.col("frame") >= 3, level="unit") # some kept frames hold no detections
splits = split_dataset(kept, stratify=True, test_frac=0.2, val_frac=0.2)
# v1.1: folds stratified on other frames' classes
# v1.2: ValueError: 5 of the 85 items this filtered Metadata keeps hold no detections, ... Pass stratify=False, ...
A grouped split that leaves fewer groups after the test split than its folds need, num_folds of them or two for a
single split, raises a ValueError that counts groups, in place of a scikit-learn error that reported n_splits=1 or
counted samples:
split_dataset(metadata, split_on=["site"], test_frac=0.5, val_frac=0.2) # 3 sites
# v1.1: ValueError: k-fold cross-validation requires at least one train/test split ... got n_splits=1.
# v1.2: ValueError: The test split leaves 1 of the 3 groups for 1 [train, val] fold(s), which need at least 2. ...
A stratified detection split, with or without split_on, now logs a warning on the dataeval.data logger when a class
misses a partition it could have reached, as a classification split already did. The message names the partition and
the class, such as “fold 2 val lacks class 1”.
Outliers judges a tracking dataset in video terms¶
Outliers().evaluate(tracking_dataset) failed with a numpy error before v1.2. It now measures every
frame with image_stats(), places the statistics on the dataset’s Metadata with
SequenceFrames.to_source(), and judges the result, as Duplicates already does. A
SequenceFrames over a tracking dataset gets the same treatment, and its selector decides which frames are
measured.
from dataeval.quality import Outliers
result = Outliers().evaluate(tracking_dataset) # the sequence and unit levels
result = Outliers().evaluate(tracking_dataset, per_target=True) # also track and instance
Levels.
sequence(a video) andunit(a frame) are judged by default.per_target=Trueaddstrackandinstance(a detection). A video’s value of a statistic is the median over its frames, and a track’s is the median over its detections.Addresses. Rows are addressed in the source dataset, not by position in the frame view:
item_indexis the video, andtarget_indexis the frame number atunit, the track id attrackand null atsequence. A row’sSourceIndexresolves withSourceLocator.result.statskeeps the frame view’s own positions.Names. Metric names follow
Metadata.add_factors():widthandheightcollide with columns the tracking metadata already holds, so they readwidth_addedandheight_added, orunit_widthand so on withper_target=True.Baselines are pooled over the dataset. To judge a night video against night videos, evaluate the subset on its own; the
Outliers.evaluate()docstring gives the recipe. Thesequencelevel is judged only with 20 or more videos.Refused with a
ValueError:extractor=(cluster mode),factors,combinations,given,kindsandneighbors, several datasets in one call, a tracking dataset mixed with image datasets, ametadata=of another dataset, and a frame view that keeps no frames.Cost. Each video’s stream is read more than once per call.
DriftKnownDomains names the known domain new data is closest to¶
DriftKnownDomains is new in v1.2 and experimental ⚠️: it raises
ExperimentalWarning and may change or be removed without a deprecation cycle. Each known
domain is a named reference dataset, or the samples of one reference dataset whose metadata factor falls in a group of
values. A classifier learns to tell the domains apart, and a sample is novel to the extent that it is not clearly any
one of them. It reports the closest known domain and whether more of the test data is novel than of held-out reference
data, and it can be saved to an ONNX file and restored to monitor without the reference data. Unlike
DriftDomainClassifier, which compares the test data with the reference as a whole, it needs at least three
domains. Import it from dataeval.shift.
Videos without tracking targets or metadata¶
Duplicates, Outliers, SequenceFrames and SourceLocator take a dataset of videos
that carries no tracking targets: each item is a video stream alone, or a (stream, target, metadata) tuple whose
target is not a tracking target, such as None. Each video is measured as video with no detections, and findings are
reported by video and frame, as on a tracking dataset. A dataset needs no metadata
attribute, with tracking targets or without. ImageDataset and
VideoDataset name the inputs the evaluators accept.
from dataeval.quality import Duplicates, Outliers
# `videos[i]` is a stream of frames that carry `pixels`, `time_s`, `pts` and `frame_index`
duplicates = Duplicates().evaluate(videos)
outliers = Outliers().evaluate(videos) # the sequence and unit levels
Levels. There are no tracks, so
Outliersraises forlevels="track", and withper_target=Trueit finds no tracks or detections to judge.Frame count. A video’s length is
len(stream)where the stream has one, and otherwise one walk of the video, made once and kept.Metadata. The evaluators build their own;
Metadata(videos)on such a dataset is not supported.
Checklist¶
Replace deprecated
Metadatamembers using the table above.Add
level=orsource_index=to everyadd_factorscall. Split anylevel="combined"array into separate factors per level.Update imports from
dataeval.selectionanddataeval.utils.datatodataeval.data. ReplaceSelectwithView.Replace
ClassFilter(...)withFilter(classes=...), andfilter_detections=Falsewithdrop="items". RemoveClassFilter([]).Remove
per_channel=arguments, includingper_channel=False. Replaceper_channel=Truewith achannels={...}dictionary.Replace
BitDepth,get_bitdepth(), andclip_and_pad()with their v1.2 equivalents.Rename
SourceIndex.targetattribute access andtarget=construction arguments tokey. Update positional construction and unpacking ofSourceIndexso the third value representslevel.Implement
is_binnedon custom implementations ofMetadataLike.Pass
discrete_features=tomutual_info. Removediscrete_features=from calls tomutual_info_classwise.Replace
ClassifierUncertaintyExtractorwithUncertaintyExtractororClasswiseUncertaintyExtractor.Update code reading
scoreto expect a single float per row. Check for missing scores usingis_null()instead of array shape. Note thatRelabel(..., reduce_detection_scores=False)affects target layout rather than column width, and alters the values extracted into metadata.Rescale any
ConstantThresholdon aPIXELstatistic passed toOutliersto stored units. For images at mixed bit depths, compute statistics withnormalize_pixel_values=Trueand callOutliers.from_stats.Pass
per_target=TruetoOutliers.from_stats, including in the defaultfrom_stats(stats)call, if you re-detect withwith_threshold,classwiseoritemwiseand expect target rows back. Passper_image=Trueif you expect item rows back.Check
result.populationsforOutlierspopulations under 20 rows, which are no longer judged by a statistical threshold. Collect more rows for them, or judge them with aConstantThreshold.Re-read any per-class outlier counts you recorded from
aggregate_by_classonper_target=Trueresults, or on images that hold more than one box of a class.Replace
compute_ratioswith therel_*statistics for box geometry, and with a division ofinstance_byunit_columns on aMetadatafor any other statistic.Replace
ParityOutput.insufficient_datawithcells.filter(pl.col("is_sparse")), andis_significantwithis_associatedorstatus. Re-read any Parity flags you recorded: they now use adjusted p-values and, on detection and tracking data, grouped tests.Give custom metadata containers
item_indicesbefore runningParity, or passassume_independent=Truewhen each row is its own item. Passgroupsorassume_independent=Trueto coreparity.Pass
expected_groupsandobserved_groupstolabel_parityfor detection or tracking labels, orassume_independent=Truefor one label per image. Removenum_classes.Expect a
<name>_originalcolumn after anadd_factors(..., overwrite=True)that replaces values. It is kept for reading but not analyzed; name it inincludewhere the replaced values should be analyzed.Recompute any stored
phashordhashof integer images with values above 255, which now hash their scaled content rather than a saturated frame.Re-baseline any stored
ber_knnlower bounds computed with an evenkor more than two classes.Re-read any stored box statistics for
Nonebox entries on non-square or 2-D images, and anyboxes_to_maskresult built from one. They covered the image with its width and height swapped, or nothing at all on a 2-D image.Make
boxesthe same length as the images passed toimage_statsorcompute_stats, passingNonefor an unannotated image.Re-split any split made with
split_onon a detection dataset, which v1.1 ignored and split by image, and expect partition sizes that follow the groups rather than the fractions. Re-split one made withsplit_onan integer factor past the level budget, which v1.1 grouped by its bins, and drop anyfactor_levelsdeclared onsequenceonly to keep one group per video.Run DataEval on Python 3.11 or later.
Make every extractor wrapped in
Embeddingsreturn one row per image. Passmetadata.dataset, not aMetadata, toEmbeddings, and re-read anyCoverageresult computed from aMetadatawithoutembeddings=.Pass
Prioritizeone class label per embedding. For a detection dataset, wrap it inDetectionCropsfirst.Correct any evaluator constructor argument that now fails validation: a wrong type, an unknown choice, or a value outside its range.
Split a chunked drift detector’s reference into at least 3 chunks, or give it a
ConstantThreshold.Scale masks stored as 0 and 1 to 0 to 255 before running
Duplicates, and recompute any storedphashordhashof integer images whose maximum is at most 1.Match
unit_andinstance_metric names onOutliers.evaluateresults, and their aggregations, from detection datasets withper_target=True.Re-read any
Balancefactorsresults andis_correlatedflags recorded on detection datasets.Read the datum
idfrom theitem_idcolumn, and expect noidrow inBalanceorDiversityresults.Check
dropped_factorsformixed_typesandinvalid_known_factor, and repair or rename those factors. Select missing values withis_null()rather than the string"None", and re-read bias results for number keys that hold missing values.Compute
(metric_value - population_mean) / population_stdwherever you read or threshold anOutlierscluster_distanceas a number of standard deviations.Expect
split_dataset(..., stratify=True)to raise where the rarest class cannot reach train, validation and test, or where a detection split leaves a partition empty. Ask for fewer partitions or less extreme fractions, or passstratify=False.Pass
stratify=Falsetosplit_datasetfor a filtered detectionMetadatathat keeps items without detections, or filter so that every kept item holds a detection.
Related¶
Measuring channel groups (guide to band groups)
Metadata levels (overview of dataset metadata hierarchy)