dataeval.protocols¶

Common type protocols used for interoperability with DataEval.

Attributes¶

`ArrayLike`	Type alias for a Union representing objects that can be coerced into an array.
`DatasetMetadata`	Metadata associated with a dataset (collection-level).
`DatumMetadata`	Metadata associated with a single datum (item-level).
`DeviceLike`	Type alias for a Union representing types that specify a torch.device.
`ImageClassificationDataset`	Type alias for an `AnnotatedDataset` of `ImageClassificationDatum` elements.
`ImageClassificationDatum`	Type alias for an image classification datum tuple.
`Model`	Model protocol specifying batch inference behavior over data.
`ModelMetadata`	Metadata associated with a model.
`MultiobjectTrackingDataset`	Type alias for an `AnnotatedDataset` of `MultiobjectTrackingDatum` elements.
`MultiobjectTrackingDatum`	Type alias for a multi-object tracking datum tuple.
`MultiobjectTrackingTarget`	Set of tracked objects over a sequence of video frames.
`ObjectDetectionDataset`	Type alias for an `AnnotatedDataset` of `ObjectDetectionDatum` elements.
`ObjectDetectionDatum`	Type alias for an object detection datum tuple.
`ObjectDetectionTarget`	Object-detection target for a single image.
`SegmentationDataset`	Type alias for an `AnnotatedDataset` of `SegmentationDatum` elements.
`SegmentationDatum`	Type alias for a segmentation datum tuple.
`SingleFrameObjectTrackingTarget`	Single-frame object-tracking target (tracked objects within one frame).
`ThresholdLike`	Type alias for threshold specifications.
`VideoFrame`	Contents of a single decoded video frame.
`VideoStream`	Iterable of `VideoFrame` representing a single video.

Classes¶

`AnnotatedDataset`	Protocol for a generic AnnotatedDataset.
`Array`	Protocol for array objects providing interoperability with DataEval.
`Chunker`	Protocol for chunking datasets into subsets by returning index arrays.
`Dataset`	Protocol for a generic Dataset.
`EvaluationSchedule`	Protocol for determining evaluation points in sufficiency analysis.
`EvaluationStrategy`	Protocol defining the interface for evaluating a trained model.
`EvidenceLowerBoundLossFn`	Protocol for Evidence Lower Bound (ELBO) loss functions.
`FeatureExtractor`	Protocol defining a feature extraction function for drift detection.
`LossFn`	Protocol for generic loss functions that can be used with PyTorch models.
`Matcher`	Protocol for an element-level matcher used in ontology alignment.
`MetadataLike`	Minimal protocol for metadata objects used in bias and quality analysis.
`ModelResetStrategy`	Protocol for resetting model parameters between training runs.
`ProgressCallback`	Protocol for a callable progress callback function.
`ReconstructionLossFn`	Protocol for reconstruction-based loss functions (Autoencoder).
`SegmentationTarget`	Protocol for targets in a Segmentation dataset.
`SequenceLike`	Protocol for sequence-like objects that can be indexed and iterated.
`Threshold`	Protocol for threshold objects used in bias and quality evaluators.
`TrainingStrategy`	Protocol defining the interface for training a model on a dataset subset.
`Transform`	Protocol defining a transform function.
`UpdateStrategy`	Protocol defining the interface for updating reference data in drift detectors.

Module Contents¶

type dataeval.protocols.ArrayLike = np.typing.ArrayLike¶: Type alias for a Union representing objects that can be coerced into an array.

See also

numpy.typing.ArrayLike

type dataeval.protocols.DatasetMetadata = maite.protocols.DatasetMetadata¶

Metadata associated with a dataset (collection-level).

A TypedDict with the following keys:

id : str (required, read-only) - Unique identifier for the dataset.
index2label : dict[int, str] (optional, read-only) - Mapping from integer class index to the corresponding human-readable label name.

Implementations may add further string-keyed entries; only id is required by the protocol. Extra keys are passed through unchanged.

type dataeval.protocols.DatumMetadata = maite.protocols.DatumMetadata¶

Metadata associated with a single datum (item-level).

A TypedDict with the following keys:

id : int | str (required, read-only) - Unique identifier for the datum.

Implementations may add further string-keyed entries; only id is required by the protocol. Extra keys are passed through unchanged.

type dataeval.protocols.DeviceLike = int | str | tuple[str, int] | torch.device¶: Type alias for a Union representing types that specify a torch.device.

See also

torch.device

type dataeval.protocols.ImageClassificationDataset = AnnotatedDataset[ImageClassificationDatum]¶: Type alias for an AnnotatedDataset of ImageClassificationDatum elements.

type dataeval.protocols.ImageClassificationDatum = tuple[ArrayLike, ArrayLike, DatumMetadata]¶

Type alias for an image classification datum tuple.

ArrayLike of shape (C, H, W) - Image data in channel, height, width format.
ArrayLike of shape (N,) - Class label as one-hot encoded ground-truth or prediction confidences.
DatumMetadata - Datum level metadata.

type dataeval.protocols.Model = maite.protocols.Model[_InputType, _TargetType]¶

Model protocol specifying batch inference behavior over data.

Re-export of the generic MAITE Model protocol. Use bare for any model, or specialize the input/target types for a concrete task — e.g. Model[ArrayLike, ObjectDetectionTarget].

type dataeval.protocols.ModelMetadata = maite.protocols.ModelMetadata¶

Metadata associated with a model.

A TypedDict with the following keys:

id : str (required, read-only) - Unique identifier for the model.
index2label : dict[int, str] (optional, read-only) - Mapping from integer class index to the corresponding human-readable label name the model predicts.

Implementations may add further string-keyed entries; only id is required by the protocol. Extra keys are passed through unchanged.

type dataeval.protocols.MultiobjectTrackingDataset = AnnotatedDataset[MultiobjectTrackingDatum]¶: Type alias for an AnnotatedDataset of MultiobjectTrackingDatum elements.

type dataeval.protocols.MultiobjectTrackingDatum = tuple[VideoStream, MultiobjectTrackingTarget, DatumMetadata]¶

Type alias for a multi-object tracking datum tuple.

VideoStream - An iterable of VideoFrame for a single video.
MultiobjectTrackingTarget - Tracked objects across the sequence of frames.
DatumMetadata - Datum level metadata.

type dataeval.protocols.MultiobjectTrackingTarget = maite.protocols.multiobject_tracking.MultiobjectTrackingTarget¶: Set of tracked objects over a sequence of video frames.

type dataeval.protocols.ObjectDetectionDataset = AnnotatedDataset[ObjectDetectionDatum]¶: Type alias for an AnnotatedDataset of ObjectDetectionDatum elements.

type dataeval.protocols.ObjectDetectionDatum = tuple[ArrayLike, ObjectDetectionTarget, DatumMetadata]¶

Type alias for an object detection datum tuple.

ArrayLike of shape (C, H, W) - Image data in channel, height, width format.
ObjectDetectionTarget - Object detection target information for the image.
DatumMetadata - Datum level metadata.

type dataeval.protocols.ObjectDetectionTarget = maite.protocols.object_detection.ObjectDetectionTarget¶

Object-detection target for a single image.

boxes : ArrayLike of shape (N, 4) - Bounding boxes in (x0, y0, x1, y1) format
labels : ArrayLike of shape (N,) - Class labels for each bounding box
scores : ArrayLike of shape (N,) - Confidence scores for each bounding box

type dataeval.protocols.SegmentationDataset = AnnotatedDataset[SegmentationDatum]¶: Type alias for an AnnotatedDataset of SegmentationDatum elements.

type dataeval.protocols.SegmentationDatum = tuple[ArrayLike, SegmentationTarget, DatumMetadata]¶

Type alias for a segmentation datum tuple.

ArrayLike of shape (C, H, W) - Image data in channel, height, width format.
SegmentationTarget - Segmentation target information for the image.
DatumMetadata - Datum level metadata.

type dataeval.protocols.SingleFrameObjectTrackingTarget = maite.protocols.multiobject_tracking.SingleFrameObjectTrackingTarget¶: Single-frame object-tracking target (tracked objects within one frame).

Type alias for threshold specifications.

Values default to modified z-score thresholds if not provided.

float: symmetric multiplier (same for lower and upper)
str: named threshold (e.g., “modzscore”) with default bounds
tuple[float | None, float | None]: (lower, upper) for asymmetric bounds
tuple[str, float | tuple[float | None, float | None]]: named threshold with optional lower and upper bounds
tuple[str, bounds, (lower_limit, upper_limit)]: named threshold with bounds and limit clamping, e.g. ("zscore", (1.0, 3.5), (0.0, 1.0)). Pass None for bounds to use defaults: ("zscore", None, (0.0, 1.0))
tuple[bounds | None, (lower_limit, upper_limit)]: default threshold with bounds and limit clamping, e.g. (2.5, (0.0, 1.0)) or (None, (0.0, 1.0)) for default multiplier
Threshold: a fully configured Threshold instance

type dataeval.protocols.VideoFrame = maite.protocols.multiobject_tracking.VideoFrame¶: Contents of a single decoded video frame.

type dataeval.protocols.VideoStream = maite.protocols.multiobject_tracking.VideoStream¶: Iterable of VideoFrame representing a single video.