dataeval.utils#

The utility classes and functions are provided by DataEval to assist users in setting up architectures that are guaranteed to work with applicable DataEval metrics. Currently DataEval supports both TensorFlow and PyTorch backends.

Functions#

dataeval.utils.merge_metadata(metadata: Iterable[Mapping[str, Any]], ignore_lists: bool = False, fully_qualified: bool = False, as_numpy: bool = False) → tuple[dict[str, list[Any]] | dict[str, ndarray[Any, dtype[Any]]], ndarray[Any, dtype[int64]]]#

Merges a collection of metadata dictionaries into a single flattened dictionary of keys and values.

Nested dictionaries are flattened, and lists are expanded. Nested lists are dropped as the expanding into multiple hierarchical trees is not supported.

Parameters:

  • metadata (Iterable[Mapping[str, Any]]) – Iterable collection of metadata dictionaries to flatten and merge

  • ignore_lists (bool, default False) – Option to skip expanding lists within metadata

  • fully_qualified (bool, default False) – Option to return dictionary keys full qualified instead of minimized

  • as_numpy (bool, default False) – Option to return results as lists or NumPy arrays

Returns:

  • dict[str, list[Any]] or dict[str, NDArray[Any]] – A single dictionary containing the flattened data as lists or NumPy arrays

  • NDArray[np.int_] – Array defining where individual images start, helpful when working with object detection metadata

Note

Nested lists of values and inconsistent keys are dropped in the merged metadata dictionary

Example

>>> list_metadata = [{"common": 1, "target": [{"a": 1, "b": 3, "c": 5}, {"a": 2, "b": 4}], "source": "example"}]
>>> reorganized_metadata, image_indicies = merge_metadata(list_metadata)
>>> reorganized_metadata
{'common': [1, 1], 'a': [1, 2], 'b': [3, 4], 'source': ['example', 'example']}
>>> image_indicies
array([0])

Submodules#

tensorflow

TensorFlow models are used in out of distribution detectors in the dataeval.detectors.ood module.

torch

PyTorch is the primary backend for metrics that require neural networks.