dataeval.scope.Representation¶

class dataeval.scope.Representation(ontology, *, expected=None, config=None)¶

Evaluate a dataset’s coverage of an ontology and prioritize what to collect.

Resolves the dataset’s class labels against the ontology, compares the observed distribution to an expected one, and returns a RepresentationOutput worklist of the leaf species to acquire or augment. The default expectation is a uniform distribution over leaf species; pass expected to assert a minimum share (a fraction of the whole dataset) for specific classes you know to be rare, which both right-sizes their target and is validated as an assertion.

Parameters:¶

ontology : Ontology¶

Ontology whose leaf species define the label space to cover.

expected : Mapping[str, float] or None, default None¶

Class name to its minimum expected share of the dataset (a fraction in [0, 1]). Named classes use this floor as their target in place of the uniform share, and are validated in RepresentationOutput.violations; unnamed classes keep the uniform target. None means a uniform expectation for every leaf.

config : Representation.Config or None, default None¶

Optional configuration object; parameters passed directly to __init__ override its values.

See also

dataeval.core.label_coverage

The observation-only coverage facts this builds on.

dataeval.core.label_reconciliation

Resolve labels against an ontology.

Notes

Targets are rounded to the nearest whole label. A class named in expected that does not resolve to exactly one concept is ignored (resolve it upstream).

Examples

>>> from dataeval import Ontology
>>> from dataeval.scope import Representation
>>> ontology = Ontology.from_hierarchy({"animal": {"mammal": ["cat", "dog"], "bird": ["owl"]}})
>>> result = Representation(ontology).evaluate(dataset)
>>> result.columns
['concept', 'label', 'parent', 'action', 'count', 'target', 'deficit']

Assert that a known-rare class need only make up 5% of the dataset:

>>> result = Representation(ontology, expected={"owl": 0.05}).evaluate(dataset)
>>> result.violations.columns
['concept', 'label', 'floor', 'actual', 'shortfall']

evaluate(data, *, index2label=None)¶

Evaluate a dataset’s coverage of the ontology.

Parameters:¶

data : AnnotatedDataset, MetadataLike, Mapping, or Sequence of int¶

The source of class labels, accepted in several forms:

• a full AnnotatedDataset — class labels and their index2label names are read from it and counted via label_stats();

• a Metadata — same, but read straight from it;

• any MetadataLike object — its class_labels are counted, named via its own index2label if present, else the index2label argument;

• a {label_name: count} mapping (string keys) — the minimal form the core consumes; class names resolve directly against the ontology, so no dataset or index2label is needed;

• a {label_index: count} mapping (integer keys) — indices are named through index2label (or their own string form when it is omitted);

• a raw sequence/array of integer class labels — counted, then named the same way as the integer mapping.

index2label : Mapping[int, str] or None, default None¶

Optional mapping from integer class index to human-readable name, used only when data is integer labels (a raw sequence or an integer-keyed mapping). Ignored for the dataset, Metadata, and {label_name: count} forms, which already carry or are their own names. When omitted, integer labels name themselves by their string form (which resolves against the ontology only if its concept ids/labels are those strings).

Returns:¶

The collection worklist (acquire / augment rows) with leaf_coverage, total_deficit, violations, and dark_branches.

Return type:¶

RepresentationOutput

Examples

>>> ontology = Ontology.from_hierarchy({
...     "vehicle": {"land": ["car", "bike"], "water": ["boat"], "air": ["plane"]}
... })
>>> evaluator = Representation(ontology)
>>> result = evaluator.evaluate(dataset)
>>> result.data()
shape: (2, 7)
┌─────────┬───────┬────────┬─────────┬───────┬────────┬─────────┐
│ concept ┆ label ┆ parent ┆ action  ┆ count ┆ target ┆ deficit │
│ ---     ┆ ---   ┆ ---    ┆ ---     ┆ ---   ┆ ---    ┆ ---     │
│ str     ┆ str   ┆ str    ┆ str     ┆ i64   ┆ i64    ┆ i64     │
╞═════════╪═══════╪════════╪═════════╪═══════╪════════╪═════════╡
│ bike    ┆ bike  ┆ land   ┆ acquire ┆ 0     ┆ 23     ┆ 23      │
│ boat    ┆ boat  ┆ water  ┆ augment ┆ 22    ┆ 23     ┆ 1       │
└─────────┴───────┴────────┴─────────┴───────┴────────┴─────────┘
>>> result.total_deficit
24
>>> result.leaf_coverage
0.75

Classes¶