dataeval.metrics.estimators.uap¶

dataeval.metrics.estimators.uap(labels, scores)¶

FR Test Statistic based estimate of the empirical mean precision for the upperbound average precision.

Parameters:¶

labels : ArrayLike¶

A term:NumPy array of n_samples of class labels with M unique classes.

scores : ArrayLike¶

A 2D array of class probabilities per image

Returns:¶

The empirical mean precision estimate, float

Return type:¶

UAPOutput

Raises:¶

ValueError – If unique classes M < 2

Note

This function calculates the empirical mean precision using the average_precision_score from scikit-learn, weighted by the class distribution.

Examples

>>> y_true = np.array([0, 0, 1, 1])
>>> y_scores = np.array([0.1, 0.4, 0.35, 0.8])
>>> uap(y_true, y_scores)
UAPOutput(uap=0.8333333333333333)

>>> y_true = np.array([0, 0, 1, 1, 2, 2])
>>> y_scores = np.array(
...     [
...         [0.7, 0.2, 0.1],
...         [0.4, 0.3, 0.3],
...         [0.1, 0.8, 0.1],
...         [0.2, 0.3, 0.5],
...         [0.4, 0.4, 0.2],
...         [0.1, 0.2, 0.7],
...     ]
... )
>>> uap(y_true, y_scores)
UAPOutput(uap=0.7777777777777777)