dataeval.detectors.drift.DriftUncertainty¶
-
class dataeval.detectors.drift.DriftUncertainty(x_ref, model, p_val=
0.05, x_ref_preprocessed=False, update_x_ref=None, preds_type='probs', batch_size=32, preprocess_batch_fn=None, device=None)¶ Test for a change in the number of instances falling into regions on which the model is uncertain.
Performs a K-S test on prediction entropies.
- Parameters:¶
- x_ref : ArrayLike¶
Data used as reference distribution.
- model : Callable¶
Classification model outputting class probabilities (or logits)
- p_val : float, default 0.05¶
P-Value used for the significance of the test.
- x_ref_preprocessed : bool, default False¶
Whether the given reference data
x_refhas been preprocessed yet. IfTrue, only the test dataxwill be preprocessed at prediction time. IfFalse, the reference data will also be preprocessed.- update_x_ref : UpdateStrategy or None, default None¶
Reference data can optionally be updated using an UpdateStrategy class. Update using the last n instances seen by the detector with LastSeenUpdateStrategy or via reservoir sampling with ReservoirSamplingUpdateStrategy.
- preds_type : "probs" or "logits", default "probs"¶
Type of prediction output by the model. Options are ‘probs’ (in [0,1]) or ‘logits’ (in [-inf,inf]).
- batch_size : int, default 32¶
Batch size used to evaluate model. Only relevant when backend has been specified for batch prediction.
- preprocess_batch_fn : Callable or None, default None¶
Optional batch preprocessing function. For example to convert a list of objects to a batch which can be processed by the model.
- device : DeviceLike or None, default None¶
Device type used. The default None tries to use the GPU and falls back on CPU if needed. Can be specified by passing either ‘cuda’ or ‘cpu’.
Example
>>> model = ClassificationModel() >>> drift = DriftUncertainty(x_ref, model=model, batch_size=20)Verify reference images have not drifted
>>> drift.predict(x_ref.copy()).drifted FalseTest incoming images for drift
>>> drift.predict(x_test).drifted True