dataeval.performance.SufficiencyConfig¶

class dataeval.performance.SufficiencyConfig¶

Configuration for sufficiency analysis execution.

training_strategy¶

Strategy for training models on dataset subsets. Must implement the train(model, dataset, indices) method.

Type:¶: TrainingStrategy

evaluation_strategy¶

Strategy for evaluating trained models. Must implement the evaluate(model, dataset) method returning metrics.

Type:¶: EvaluationStrategy

runs¶

Number of independent training runs to perform. Each run trains a fresh model from scratch.

Type:¶: int, default 1

substeps¶

Number of evaluation steps per run. Used for default geometric schedule if no custom schedule is provided.

Type:¶: int, default 5

unit_interval¶

Whether metrics are constrained to [0, 1]. Set True for metrics like accuracy, precision, recall. Set False for unbounded metrics like loss or error.

Type:¶: bool, default True

Raises:¶: ValueError – If runs or substeps is not greater than 1

Examples

Basic configuration:

>>> training = CustomTrainingStrategy(learning_rate=0.001, epochs=10)
>>> evaluation = CustomEvaluationStrategy(batch_size=32)
>>> config = SufficiencyConfig(training, evaluation, runs=3, substeps=5)

Configuration for unbounded metrics (e.g., loss):

>>> config = SufficiencyConfig(
...     training,
...     evaluation,
...     runs=5,
...     unit_interval=False,  # For loss metrics
... )

Notes

This class is immutable (frozen=True) to ensure configuration cannot be accidentally modified during analysis.