dataeval.performance.Sufficiency.Config¶

class dataeval.performance.Sufficiency.Config¶

Configuration for sufficiency analysis execution.

training_strategy¶

Strategy for training models on dataset subsets. Must implement the train(model, dataset, indices) method.

Type:¶: TrainingStrategy

evaluation_strategy¶

Strategy for evaluating trained models. Must implement the evaluate(model, dataset) method returning metrics.

Type:¶: EvaluationStrategy

reset_strategy¶

Strategy for resetting model parameters between runs. Required when using Sufficiency. Must be a callable that takes the model and returns a reset model (e.g., with re-initialized weights).

Type:¶: Callable[[Any], Any] or None, default None

runs¶

Number of independent training runs to perform. Each run trains a fresh model from scratch.

Type:¶: int, default 1

substeps¶

Number of evaluation steps per run. Used for default geometric schedule if no custom schedule is provided.

Type:¶: int, default 5

unit_interval¶

Whether metrics are constrained to [0, 1]. Set True for metrics like accuracy, precision, recall. Set False for unbounded metrics like loss or error.

Type:¶: bool, default True

Raises:¶: ValueError – If runs or substeps is not at least 1

See also

-: class:.TrainingStrategy
-: class:.EvaluationStrategy
-: class:.ModelResetStrategy

Examples

Basic configuration:

>>> training = CustomTrainingStrategy(learning_rate=0.001, epochs=10)
>>> evaluation = CustomEvaluationStrategy(batch_size=32)
>>> config = Sufficiency.Config(
...     training_strategy=training,
...     evaluation_strategy=evaluation,
...     runs=3,
...     substeps=5,
... )

Configuration for unbounded metrics (e.g., loss):

>>> config = Sufficiency.Config(
...     training_strategy=training,
...     evaluation_strategy=evaluation,
...     runs=5,
...     unit_interval=False,  # For loss metrics
... )

Configuration with custom reset strategy:

>>> def custom_reset(model):
...     # Custom logic to reset model parameters
...     return model

>>> config = Sufficiency.Config(
...     training_strategy=training,
...     evaluation_strategy=evaluation,
...     reset_strategy=custom_reset,
... )