How to measure dataset sufficiency for image classification¶

This guide walks you through analyzing an image classification model’s hypothetical performance.

Estimated time to complete: 10 minutes

Relevant personas: ML Engineer

Problem statement¶

For machine learning tasks, often we would like to evaluate the performance of a model on a small, preliminary dataset. In situations where data collection is expensive, we would like to extrapolate hypothetical performance out to a larger dataset.

DataEval has introduced a method projecting performance via sufficiency curves.

When to use¶

The Sufficiency class should be used when you would like to extrapolate hypothetical performance. For example, if you have a small dataset, and would like to know if it is worthwhile to collect more data.

What you will need¶

A particular model architecture.
Metric(s) that we would like to evaluate.
A dataset of interest.
A Python environment with the following packages installed:
- tabulate

Getting started¶

Let’s import the required libraries needed to set up a minimal working example

import os
from collections.abc import Sequence
from typing import Any, cast

import dataeval_plots as dep
import numpy as np
import plotly.io as pio
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torchmetrics
from IPython.display import display
from maite_datasets.image_classification import MNIST
from numpy.typing import NDArray
from torch.utils.data import DataLoader, Subset
from torch.utils.data import Dataset as TorchDataset

from dataeval import config
from dataeval.data import Limit, Select
from dataeval.performance import Sufficiency
from dataeval.protocols import Dataset, DatumMetadata

DatumType = tuple[NDArray[np.number[Any]], NDArray[np.number[Any]], DatumMetadata]

# Set seed for reproducibility
config.set_seed(0, all_generators=True)

# Set hardware based on system
device = "cuda" if torch.cuda.is_available() else "cpu"
config.set_device(device=device)

# Additional reproducibility and printing options
np.set_printoptions(formatter={"float": lambda x: f"{x:0.4f}"})
torch.set_float32_matmul_precision("high")
torch._dynamo.config.suppress_errors = True
torch.use_deterministic_algorithms(True)
os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"

# Use plotly to render plots
dep.set_default_backend("plotly")

# Use the notebook renderer so JS is embedded
pio.renderers.default = "notebook"

Load data and create model¶

Before calculating the sufficiency of a dataset, the dataset must be loaded and the model architecture defined. We will walk through these in the following steps.

Loading MNIST data¶

Load the MNIST data and split it into training and test datasets. For this notebook, we will use subsets of the training (2500) and test (500) data.

# Configure the dataset transforms

transforms = [
    lambda x: x / 255.0,  # scale to [0, 1]
    lambda x: x.astype(np.float32),  # convert to float32
]

# Download the mnist dataset and apply the transforms and subset the data
train_ds = Select(MNIST("./data", image_set="train", transforms=transforms, download=True), selections=[Limit(2500)])
test_ds = Select(MNIST("./data", image_set="test", transforms=transforms, download=True), selections=[Limit(500)])

Creating a PyTorch model¶

Next, we define the network architecture that will be trained and then evaluated throughout the sufficiency calculation.

# Define our network architecture
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(6400, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        x = torch.flatten(x, 1)  # flatten all dimensions except batch
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x


# Compile the model (cast sets the type to Net as compile returns an Unknown)
model: Net = cast(Net, torch.compile(Net().to(device)))

Strategy protocols¶

Training and evaluation functions are heavily dependent on the hyperparameters defined by a user. These can include metrics, loss functions, optimizers, model architectures, input sizes, etc.

To allow the Sufficiency class to handle this situation, DataEval uses Protocols. Sufficiency requires two specific protocols called TrainingStrategy and EvaluationStrategy.
Below we will define the strategies that align with this notebook and combine them into a Sufficiency.Config that can be given to the Sufficiency class.

Training strategy¶

class MNISTTrainingStrategy:
    def train(self, model: nn.Module, dataset: Dataset[DatumType], indices: Sequence[int]):
        # Defined only for this testing scenario
        criterion = torch.nn.CrossEntropyLoss().to(device)
        optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)
        epochs = 10

        # Define the dataloader for training
        dataloader = DataLoader(Subset(cast(TorchDataset, dataset), indices), batch_size=8)

        for _epoch in range(epochs):
            for batch in dataloader:
                # Load data/images to device
                X = torch.Tensor(batch[0]).to(device)
                # Load one-hot encoded targets/labels to device
                y = torch.argmax(torch.asarray(batch[1], dtype=torch.int).to(device), dim=1)
                # Zero out gradients
                optimizer.zero_grad()
                # Forward propagation
                outputs = model(X)
                # Compute loss
                loss = criterion(outputs, y)
                # Back prop
                loss.backward()
                # Update weights/parameters
                optimizer.step()

Evaluation strategy¶

class MNISTEvaluationStrategy:
    def evaluate(self, model: nn.Module, dataset: Dataset[DatumType]) -> dict[str, float]:
        # Metrics of interest
        metrics = {
            "Accuracy": torchmetrics.Accuracy(task="multiclass", num_classes=10).to(device),
            "AUROC": torchmetrics.AUROC(task="multiclass", num_classes=10).to(device),
            "TPR at 0.5 Fixed FPR": torchmetrics.ROC(task="multiclass", average="macro", num_classes=10).to(device),
        }
        result = {}
        # Set model layers into evaluation mode
        model.eval()
        dataloader = DataLoader(cast(TorchDataset, dataset), batch_size=8)
        # Tell PyTorch to not track gradients, greatly speeds up processing
        with torch.no_grad():
            for batch in dataloader:
                # Load data/images to device
                X = torch.Tensor(batch[0]).to(device)
                # Load one-hot encoded targets/labels to device
                y = torch.argmax(torch.asarray(batch[1], dtype=torch.int).to(device), dim=1)
                preds = model(X)
                for metric in metrics.values():
                    metric.update(preds, y)
            # Compute ROC curve
            false_positive_rate, true_positive_rate, _ = metrics["TPR at 0.5 Fixed FPR"].compute()
            # determine interval to examine
            desired_rate = 0.5
            closest_desired_index = torch.argmin(torch.abs(false_positive_rate - desired_rate)).item()
            # return corresponding tpr value
            result["TPR at 0.5 Fixed FPR"] = true_positive_rate[closest_desired_index].cpu()
            result["Accuracy"] = metrics["Accuracy"].compute().cpu()
            result["AUROC"] = metrics["AUROC"].compute().cpu()
        return result

Reset strategy¶

The Sufficiency class requires a reset_strategy that resets the model’s parameters between runs. This ensures each run starts from a fresh initialization. Here’s a simple implementation for PyTorch models:

def reset_model(model: nn.Module) -> nn.Module:
    """Reset all parameters in a PyTorch model."""

    @torch.no_grad()
    def weight_reset(m: nn.Module) -> None:
        reset_fn = getattr(m, "reset_parameters", None)
        if callable(reset_fn):
            reset_fn()

    return model.apply(fn=weight_reset)

Sufficiency config¶

Do not forget to initialize your strategy classes!

mnist_config = Sufficiency.Config(
    training_strategy=MNISTTrainingStrategy(),
    evaluation_strategy=MNISTEvaluationStrategy(),
    reset_strategy=reset_model,
    runs=5,
    substeps=10,
)

Initialize sufficiency metric¶

Attach the custom training and evaluation functions to the Sufficiency metric and define the number of models to train in parallel (stability), as well as the number of steps along the learning curve to evaluate.

# Instantiate sufficiency metric
suff = Sufficiency(
    model=model,
    config=mnist_config,
)

Evaluate sufficiency¶

Now we can evaluate the metric to train the models and produce the learning curve.

# Train & test model
output = suff.evaluate(train_ds, test_ds)

W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035] WON'T CONVERT forward /tmp/ipykernel_5110/1189723111.py line 11 
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035] due to: 
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035] Traceback (most recent call last):
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 1945, in __call__
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     result = self._inner_convert(
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         frame, cache_entry, hooks, frame_state, skip=skip + 1
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     )
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 707, in __call__
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     result = _compile(
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         frame.f_code,
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ...<16 lines>...
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         convert_frame_box=self._box,
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     )
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 1752, in _compile
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     guarded_code, tracer_output = compile_inner(code, one_graph, hooks)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]                                   ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_utils_internal.py", line 97, in wrapper_function
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     return function(*args, **kwargs)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 1433, in compile_inner
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     return _compile_inner(code, one_graph, hooks)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 1467, in _compile_inner
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     dynamo_output = compile_frame(
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         code,
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ...<11 lines>...
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         package=package,
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     )
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 1341, in compile_frame
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     bytecode, tracer_output = transform_code_object(code, transform)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]                               ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/bytecode_transformation.py", line 1600, in transform_code_object
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     tracer_output = transformations(instructions, code_options)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 1313, in transform
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     tracer_output = trace_frame(
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         code,
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ...<14 lines>...
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         package=package,
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     )
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 328, in _fn
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     return fn(*args, **kwargs)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 838, in trace_frame
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     run_tracer()
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 819, in run_tracer
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     tracer.run()
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/symbolic_convert.py", line 1654, in run
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     while self.step():
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]           ~~~~~~~~~^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/symbolic_convert.py", line 1334, in step
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     self.dispatch_table[inst.opcode](self, inst)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/symbolic_convert.py", line 4692, in RETURN_VALUE
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     self._return(inst)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~~~^^^^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/symbolic_convert.py", line 4674, in _return
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     all_stack_locals_metadata = self.output.compile_subgraph(
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         self,
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ...<4 lines>...
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         stack_pops=1 if inst.opname == "RETURN_VALUE" else 0,
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     )
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/output_graph.py", line 1700, in compile_subgraph
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/output_graph.py", line 2225, in compile_and_call_fx_graph
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     compiled_fn = self.call_user_compiler(gm, self.example_inputs())
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/output_graph.py", line 2377, in call_user_compiler
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     return self._call_user_compiler(gm, example_inputs)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]            ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/output_graph.py", line 2412, in _call_user_compiler
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     compiled_fn = compiler_fn(gm, example_inputs)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/repro/after_dynamo.py", line 156, in __call__
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     compiled_gm = compiler_fn(gm, example_inputs)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/__init__.py", line 2435, in __call__
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     return compile_fx(model_, inputs_, config_patches=self.config)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 2537, in compile_fx
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     return _maybe_wrap_and_compile_fx_main(
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         model_,
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ...<3 lines>...
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         ignore_shape_env,
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     )
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 2614, in _maybe_wrap_and_compile_fx_main
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     return _compile_fx_main(
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         model_,
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ...<3 lines>...
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         ignore_shape_env,
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     )
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 2823, in _compile_fx_main
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     raise e.remove_dynamo_frames() from None  # see TORCHDYNAMO_VERBOSE=1
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 1019, in _compile_fx_inner
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     raise InductorError(e, currentframe()).with_traceback(
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         e.__traceback__
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ) from None
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 1003, in _compile_fx_inner
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     mb_compiled_graph = fx_codegen_and_compile(
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         gm, example_inputs, inputs_to_check, **graph_kwargs
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     )
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 1766, in fx_codegen_and_compile
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     return scheme.codegen_and_compile(gm, example_inputs, inputs_to_check, graph_kwargs)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]            ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 1537, in codegen_and_compile
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     compiled_module = graph.compile_to_module()
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/graph.py", line 2416, in compile_to_module
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     return self._compile_to_module()
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]            ~~~~~~~~~~~~~~~~~~~~~~~^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/graph.py", line 2422, in _compile_to_module
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     self.codegen_with_cpp_wrapper() if self.cpp_wrapper else self.codegen()
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]                                                              ~~~~~~~~~~~~^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/graph.py", line 2358, in codegen
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     self.scheduler.codegen()
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~~~~~~~~~~~~~^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/scheduler.py", line 6019, in codegen
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     self._codegen_partitions()
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~~~~~~~~~~~~~~~^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/scheduler.py", line 6161, in _codegen_partitions
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     self._codegen(partition)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~~~~^^^^^^^^^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/scheduler.py", line 6266, in _codegen
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     self.get_backend(device).codegen_node(node)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/codegen/cpp.py", line 5295, in codegen_node
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     cpp_kernel_proxy = self.kernel_proxy_cls(kernel_group)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/codegen/cpp.py", line 4018, in __init__
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     self.picked_vec_isa: cpu_vec_isa.VecISA = cpu_vec_isa.pick_vec_isa()
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]                                               ~~~~~~~~~~~~~~~~~~~~~~~~^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/cpu_vec_isa.py", line 551, in pick_vec_isa
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     _valid_vec_isa_list: list[VecISA] = valid_vec_isa_list()
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]                                         ~~~~~~~~~~~~~~~~~~^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/cpu_vec_isa.py", line 538, in valid_vec_isa_list
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     isa_list.extend(
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~~~~~~^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         isa
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         ^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         for isa in supported_vec_isa_list
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         if all(flag in _cpu_supported_x86_isa for flag in str(isa).split()) and isa
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     )
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/cpu_vec_isa.py", line 541, in <genexpr>
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     if all(flag in _cpu_supported_x86_isa for flag in str(isa).split()) and isa
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]                                                                             ^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/cpu_vec_isa.py", line 284, in __bool__
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     if super().__bool__():
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]        ~~~~~~~~~~~~~~~~^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/cpu_vec_isa.py", line 223, in __bool__
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     if super().__bool__():
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]        ~~~~~~~~~~~~~~~~^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/cpu_vec_isa.py", line 144, in __bool__
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     return self.__bool__impl(config.cpp.vec_isa_ok)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]            ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/cpu_vec_isa.py", line 154, in __bool__impl
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     return self.check_build(VecISA._avx_code)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]            ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/cpu_vec_isa.py", line 104, in check_build
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     extra=_get_isa_dry_compile_fingerprint(self._arch_flags),
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/cpu_vec_isa.py", line 30, in _get_isa_dry_compile_fingerprint
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     compiler_info = get_compiler_version_info(get_cpp_compiler())
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]                                               ~~~~~~~~~~~~~~~~^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/cpp_builder.py", line 392, in get_cpp_compiler
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     compiler = cpp_compiler_search(search)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/cpp_builder.py", line 105, in cpp_compiler_search
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     raise exc.InvalidCxxCompiler
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035] torch._inductor.exc.InductorError: InvalidCxxCompiler: No working C++ compiler found in torch._inductor.config.cpp.cxx: (None, 'g++')
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035] 
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035] Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035] 
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035] Traceback (most recent call last):
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 1945, in __call__
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     result = self._inner_convert(
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         frame, cache_entry, hooks, frame_state, skip=skip + 1
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     )
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 707, in __call__
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     result = _compile(
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         frame.f_code,
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ...<16 lines>...
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         convert_frame_box=self._box,
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     )
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 1752, in _compile
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     guarded_code, tracer_output = compile_inner(code, one_graph, hooks)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]                                   ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_utils_internal.py", line 97, in wrapper_function
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     return function(*args, **kwargs)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 1433, in compile_inner
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     return _compile_inner(code, one_graph, hooks)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 1467, in _compile_inner
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     dynamo_output = compile_frame(
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         code,
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ...<11 lines>...
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         package=package,
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     )
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 1341, in compile_frame
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     bytecode, tracer_output = transform_code_object(code, transform)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]                               ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/bytecode_transformation.py", line 1600, in transform_code_object
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     tracer_output = transformations(instructions, code_options)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 1313, in transform
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     tracer_output = trace_frame(
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         code,
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ...<14 lines>...
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         package=package,
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     )
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 328, in _fn
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     return fn(*args, **kwargs)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 838, in trace_frame
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     run_tracer()
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 819, in run_tracer
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     tracer.run()
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/symbolic_convert.py", line 1654, in run
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     while self.step():
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]           ~~~~~~~~~^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/symbolic_convert.py", line 1334, in step
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     self.dispatch_table[inst.opcode](self, inst)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/symbolic_convert.py", line 4692, in RETURN_VALUE
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     self._return(inst)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~~~^^^^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/symbolic_convert.py", line 4674, in _return
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     all_stack_locals_metadata = self.output.compile_subgraph(
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         self,
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ...<4 lines>...
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         stack_pops=1 if inst.opname == "RETURN_VALUE" else 0,
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     )
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/output_graph.py", line 1700, in compile_subgraph
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/output_graph.py", line 2225, in compile_and_call_fx_graph
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     compiled_fn = self.call_user_compiler(gm, self.example_inputs())
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/output_graph.py", line 2377, in call_user_compiler
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     return self._call_user_compiler(gm, example_inputs)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]            ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/output_graph.py", line 2412, in _call_user_compiler
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     compiled_fn = compiler_fn(gm, example_inputs)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/repro/after_dynamo.py", line 156, in __call__
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     compiled_gm = compiler_fn(gm, example_inputs)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/__init__.py", line 2435, in __call__
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     return compile_fx(model_, inputs_, config_patches=self.config)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 2537, in compile_fx
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     return _maybe_wrap_and_compile_fx_main(
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         model_,
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ...<3 lines>...
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         ignore_shape_env,
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     )
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 2614, in _maybe_wrap_and_compile_fx_main
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     return _compile_fx_main(
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         model_,
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ...<3 lines>...
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         ignore_shape_env,
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     )
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 2823, in _compile_fx_main
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     raise e.remove_dynamo_frames() from None  # see TORCHDYNAMO_VERBOSE=1
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 1019, in _compile_fx_inner
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     raise InductorError(e, currentframe()).with_traceback(
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         e.__traceback__
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ) from None
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 1003, in _compile_fx_inner
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     mb_compiled_graph = fx_codegen_and_compile(
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         gm, example_inputs, inputs_to_check, **graph_kwargs
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     )
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 1766, in fx_codegen_and_compile
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     return scheme.codegen_and_compile(gm, example_inputs, inputs_to_check, graph_kwargs)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]            ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 1537, in codegen_and_compile
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     compiled_module = graph.compile_to_module()
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/graph.py", line 2416, in compile_to_module
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     return self._compile_to_module()
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]            ~~~~~~~~~~~~~~~~~~~~~~~^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/graph.py", line 2422, in _compile_to_module
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     self.codegen_with_cpp_wrapper() if self.cpp_wrapper else self.codegen()
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]                                                              ~~~~~~~~~~~~^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/graph.py", line 2358, in codegen
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     self.scheduler.codegen()
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~~~~~~~~~~~~~^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/scheduler.py", line 6019, in codegen
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     self._codegen_partitions()
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~~~~~~~~~~~~~~~^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/scheduler.py", line 6161, in _codegen_partitions
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     self._codegen(partition)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~~~~^^^^^^^^^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/scheduler.py", line 6266, in _codegen
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     self.get_backend(device).codegen_node(node)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/codegen/cpp.py", line 5295, in codegen_node
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     cpp_kernel_proxy = self.kernel_proxy_cls(kernel_group)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/codegen/cpp.py", line 4018, in __init__
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     self.picked_vec_isa: cpu_vec_isa.VecISA = cpu_vec_isa.pick_vec_isa()
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]                                               ~~~~~~~~~~~~~~~~~~~~~~~~^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/cpu_vec_isa.py", line 551, in pick_vec_isa
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     _valid_vec_isa_list: list[VecISA] = valid_vec_isa_list()
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]                                         ~~~~~~~~~~~~~~~~~~^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/cpu_vec_isa.py", line 538, in valid_vec_isa_list
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     isa_list.extend(
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~~~~~~^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         isa
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         ^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         for isa in supported_vec_isa_list
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         if all(flag in _cpu_supported_x86_isa for flag in str(isa).split()) and isa
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     )
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     ^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/cpu_vec_isa.py", line 541, in <genexpr>
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     if all(flag in _cpu_supported_x86_isa for flag in str(isa).split()) and isa
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]                                                                             ^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/cpu_vec_isa.py", line 284, in __bool__
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     if super().__bool__():
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]        ~~~~~~~~~~~~~~~~^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/cpu_vec_isa.py", line 223, in __bool__
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     if super().__bool__():
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]        ~~~~~~~~~~~~~~~~^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/cpu_vec_isa.py", line 144, in __bool__
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     return self.__bool__impl(config.cpp.vec_isa_ok)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]            ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/cpu_vec_isa.py", line 154, in __bool__impl
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     return self.check_build(VecISA._avx_code)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]            ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/cpu_vec_isa.py", line 104, in check_build
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     extra=_get_isa_dry_compile_fingerprint(self._arch_flags),
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/cpu_vec_isa.py", line 30, in _get_isa_dry_compile_fingerprint
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     compiler_info = get_compiler_version_info(get_cpp_compiler())
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]                                               ~~~~~~~~~~~~~~~~^^
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/cpp_builder.py", line 392, in get_cpp_compiler
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     compiler = cpp_compiler_search(search)
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/cpp_builder.py", line 105, in cpp_compiler_search
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035]     raise exc.InvalidCxxCompiler
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035] torch._inductor.exc.InductorError: InvalidCxxCompiler: No working C++ compiler found in torch._inductor.config.cpp.cxx: (None, 'g++')
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035] 
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035] Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
W0720 07:24:17.767000 5110 torch/_dynamo/convert_frame.py:2035] 

# Print out sufficiency output in a table format
output.to_dataframe()

shape: (10, 4)

step	TPR at 0.5 Fixed FPR	Accuracy	AUROC
i64	f64	f64	f64
25	0.752511	0.1644	0.706771
41	0.894779	0.4876	0.844997
69	0.96185	0.636	0.917065
116	0.973859	0.698	0.938911
193	0.988418	0.764	0.961014
322	0.991857	0.8428	0.978089
538	0.997422	0.8632	0.985359
898	0.999135	0.898	0.990626
1498	0.99808	0.9044	0.992273
2500	1.0	0.9348	0.995885

# Print out projected output values
output.project([1000, 2500, 5000])

shape: (3, 4)

step	TPR at 0.5 Fixed FPR	Accuracy	AUROC
i64	f64	f64	f64
1000	0.996978	0.894054	0.988713
2500	0.997275	0.910998	0.990752
5000	0.997327	0.917355	0.991312

# Plot the output using dataeval-plots library
for plot in dep.plot(output, backend="plotly"):
    display(plot)

Results¶

Using these learning curves, we can project performance under much larger datasets (with the same models).

Predicting sample requirements¶

We can also predict the amount of training samples required to achieve specific performance thresholds.

Let’s say we wanted to see how many samples are needed to hit 90%, 93%, and 99% accuracy, area under the receiver operating characteristic, and true positive rate at a fixed false positive rate of 0.5.

# Initialize the array of desired thresholds to apply to all metrics
output.inv_project([0.90, 0.93, 0.99])

shape: (3, 4)

target	TPR at 0.5 Fixed FPR	Accuracy	AUROC
f64	i64	i64	i64
0.9	42	1280	62
0.93	52	-1	85
0.99	183	-1	1572

With a value of “-1” samples, the projection shows that given the current model, hitting an accuracy of 99% is improbable.

How to measure dataset sufficiency for image classification¶

Problem statement¶

When to use¶

What you will need¶

Getting started¶

Load data and create model¶

Loading MNIST data¶

Creating a PyTorch model¶

Strategy protocols¶

Training strategy¶

Evaluation strategy¶

Reset strategy¶

Sufficiency config¶

Initialize sufficiency metric¶

Evaluate sufficiency¶

Results¶

Predicting sample requirements¶

See also¶

How-to guides¶

Tutorials¶

How to measure dataset sufficiency for image classification¶

Problem statement¶

When to use¶

What you will need¶

Getting started¶

Load data and create model¶

Loading MNIST data¶

Creating a PyTorch model¶

Strategy protocols¶

Training strategy¶

Evaluation strategy¶

Reset strategy¶

Sufficiency config¶

Initialize sufficiency metric¶

Evaluate sufficiency¶

Results¶

Predicting sample requirements¶

Related concepts¶

See also¶

How-to guides¶

Tutorials¶