How to measure dataset sufficiency for image classification¶

This guide walks you through analyzing an image classification model’s hypothetical performance.

Estimated time to complete: 10 minutes

Relevant personas: ML Engineer

Problem statement¶

For machine learning tasks, often we would like to evaluate the performance of a model on a small, preliminary dataset. In situations where data collection is expensive, we would like to extrapolate hypothetical performance out to a larger dataset.

DataEval has introduced a method projecting performance via sufficiency curves.

When to use¶

The Sufficiency class should be used when you would like to extrapolate hypothetical performance. For example, if you have a small dataset, and would like to know if it is worthwhile to collect more data.

What you will need¶

A particular model architecture.
Metric(s) that we would like to evaluate.
A dataset of interest.
A Python environment with the following packages installed:
- tabulate

Getting started¶

Let’s import the required libraries needed to set up a minimal working example

import os
from collections.abc import Sequence
from typing import Any, cast

import dataeval_plots as dep
import numpy as np
import plotly.io as pio
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torchmetrics
from IPython.display import display
from maite_datasets.image_classification import MNIST
from numpy.typing import NDArray
from torch.utils.data import DataLoader, Subset
from torch.utils.data import Dataset as TorchDataset

from dataeval import config
from dataeval.performance import Sufficiency
from dataeval.protocols import Dataset, DatumMetadata
from dataeval.selection import Limit, Select

DatumType = tuple[NDArray[np.number[Any]], NDArray[np.number[Any]], DatumMetadata]

# Set seed for reproducibility
config.set_seed(0, all_generators=True)

# Set hardware based on system
device = "cuda" if torch.cuda.is_available() else "cpu"
config.set_device(device=device)

# Additional reproducibility and printing options
np.set_printoptions(formatter={"float": lambda x: f"{x:0.4f}"})
torch.set_float32_matmul_precision("high")
torch._dynamo.config.suppress_errors = True
torch.use_deterministic_algorithms(True)
os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"

# Use plotly to render plots
dep.set_default_backend("plotly")

# Use the notebook renderer so JS is embedded
pio.renderers.default = "notebook"

Load data and create model¶

Before calculating the sufficiency of a dataset, the dataset must be loaded and the model architecture defined. We will walk through these in the following steps.

Loading MNIST data¶

Load the MNIST data and split it into training and test datasets. For this notebook, we will use subsets of the training (2500) and test (500) data.

# Configure the dataset transforms

transforms = [
    lambda x: x / 255.0,  # scale to [0, 1]
    lambda x: x.astype(np.float32),  # convert to float32
]

# Download the mnist dataset and apply the transforms and subset the data
train_ds = Select(MNIST("./data", image_set="train", transforms=transforms, download=True), selections=[Limit(2500)])
test_ds = Select(MNIST("./data", image_set="test", transforms=transforms, download=True), selections=[Limit(500)])

Creating a PyTorch model¶

Next, we define the network architecture that will be trained and then evaluated throughout the sufficiency calculation.

# Define our network architecture
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(6400, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        x = torch.flatten(x, 1)  # flatten all dimensions except batch
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x


# Compile the model (cast sets the type to Net as compile returns an Unknown)
model: Net = cast(Net, torch.compile(Net().to(device)))

Strategy protocols¶

Training and evaluation functions are heavily dependent on the hyperparameters defined by a user. These can include metrics, loss functions, optimizers, model architectures, input sizes, etc.

To allow the Sufficiency class to handle this situation, DataEval uses Protocols. Sufficiency requires two specific protocols called TrainingStrategy and EvaluationStrategy.
Below we will define the strategies that align with this notebook and combine them into a Sufficiency.Config that can be given to the Sufficiency class.

Training strategy¶

class MNISTTrainingStrategy:
    def train(self, model: nn.Module, dataset: Dataset[DatumType], indices: Sequence[int]):
        # Defined only for this testing scenario
        criterion = torch.nn.CrossEntropyLoss().to(device)
        optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)
        epochs = 10

        # Define the dataloader for training
        dataloader = DataLoader(Subset(cast(TorchDataset, dataset), indices), batch_size=8)

        for _epoch in range(epochs):
            for batch in dataloader:
                # Load data/images to device
                X = torch.Tensor(batch[0]).to(device)
                # Load one-hot encoded targets/labels to device
                y = torch.argmax(torch.asarray(batch[1], dtype=torch.int).to(device), dim=1)
                # Zero out gradients
                optimizer.zero_grad()
                # Forward propagation
                outputs = model(X)
                # Compute loss
                loss = criterion(outputs, y)
                # Back prop
                loss.backward()
                # Update weights/parameters
                optimizer.step()

Evaluation strategy¶

class MNISTEvaluationStrategy:
    def evaluate(self, model: nn.Module, dataset: Dataset[DatumType]) -> dict[str, float]:
        # Metrics of interest
        metrics = {
            "Accuracy": torchmetrics.Accuracy(task="multiclass", num_classes=10).to(device),
            "AUROC": torchmetrics.AUROC(task="multiclass", num_classes=10).to(device),
            "TPR at 0.5 Fixed FPR": torchmetrics.ROC(task="multiclass", average="macro", num_classes=10).to(device),
        }
        result = {}
        # Set model layers into evaluation mode
        model.eval()
        dataloader = DataLoader(cast(TorchDataset, dataset), batch_size=8)
        # Tell PyTorch to not track gradients, greatly speeds up processing
        with torch.no_grad():
            for batch in dataloader:
                # Load data/images to device
                X = torch.Tensor(batch[0]).to(device)
                # Load one-hot encoded targets/labels to device
                y = torch.argmax(torch.asarray(batch[1], dtype=torch.int).to(device), dim=1)
                preds = model(X)
                for metric in metrics.values():
                    metric.update(preds, y)
            # Compute ROC curve
            false_positive_rate, true_positive_rate, _ = metrics["TPR at 0.5 Fixed FPR"].compute()
            # determine interval to examine
            desired_rate = 0.5
            closest_desired_index = torch.argmin(torch.abs(false_positive_rate - desired_rate)).item()
            # return corresponding tpr value
            result["TPR at 0.5 Fixed FPR"] = true_positive_rate[closest_desired_index].cpu()
            result["Accuracy"] = metrics["Accuracy"].compute().cpu()
            result["AUROC"] = metrics["AUROC"].compute().cpu()
        return result

Reset strategy¶

The Sufficiency class requires a reset_strategy that resets the model’s parameters between runs. This ensures each run starts from a fresh initialization. Here’s a simple implementation for PyTorch models:

def reset_model(model: nn.Module) -> nn.Module:
    """Reset all parameters in a PyTorch model."""

    @torch.no_grad()
    def weight_reset(m: nn.Module) -> None:
        reset_fn = getattr(m, "reset_parameters", None)
        if callable(reset_fn):
            reset_fn()

    return model.apply(fn=weight_reset)

Sufficiency config¶

Do not forget to initialize your strategy classes!

mnist_config = Sufficiency.Config(
    training_strategy=MNISTTrainingStrategy(),
    evaluation_strategy=MNISTEvaluationStrategy(),
    reset_strategy=reset_model,
    runs=5,
    substeps=10,
)

Initialize sufficiency metric¶

Attach the custom training and evaluation functions to the Sufficiency metric and define the number of models to train in parallel (stability), as well as the number of steps along the learning curve to evaluate.

# Instantiate sufficiency metric
suff = Sufficiency(
    model=model,
    config=mnist_config,
)

Evaluate sufficiency¶

Now we can evaluate the metric to train the models and produce the learning curve.

# Train & test model
output = suff.evaluate(train_ds, test_ds)

W0603 23:18:58.825000 6980 torch/_inductor/utils.py:1679] [0/0] Not enough SMs to use max_autotune_gemm mode

W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035] WON'T CONVERT forward /tmp/ipykernel_6980/1189723111.py line 11 
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035] due to: 
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035] Traceback (most recent call last):
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 1945, in __call__
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     result = self._inner_convert(
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         frame, cache_entry, hooks, frame_state, skip=skip + 1
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     )
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 707, in __call__
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     result = _compile(
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         frame.f_code,
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ...<16 lines>...
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         convert_frame_box=self._box,
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     )
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 1752, in _compile
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     guarded_code, tracer_output = compile_inner(code, one_graph, hooks)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]                                   ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_utils_internal.py", line 97, in wrapper_function
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     return function(*args, **kwargs)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 1433, in compile_inner
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     return _compile_inner(code, one_graph, hooks)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 1467, in _compile_inner
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     dynamo_output = compile_frame(
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         code,
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ...<11 lines>...
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         package=package,
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     )
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 1341, in compile_frame
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     bytecode, tracer_output = transform_code_object(code, transform)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]                               ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/bytecode_transformation.py", line 1600, in transform_code_object
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     tracer_output = transformations(instructions, code_options)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 1313, in transform
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     tracer_output = trace_frame(
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         code,
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ...<14 lines>...
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         package=package,
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     )
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 328, in _fn
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     return fn(*args, **kwargs)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 838, in trace_frame
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     run_tracer()
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 819, in run_tracer
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     tracer.run()
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/symbolic_convert.py", line 1654, in run
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     while self.step():
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]           ~~~~~~~~~^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/symbolic_convert.py", line 1334, in step
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     self.dispatch_table[inst.opcode](self, inst)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/symbolic_convert.py", line 4692, in RETURN_VALUE
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     self._return(inst)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~~~^^^^^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/symbolic_convert.py", line 4674, in _return
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     all_stack_locals_metadata = self.output.compile_subgraph(
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         self,
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ...<4 lines>...
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         stack_pops=1 if inst.opname == "RETURN_VALUE" else 0,
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     )
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/output_graph.py", line 1700, in compile_subgraph
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/output_graph.py", line 2225, in compile_and_call_fx_graph
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     compiled_fn = self.call_user_compiler(gm, self.example_inputs())
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/output_graph.py", line 2377, in call_user_compiler
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     return self._call_user_compiler(gm, example_inputs)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]            ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/output_graph.py", line 2412, in _call_user_compiler
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     compiled_fn = compiler_fn(gm, example_inputs)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/repro/after_dynamo.py", line 156, in __call__
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     compiled_gm = compiler_fn(gm, example_inputs)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/__init__.py", line 2435, in __call__
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     return compile_fx(model_, inputs_, config_patches=self.config)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 2537, in compile_fx
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     return _maybe_wrap_and_compile_fx_main(
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         model_,
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ...<3 lines>...
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         ignore_shape_env,
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     )
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 2614, in _maybe_wrap_and_compile_fx_main
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     return _compile_fx_main(
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         model_,
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ...<3 lines>...
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         ignore_shape_env,
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     )
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 2823, in _compile_fx_main
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     raise e.remove_dynamo_frames() from None  # see TORCHDYNAMO_VERBOSE=1
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 1019, in _compile_fx_inner
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     raise InductorError(e, currentframe()).with_traceback(
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         e.__traceback__
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ) from None
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 1003, in _compile_fx_inner
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     mb_compiled_graph = fx_codegen_and_compile(
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         gm, example_inputs, inputs_to_check, **graph_kwargs
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     )
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 1766, in fx_codegen_and_compile
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     return scheme.codegen_and_compile(gm, example_inputs, inputs_to_check, graph_kwargs)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]            ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 1537, in codegen_and_compile
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     compiled_module = graph.compile_to_module()
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/graph.py", line 2416, in compile_to_module
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     return self._compile_to_module()
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]            ~~~~~~~~~~~~~~~~~~~~~~~^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/graph.py", line 2422, in _compile_to_module
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     self.codegen_with_cpp_wrapper() if self.cpp_wrapper else self.codegen()
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]                                                              ~~~~~~~~~~~~^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/graph.py", line 2358, in codegen
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     self.scheduler.codegen()
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~~~~~~~~~~~~~^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/scheduler.py", line 6019, in codegen
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     self._codegen_partitions()
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~~~~~~~~~~~~~~~^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/scheduler.py", line 6161, in _codegen_partitions
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     self._codegen(partition)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~~~~^^^^^^^^^^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/scheduler.py", line 6266, in _codegen
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     self.get_backend(device).codegen_node(node)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/codegen/cuda_combined_scheduling.py", line 130, in codegen_node
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     return self._triton_scheduling.codegen_node(node)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/codegen/simd.py", line 1822, in codegen_node
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     return self._codegen_nodes(nodes, coalesce_analysis)  # type: ignore[arg-type]
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]            ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/codegen/simd.py", line 1795, in _codegen_nodes
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     return self.codegen_node_schedule(
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]            ~~~~~~~~~~~~~~~~~~~~~~~~~~^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         SIMDKernelFeatures(node_schedule, numel, rnumel, coalesce_analysis)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     )
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/codegen/simd.py", line 1885, in codegen_node_schedule
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     src_code = kernel.codegen_kernel()
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/codegen/triton.py", line 5295, in codegen_kernel
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     **self.inductor_meta_common(),
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]       ~~~~~~~~~~~~~~~~~~~~~~~~~^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/codegen/triton.py", line 5097, in inductor_meta_common
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     "backend_hash": torch.utils._triton.triton_hash_with_backend(),
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/utils/_triton.py", line 200, in triton_hash_with_backend
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     backend = triton_backend()
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/utils/_triton.py", line 192, in triton_backend
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     target = driver.active.get_current_target()
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]              ^^^^^^^^^^^^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/triton/runtime/driver.py", line 28, in active
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     self._active = self.default
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]                    ^^^^^^^^^^^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/triton/runtime/driver.py", line 22, in default
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     self._default = _create_driver()
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]                     ~~~~~~~~~~~~~~^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/triton/runtime/driver.py", line 10, in _create_driver
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     return active_drivers[0]()
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]            ~~~~~~~~~~~~~~~~~^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/triton/backends/nvidia/driver.py", line 720, in __init__
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     self.utils = CudaUtils()  # TODO: make static
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]                  ~~~~~~~~~^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/triton/backends/nvidia/driver.py", line 62, in __init__
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     mod = compile_module_from_src(
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         src=Path(os.path.join(dirname, "driver.c")).read_text(),
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ...<3 lines>...
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         libraries=libraries,
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     )
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/triton/runtime/build.py", line 93, in compile_module_from_src
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     so = _build(name, src_path, tmpdir, library_dirs or [], include_dirs or [], libraries or [], ccflags or [])
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/triton/runtime/build.py", line 32, in _build
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     raise RuntimeError(
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         "Failed to find C compiler. Please specify via CC environment variable or set triton.knobs.build.impl.")
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035] torch._inductor.exc.InductorError: RuntimeError: Failed to find C compiler. Please specify via CC environment variable or set triton.knobs.build.impl.
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035] 
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035] Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035] 
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035] Traceback (most recent call last):
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 1945, in __call__
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     result = self._inner_convert(
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         frame, cache_entry, hooks, frame_state, skip=skip + 1
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     )
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 707, in __call__
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     result = _compile(
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         frame.f_code,
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ...<16 lines>...
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         convert_frame_box=self._box,
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     )
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 1752, in _compile
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     guarded_code, tracer_output = compile_inner(code, one_graph, hooks)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]                                   ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_utils_internal.py", line 97, in wrapper_function
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     return function(*args, **kwargs)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 1433, in compile_inner
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     return _compile_inner(code, one_graph, hooks)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 1467, in _compile_inner
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     dynamo_output = compile_frame(
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         code,
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ...<11 lines>...
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         package=package,
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     )
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 1341, in compile_frame
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     bytecode, tracer_output = transform_code_object(code, transform)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]                               ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/bytecode_transformation.py", line 1600, in transform_code_object
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     tracer_output = transformations(instructions, code_options)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 1313, in transform
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     tracer_output = trace_frame(
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         code,
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ...<14 lines>...
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         package=package,
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     )
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 328, in _fn
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     return fn(*args, **kwargs)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 838, in trace_frame
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     run_tracer()
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/convert_frame.py", line 819, in run_tracer
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     tracer.run()
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/symbolic_convert.py", line 1654, in run
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     while self.step():
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]           ~~~~~~~~~^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/symbolic_convert.py", line 1334, in step
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     self.dispatch_table[inst.opcode](self, inst)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/symbolic_convert.py", line 4692, in RETURN_VALUE
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     self._return(inst)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~~~^^^^^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/symbolic_convert.py", line 4674, in _return
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     all_stack_locals_metadata = self.output.compile_subgraph(
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         self,
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ...<4 lines>...
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         stack_pops=1 if inst.opname == "RETURN_VALUE" else 0,
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     )
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/output_graph.py", line 1700, in compile_subgraph
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/output_graph.py", line 2225, in compile_and_call_fx_graph
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     compiled_fn = self.call_user_compiler(gm, self.example_inputs())
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/output_graph.py", line 2377, in call_user_compiler
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     return self._call_user_compiler(gm, example_inputs)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]            ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/output_graph.py", line 2412, in _call_user_compiler
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     compiled_fn = compiler_fn(gm, example_inputs)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_dynamo/repro/after_dynamo.py", line 156, in __call__
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     compiled_gm = compiler_fn(gm, example_inputs)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/__init__.py", line 2435, in __call__
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     return compile_fx(model_, inputs_, config_patches=self.config)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 2537, in compile_fx
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     return _maybe_wrap_and_compile_fx_main(
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         model_,
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ...<3 lines>...
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         ignore_shape_env,
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     )
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 2614, in _maybe_wrap_and_compile_fx_main
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     return _compile_fx_main(
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         model_,
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ...<3 lines>...
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         ignore_shape_env,
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     )
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 2823, in _compile_fx_main
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     raise e.remove_dynamo_frames() from None  # see TORCHDYNAMO_VERBOSE=1
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 1019, in _compile_fx_inner
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     raise InductorError(e, currentframe()).with_traceback(
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         e.__traceback__
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ) from None
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 1003, in _compile_fx_inner
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     mb_compiled_graph = fx_codegen_and_compile(
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         gm, example_inputs, inputs_to_check, **graph_kwargs
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     )
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 1766, in fx_codegen_and_compile
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     return scheme.codegen_and_compile(gm, example_inputs, inputs_to_check, graph_kwargs)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]            ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/compile_fx.py", line 1537, in codegen_and_compile
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     compiled_module = graph.compile_to_module()
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/graph.py", line 2416, in compile_to_module
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     return self._compile_to_module()
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]            ~~~~~~~~~~~~~~~~~~~~~~~^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/graph.py", line 2422, in _compile_to_module
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     self.codegen_with_cpp_wrapper() if self.cpp_wrapper else self.codegen()
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]                                                              ~~~~~~~~~~~~^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/graph.py", line 2358, in codegen
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     self.scheduler.codegen()
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~~~~~~~~~~~~~^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/scheduler.py", line 6019, in codegen
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     self._codegen_partitions()
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~~~~~~~~~~~~~~~^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/scheduler.py", line 6161, in _codegen_partitions
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     self._codegen(partition)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~~~~^^^^^^^^^^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/scheduler.py", line 6266, in _codegen
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     self.get_backend(device).codegen_node(node)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/codegen/cuda_combined_scheduling.py", line 130, in codegen_node
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     return self._triton_scheduling.codegen_node(node)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/codegen/simd.py", line 1822, in codegen_node
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     return self._codegen_nodes(nodes, coalesce_analysis)  # type: ignore[arg-type]
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]            ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/codegen/simd.py", line 1795, in _codegen_nodes
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     return self.codegen_node_schedule(
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]            ~~~~~~~~~~~~~~~~~~~~~~~~~~^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         SIMDKernelFeatures(node_schedule, numel, rnumel, coalesce_analysis)
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     )
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/codegen/simd.py", line 1885, in codegen_node_schedule
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     src_code = kernel.codegen_kernel()
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/codegen/triton.py", line 5295, in codegen_kernel
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     **self.inductor_meta_common(),
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]       ~~~~~~~~~~~~~~~~~~~~~~~~~^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/_inductor/codegen/triton.py", line 5097, in inductor_meta_common
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     "backend_hash": torch.utils._triton.triton_hash_with_backend(),
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/utils/_triton.py", line 200, in triton_hash_with_backend
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     backend = triton_backend()
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/torch/utils/_triton.py", line 192, in triton_backend
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     target = driver.active.get_current_target()
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]              ^^^^^^^^^^^^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/triton/runtime/driver.py", line 28, in active
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     self._active = self.default
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]                    ^^^^^^^^^^^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/triton/runtime/driver.py", line 22, in default
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     self._default = _create_driver()
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]                     ~~~~~~~~~~~~~~^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/triton/runtime/driver.py", line 10, in _create_driver
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     return active_drivers[0]()
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]            ~~~~~~~~~~~~~~~~~^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/triton/backends/nvidia/driver.py", line 720, in __init__
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     self.utils = CudaUtils()  # TODO: make static
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]                  ~~~~~~~~~^^
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/triton/backends/nvidia/driver.py", line 62, in __init__
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     mod = compile_module_from_src(
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         src=Path(os.path.join(dirname, "driver.c")).read_text(),
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     ...<3 lines>...
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         libraries=libraries,
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     )
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/triton/runtime/build.py", line 93, in compile_module_from_src
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     so = _build(name, src_path, tmpdir, library_dirs or [], include_dirs or [], libraries or [], ccflags or [])
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]   File "/builds/jatic/aria/dataeval/.nox/docs-3-13/lib/python3.13/site-packages/triton/runtime/build.py", line 32, in _build
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]     raise RuntimeError(
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035]         "Failed to find C compiler. Please specify via CC environment variable or set triton.knobs.build.impl.")
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035] torch._inductor.exc.InductorError: RuntimeError: Failed to find C compiler. Please specify via CC environment variable or set triton.knobs.build.impl.
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035] 
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035] Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
W0603 23:18:59.017000 6980 torch/_dynamo/convert_frame.py:2035] 

# Print out sufficiency output in a table format
output.to_dataframe()

shape: (10, 4)

step	TPR at 0.5 Fixed FPR	Accuracy	AUROC
i64	f64	f64	f64
25	0.785096	0.226	0.745275
41	0.90888	0.5144	0.851819
69	0.940229	0.6012	0.891308
116	0.975251	0.7124	0.94178
193	0.989772	0.7832	0.963517
322	0.994541	0.8288	0.975333
538	0.996286	0.8832	0.98551
898	0.998319	0.898	0.989946
1498	1.0	0.9212	0.994204
2500	0.9995	0.9304	0.995371

# Print out projected output values
output.project([1000, 2500, 5000])

shape: (3, 4)

step	TPR at 0.5 Fixed FPR	Accuracy	AUROC
i64	f64	f64	f64
1000	0.997891	0.902229	0.991013
2500	0.998557	0.924854	0.995513
5000	0.998705	0.934231	0.997089

# Plot the output using dataeval-plots library
for plot in dep.plot(output, backend="plotly"):
    display(plot)

Results¶

Using these learning curves, we can project performance under much larger datasets (with the same models).

Predicting sample requirements¶

We can also predict the amount of training samples required to achieve specific performance thresholds.

Let’s say we wanted to see how many samples are needed to hit 90%, 93%, and 99% accuracy, area under the receiver operating characteristic, and true positive rate at a fixed false positive rate of 0.5.

# Initialize the array of desired thresholds to apply to all metrics
output.inv_project([0.90, 0.93, 0.99])

shape: (3, 4)

target	TPR at 0.5 Fixed FPR	Accuracy	AUROC
f64	i64	i64	i64
0.9	41	937	67
0.93	53	3499	98
0.99	214	-1	878

With a value of “-1” samples, the projection shows that given the current model, hitting an accuracy of 99% is improbable.

How to measure dataset sufficiency for image classification¶

Problem statement¶

When to use¶

What you will need¶

Getting started¶

Load data and create model¶

Loading MNIST data¶

Creating a PyTorch model¶

Strategy protocols¶

Training strategy¶

Evaluation strategy¶

Reset strategy¶

Sufficiency config¶

Initialize sufficiency metric¶

Evaluate sufficiency¶

Results¶

Predicting sample requirements¶

See also¶

How-to guides¶

Tutorials¶

How to measure dataset sufficiency for image classification¶

Problem statement¶

When to use¶

What you will need¶

Getting started¶

Load data and create model¶

Loading MNIST data¶

Creating a PyTorch model¶

Strategy protocols¶

Training strategy¶

Evaluation strategy¶

Reset strategy¶

Sufficiency config¶

Initialize sufficiency metric¶

Evaluate sufficiency¶

Results¶

Predicting sample requirements¶

Related concepts¶

See also¶

How-to guides¶

Tutorials¶