DataEval documentation¶
Version: 1.0.0 | Date: Mar 09, 2026
DataEval analyzes datasets and models to give users the ability to train and test performant, unbiased, and reliable AI models and monitor data for impactful shifts to deployed models.
DataEval is an open-source software designed to help data scientists, developers, and T&E engineers develop and analyze computer vision datasets and the resulting impact on models.
Getting Started
New to DataEval? Check out the Beginner’s Guide. It contains an introduction to DataEval’s main concepts.
Tutorials
Learn DataEval through guided end-to-end examples.
Explanations
Learn about the key concepts related to using data in computer-vision AI applications.
How-to Guides
Task-based instructions and workflow recipes.
Reference
The reference guide contains a detailed description the DataEval API. The reference describes how the methods work and which parameters can be used. It assumes that you have an understanding of the key concepts.
Roles Guide
Not sure how DataEval fits into your daily routine? Check out our Roles Guide which shows some of the ways in which we use DataEval.
Why DataEval?¶
DataEval addresses the critical need underlying every AI model – the data. The difference between a great dataset and a poor dataset can have drastic consequences on AI model performance. Data collected in the wild is noisy, often imbalanced, and doesn’t always cover the entire spectrum of conditions need for deployment. DataEval provides AI practitioners with a library of rigorous, algorithm-backed metrics for performance estimation, bias analysis, dataset cleaning and assessment, and data distribution shifts. Throughout all stages of the machine learning lifecycle – from initial data collection through operational monitoring – DataEval identifies data problems before they become model failures.
DataEval is easy to install, supports a wide range of Python versions, and is compatible with many of the most popular packages in the scientific and T&E communities.
Key Features¶
DataEval empowers professionals across domains with tools designed to enhance their workflows. Explore capabilities specific to your role:
Metafeatures: Leverage metrics to analyze data complexity and improve data-driven decisions based on metadata features.
Real-World Insights: Improve dataset sampling and improve Balance, Completeness, and Coverage.
Model-Specific Metrics: Evaluate dataset Sufficiency and detect data/model complexity mismatches.
Performance Optimization: Establish bounds on real-world model performance for improved training strategies.
Responsive metrics: Optimize evaluation with tailored guidance for error assessment and retraining.
Robust Testing: Reduce errors with metrics that reliably work with state of the art image classification and object detection datasets
Post-Deployment Monitoring: Keep models on track with easy-to-implement logging of Operational Drift metrics
Drift Detection: Rapidly diagnose model degradation under Operational Drift to maintain model accuracy and stability.
Complete Shift Analysis: Quantify impactful changes in data due to Covariate Shift, Label Shift, and Concept Drift before they impact your model.
By incorporating DataEval into their workflows, data scientists, developers, and T&E engineers can thoroughly analyze and evaluate their datasets to maximize generalization and performance.
Acknowledgement¶
This material is based upon work supported by the Chief Digital and Artificial Intelligence Office under Contract No. W519TC-23-9-2033. The views and conclusions contained herein are those of the author(s) and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the U.S. Government.