DataEval

_images/DataEval_Logo.png

DataEval curates datasets to train and test performant, robust, unbiased and reliable AI models and monitors for data shifts that impact performance of deployed models.

Our Mission

DataEval is an effective, powerful, and reliable set of tools for any T&E engineer. Throughout all stages of the machine learning lifecycle, DataEval supports model development, data analysis, and monitoring with state-of-the-art algorithms to help you solve difficult problems. With a focus on computer vision tasks, DataEval provides simple, but effective metrics for performance estimation, bias detection, and dataset linting.

DataEval is easy to install, supports a wide range of Python versions, and is compatible with many of the most popular packages in the scientific and T&E communities.

DataEval also has native interopability between JATIC’s suite of tools when using MAITE-compliant datasets and models.

Key Features

DataEval provides many powerful tools to assist in the following T&E tasks:

  • Model-agnostic metrics that bound real-world performance

    • relevance/completeness/coverage

    • metafeatures (data complexity)

  • Model-specific metrics that guide model selection and training

    • dataset sufficiency

    • data/model complexity mismatch

  • Metrics for post-deployment monitoring of data with bounds on model performance to guide retraining

    • dataset-shift metrics

    • model performance bounds under covariate shift

    • guidance on sampling to assess model error and model retraining

Acknowledgement

CDAO Funding Acknowledgement

This material is based upon work supported by the Chief Digital and Artificial Intelligence Office under Contract No. W519TC-23-9-2033. The views and conclusions contained herein are those of the author(s) and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the U.S. Government.