dataeval.quality.Prioritize¶

class dataeval.quality.Prioritize(extractor=None, method=None, k=None, c=None, n_init=None, max_cluster_size=None, order=None, policy=None, num_bins=None, reference=None, config=None)¶

Prioritize dataset samples based on their position in the embedding space.

This class provides factory methods for common configurations and supports both direct instantiation and fluent policy configuration via the Output.

Parameters:¶

extractor : FeatureExtractor¶

Feature extractor instance to use for extracting embeddings from data.

method : {"knn", "kmeans_distance", "kmeans_complexity", "hdbscan_distance", "hdbscan_complexity"}, default "knn"¶

Ranking method to use:

• ”knn”: K-nearest neighbors distance ranking

• ”kmeans_distance”: Distance to assigned K-means cluster center

• ”kmeans_complexity”: Weighted sampling based on K-means cluster structure

• ”hdbscan_distance”: Distance to assigned HDBSCAN cluster center

• ”hdbscan_complexity”: Weighted sampling based on HDBSCAN cluster structure

k : int or None, default None¶

Number of nearest neighbors for “knn” method. If None, uses sqrt(n_samples).

c : int or None, default None¶

Number of clusters for clustering methods. If None, uses sqrt(n_samples).

n_init : int or "auto", default "auto"¶

Number of K-means initializations (kmeans methods only).

max_cluster_size : int or None, default None¶

Maximum cluster size for HDBSCAN methods.

order : {"easy_first", "hard_first"}, default "easy_first"¶

Sort direction for output indices:

• ”easy_first”: Prototypical samples first (low distance)

• ”hard_first”: Challenging samples first (high distance)

policy : {"difficulty", "stratified", "class_balanced"}, default "difficulty"¶

Selection policy:

• ”difficulty”: Direct ordering by ranking results (no additional reordering)

• ”stratified”: Balanced selection across difficulty bins

• ”class_balanced”: Balanced selection across class labels

num_bins : int, default 50¶

Number of bins for “stratified” policy.

reference : AnnotatedDataset or Array or None, default None¶

Optional reference dataset or pre-computed embeddings. When provided, incoming datasets will be prioritized relative to this reference set. Useful for active learning (reference = labeled data) or quality filtering (reference = high-quality corpus).

config : Prioritize.Config or None, default None¶

Optional configuration object with default parameters. Parameters specified directly in __init__ will override config defaults.

See also

Outliers, Indices, dataeval.core.rank_knn(), dataeval.core.rank_kmeans_distance(), dataeval.core.rank_kmeans_complexity(), dataeval.core.rank_hdbscan_distance(), dataeval.core.rank_hdbscan_complexity()

Examples

Using factory methods (recommended):

>>> from dataeval.quality import Prioritize
>>>
>>> # KNN with default policy (difficulty/easy_first)
>>> result = Prioritize.knn(extractor, k=10).evaluate(unlabeled_data)
>>>
>>> # Configure specific policy in factory
>>> result = Prioritize.knn(extractor, k=10).evaluate(unlabeled_data)
>>>
>>> # Re-bucket results (Cheap operation)
>>> stratified_res = result.stratified(num_bins=20)
>>> hard_res = result.hard_first()

Direct instantiation:

>>> prioritizer = Prioritize(
...     extractor=extractor,
...     method="knn",
...     k=10,
...     policy="stratified",
...     num_bins=20,
... )
>>> result = prioritizer.evaluate(unlabeled_data)

Active learning with reference data:

>>> # Prioritize unlabeled data based on distance to labeled data
>>> prioritizer = Prioritize.knn(extractor, k=10, reference=labeled_data)
>>> result = prioritizer.evaluate(unlabeled_data)
>>> # Get the items most unlike the reference data
>>> most_novel = result.hard_first().indices

Using configuration:

>>> config = Prioritize.Config(extractor=extractor, method="knn", k=10)
>>> prioritizer = Prioritize(config=config)

Applying class-balanced policy with class labels from metadata:

>>> prioritizer = Prioritize.knn(extractor, k=5)
>>> # evaluate() extracts labels from dataset metadata automatically
>>> result = prioritizer.evaluate(unlabeled_data)
>>> # Apply balancing
>>> balanced = result.class_balanced()

evaluate(dataset, class_labels=None)¶

Evaluate the dataset and return prioritized indices.

Uses the configured method and policy to rank samples.

Parameters:¶

dataset : AnnotatedDataset[Any] | Array¶

The incoming dataset to prioritize. Can be either:

• AnnotatedDataset: Will compute embeddings using the extractor

• Array: Pre-computed embeddings (e.g. from Embeddings or numpy)

class_labels : NDArray[np.integer] | None, default None¶

Optional class labels for class_balanced policy. If not provided, will attempt to extract from dataset metadata.

Returns:¶

Output containing prioritized indices, scores (if available), and configuration information.

Return type:¶

PriorityOutput

Raises:¶

• ValueError – If class_balanced policy is used with a dataset that lacks metadata (e.g., raw arrays). If stratified policy is used with complexity methods (no scores).

• TypeError – If dataset is neither an AnnotatedDataset nor Array.

Examples

Using factory methods:

>>> result = Prioritize.knn(extractor, k=5).evaluate(unlabeled_data)

Using direct instantiation:

>>> prioritizer = Prioritize(extractor=extractor, method="knn", k=5, order="hard_first")
>>> result = prioritizer.evaluate(unlabeled_data)

classmethod hdbscan_complexity(extractor, c=None, max_cluster_size=None, reference=None)¶

Create a Prioritize instance using HDBSCAN complexity method.

Uses weighted sampling based on intra/inter-cluster distances from HDBSCAN clustering. Returns samples in easy-first order.

Note: This method does not produce scores, so “stratified” policy is not available.

Parameters:¶

extractor : FeatureExtractor¶

Feature extractor instance for embedding extraction.

c : int or None, default None¶

Expected number of clusters (used as hint for min_cluster_size). If None, uses sqrt(n_samples).

max_cluster_size : int or None, default None¶

Maximum size limit for identified clusters.

reference : AnnotatedDataset or Array or None, default None¶

Optional reference dataset for relative prioritization.

Returns:¶

Configured instance ready for evaluation.

Return type:¶

Prioritize

Examples

>>> result = Prioritize.hdbscan_complexity(extractor, c=10).evaluate(unlabeled_data)

classmethod hdbscan_distance(extractor, c=None, max_cluster_size=None, reference=None)¶

Create a Prioritize instance using HDBSCAN distance method.

Clusters embeddings using HDBSCAN and ranks by distance to assigned cluster centers. Returns samples in easy-first order (low distance = prototypical).

Parameters:¶

extractor : FeatureExtractor¶

Feature extractor instance for embedding extraction.

c : int or None, default None¶

Expected number of clusters (used as hint for min_cluster_size). If None, uses sqrt(n_samples).

max_cluster_size : int or None, default None¶

Maximum size limit for identified clusters.

reference : AnnotatedDataset or Array or None, default None¶

Optional reference dataset for relative prioritization.

Returns:¶

Configured instance ready for evaluation.

Return type:¶

Prioritize

Examples

>>> result = Prioritize.hdbscan_distance(extractor, c=15).evaluate(unlabeled_data)

classmethod kmeans_complexity(extractor, c=None, n_init='auto', reference=None)¶

Create a Prioritize instance using K-means complexity method.

Uses weighted sampling based on intra/inter-cluster distances. Returns samples in easy-first order.

Note: This method does not produce scores, so “stratified” policy is not available.

Parameters:¶

extractor : FeatureExtractor¶

Feature extractor instance for embedding extraction.

c : int or None, default None¶

Number of clusters. If None, uses sqrt(n_samples).

n_init : int or "auto", default "auto"¶

Number of K-means initializations.

reference : AnnotatedDataset or Array or None, default None¶

Optional reference dataset for relative prioritization.

Returns:¶

Configured instance ready for evaluation.

Return type:¶

Prioritize

Examples

>>> result = Prioritize.kmeans_complexity(extractor, c=10).evaluate(unlabeled_data)

classmethod kmeans_distance(extractor, c=None, n_init='auto', reference=None)¶

Create a Prioritize instance using K-means distance method.

Ranks samples by distance to their assigned cluster centers. Returns samples in easy-first order (low distance = prototypical).

Parameters:¶

extractor : FeatureExtractor¶

Feature extractor instance for embedding extraction.

c : int or None, default None¶

Number of clusters. If None, uses sqrt(n_samples).

n_init : int or "auto", default "auto"¶

Number of K-means initializations.

reference : AnnotatedDataset or Array or None, default None¶

Optional reference dataset for relative prioritization.

Returns:¶

Configured instance ready for evaluation.

Return type:¶

Prioritize

Examples

>>> result = Prioritize.kmeans_distance(extractor, c=15).evaluate(unlabeled_data)

classmethod knn(extractor, k=None, reference=None)¶

Create a Prioritize instance using k-nearest neighbors method.

Parameters:¶

extractor : FeatureExtractor¶

Feature extractor instance for embedding extraction.

k : int or None, default None¶

Number of nearest neighbors. If None, uses sqrt(n_samples).

reference : AnnotatedDataset or Array or None, default None¶

Optional reference dataset for relative prioritization.

Returns:¶

Configured instance ready for evaluation.

Return type:¶

Prioritize

Examples

>>> # Default KNN
>>> result = Prioritize.knn(extractor, k=10).evaluate(unlabeled_data)
>>> # KNN relative to a reference (Active Learning)
>>> result = Prioritize.knn(extractor, k=5, reference=labeled_data).evaluate(unlabeled_data)

Classes¶