org.tribuo.evaluation

Interface Evaluator<T extends Output<T>,E extends Evaluation<T>>

Type Parameters:

T - The output type.

E - The evaluation type.

All Known Implementing Classes:

AbstractEvaluator, AnomalyEvaluator, ClusteringEvaluator, LabelEvaluator, MultiLabelEvaluator, RegressionEvaluator

public interface Evaluator<T extends Output<T>,E extends Evaluation<T>>

An evaluation factory which produces immutable Evaluations of a given Dataset using the given Model.

If the dataset contains an unknown output (as generated by OutputFactory.getUnknownOutput()) or a valid output which is outside of the domain of the Model then the evaluate methods will throw IllegalArgumentException with an appropriate message.

Method Summary

Modifier and Type	Method and Description
default OnlineEvaluator<T,E>	createOnlineEvaluator(Model<T> model, DataProvenance provenance) Creates an online evaluator that maintains a list of all the predictions it has seen and can evaluate them upon request.
E	evaluate(Model<T> model, Dataset<T> dataset) Evaluates the dataset using the supplied model, returning an immutable Evaluation of the appropriate type.
E	evaluate(Model<T> model, DataSource<T> datasource) Evaluates the dataset using the supplied model, returning an immutable Evaluation of the appropriate type.
E	evaluate(Model<T> model, List<Prediction<T>> predictions, DataProvenance dataProvenance) Evaluates the model performance using the supplied predictions, returning an immutable Evaluation of the appropriate type.
default E	evaluate(Model<T> model, List<Prediction<T>> predictions, List<T> groundTruth, DataProvenance dataProvenance) Evaluates the model performance using the supplied predictions, returning an immutable Evaluation of the appropriate type.

Method Detail

evaluate

E evaluate(Model<T> model,
           Dataset<T> dataset)

Evaluates the dataset using the supplied model, returning an immutable Evaluation of the appropriate type.

Parameters:

model - The model to use.

dataset - The dataset to evaluate.

Returns:

An evaluation.

evaluate

E evaluate(Model<T> model,
           DataSource<T> datasource)

Evaluates the dataset using the supplied model, returning an immutable Evaluation of the appropriate type.

Parameters:

model - The model to use.

datasource - The data to evaluate.

Returns:

An evaluation.

evaluate

E evaluate(Model<T> model,
           List<Prediction<T>> predictions,
           DataProvenance dataProvenance)

Evaluates the model performance using the supplied predictions, returning an immutable Evaluation of the appropriate type.

It does not validate that the model produced the supplied predictions, or that the dataProvenance matches the input examples. Supplying arguments which do not meet these invariants will produce an invalid Evaluation.

Parameters:

model - The model to use.

predictions - The predictions to evaluate.

dataProvenance - The provenance of the predicted dataset.

Returns:

An evaluation.

evaluate

default E evaluate(Model<T> model,
                   List<Prediction<T>> predictions,
                   List<T> groundTruth,
                   DataProvenance dataProvenance)

Evaluates the model performance using the supplied predictions, returning an immutable Evaluation of the appropriate type.

This method is used when the predictions do not contain the correct ground truth labels (e.g., if they were collected separately from the examples constructed for prediction). First it creates a new set of predictions, containing the same examples with the matched ground truth prediction. Then it calls evaluate(Model, List, DataProvenance) with the updated predictions.

It does not validate that the model produced the supplied predictions, or that the dataProvenance matches the input examples. Supplying arguments which do not meet these invariants will produce an invalid Evaluation.

Parameters:

model - The model to use.

predictions - The predictions to evaluate.

groundTruth - The ground truth outputs to use.

dataProvenance - The provenance of the predicted dataset.

Returns:

An evaluation.

createOnlineEvaluator

default OnlineEvaluator<T,E> createOnlineEvaluator(Model<T> model,
                                                   DataProvenance provenance)

Creates an online evaluator that maintains a list of all the predictions it has seen and can evaluate them upon request.

Parameters:

model - The model to use for online evaluation.

provenance - The provenance of the data.

Returns:

An online evaluator.