Class AbstractSequenceEvaluator<T extends Output<T>, C extends MetricContext<T>, E extends SequenceEvaluation<T>, M extends EvaluationMetric<T,C>>

java.lang.Object

org.tribuo.sequence.AbstractSequenceEvaluator<T,C,E,M>

All Implemented Interfaces:

SequenceEvaluator<T,E>

Direct Known Subclasses:

LabelSequenceEvaluator

public abstract class AbstractSequenceEvaluator<T extends Output<T>, C extends MetricContext<T>, E extends SequenceEvaluation<T>, M extends EvaluationMetric<T,C>>
extends Object
implements SequenceEvaluator<T,E>

Base class for sequence evaluators.

Constructor Summary

Constructors

Constructor	Description
AbstractSequenceEvaluator()

Method Summary

Modifier and Type	Method	Description
protected Map<MetricID<T>, Double>	computeResults(C ctx, Set<? extends EvaluationMetric<T,C>> metrics)	Computes each metric given the context.
protected abstract C	createContext(SequenceModel<T> model, List<List<Prediction<T>>> predictions)	Create the context needed for evaluation.
protected abstract E	createEvaluation(C context, Map<MetricID<T>, Double> results, EvaluationProvenance provenance)	Create an evaluation for the given results
protected abstract Set<M>	createMetrics(SequenceModel<T> model)	Creates the appropriate set of metrics for this model, by querying for it's OutputInfo.
final E	evaluate(SequenceModel<T> model, List<List<Prediction<T>>> predictions, DataProvenance dataProvenance)	Produces an evaluation for the supplied model and predictions by aggregating the appropriate statistics.
final E	evaluate(SequenceModel<T> model, SequenceDataset<T> dataset)	Produces an evaluation for the supplied model and dataset, by calling SequenceModel.predict(org.tribuo.sequence.SequenceExample<T>) to create the predictions, then aggregating the appropriate statistics.
final E	evaluate(SequenceModel<T> model, SequenceDataSource<T> datasource)	Produces an evaluation for the supplied model and datasource, by calling SequenceModel.predict(org.tribuo.sequence.SequenceExample<T>) to create the predictions, then aggregating the appropriate statistics.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

AbstractSequenceEvaluator

public AbstractSequenceEvaluator()

Method Details

evaluate

public final E evaluate(SequenceModel<T> model,
 SequenceDataset<T> dataset)

Produces an evaluation for the supplied model and dataset, by calling SequenceModel.predict(org.tribuo.sequence.SequenceExample<T>) to create the predictions, then aggregating the appropriate statistics.

Specified by:

evaluate in interface SequenceEvaluator<T extends Output<T>, C extends MetricContext<T>>

Parameters:

model - The model to use.

dataset - The dataset to make predictions for.

Returns:

An evaluation of the dataset on the model.

evaluate

public final E evaluate(SequenceModel<T> model,
 SequenceDataSource<T> datasource)

Produces an evaluation for the supplied model and datasource, by calling SequenceModel.predict(org.tribuo.sequence.SequenceExample<T>) to create the predictions, then aggregating the appropriate statistics.

Specified by:

evaluate in interface SequenceEvaluator<T extends Output<T>, C extends MetricContext<T>>

Parameters:

model - The model to use.

datasource - The datasource to make predictions for.

Returns:

An evaluation of the datasource on the model.

evaluate

public final E evaluate(SequenceModel<T> model,
 List<List<Prediction<T>>> predictions,
 DataProvenance dataProvenance)

Produces an evaluation for the supplied model and predictions by aggregating the appropriate statistics.

Warning, this method cannot validate that the predictions were returned by the model in question.

Specified by:

evaluate in interface SequenceEvaluator<T extends Output<T>, C extends MetricContext<T>>

Parameters:

model - The model to use.

predictions - The predictions to use.

dataProvenance - The provenance of the test data.

Returns:

An evaluation of the predictions.

computeResults

protected Map<MetricID<T>, Double> computeResults(C ctx,
 Set<? extends EvaluationMetric<T,C>> metrics)

Computes each metric given the context.

Parameters:

ctx - The metric context (i.e., the sufficient statistics).

metrics - The metrics to compute.

Returns:

The value of each requested metric.

createMetrics

protected abstract Set<M> createMetrics(SequenceModel<T> model)

Creates the appropriate set of metrics for this model, by querying for it's OutputInfo.

Parameters:

model - The model to inspect.

Returns:

The set of metrics.

createContext

protected abstract C createContext(SequenceModel<T> model,
 List<List<Prediction<T>>> predictions)

Create the context needed for evaluation. The context might store global properties or cache computation.

Parameters:

model - the model that will be evaluated

predictions - the predictions that will be evaluated

Returns:

the context for this model and its predictions

createEvaluation

protected abstract E createEvaluation(C context,
 Map<MetricID<T>, Double> results,
 EvaluationProvenance provenance)

Create an evaluation for the given results

Parameters:

context - the context that was used to compute these results

results - the results

provenance - the provenance of the results (including information about the model and dataset)

Returns:

the evaluation