Package org.tribuo.sequence

Class AbstractSequenceEvaluator<T extends Output<T>,C extends MetricContext<T>,E extends SequenceEvaluation<T>,M extends EvaluationMetric<T,C>>

java.lang.Object
org.tribuo.sequence.AbstractSequenceEvaluator<T,C,E,M>
All Implemented Interfaces:
SequenceEvaluator<T,E>
Direct Known Subclasses:
LabelSequenceEvaluator
public abstract class AbstractSequenceEvaluator<T extends Output<T>,C extends MetricContext<T>,E extends SequenceEvaluation<T>,M extends EvaluationMetric<T,C>> extends Object implements SequenceEvaluator<T,E>
Base class for sequence evaluators.

  • Constructor Details

    • AbstractSequenceEvaluator

      public AbstractSequenceEvaluator()

  • Method Details

    • evaluate

      public final E evaluate(SequenceModel<T> model, SequenceDataset<T> dataset)
      Produces an evaluation for the supplied model and dataset, by calling SequenceModel.predict(org.tribuo.sequence.SequenceExample<T>) to create the predictions, then aggregating the appropriate statistics.
      Specified by:
      evaluate in interface SequenceEvaluator<T extends Output<T>,C extends MetricContext<T>>
      Parameters:
      model - The model to use.
      dataset - The dataset to make predictions for.
      Returns:
      An evaluation of the dataset on the model.

    • evaluate

      public final E evaluate(SequenceModel<T> model, SequenceDataSource<T> datasource)
      Produces an evaluation for the supplied model and datasource, by calling SequenceModel.predict(org.tribuo.sequence.SequenceExample<T>) to create the predictions, then aggregating the appropriate statistics.
      Specified by:
      evaluate in interface SequenceEvaluator<T extends Output<T>,C extends MetricContext<T>>
      Parameters:
      model - The model to use.
      datasource - The datasource to make predictions for.
      Returns:
      An evaluation of the datasource on the model.

    • evaluate

      public final E evaluate(SequenceModel<T> model, List<List<Prediction<T>>> predictions, DataProvenance dataProvenance)
      Produces an evaluation for the supplied model and predictions by aggregating the appropriate statistics.

      Warning, this method cannot validate that the predictions were returned by the model in question.

      Specified by:
      evaluate in interface SequenceEvaluator<T extends Output<T>,C extends MetricContext<T>>
      Parameters:
      model - The model to use.
      predictions - The predictions to use.
      dataProvenance - The provenance of the test data.
      Returns:
      An evaluation of the predictions.

    • computeResults

      protected Map<MetricID<T>,Double> computeResults(C ctx, Set<? extends EvaluationMetric<T,C>> metrics)
      Computes each metric given the context.
      Parameters:
      ctx - The metric context (i.e., the sufficient statistics).
      metrics - The metrics to compute.
      Returns:
      The value of each requested metric.

    • createMetrics

      protected abstract Set<M> createMetrics(SequenceModel<T> model)
      Creates the appropriate set of metrics for this model, by querying for it's OutputInfo.
      Parameters:
      model - The model to inspect.
      Returns:
      The set of metrics.

    • createContext

      protected abstract C createContext(SequenceModel<T> model, List<List<Prediction<T>>> predictions)
      Create the context needed for evaluation. The context might store global properties or cache computation.
      Parameters:
      model - the model that will be evaluated
      predictions - the predictions that will be evaluated
      Returns:
      the context for this model and its predictions

    • createEvaluation

      protected abstract E createEvaluation(C context, Map<MetricID<T>,Double> results, EvaluationProvenance provenance)
      Create an evaluation for the given results
      Parameters:
      context - the context that was used to compute these results
      results - the results
      provenance - the provenance of the results (including information about the model and dataset)
      Returns:
      the evaluation