Classification Tutorial¶
This tutorial will show how to use Tribuo's classification models to predict Iris species using Fisher's well known Irises dataset (it's 2020 and we're still using a dataset from 1936 in demos, but not to worry we'll use MNIST from the 90s next time). We'll focus on a simple logistic regression, and investigate the provenance and metadata that Tribuo stores inside each model.
Setup¶
You'll need to get a copy of the irises dataset.
wget https://archive.ics.uci.edu/ml/machine-learning-databases/iris/bezdekIris.data
It's Java, so first we load in the necessary Tribuo jars. Here we're using the classification experiments jar, along with the json interop jar to read and write the provenance information.
%jars ./tribuo-classification-experiments-4.1.0-jar-with-dependencies.jar
%jars ./tribuo-json-4.1.0-jar-with-dependencies.jar
import java.nio.file.Paths;
import java.nio.file.Files;
We import everything from the base org.tribuo package, along with the simple CSV loader, and the classification packages. We're going to build a logistic regression, so we'll need that too.
import org.tribuo.*;
import org.tribuo.evaluation.TrainTestSplitter;
import org.tribuo.data.csv.CSVLoader;
import org.tribuo.classification.*;
import org.tribuo.classification.evaluation.*;
import org.tribuo.classification.sgd.linear.LogisticRegressionTrainer;
These imports are for the provenance system, which we'll get to in a minute.
import com.fasterxml.jackson.databind.*;
import com.oracle.labs.mlrg.olcut.provenance.ProvenanceUtil;
import com.oracle.labs.mlrg.olcut.config.json.*;
Loading the data¶
In Tribuo, all the prediction types have an associated OutputFactory implementation, which can create the appropriate Output subclasses from an input. Here we're going to use LabelFactory as we're performing multi-class classification. We then pass the labelFactory into the simple CSVLoader which reads all the columns into a DataSource.
var labelFactory = new LabelFactory();
var csvLoader = new CSVLoader<>(labelFactory);
Our copy of irises doesn't have any column headers, so we create the headers and supply them to the load method along with the path, and which variable is the output (in this case \"species\"). Irises doesn't have a pre-defined train/test split, so we're going to create one, with 70% of the data used for training. Note if your csv file is more complicated than a table of numbers and a response column then you should use CSVDataSource to load it in, and you might want to read the columnar data tutorial too.
var irisHeaders = new String[]{"sepalLength", "sepalWidth", "petalLength", "petalWidth", "species"};
var irisesSource = csvLoader.loadDataSource(Paths.get("bezdekIris.data"),"species",irisHeaders);
var irisSplitter = new TrainTestSplitter<>(irisesSource,0.7,1L);
We feed the training datasource and the test datasource into their respective datasets. These datasets compute all the necessary metadata, like the feature domain and the output domain. For training datasets it's best to use a MutableDataset as it can have transformations applied to it, and the domains grow as more examples are added. Now we have datasets we're ready to train some models.
var trainingDataset = new MutableDataset<>(irisSplitter.getTrain());
var testingDataset = new MutableDataset<>(irisSplitter.getTest());
System.out.println(String.format("Training data size = %d, number of features = %d, number of classes = %d",trainingDataset.size(),trainingDataset.getFeatureMap().size(),trainingDataset.getOutputInfo().size()));
System.out.println(String.format("Testing data size = %d, number of features = %d, number of classes = %d",testingDataset.size(),testingDataset.getFeatureMap().size(),testingDataset.getOutputInfo().size()));
Training the model¶
Now let's instantiate the trainer, and see what it's default hyperparameters are. For full control over these parameters you can directly use LinearSGDTrainer which is fully configurable.
Trainer<Label> trainer = new LogisticRegressionTrainer();
System.out.println(trainer.toString());
So that's a linear model, using a logistic loss, trained with AdaGrad for 5 epochs.
Now let's train the model. As with other packages, training is pretty simple when you have the training algorithm and training data.
Model<Label> irisModel = trainer.train(trainingDataset);
Evaluating the model¶
Once we've trained a model, it's time to figure out how good it is. For this can we ask the labelFactory what the appropriate Evaluator is or instantiate it directly to get a sharper type, then pass the evaluator the model and the test dataset. You can also supply a datasource instead of the dataset, or even a list of predictions if you've already generated them. The LabelEvaluator class implements most of the common classification metrics, each of which can be individually inspected. These metrics can be inspected on a per label basis or averaged across all the possible labels. LabelEvaluator.toString() produces a nicely formatted summary of the metrics.
var evaluator = new LabelEvaluator();
var evaluation = evaluator.evaluate(irisModel,testingDataset);
System.out.println(evaluation.toString());
This output lists:
- the different classes in the test set
- n, the number of ground truth labels of that class
- tp, the number of true positives (i.e., the number of times the classifier correctly predicted this class)
- fn, the number of false negatives (i.e., the number of times the classifier predicted this class as another class)
- fp, the number of false positives (i.e., the number of times the classifier incorrectly predicted this class when it was another class)
- recall, the true positives divided by the number of ground truth labels (i.e., the fraction of this class that the classifier can detect)
- precision, the true positives divided by the predicted positives (i.e, the fraction of the time that this class is predicted correctly)
- accuracy, the sum of the true positives divided by the total number of test examples
- balanced error rate, the average of the per class error rates
For probabilistic classifiers it's also possible to compute the ROC curve and precision-recall curve.
We can also print the confusion matrix.
System.out.println(evaluation.getConfusionMatrix().toString());
We can see there is a single misclassification, where an Iris-virginica is misclassified as an Iris-versicolor.
Model Metadata¶
Tribuo tracks the feature and output domains of all constructed models. This means it's possible to run techniques like LIME without access to the original training data, and also to add checks that a particular input is within the bounds seen by the trained model.
Let's look at the feature domain from our Irises model.
var featureMap = irisModel.getFeatureIDMap();
for (var v : featureMap) {
System.out.println(v.toString());
System.out.println();
}
We can see the 4 features, along with a histogram of their values. This information can be used to sample from each feature, to build candidate examples for local explainers like LIME, or to check the range. The feature information is frozen at model training time, so it can also be used to check the number of times a feature occurred in the training set, when the feature set is sparse (as is commonly the case in NLP problems).
Model Provenance¶
Modern applications deploy many different kinds of ML models, helping with many different aspects of the application. However most ML packages don't provide good support for tracking and rebuilding models. In Tribuo each model tracks it's provenance. It knows how it was created, when it was created, and what data was involved. Let's look at the data provenance for our irises model. By default Tribuo prints the provenance in a moderately human readable format in each provenance object's toString(), but all the information is accessible programmatically.
var provenance = irisModel.getProvenance();
System.out.println(ProvenanceUtil.formattedProvenanceString(provenance.getDatasetProvenance().getSourceProvenance()));
We can see the model was trained on a datasource which was split in two, using a specific random seed & split percentage. The original datasource was a CSV file, and the file modified time and SHA-256 hash are recorded too.
We can similarly inspect the trainer provenance to find out about the training algorithm.
System.out.println(ProvenanceUtil.formattedProvenanceString(provenance.getTrainerProvenance()));
Here we see as expected that our model was trained using a LogisticRegressionTrainer which used AdaGrad as the gradient descent algorithm.
Provenance can be extracted from models and stored as json files, if you wish to keep a separate record (or redact the provenance from a deployed model).
ObjectMapper objMapper = new ObjectMapper();
objMapper.registerModule(new JsonProvenanceModule());
objMapper = objMapper.enable(SerializationFeature.INDENT_OUTPUT);
The json provenance is verbose, but provides an alternative human readable serialization format.
String jsonProvenance = objMapper.writeValueAsString(ProvenanceUtil.marshalProvenance(provenance));
System.out.println(jsonProvenance);
Alternatively the model provenance is also present in the output of Model.toString(), though this format is not machine readable (or particularly human readable for that matter).
System.out.println(irisModel.toString());
Evaluations also have a provenance that records the model provenance along with the test data provenance. We're using an alternate form of the JSON provenance that's easier to read, though a little less precise. This form is suitable for reference but can't be used to reconstruct the original provenance object as it's converted everything into Strings.
String jsonEvaluationProvenance = objMapper.writeValueAsString(ProvenanceUtil.convertToMap(evaluation.getProvenance()));
System.out.println(jsonEvaluationProvenance);
We can see that this provenance includes all the fields from the models' provenance, along with the test data, it's split, and the CSV it came from.
This provenance information is useful on it's own for tracking models, but when combined with the config system described in the configuration tutorial it becomes a powerful way of rebuilding models and experiments, allowing near perfect replicability of any ML model.
Loading and saving models¶
Tribuo uses Java Serialization to save and load models. Models and Datasets are java.io.Serializable and can be written to input and output streams in the usual manner. Here we'll go through saving and loading the model we just trained, but the procedure is the same for all other Tribuo models.
First we save the model out using an ObjectOutputStream.
File tmpFile = File.createTempFile("irisModel","ser");
try (ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream(tmpFile))) {
oos.writeObject(irisModel);
}
Now we can load in the saved model. We're going to use the serialization allow list that comes with Tribuo, to ensure we only load in Tribuo related classes (this is described in JEP 290). This feature is available in Java 9 onwards, and as a process wide feature in Java 8 from 8u121. Usually the pattern would be stored in code or as a classpath resource, here we're going to read it out of the Tribuo repository (assuming this notebook is in Tribuo/tutorials).
String filterPattern = Files.readAllLines(Paths.get("../docs/jep-290-allowlist.txt")).get(0);
ObjectInputFilter filter = ObjectInputFilter.Config.createFilter(filterPattern);
Model<?> loadedModel;
try (ObjectInputStream ois = new ObjectInputStream(new BufferedInputStream(new FileInputStream(tmpFile)))) {
ois.setObjectInputFilter(filter);
loadedModel = (Model<?>) ois.readObject();
}
As Tribuo's models are generically typed, and Java's generics are erased, this requires an unchecked cast to apply the right type to the model. Tribuo has a mechanism for validating that the type is correct, model.validate(Class<? extends Output<?>>) which returns true if the supplied class is the same as the internal output type stored in this model.
if (loadedModel.validate(Label.class)) {
System.out.println("It's a Model<Label>!");
} else {
System.out.println("It's some other kind of Model.");
}
You can use this check to guard a cast to the appropriate generic type before using the model as normal.
We'll check that the models are the same by comparing their provenances.
loadedModel.getProvenance().equals(irisModel.getProvenance())
Conclusion¶
We looked at Tribuo's csv loading mechanism, how to train a simple classifier, how to evaluate a classifier on test data, what metadata and provenance information is stored inside Tribuo's Model and Evaluation objects, and finally how to save and load Tribuo's models.