Uses of Interface
org.tribuo.util.tokens.Tokenizer

Packages that use Tokenizer

Package	Description
org.tribuo.classification.explanations.lime	Provides an implementation of LIME (Locally Interpretable Model Explanations).
org.tribuo.data.text.impl	Provides implementations of text data processors.
org.tribuo.util.tokens	Core definitions for tokenization.
org.tribuo.util.tokens.impl	Simple fixed rule tokenizers.
org.tribuo.util.tokens.options	OLCUT Options implementations which can construct Tokenizers of various types.
org.tribuo.util.tokens.universal

Uses of Tokenizer in org.tribuo.classification.explanations.lime

Constructors in org.tribuo.classification.explanations.lime with parameters of type Tokenizer

Constructor and Description
LIMEColumnar(SplittableRandom rng, Model<Label> innerModel, SparseTrainer<Regressor> explanationTrainer, int numSamples, RowProcessor<Label> exampleGenerator, Tokenizer tokenizer) Constructs a LIME explainer for a model which uses the columnar data processing system.
LIMEText(SplittableRandom rng, Model<Label> innerModel, SparseTrainer<Regressor> explanationTrainer, int numSamples, TextFeatureExtractor<Label> extractor, Tokenizer tokenizer) Constructs a LIME explainer for a model which uses text data.

Uses of Tokenizer in org.tribuo.data.text.impl

Constructors in org.tribuo.data.text.impl with parameters of type Tokenizer

Constructor and Description
BasicPipeline(Tokenizer tokenizer, int ngram)
NgramProcessor(Tokenizer tokenizer, int n, double value) Creates a processor that will generate token ngrams of size n.
TokenPipeline(Tokenizer tokenizer, int ngram, boolean termCounting) Creates a new token pipeline.
TokenPipeline(Tokenizer tokenizer, int ngram, boolean termCounting, int dimension) Creates a new token pipeline.

Uses of Tokenizer in org.tribuo.util.tokens

Methods in org.tribuo.util.tokens that return Tokenizer

Modifier and Type	Method and Description
Tokenizer	Tokenizer.clone() Clones a tokenizer with it's configuration.

Methods in org.tribuo.util.tokens that return types with arguments of type Tokenizer

Modifier and Type	Method and Description
static Supplier<Tokenizer>	Tokenizer.createSupplier(Tokenizer tokenizer)
static ThreadLocal<Tokenizer>	Tokenizer.createThreadLocal(Tokenizer tokenizer)

Methods in org.tribuo.util.tokens with parameters of type Tokenizer

Modifier and Type	Method and Description
static Supplier<Tokenizer>	Tokenizer.createSupplier(Tokenizer tokenizer)
static ThreadLocal<Tokenizer>	Tokenizer.createThreadLocal(Tokenizer tokenizer)

Uses of Tokenizer in org.tribuo.util.tokens.impl

Classes in org.tribuo.util.tokens.impl that implement Tokenizer

Modifier and Type	Class and Description
class	BreakIteratorTokenizer A tokenizer wrapping a BreakIterator instance.
class	NonTokenizer A convenience class for when you are required to provide a tokenizer but you don't actually want to split up the text into tokens.
class	ShapeTokenizer This tokenizer is loosely based on the notion of word shape which is a common feature used in NLP.
class	SplitCharactersTokenizer This implementation of Tokenizer is instantiated with an array of characters that are considered split characters.
class	SplitPatternTokenizer This implementation of Tokenizer is instantiated with a regular expression pattern which determines how to split a string into tokens.

Uses of Tokenizer in org.tribuo.util.tokens.options

Methods in org.tribuo.util.tokens.options that return Tokenizer

Modifier and Type	Method and Description
Tokenizer	TokenizerOptions.getTokenizer() Creates the appropriately configured tokenizer.
Tokenizer	SplitPatternTokenizerOptions.getTokenizer()
Tokenizer	SplitCharactersTokenizerOptions.getTokenizer()
Tokenizer	CoreTokenizerOptions.getTokenizer()
Tokenizer	BreakIteratorTokenizerOptions.getTokenizer()

Uses of Tokenizer in org.tribuo.util.tokens.universal

Classes in org.tribuo.util.tokens.universal that implement Tokenizer

Modifier and Type	Class and Description
class	UniversalTokenizer This class was originally written for the purpose of document indexing in an information retrieval context (principally used in Sun Labs' Minion search engine).

Methods in org.tribuo.util.tokens.universal that return Tokenizer

Modifier and Type	Method and Description
Tokenizer	UniversalTokenizer.clone()