Uses of Package
org.tribuo.util.tokens.impl

Packages that use org.tribuo.util.tokens.impl

Package	Description
org.tribuo.util.tokens.impl	Simple fixed rule tokenizers.

Classes in org.tribuo.util.tokens.impl used by org.tribuo.util.tokens.impl

Class and Description
BreakIteratorTokenizer A tokenizer wrapping a BreakIterator instance.
NonTokenizer A convenience class for when you are required to provide a tokenizer but you don't actually want to split up the text into tokens.
ShapeTokenizer This tokenizer is loosely based on the notion of word shape which is a common feature used in NLP.
SplitCharactersTokenizer This implementation of Tokenizer is instantiated with an array of characters that are considered split characters.
SplitPatternTokenizer This implementation of Tokenizer is instantiated with a regular expression pattern which determines how to split a string into tokens.