Uses of Class
org.tribuo.util.tokens.impl.SplitFunctionTokenizer

Packages that use SplitFunctionTokenizer

Package	Description
org.tribuo.util.tokens.impl	Simple fixed rule tokenizers.
org.tribuo.util.tokens.impl.wordpiece	Provides an implementation of a Wordpiece tokenizer which implements to the Tribuo Tokenizer API.

Uses of SplitFunctionTokenizer in org.tribuo.util.tokens.impl

Subclasses of SplitFunctionTokenizer in org.tribuo.util.tokens.impl

Modifier and Type	Class and Description
class	SplitCharactersTokenizer This implementation of Tokenizer is instantiated with an array of characters that are considered split characters.
class	WhitespaceTokenizer A simple tokenizer that splits on whitespace.

Uses of SplitFunctionTokenizer in org.tribuo.util.tokens.impl.wordpiece

Subclasses of SplitFunctionTokenizer in org.tribuo.util.tokens.impl.wordpiece

Modifier and Type	Class and Description
class	WordpieceBasicTokenizer This is a tokenizer that is used "upstream" of WordpieceTokenizer and implements much of the functionality of the 'BasicTokenizer' implementation in huggingface.