Uses of Interface
org.tribuo.util.tokens.impl.SplitFunctionTokenizer.SplitFunction

Packages that use SplitFunctionTokenizer.SplitFunction

Package	Description
org.tribuo.util.tokens.impl	Simple fixed rule tokenizers.
org.tribuo.util.tokens.impl.wordpiece	Provides an implementation of a Wordpiece tokenizer which implements to the Tribuo Tokenizer API.

Uses of SplitFunctionTokenizer.SplitFunction in org.tribuo.util.tokens.impl

Classes in org.tribuo.util.tokens.impl that implement SplitFunctionTokenizer.SplitFunction

Modifier and Type	Class and Description
static class	SplitCharactersTokenizer.SplitCharactersSplitterFunction Splits tokens at the supplied characters.

Fields in org.tribuo.util.tokens.impl declared as SplitFunctionTokenizer.SplitFunction

Modifier and Type	Field and Description
protected SplitFunctionTokenizer.SplitFunction	SplitFunctionTokenizer.splitFunction
static SplitFunctionTokenizer.SplitFunction	WhitespaceTokenizer.whitespaceSplitCharacterFunction

Constructors in org.tribuo.util.tokens.impl with parameters of type SplitFunctionTokenizer.SplitFunction

Constructor and Description
SplitFunctionTokenizer(SplitFunctionTokenizer.SplitFunction splitFunction) Creates a new tokenizer using the supplied split function.

Uses of SplitFunctionTokenizer.SplitFunction in org.tribuo.util.tokens.impl.wordpiece

Methods in org.tribuo.util.tokens.impl.wordpiece that return SplitFunctionTokenizer.SplitFunction

Modifier and Type	Method and Description
static SplitFunctionTokenizer.SplitFunction	WordpieceBasicTokenizer.createSplitFunction(boolean tokenizeChineseChars) Creates a SplitFunction that is used by the super class SplitFunctionTokenizer to determine how and where the tokenizer splits the input.