Uses of Interface
org.tribuo.util.tokens.impl.SplitFunctionTokenizer.SplitFunction

Packages that use SplitFunctionTokenizer.SplitFunction

Package	Description
org.tribuo.util.tokens.impl	Simple fixed rule tokenizers.
org.tribuo.util.tokens.impl.wordpiece	Provides an implementation of a Wordpiece tokenizer which implements to the Tribuo Tokenizer API.

Uses of SplitFunctionTokenizer.SplitFunction in org.tribuo.util.tokens.impl

Classes in org.tribuo.util.tokens.impl that implement SplitFunctionTokenizer.SplitFunction

Modifier and Type	Class	Description
static class	SplitCharactersTokenizer.SplitCharactersSplitterFunction	Splits tokens at the supplied characters.

Fields in org.tribuo.util.tokens.impl declared as SplitFunctionTokenizer.SplitFunction

Modifier and Type	Field	Description
protected SplitFunctionTokenizer.SplitFunction	SplitFunctionTokenizer.splitFunction	The splitting function.
static final SplitFunctionTokenizer.SplitFunction	WhitespaceTokenizer.whitespaceSplitCharacterFunction	The splitting function for whitespace, using Character.isWhitespace(char).

Constructors in org.tribuo.util.tokens.impl with parameters of type SplitFunctionTokenizer.SplitFunction

Modifier	Constructor	Description
	SplitFunctionTokenizer(SplitFunctionTokenizer.SplitFunction splitFunction)	Creates a new tokenizer using the supplied split function.

Uses of SplitFunctionTokenizer.SplitFunction in org.tribuo.util.tokens.impl.wordpiece

Methods in org.tribuo.util.tokens.impl.wordpiece that return SplitFunctionTokenizer.SplitFunction

Modifier and Type	Method	Description
static SplitFunctionTokenizer.SplitFunction	WordpieceBasicTokenizer.createSplitFunction(boolean tokenizeChineseChars)	Creates a SplitFunctionTokenizer.SplitFunction that is used by the super class SplitFunctionTokenizer to determine how and where the tokenizer splits the input.