Uses of Interface
org.tribuo.util.tokens.impl.SplitFunctionTokenizer.SplitFunction
Packages that use SplitFunctionTokenizer.SplitFunction
Package
Description
Simple fixed rule tokenizers.
Provides an implementation of a Wordpiece tokenizer which implements
to the Tribuo
Tokenizer
API.-
Uses of SplitFunctionTokenizer.SplitFunction in org.tribuo.util.tokens.impl
Classes in org.tribuo.util.tokens.impl that implement SplitFunctionTokenizer.SplitFunctionModifier and TypeClassDescriptionstatic class
Splits tokens at the supplied characters.Fields in org.tribuo.util.tokens.impl declared as SplitFunctionTokenizer.SplitFunctionModifier and TypeFieldDescriptionprotected SplitFunctionTokenizer.SplitFunction
SplitFunctionTokenizer.splitFunction
WhitespaceTokenizer.whitespaceSplitCharacterFunction
Constructors in org.tribuo.util.tokens.impl with parameters of type SplitFunctionTokenizer.SplitFunctionModifierConstructorDescriptionSplitFunctionTokenizer
(SplitFunctionTokenizer.SplitFunction splitFunction) Creates a new tokenizer using the supplied split function. -
Uses of SplitFunctionTokenizer.SplitFunction in org.tribuo.util.tokens.impl.wordpiece
Methods in org.tribuo.util.tokens.impl.wordpiece that return SplitFunctionTokenizer.SplitFunctionModifier and TypeMethodDescriptionWordpieceBasicTokenizer.createSplitFunction
(boolean tokenizeChineseChars) Creates aSplitFunctionTokenizer.SplitFunction
that is used by the super classSplitFunctionTokenizer
to determine how and where the tokenizer splits the input.