Uses of Enum Class
org.tribuo.util.tokens.Token.TokenType
Package
Description
Core definitions for tokenization.
Simple fixed rule tokenizers.
Provides an implementation of a Wordpiece tokenizer which implements
to the Tribuo
Tokenizer
API.An implementation of a "universal" tokenizer which will split
on word boundaries or character boundaries for languages where
word boundaries are contextual.
-
Uses of Token.TokenType in org.tribuo.util.tokens
Modifier and TypeMethodDescriptionTokenizer.getType()
Gets the type of the current token.static Token.TokenType
Returns the enum constant of this class with the specified name.static Token.TokenType[]
Token.TokenType.values()
Returns an array containing the constants of this enum class, in the order they are declared.ModifierConstructorDescriptionToken
(String text, int start, int end, Token.TokenType type) Constructs a token. -
Uses of Token.TokenType in org.tribuo.util.tokens.impl
Modifier and TypeFieldDescriptionfinal Token.TokenType
SplitFunctionTokenizer.SplitResult.tokenType
The token type. -
Uses of Token.TokenType in org.tribuo.util.tokens.impl.wordpiece
-
Uses of Token.TokenType in org.tribuo.util.tokens.universal