Package | Description |
---|---|
org.tribuo.util.tokens |
Core definitions for tokenization.
|
org.tribuo.util.tokens.impl |
Simple fixed rule tokenizers.
|
org.tribuo.util.tokens.impl.wordpiece |
Provides an implementation of a Wordpiece tokenizer which implements
to the Tribuo
Tokenizer API. |
org.tribuo.util.tokens.universal |
An implementation of a "universal" tokenizer which will split
on word boundaries or character boundaries for languages where
word boundaries are contextual.
|
Modifier and Type | Field and Description |
---|---|
Token.TokenType |
Token.type |
Modifier and Type | Method and Description |
---|---|
Token.TokenType |
Tokenizer.getType()
Gets the type of the current token.
|
static Token.TokenType |
Token.TokenType.valueOf(String name)
Returns the enum constant of this type with the specified name.
|
static Token.TokenType[] |
Token.TokenType.values()
Returns an array containing the constants of this enum type, in
the order they are declared.
|
Constructor and Description |
---|
Token(String text,
int start,
int end,
Token.TokenType type)
Constructs a token.
|
Modifier and Type | Field and Description |
---|---|
Token.TokenType |
SplitFunctionTokenizer.SplitResult.tokenType |
Modifier and Type | Method and Description |
---|---|
Token.TokenType |
SplitPatternTokenizer.getType() |
Token.TokenType |
SplitFunctionTokenizer.getType() |
Token.TokenType |
ShapeTokenizer.getType() |
Token.TokenType |
NonTokenizer.getType() |
Token.TokenType |
BreakIteratorTokenizer.getType() |
Modifier and Type | Method and Description |
---|---|
Token.TokenType |
WordpieceTokenizer.getType() |
Modifier and Type | Field and Description |
---|---|
Token.TokenType |
Range.type |
Modifier and Type | Method and Description |
---|---|
Token.TokenType |
UniversalTokenizer.getType() |
Modifier and Type | Method and Description |
---|---|
void |
Range.setType(Token.TokenType type) |
Copyright © 2015–2021 Oracle and/or its affiliates. All rights reserved.