Uses of Class
org.tribuo.util.tokens.Token

Packages that use Token

Package	Description
org.tribuo.classification.explanations.lime	Provides an implementation of LIME (Locally Interpretable Model Explanations).
org.tribuo.util.tokens	Core definitions for tokenization.
org.tribuo.util.tokens.impl.wordpiece	Provides an implementation of a Wordpiece tokenizer which implements to the Tribuo Tokenizer API.

Uses of Token in org.tribuo.classification.explanations.lime

Method parameters in org.tribuo.classification.explanations.lime with type arguments of type Token

Modifier and Type	Method and Description
protected List<Example<Regressor>>	LIMEText.sampleData(String inputText, List<Token> tokens) Samples a new dataset from the input text.

Uses of Token in org.tribuo.util.tokens

Methods in org.tribuo.util.tokens that return Token

Modifier and Type	Method and Description
default Token	Tokenizer.getToken() Generates a Token object from the current state of the tokenizer.

Methods in org.tribuo.util.tokens that return types with arguments of type Token

Modifier and Type	Method and Description
default List<Token>	Tokenizer.tokenize(CharSequence cs) Uses this tokenizer to tokenize a string and return the list of tokens that were generated.

Uses of Token in org.tribuo.util.tokens.impl.wordpiece

Methods in org.tribuo.util.tokens.impl.wordpiece that return Token

Modifier and Type	Method and Description
Token	WordpieceTokenizer.getToken()