Package org.tribuo.util.tokens

package org.tribuo.util.tokens

Core definitions for tokenization. Tokens are represented by instances of Token, and emitted by implementations of Tokenizer.

Related Packages

Package	Description
org.tribuo.util	Provides utilities which don't have other Tribuo dependencies.
org.tribuo.util.tokens.impl	Simple fixed rule tokenizers.
org.tribuo.util.tokens.options	OLCUT Options implementations which can construct Tokenizers of various types.
org.tribuo.util.tokens.universal	An implementation of a "universal" tokenizer which will split on word boundaries or character boundaries for languages where word boundaries are contextual.
org.tribuo.util.infotheory	This package provides static classes of information theoretic functions.
org.tribuo.util.onnx	Interfaces and utilities for writing ONNX models from Java.

Class	Description
Token	A single token extracted from a String.
Token.TokenType	Tokenizers may product multiple kinds of tokens, depending on the application to which they're being put.
TokenizationException	Wraps exceptions thrown by tokenizers.
Tokenizer	An interface for things that tokenize text: breaking it into words according to some set of rules.