Package org.tribuo.util.tokens

Core definitions for tokenization.

See: Description

Interface Summary

Interface	Description
Tokenizer	An interface for things that tokenize text: breaking it into words according to some set of rules.

Class Summary

Class	Description
Token	A single token extracted from a String.

Enum Summary

Enum	Description
Token.TokenType	Tokenizers may product multiple kinds of tokens, depending on the application to which they're being put.

Exception Summary

Exception	Description
TokenizationException	Wraps exceptions thrown by tokenizers.

Package org.tribuo.util.tokens Description

Core definitions for tokenization. Tokens are represented by instances of Token, and emitted by implementations of Tokenizer.