Package org.tribuo.util.tokens

package org.tribuo.util.tokens

Core definitions for tokenization. Tokens are represented by instances of Token, and emitted by implementations of Tokenizer.

Related Packages

Package	Description
org.tribuo.util	Provides utilities which don't have other Tribuo dependencies.
org.tribuo.util.tokens.impl	Simple fixed rule tokenizers.
org.tribuo.util.tokens.options	OLCUT Options implementations which can construct Tokenizers of various types.
org.tribuo.util.tokens.universal
org.tribuo.util.infotheory	This package provides static classes of information theoretic functions.

Class	Description
Token	A single token extracted from a String.
Token.TokenType	Tokenizers may product multiple kinds of tokens, depending on the application to which they're being put.
TokenizationException	Wraps exceptions thrown by tokenizers.
Tokenizer	An interface for things that tokenize text: breaking it into words according to some set of rules.