Package org.tribuo.util.tokens.universal

package org.tribuo.util.tokens.universal

Related Packages

Package	Description
org.tribuo.util.tokens	Core definitions for tokenization.
org.tribuo.util.tokens.impl	Simple fixed rule tokenizers.
org.tribuo.util.tokens.options	OLCUT Options implementations which can construct Tokenizers of various types.

Classes

Class	Description
Range	A range currently being segmented.
UniversalTokenizer	This class was originally written for the purpose of document indexing in an information retrieval context (principally used in Sun Labs' Minion search engine).