Package org.tribuo.util.tokens.universal

An implementation of a "universal" tokenizer which will split on word boundaries or character boundaries for languages where word boundaries are contextual.

See: Description

Class Summary

Class	Description
Range	A range currently being segmented.
UniversalTokenizer	This class was originally written for the purpose of document indexing in an information retrieval context (principally used in Sun Labs' Minion search engine).

Package org.tribuo.util.tokens.universal Description

An implementation of a "universal" tokenizer which will split on word boundaries or character boundaries for languages where word boundaries are contextual.

It was originally developed to support information retrieval and forms a useful baseline tokenizer for generating features for machine learning.