Package org.tribuo.util.tokens.options

package org.tribuo.util.tokens.options

OLCUT Options implementations which can construct Tokenizers of various types.

Related Packages

Package	Description
org.tribuo.util.tokens	Core definitions for tokenization.
org.tribuo.util.tokens.impl	Simple fixed rule tokenizers.
org.tribuo.util.tokens.universal	An implementation of a "universal" tokenizer which will split on word boundaries or character boundaries for languages where word boundaries are contextual.

Class	Description
BreakIteratorTokenizerOptions	CLI options for a BreakIteratorTokenizer.
CoreTokenizerOptions	CLI Options for all the tokenizers in the core package.
CoreTokenizerOptions.CoreTokenizerType	Tokenizer type.
SplitCharactersTokenizerOptions	CLI options for a SplitCharactersTokenizer.
SplitPatternTokenizerOptions	CLI options for a SplitPatternTokenizer.
TokenizerOptions	CLI Options for creating a tokenizer.