Class CoreTokenizerOptions

java.lang.Object

org.tribuo.util.tokens.options.CoreTokenizerOptions

All Implemented Interfaces:

com.oracle.labs.mlrg.olcut.config.Options, TokenizerOptions

public class CoreTokenizerOptions
extends Object
implements TokenizerOptions

CLI Options for all the tokenizers in the core package.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static enum	CoreTokenizerOptions.CoreTokenizerType	Tokenizer type.

Field Summary

Fields

Modifier and Type	Field	Description
BreakIteratorTokenizerOptions	breakIteratorOptions	Options for the break iterator tokenizer.
CoreTokenizerOptions.CoreTokenizerType	coreTokenizerType	Type of tokenizer
SplitCharactersTokenizerOptions	splitCharactersTokenizerOptions	Options for the split characters tokenizer.
SplitPatternTokenizerOptions	splitPatternTokenizerOptions	Options for the split pattern tokenizer.

Fields inherited from interface com.oracle.labs.mlrg.olcut.config.Options

header

Constructor Summary

Constructors

Constructor	Description
CoreTokenizerOptions()

Method Summary

Modifier and Type	Method	Description
Tokenizer	getTokenizer()	Creates the appropriately configured tokenizer.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.oracle.labs.mlrg.olcut.config.Options

getOptionsDescription

Field Details

breakIteratorOptions

public BreakIteratorTokenizerOptions breakIteratorOptions

Options for the break iterator tokenizer.

splitCharactersTokenizerOptions

public SplitCharactersTokenizerOptions splitCharactersTokenizerOptions

Options for the split characters tokenizer.

splitPatternTokenizerOptions

public SplitPatternTokenizerOptions splitPatternTokenizerOptions

Options for the split pattern tokenizer.

coreTokenizerType

@Option(longName="core-tokenizer-type",
        usage="Type of tokenizer")
public CoreTokenizerOptions.CoreTokenizerType coreTokenizerType

Type of tokenizer

Constructor Details

CoreTokenizerOptions

public CoreTokenizerOptions()

Method Details

getTokenizer

public Tokenizer getTokenizer()

Description copied from interface: TokenizerOptions

Creates the appropriately configured tokenizer.

Specified by:

getTokenizer in interface TokenizerOptions

Returns:

The configured tokenizer.