org.tribuo.util.tokens.impl

Class WhitespaceTokenizer

java.lang.Object

org.tribuo.util.tokens.impl.SplitFunctionTokenizer

org.tribuo.util.tokens.impl.WhitespaceTokenizer

All Implemented Interfaces:

com.oracle.labs.mlrg.olcut.config.Configurable, com.oracle.labs.mlrg.olcut.provenance.Provenancable<com.oracle.labs.mlrg.olcut.provenance.ConfiguredObjectProvenance>, Cloneable, Tokenizer

public class WhitespaceTokenizer
extends SplitFunctionTokenizer

A simple tokenizer that splits on whitespace. This tokenizer does not create tokens that corresond to whitespace - only those spans of text delimited by whitespace. For example, the text "a b" will result in two tokens "a" and "b".

Nested Class Summary

Nested classes/interfaces inherited from class org.tribuo.util.tokens.impl.SplitFunctionTokenizer

SplitFunctionTokenizer.SplitFunction, SplitFunctionTokenizer.SplitResult, SplitFunctionTokenizer.SplitType

Field Summary

Fields

Modifier and Type	Field and Description
static SplitFunctionTokenizer.SplitFunction	whitespaceSplitCharacterFunction

Fields inherited from class org.tribuo.util.tokens.impl.SplitFunctionTokenizer

splitFunction

Constructor Summary

Constructors

Constructor and Description
WhitespaceTokenizer()

Method Summary

Modifier and Type	Method and Description
WhitespaceTokenizer	clone() Clones a tokenizer with it's configuration.
com.oracle.labs.mlrg.olcut.provenance.ConfiguredObjectProvenance	getProvenance()

Methods inherited from class org.tribuo.util.tokens.impl.SplitFunctionTokenizer

advance, getEnd, getStart, getText, getType, reset

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.tribuo.util.tokens.Tokenizer

createSupplier, createThreadLocal, getToken, split, tokenize

Methods inherited from interface com.oracle.labs.mlrg.olcut.config.Configurable

postConfig

Field Detail

whitespaceSplitCharacterFunction

public static SplitFunctionTokenizer.SplitFunction whitespaceSplitCharacterFunction

Constructor Detail

WhitespaceTokenizer

public WhitespaceTokenizer()

Method Detail

getProvenance

public com.oracle.labs.mlrg.olcut.provenance.ConfiguredObjectProvenance getProvenance()

clone

public WhitespaceTokenizer clone()

Description copied from interface: Tokenizer

Clones a tokenizer with it's configuration. Cloned tokenizers are not processing the same text as the original tokenizer and need to be reset with a fresh CharSequence.

Specified by:

clone in interface Tokenizer

Overrides:

clone in class SplitFunctionTokenizer

Returns:

A tokenizer with the same configuration, but independent state.