org.tribuo.util.tokens

Interface Tokenizer

All Superinterfaces:

Cloneable, com.oracle.labs.mlrg.olcut.config.Configurable, com.oracle.labs.mlrg.olcut.provenance.Provenancable<com.oracle.labs.mlrg.olcut.provenance.ConfiguredObjectProvenance>

All Known Implementing Classes:

BreakIteratorTokenizer, NonTokenizer, ShapeTokenizer, SplitCharactersTokenizer, SplitFunctionTokenizer, SplitPatternTokenizer, UniversalTokenizer, WhitespaceTokenizer, WordpieceBasicTokenizer, WordpieceTokenizer

public interface Tokenizer
extends com.oracle.labs.mlrg.olcut.config.Configurable, Cloneable, com.oracle.labs.mlrg.olcut.provenance.Provenancable<com.oracle.labs.mlrg.olcut.provenance.ConfiguredObjectProvenance>

An interface for things that tokenize text: breaking it into words according to some set of rules.

Note that tokenizers are not guaranteed to be thread safe! Using the same tokenizer from multiple threads may result in strange behavior.

Tokenizers which are not ready throw IllegalStateException when advance() or any get method is called.

Most Tokenizers are Cloneable, and implement the Cloneable interface.

Method Summary

Modifier and Type	Method and Description
boolean	advance() Advances the tokenizer to the next token.
Tokenizer	clone() Clones a tokenizer with it's configuration.
static Supplier<Tokenizer>	createSupplier(Tokenizer tokenizer)
static ThreadLocal<Tokenizer>	createThreadLocal(Tokenizer tokenizer)
int	getEnd() Gets the ending offset (exclusive) of the current token in the character sequence
int	getStart() Gets the starting character offset of the current token in the character sequence
String	getText() Gets the text of the current token, as a string
default Token	getToken() Generates a Token object from the current state of the tokenizer.
Token.TokenType	getType() Gets the type of the current token.
void	reset(CharSequence cs) Resets the tokenizer so that it operates on a new sequence of characters.
default List<String>	split(CharSequence cs) Uses this tokenizer to split a string into it's component substrings.
default List<Token>	tokenize(CharSequence cs) Uses this tokenizer to tokenize a string and return the list of tokens that were generated.

Methods inherited from interface com.oracle.labs.mlrg.olcut.config.Configurable

postConfig

Methods inherited from interface com.oracle.labs.mlrg.olcut.provenance.Provenancable

getProvenance

Method Detail

createSupplier

static Supplier<Tokenizer> createSupplier(Tokenizer tokenizer)

createThreadLocal

static ThreadLocal<Tokenizer> createThreadLocal(Tokenizer tokenizer)

reset

void reset(CharSequence cs)

Resets the tokenizer so that it operates on a new sequence of characters.

Parameters:

cs - a character sequence to tokenize

advance

boolean advance()

Advances the tokenizer to the next token.

Returns:

true if there is such a token, false otherwise.

getText

String getText()

Gets the text of the current token, as a string

Returns:

the text of the current token

getStart

int getStart()

Gets the starting character offset of the current token in the character sequence

Returns:

the starting character offset of the token

getEnd

int getEnd()

Gets the ending offset (exclusive) of the current token in the character sequence

Returns:

the exclusive ending character offset for the current token.

getType

Token.TokenType getType()

Gets the type of the current token.

Returns:

the type of the current token.

clone

Tokenizer clone()
         throws CloneNotSupportedException

Clones a tokenizer with it's configuration. Cloned tokenizers are not processing the same text as the original tokenizer and need to be reset with a fresh CharSequence.

Returns:

A tokenizer with the same configuration, but independent state.

Throws:

CloneNotSupportedException - if the tokenizer isn't cloneable.

getToken

default Token getToken()

Generates a Token object from the current state of the tokenizer.

Returns:

The token object from the current state.

tokenize

default List<Token> tokenize(CharSequence cs)

Uses this tokenizer to tokenize a string and return the list of tokens that were generated. Many applications will simply want to take a character sequence and get a list of tokens, so this will do that for them.

Here is the contract of the tokenize function:

• all returned tokens correspond to substrings of the input text
• the tokens do not overlap
• the tokens are returned in the order that they appear in the text
• the value of Token.text should be the same as calling text.substring(token.start, token.end)

Parameters:

cs - a sequence of characters to tokenize

Returns:

the tokens discovered in the character sequence, in order (true?).

split

default List<String> split(CharSequence cs)

Uses this tokenizer to split a string into it's component substrings. Many applications will simply want the component strings making up a larger character sequence.

Parameters:

cs - the character sequence to tokenize

Returns:

a list of strings making up the character sequence.