org.tribuo.util.tokens

Enum Token.TokenType

java.lang.Object

java.lang.Enum<Token.TokenType>

org.tribuo.util.tokens.Token.TokenType

All Implemented Interfaces:

Serializable, Comparable<Token.TokenType>

Enclosing class:

Token

public static enum Token.TokenType
extends Enum<Token.TokenType>

Tokenizers may product multiple kinds of tokens, depending on the application to which they're being put. For example, when processing a document for highlighting during querying, we need to send through whitespace and punctuation so that the document looks as it did in it's original form. For most tokenizer applications, they will only send word tokens.

Enum Constant Summary

Enum Constants

Enum Constant and Description
NGRAM
PUNCTUATION
WHITESPACE
WORD

Method Summary

Modifier and Type	Method and Description
static Token.TokenType	valueOf(String name) Returns the enum constant of this type with the specified name.
static Token.TokenType[]	values() Returns an array containing the constants of this enum type, in the order they are declared.

Methods inherited from class java.lang.Enum

clone, compareTo, equals, finalize, getDeclaringClass, hashCode, name, ordinal, toString, valueOf

Methods inherited from class java.lang.Object

getClass, notify, notifyAll, wait, wait, wait

Enum Constant Detail

WORD

public static final Token.TokenType WORD

NGRAM

public static final Token.TokenType NGRAM

PUNCTUATION

public static final Token.TokenType PUNCTUATION

WHITESPACE

public static final Token.TokenType WHITESPACE

Method Detail

values

public static Token.TokenType[] values()

Returns an array containing the constants of this enum type, in the order they are declared. This method may be used to iterate over the constants as follows:

for (Token.TokenType c : Token.TokenType.values())
    System.out.println(c);

Returns:

an array containing the constants of this enum type, in the order they are declared

valueOf

public static Token.TokenType valueOf(String name)

Returns the enum constant of this type with the specified name. The string must match exactly an identifier used to declare an enum constant in this type. (Extraneous whitespace characters are not permitted.)

Parameters:

name - the name of the enum constant to be returned.

Returns:

the enum constant with the specified name

Throws:

IllegalArgumentException - if this enum type has no constant with the specified name

NullPointerException - if the argument is null