org.tribuo.util.tokens.impl

Enum SplitFunctionTokenizer.SplitType

java.lang.Object

java.lang.Enum<SplitFunctionTokenizer.SplitType>

org.tribuo.util.tokens.impl.SplitFunctionTokenizer.SplitType

All Implemented Interfaces:

Serializable, Comparable<SplitFunctionTokenizer.SplitType>

Enclosing class:

SplitFunctionTokenizer

public static enum SplitFunctionTokenizer.SplitType
extends Enum<SplitFunctionTokenizer.SplitType>

Defines different ways that a tokenizer can split the input text at a given character.

Enum Constant Summary

Enum Constants

Enum Constant and Description
NO_SPLIT the current character is added to the in-progress token (i.e.
SPLIT_AFTER The current character will cause the in-progress token to be completed after the current character is appended to the in-progress token.
SPLIT_AT The current character will cause the in-progress token to be completed.
SPLIT_BEFORE The current character will cause the in-progress token to be completed the current character will be included in the next token.
SPLIT_BEFORE_AND_AFTER The current character should cause the in-progress token to be completed.

Method Summary

Modifier and Type	Method and Description
static SplitFunctionTokenizer.SplitType	valueOf(String name) Returns the enum constant of this type with the specified name.
static SplitFunctionTokenizer.SplitType[]	values() Returns an array containing the constants of this enum type, in the order they are declared.

Methods inherited from class java.lang.Enum

clone, compareTo, equals, finalize, getDeclaringClass, hashCode, name, ordinal, toString, valueOf

Methods inherited from class java.lang.Object

getClass, notify, notifyAll, wait, wait, wait

Enum Constant Detail

NO_SPLIT

public static final SplitFunctionTokenizer.SplitType NO_SPLIT

the current character is added to the in-progress token (i.e. do not split on the current character)

SPLIT_AT

public static final SplitFunctionTokenizer.SplitType SPLIT_AT

The current character will cause the in-progress token to be completed. the current character will not be included in any returned token and the token type of the corresponding SplitResult is ignored (See SplitFunctionTokenizer.SplitResult.SPLIT_AT). This SplitType may be useful for whitespace.

SPLIT_BEFORE

public static final SplitFunctionTokenizer.SplitType SPLIT_BEFORE

The current character will cause the in-progress token to be completed the current character will be included in the next token. The token type of the corresponding SplitResult is ignored (See SplitFunctionTokenizer.SplitResult.SPLIT_BEFORE). This SplitType may be useful for e.g. capitalized letters when CamelCase splitting of digits when separating out a currency symbol.

SPLIT_AFTER

public static final SplitFunctionTokenizer.SplitType SPLIT_AFTER

The current character will cause the in-progress token to be completed after the current character is appended to the in-progress token. The token type of the created token (that includes the current character) will be assigned the type included with the SplitFunctionTokenizer.SplitResult.

SPLIT_BEFORE_AND_AFTER

public static final SplitFunctionTokenizer.SplitType SPLIT_BEFORE_AND_AFTER

The current character should cause the in-progress token to be completed. The token assigned to the in-progress token will be whatever was previously assigned to the previous character. This token will be followed by a second single-character token consisting of the current character. The token type assigned to this second token will be provided with the SplitFunctionTokenizer.SplitResult.

Method Detail

values

public static SplitFunctionTokenizer.SplitType[] values()

Returns an array containing the constants of this enum type, in the order they are declared. This method may be used to iterate over the constants as follows:

for (SplitFunctionTokenizer.SplitType c : SplitFunctionTokenizer.SplitType.values())
    System.out.println(c);

Returns:

an array containing the constants of this enum type, in the order they are declared

valueOf

public static SplitFunctionTokenizer.SplitType valueOf(String name)

Returns the enum constant of this type with the specified name. The string must match exactly an identifier used to declare an enum constant in this type. (Extraneous whitespace characters are not permitted.)

Parameters:

name - the name of the enum constant to be returned.

Returns:

the enum constant with the specified name

Throws:

IllegalArgumentException - if this enum type has no constant with the specified name

NullPointerException - if the argument is null