org.tribuo.util.tokens

Class Token

java.lang.Object

org.tribuo.util.tokens.Token

public class Token
extends Object

A single token extracted from a String.

Tokens are immutable.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Token.TokenType Tokenizers may product multiple kinds of tokens, depending on the application to which they're being put.

Field Summary

Fields

Modifier and Type	Field and Description
int	end
int	start
String	text
Token.TokenType	type

Constructor Summary

Constructors

Constructor and Description
Token(String text, int start, int end) Constructs a token.
Token(String text, int start, int end, Token.TokenType type) Constructs a token.

Method Summary

Modifier and Type	Method and Description
int	length() The number of characters in this token.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

text

public final String text

start

public final int start

end

public final int end

type

public final Token.TokenType type

Constructor Detail

Token

public Token(String text,
             int start,
             int end)

Constructs a token.

Parameters:

text - should be equivalent to the substring of the original tokenized text for the given character offsets start and end

start - the starting offset of the token

end - the ending offset of the token (exclusive or inclusive?)

Token

public Token(String text,
             int start,
             int end,
             Token.TokenType type)

Constructs a token.

Parameters:

text - should be equivalent to the substring of the original tokenized text for the given character offsets start and end

start - the starting offset of the token

end - the ending offset of the token (exclusive or inclusive?)

type - the type of the token

Method Detail

length

public int length()

The number of characters in this token.

Returns:

The number of characters.

toString

public String toString()

Overrides:

toString in class Object