Class Token

java.lang.Object

org.tribuo.util.tokens.Token

public class Token
extends Object

A single token extracted from a String.

Tokens are immutable.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static enum	Token.TokenType	Tokenizers may product multiple kinds of tokens, depending on the application to which they're being put.

Field Summary

Fields

Modifier and Type	Field	Description
final int	end
final int	start
final String	text
final Token.TokenType	type

Constructor Summary

Constructors

Constructor	Description
Token(String text, int start, int end)	Constructs a token.
Token(String text, int start, int end, Token.TokenType type)	Constructs a token.

Method Summary

Modifier and Type	Method	Description
int	length()	The number of characters in this token.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details

text

public final String text

start

public final int start

end

public final int end

type

public final Token.TokenType type

Constructor Details

Token

public Token(String text,
 int start,
 int end)

Constructs a token.

Parameters:

text - should be equivalent to the substring of the original tokenized text for the given character offsets start and end

start - the starting offset of the token

end - the ending offset of the token (exclusive or inclusive?)

Token

public Token(String text,
 int start,
 int end,
 Token.TokenType type)

Constructs a token.

Parameters:

text - should be equivalent to the substring of the original tokenized text for the given character offsets start and end

start - the starting offset of the token

end - the ending offset of the token (exclusive or inclusive?)

type - the type of the token

Method Details

length

public int length()

The number of characters in this token.

Returns:

The number of characters.

toString

public String toString()

Overrides:

toString in class Object