Package org.tribuo.util.tokens

Class Token

java.lang.Object
org.tribuo.util.tokens.Token
public class Token extends Object
A single token extracted from a String.

Tokens are immutable, and may be records one day.

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static enum 
    Token.TokenType
    Tokenizers may product multiple kinds of tokens, depending on the application to which they're being put.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    final int
    end
    The end index.
    final int
    start
    The start index.
    final String
    text
    The token text.
    final Token.TokenType
    type
    The token type.

  • Constructor Summary

    Constructors
    Constructor
    Description
    Token(String text, int start, int end)
    Constructs a token.
    Token(String text, int start, int end, Token.TokenType type)
    Constructs a token.

  • Method Summary

    Modifier and Type
    Method
    Description
    int
    length()
    The number of characters in this token.
    String
    toString()
     

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

  • Field Details

    • text

      public final String text
      The token text.

    • start

      public final int start
      The start index.

    • end

      public final int end
      The end index.

    • type

      public final Token.TokenType type
      The token type.

  • Constructor Details

    • Token

      public Token(String text, int start, int end)
      Constructs a token.
      Parameters:
      text - should be equivalent to the substring of the original tokenized text for the given character offsets start and end
      start - the starting offset of the token
      end - the ending offset of the token (exclusive or inclusive?)

    • Token

      public Token(String text, int start, int end, Token.TokenType type)
      Constructs a token.
      Parameters:
      text - should be equivalent to the substring of the original tokenized text for the given character offsets start and end
      start - the starting offset of the token
      end - the ending offset of the token (exclusive or inclusive?)
      type - the type of the token

  • Method Details

    • length

      public int length()
      The number of characters in this token.
      Returns:
      The number of characters.

    • toString

      public String toString()
      Overrides:
      toString in class Object