Package org.tribuo.util.tokens

package org.tribuo.util.tokens
Core definitions for tokenization. Tokens are represented by instances of Token, and emitted by implementations of Tokenizer.
  • Class
    A single token extracted from a String.
    Tokenizers may product multiple kinds of tokens, depending on the application to which they're being put.
    Wraps exceptions thrown by tokenizers.
    An interface for things that tokenize text: breaking it into words according to some set of rules.