Package org.tribuo.util.tokens


package org.tribuo.util.tokens
Core definitions for tokenization. Tokens are represented by instances of Token, and emitted by implementations of Tokenizer.