Class MinimumCardinalityDataset<T extends Output<T>>

java.lang.Object
org.tribuo.Dataset<T>
org.tribuo.ImmutableDataset<T>
org.tribuo.dataset.MinimumCardinalityDataset<T>
Type Parameters:
T - The type of the outputs in this Dataset.
All Implemented Interfaces:
com.oracle.labs.mlrg.olcut.provenance.Provenancable<DatasetProvenance>, Serializable, Iterable<Example<T>>, ProtoSerializable<org.tribuo.protos.core.DatasetProto>

public class MinimumCardinalityDataset<T extends Output<T>> extends ImmutableDataset<T>
This class creates a pruned dataset in which low frequency features that occur less than the provided minimum cardinality have been removed. This can be useful when the dataset is very large due to many low-frequency features. For example, this class can be used to remove low frequency words from a BoW formatted dataset. Here, a new dataset is created so that the feature counts are recalculated and so that the original, passed-in dataset is not modified. The returned dataset may have fewer examples because if any of the examples have no features after the minimum cardinality has been applied, then those examples will not be added to the constructed dataset.
See Also:
  • Field Details

    • CURRENT_VERSION

      public static final int CURRENT_VERSION
      Protobuf serialization version.
      See Also:
  • Constructor Details

    • MinimumCardinalityDataset

      public MinimumCardinalityDataset(Dataset<T> dataset, int minCardinality)
      Parameters:
      dataset - this dataset is left untouched and is used to populate the constructed dataset.
      minCardinality - features with a frequency less than minCardinality will be removed.
  • Method Details

    • deserializeFromProto

      public static MinimumCardinalityDataset<?> deserializeFromProto(int version, String className, com.google.protobuf.Any message) throws com.google.protobuf.InvalidProtocolBufferException
      Deserialization factory.
      Parameters:
      version - The serialized object version.
      className - The class name.
      message - The serialized data.
      Returns:
      The deserialized object.
      Throws:
      com.google.protobuf.InvalidProtocolBufferException - If the protobuf could not be parsed from the message.
    • getRemoved

      public Set<String> getRemoved()
      The feature names that were removed.
      Returns:
      The feature names.
    • getNumExamplesRemoved

      public int getNumExamplesRemoved()
      The number of examples removed due to a lack of features.
      Returns:
      The number of removed examples.
    • getMinCardinality

      public int getMinCardinality()
      The minimum cardinality threshold for the features.
      Returns:
      The cardinality threshold.
    • getProvenance

      public DatasetProvenance getProvenance()
      Specified by:
      getProvenance in interface com.oracle.labs.mlrg.olcut.provenance.Provenancable<T extends Output<T>>
      Overrides:
      getProvenance in class ImmutableDataset<T extends Output<T>>
    • serialize

      public org.tribuo.protos.core.DatasetProto serialize()
      Description copied from interface: ProtoSerializable
      Serializes this object to a protobuf.
      Specified by:
      serialize in interface ProtoSerializable<T extends Output<T>>
      Overrides:
      serialize in class ImmutableDataset<T extends Output<T>>
      Returns:
      The protobuf.