org.tribuo.data.columnar

Class RowProcessor<T extends Output<T>>

java.lang.Object

org.tribuo.data.columnar.RowProcessor<T>

All Implemented Interfaces:

com.oracle.labs.mlrg.olcut.config.Configurable, com.oracle.labs.mlrg.olcut.provenance.Provenancable<com.oracle.labs.mlrg.olcut.provenance.ConfiguredObjectProvenance>

public class RowProcessor<T extends Output<T>>
extends Object
implements com.oracle.labs.mlrg.olcut.config.Configurable, com.oracle.labs.mlrg.olcut.provenance.Provenancable<com.oracle.labs.mlrg.olcut.provenance.ConfiguredObjectProvenance>

A processor which takes a Map of String to String and returns an Example.

It accepts a ResponseProcessor which converts the response field into an Output, a Map of FieldProcessors which converts fields into ColumnarFeatures, and a Set of FeatureProcessors which processes the list of ColumnarFeatures before Example construction. Optionally metadata and weights can be extracted using FieldExtractors and written into each example as they are constructed.

If the metadata extractors are invalid (i.e., two extractors write to the same metadata key), the RowProcessor throws PropertyException.

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	configured
protected Map<String,FieldProcessor>	fieldProcessorMap
protected Map<String,FieldProcessor>	regexMappingProcessors
protected ResponseProcessor<T>	responseProcessor
protected FieldExtractor<Float>	weightExtractor

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	RowProcessor() For olcut.
	RowProcessor(List<FieldExtractor<?>> metadataExtractors, FieldExtractor<Float> weightExtractor, ResponseProcessor<T> responseProcessor, Map<String,FieldProcessor> fieldProcessorMap, Map<String,FieldProcessor> regexMappingProcessors, Set<FeatureProcessor> featureProcessors) Constructs a RowProcessor using the supplied responseProcessor to extract the response variable, and the supplied fieldProcessorMap to control which fields are parsed and how they are parsed.
	RowProcessor(List<FieldExtractor<?>> metadataExtractors, FieldExtractor<Float> weightExtractor, ResponseProcessor<T> responseProcessor, Map<String,FieldProcessor> fieldProcessorMap, Set<FeatureProcessor> featureProcessors) Constructs a RowProcessor using the supplied responseProcessor to extract the response variable, and the supplied fieldProcessorMap to control which fields are parsed and how they are parsed.
	RowProcessor(ResponseProcessor<T> responseProcessor, Map<String,FieldProcessor> fieldProcessorMap) Constructs a RowProcessor using the supplied responseProcessor to extract the response variable, and the supplied fieldProcessorMap to control which fields are parsed and how they are parsed.
	RowProcessor(ResponseProcessor<T> responseProcessor, Map<String,FieldProcessor> fieldProcessorMap, Set<FeatureProcessor> featureProcessors) Constructs a RowProcessor using the supplied responseProcessor to extract the response variable, and the supplied fieldProcessorMap to control which fields are parsed and how they are parsed.

Method Summary

Modifier and Type	Method and Description
RowProcessor<T>	copy() Deprecated. In a future release this API will change, in the meantime this is the correct way to get a row processor with clean state. When using regexMappingProcessors, RowProcessor is stateful in a way that can sometimes make it fail the second time it is used. Concretely: RowProcessor rp; Dataset ds1 = new MutableDataset(new CSVDataSource(csvfile1, rp)); Dataset ds2 = new MutableDataset(new CSVDataSource(csvfile2, rp)); // this may fail due to state in rp This method returns a RowProcessor with clean state and the same configuration as this row processor.
void	expandRegexMapping(Collection<String> fieldNames) Uses similar logic to TransformationMap.validateTransformations(org.tribuo.FeatureMap) to check the regexes against the supplied list of field names.
void	expandRegexMapping(ImmutableFeatureMap featureMap) Uses similar logic to TransformationMap.validateTransformations(org.tribuo.FeatureMap) to check the regexes against the supplied feature map.
void	expandRegexMapping(Model<T> model) Uses similar logic to TransformationMap.validateTransformations(org.tribuo.FeatureMap) to check the regexes against the ImmutableFeatureMap contained in the supplied Model.
Optional<Example<T>>	generateExample(ColumnarIterator.Row row, boolean outputRequired) Generate an Example from the supplied row.
Optional<Example<T>>	generateExample(long idx, Map<String,String> row, boolean outputRequired) Generate an Example from the supplied row.
Optional<Example<T>>	generateExample(Map<String,String> row, boolean outputRequired) Generate an Example from the supplied row.
List<ColumnarFeature>	generateFeatures(Map<String,String> row) Generates the features from the supplied row.
Map<String,Object>	generateMetadata(ColumnarIterator.Row row) Generates the example metadata from the supplied row and index.
Set<String>	getColumnNames() The set of column names this will use for the feature processing.
String	getDescription() Returns a description of the row processor and it's fields.
Set<FeatureProcessor>	getFeatureProcessors() Returns the set of FeatureProcessors this RowProcessor uses.
Map<String,FieldProcessor>	getFieldProcessors() Returns the map of FieldProcessors this RowProcessor uses.
Map<String,Class<?>>	getMetadataTypes() Returns the metadata keys and value types that are extracted by this RowProcessor.
com.oracle.labs.mlrg.olcut.provenance.ConfiguredObjectProvenance	getProvenance()
ResponseProcessor<T>	getResponseProcessor() Returns the response processor this RowProcessor uses.
boolean	isConfigured() Returns true if the regexes have been expanded into field processors.
protected Set<String>	partialExpandRegexMapping(Collection<String> fieldNames) Caveat Implementor! This method contains the logic of expandRegexMapping(org.tribuo.Model<T>) without any of the checks that ensure the RowProcessor is in a valid state.
void	postConfig() Used by the OLCUT configuration system, and should not be called by external code.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

weightExtractor

@Config(description="Extractor for the example weight.")
protected FieldExtractor<Float> weightExtractor

responseProcessor

@Config(mandatory=true,
        description="Processor which extracts the response.")
protected ResponseProcessor<T extends Output<T>> responseProcessor

fieldProcessorMap

protected Map<String,FieldProcessor> fieldProcessorMap

regexMappingProcessors

@Config(description="A map from a regex to field processors to apply to fields matching the regex.")
protected Map<String,FieldProcessor> regexMappingProcessors

configured

protected boolean configured

Constructor Detail

RowProcessor

public RowProcessor(ResponseProcessor<T> responseProcessor,
                    Map<String,FieldProcessor> fieldProcessorMap)

Constructs a RowProcessor using the supplied responseProcessor to extract the response variable, and the supplied fieldProcessorMap to control which fields are parsed and how they are parsed.

This processor does not generate any additional metadata for the examples, nor does it set the weight value on generated examples.

Parameters:

responseProcessor - The response processor to use.

fieldProcessorMap - The keys are the field names and the values are the field processors to apply to those fields.

RowProcessor

public RowProcessor(ResponseProcessor<T> responseProcessor,
                    Map<String,FieldProcessor> fieldProcessorMap,
                    Set<FeatureProcessor> featureProcessors)

Constructs a RowProcessor using the supplied responseProcessor to extract the response variable, and the supplied fieldProcessorMap to control which fields are parsed and how they are parsed.

After extraction the features are then processed using the supplied set of feature processors. These processors can be used to insert conjunction features which are triggered when multiple features appear, or to filter out unnecessary features.

This processor does not generate any additional metadata for the examples, nor does it set the weight value on generated examples.

Parameters:

responseProcessor - The response processor to use.

fieldProcessorMap - The keys are the field names and the values are the field processors to apply to those fields.

featureProcessors - The feature processors to run on each extracted feature list.

RowProcessor

public RowProcessor(List<FieldExtractor<?>> metadataExtractors,
                    FieldExtractor<Float> weightExtractor,
                    ResponseProcessor<T> responseProcessor,
                    Map<String,FieldProcessor> fieldProcessorMap,
                    Set<FeatureProcessor> featureProcessors)

Constructs a RowProcessor using the supplied responseProcessor to extract the response variable, and the supplied fieldProcessorMap to control which fields are parsed and how they are parsed.

After extraction the features are then processed using the supplied set of feature processors. These processors can be used to insert conjunction features which are triggered when multiple features appear, or to filter out unnecessary features.

Additionally this processor can extract a weight from each row and insert it into the example, along with more general metadata fields (e.g., the row number, date stamps). The weightExtractor can be null, and if so the weights are left unset.

Parameters:

metadataExtractors - The metadata extractors to run per example. If two metadata extractors emit the same metadata name then the constructor throws a PropertyException.

weightExtractor - The weight extractor, if null the weights are left unset at their default.

responseProcessor - The response processor to use.

fieldProcessorMap - The keys are the field names and the values are the field processors to apply to those fields.

featureProcessors - The feature processors to run on each extracted feature list.

RowProcessor

public RowProcessor(List<FieldExtractor<?>> metadataExtractors,
                    FieldExtractor<Float> weightExtractor,
                    ResponseProcessor<T> responseProcessor,
                    Map<String,FieldProcessor> fieldProcessorMap,
                    Map<String,FieldProcessor> regexMappingProcessors,
                    Set<FeatureProcessor> featureProcessors)

Constructs a RowProcessor using the supplied responseProcessor to extract the response variable, and the supplied fieldProcessorMap to control which fields are parsed and how they are parsed.

In addition this processor can instantiate field processors which match the regexes supplied in the regexMappingProcessors. If a regex matches a field which already has a fieldProcessor assigned to it, it throws an IllegalArgumentException.

After extraction the features are then processed using the supplied set of feature processors. These processors can be used to insert conjunction features which are triggered when multiple features appear, or to filter out unnecessary features.

Additionally this processor can extract a weight from each row and insert it into the example, along with more general metadata fields (e.g., the row number, date stamps). The weightExtractor can be null, and if so the weights are left unset.

Parameters:

metadataExtractors - The metadata extractors to run per example. If two metadata extractors emit the same metadata name then the constructor throws a PropertyException.

weightExtractor - The weight extractor, if null the weights are left unset at their default.

responseProcessor - The response processor to use.

fieldProcessorMap - The keys are the field names and the values are the field processors to apply to those fields.

regexMappingProcessors - A set of field processors which can be instantiated if the regexes match the field names.

featureProcessors - The feature processors to run on each extracted feature list.

RowProcessor

protected RowProcessor()

For olcut.

Method Detail

postConfig

public void postConfig()

Used by the OLCUT configuration system, and should not be called by external code.

Specified by:

postConfig in interface com.oracle.labs.mlrg.olcut.config.Configurable

getResponseProcessor

public ResponseProcessor<T> getResponseProcessor()

Returns the response processor this RowProcessor uses.

Returns:

The response processor.

getFieldProcessors

public Map<String,FieldProcessor> getFieldProcessors()

Returns the map of FieldProcessors this RowProcessor uses.

Returns:

The field processors.

getFeatureProcessors

public Set<FeatureProcessor> getFeatureProcessors()

Returns the set of FeatureProcessors this RowProcessor uses.

Returns:

The feature processors.

generateExample

public Optional<Example<T>> generateExample(ColumnarIterator.Row row,
                                            boolean outputRequired)

Generate an Example from the supplied row. Returns an empty Optional if there are no features, or the response is required but it was not found. The latter case is used at training time.

Parameters:

row - The row to process.

outputRequired - If an Output must be found in the row to return an Example.

Returns:

An Optional containing an Example if the row was valid, an empty Optional otherwise.

generateExample

public Optional<Example<T>> generateExample(Map<String,String> row,
                                            boolean outputRequired)

Generate an Example from the supplied row. Returns an empty Optional if there are no features, or the response is required but it was not found.

Supplies -1 as the example index, used in cases where the index isn't meaningful.

Parameters:

row - The row to process.

outputRequired - If an Output must be found in the row to return an Example.

Returns:

An Optional containing an Example if the row was valid, an empty Optional otherwise.

generateExample

public Optional<Example<T>> generateExample(long idx,
                                            Map<String,String> row,
                                            boolean outputRequired)

Generate an Example from the supplied row. Returns an empty Optional if there are no features, or the response is required but it was not found. The latter case is used at training time.

Parameters:

idx - The index for use in the example metadata if desired.

row - The row to process.

outputRequired - If an Output must be found in the row to return an Example.

Returns:

An Optional containing an Example if the row was valid, an empty Optional otherwise.

generateMetadata

public Map<String,Object> generateMetadata(ColumnarIterator.Row row)

Generates the example metadata from the supplied row and index.

Parameters:

row - The row to process.

Returns:

A (possibly empty) map containing the metadata.

generateFeatures

public List<ColumnarFeature> generateFeatures(Map<String,String> row)

Generates the features from the supplied row.

Parameters:

row - The row to process.

Returns:

A (possibly empty) list of ColumnarFeatures.

getColumnNames

public Set<String> getColumnNames()

The set of column names this will use for the feature processing.

Returns:

The set of column names it processes.

getDescription

public String getDescription()

Returns a description of the row processor and it's fields.

Returns:

A String description of the RowProcessor.

toString

public String toString()

Overrides:

toString in class Object

getMetadataTypes

public Map<String,Class<?>> getMetadataTypes()

Returns the metadata keys and value types that are extracted by this RowProcessor.

Returns:

The metadata keys and value types.

isConfigured

public boolean isConfigured()

Returns true if the regexes have been expanded into field processors.

Returns:

True if the RowProcessor has seen the set of input fields.

expandRegexMapping

public void expandRegexMapping(Model<T> model)

Uses similar logic to TransformationMap.validateTransformations(org.tribuo.FeatureMap) to check the regexes against the ImmutableFeatureMap contained in the supplied Model. Throws an IllegalArgumentException if any regexes overlap with themselves, or with the currently defined set of fieldProcessorMap.

Parameters:

model - The model to use to expand the regexes.

expandRegexMapping

public void expandRegexMapping(ImmutableFeatureMap featureMap)

Uses similar logic to TransformationMap.validateTransformations(org.tribuo.FeatureMap) to check the regexes against the supplied feature map. Throws an IllegalArgumentException if any regexes overlap with themselves, or with the currently defined set of fieldProcessorMap.

Parameters:

featureMap - The feature map to use to expand the regexes.

expandRegexMapping

public void expandRegexMapping(Collection<String> fieldNames)

Uses similar logic to TransformationMap.validateTransformations(org.tribuo.FeatureMap) to check the regexes against the supplied list of field names. Throws an IllegalArgumentException if any regexes overlap with themselves, or with the currently defined set of fieldProcessorMap or if there are unmatched regexes after processing.

Parameters:

fieldNames - The list of field names.

partialExpandRegexMapping

protected Set<String> partialExpandRegexMapping(Collection<String> fieldNames)

Caveat Implementor! This method contains the logic of expandRegexMapping(org.tribuo.Model<T>) without any of the checks that ensure the RowProcessor is in a valid state. This can be used in a subclass to expand a regex mapping several times for a single instance of RowProcessor. The caller is responsible for ensuring that fieldNames are not duplicated within or between calls.

Parameters:

fieldNames - The list of field names - should contain only previously unseen field names.

Returns:

the set of regexes that were matched by fieldNames.

copy

@Deprecated
public RowProcessor<T> copy()

Deprecated. In a future release this API will change, in the meantime this is the correct way to get a row processor with clean state.

When using regexMappingProcessors, RowProcessor is stateful in a way that can sometimes make it fail the second time it is used. Concretely:

     RowProcessor rp;
     Dataset ds1 = new MutableDataset(new CSVDataSource(csvfile1, rp));
     Dataset ds2 = new MutableDataset(new CSVDataSource(csvfile2, rp)); // this may fail due to state in rp
 

This method returns a RowProcessor with clean state and the same configuration as this row processor.

Returns:

a RowProcessor instance with clean state and the same configuration as this row processor.

getProvenance

public com.oracle.labs.mlrg.olcut.provenance.ConfiguredObjectProvenance getProvenance()

Specified by:

getProvenance in interface com.oracle.labs.mlrg.olcut.provenance.Provenancable<com.oracle.labs.mlrg.olcut.provenance.ConfiguredObjectProvenance>