RowProcessor (Tribuo 4.1.1 API)

java.lang.Object
- org.tribuo.data.columnar.RowProcessor<T>

All Implemented Interfaces:

com.oracle.labs.mlrg.olcut.config.Configurable, com.oracle.labs.mlrg.olcut.provenance.Provenancable<com.oracle.labs.mlrg.olcut.provenance.ConfiguredObjectProvenance>
```
public class RowProcessor<T extends Output<T>>
extends Object
implements com.oracle.labs.mlrg.olcut.config.Configurable, com.oracle.labs.mlrg.olcut.provenance.Provenancable<com.oracle.labs.mlrg.olcut.provenance.ConfiguredObjectProvenance>
```
A processor which takes a Map of String to String and returns an Example.
It accepts a ResponseProcessor which converts the response field into an Output, a Map of FieldProcessors which converts fields into ColumnarFeatures, and a Set of FeatureProcessors which processes the list of ColumnarFeatures before Example construction. Optionally metadata and weights can be extracted using FieldExtractors and written into each example as they are constructed.
If the metadata extractors are invalid (i.e., two extractors write to the same metadata key), the RowProcessor throws PropertyException.

Field Summary

Fields
Modifier and Type	Field and Description
`protected boolean`	`configured`
`protected Map<String,FieldProcessor>`	`fieldProcessorMap`
`protected Map<String,FieldProcessor>`	`regexMappingProcessors`
`protected boolean`	`replaceNewlinesWithSpaces`
`protected ResponseProcessor<T>`	`responseProcessor`
`protected FieldExtractor<Float>`	`weightExtractor`

Constructor Summary

Constructors
Modifier	Constructor and Description
`protected`	`RowProcessor()` For olcut.
	`RowProcessor(List<FieldExtractor<?>> metadataExtractors, FieldExtractor<Float> weightExtractor, ResponseProcessor<T> responseProcessor, Map<String,FieldProcessor> fieldProcessorMap, Map<String,FieldProcessor> regexMappingProcessors, Set<FeatureProcessor> featureProcessors)` Constructs a RowProcessor using the supplied responseProcessor to extract the response variable, and the supplied fieldProcessorMap to control which fields are parsed and how they are parsed.
	`RowProcessor(List<FieldExtractor<?>> metadataExtractors, FieldExtractor<Float> weightExtractor, ResponseProcessor<T> responseProcessor, Map<String,FieldProcessor> fieldProcessorMap, Map<String,FieldProcessor> regexMappingProcessors, Set<FeatureProcessor> featureProcessors, boolean replaceNewlinesWithSpaces)` Constructs a RowProcessor using the supplied responseProcessor to extract the response variable, and the supplied fieldProcessorMap to control which fields are parsed and how they are parsed.
	`RowProcessor(List<FieldExtractor<?>> metadataExtractors, FieldExtractor<Float> weightExtractor, ResponseProcessor<T> responseProcessor, Map<String,FieldProcessor> fieldProcessorMap, Set<FeatureProcessor> featureProcessors)` Constructs a RowProcessor using the supplied responseProcessor to extract the response variable, and the supplied fieldProcessorMap to control which fields are parsed and how they are parsed.
	`RowProcessor(List<FieldExtractor<?>> metadataExtractors, ResponseProcessor<T> responseProcessor, Map<String,FieldProcessor> fieldProcessorMap)` Constructs a RowProcessor using the supplied responseProcessor to extract the response variable, and the supplied fieldProcessorMap to control which fields are parsed and how they are parsed.
	`RowProcessor(ResponseProcessor<T> responseProcessor, Map<String,FieldProcessor> fieldProcessorMap)` Constructs a RowProcessor using the supplied responseProcessor to extract the response variable, and the supplied fieldProcessorMap to control which fields are parsed and how they are parsed.
	`RowProcessor(ResponseProcessor<T> responseProcessor, Map<String,FieldProcessor> fieldProcessorMap, Set<FeatureProcessor> featureProcessors)` Constructs a RowProcessor using the supplied responseProcessor to extract the response variable, and the supplied fieldProcessorMap to control which fields are parsed and how they are parsed.

Method Summary

All Methods Instance Methods Concrete Methods Deprecated Methods
Modifier and Type	Method and Description
`RowProcessor<T>`	`copy()` Deprecated. In a future release this API will change, in the meantime this is the correct way to get a row processor with clean state. When using regexMappingProcessors, RowProcessor is stateful in a way that can sometimes make it fail the second time it is used. Concretely: RowProcessor rp; Dataset ds1 = new MutableDataset(new CSVDataSource(csvfile1, rp)); Dataset ds2 = new MutableDataset(new CSVDataSource(csvfile2, rp)); // this may fail due to state in rp This method returns a RowProcessor with clean state and the same configuration as this row processor.
`void`	`expandRegexMapping(Collection<String> fieldNames)` Uses similar logic to `TransformationMap.validateTransformations(org.tribuo.FeatureMap)` to check the regexes against the supplied list of field names.
`void`	`expandRegexMapping(ImmutableFeatureMap featureMap)` Uses similar logic to `TransformationMap.validateTransformations(org.tribuo.FeatureMap)` to check the regexes against the supplied feature map.
`void`	`expandRegexMapping(Model<T> model)` Uses similar logic to `TransformationMap.validateTransformations(org.tribuo.FeatureMap)` to check the regexes against the `ImmutableFeatureMap` contained in the supplied `Model`.
`Optional<Example<T>>`	`generateExample(ColumnarIterator.Row row, boolean outputRequired)` Generate an `Example` from the supplied row.
`Optional<Example<T>>`	`generateExample(long idx, Map<String,String> row, boolean outputRequired)` Generate an `Example` from the supplied row.
`Optional<Example<T>>`	`generateExample(Map<String,String> row, boolean outputRequired)` Generate an `Example` from the supplied row.
`List<ColumnarFeature>`	`generateFeatures(Map<String,String> row)` Generates the features from the supplied row.
`Map<String,Object>`	`generateMetadata(ColumnarIterator.Row row)` Generates the example metadata from the supplied row and index.
`Set<String>`	`getColumnNames()` The set of column names this will use for the feature processing.
`String`	`getDescription()` Returns a description of the row processor and it's fields.
`Set<FeatureProcessor>`	`getFeatureProcessors()` Returns the set of `FeatureProcessor`s this RowProcessor uses.
`Map<String,FieldProcessor>`	`getFieldProcessors()` Returns the map of `FieldProcessor`s this RowProcessor uses.
`Map<String,Class<?>>`	`getMetadataTypes()` Returns the metadata keys and value types that are extracted by this RowProcessor.
`com.oracle.labs.mlrg.olcut.provenance.ConfiguredObjectProvenance`	`getProvenance()`
`ResponseProcessor<T>`	`getResponseProcessor()` Returns the response processor this RowProcessor uses.
`boolean`	`isConfigured()` Returns true if the regexes have been expanded into field processors.
`protected Set<String>`	`partialExpandRegexMapping(Collection<String> fieldNames)` Caveat Implementor! This method contains the logic of `expandRegexMapping(org.tribuo.Model<T>)` without any of the checks that ensure the RowProcessor is in a valid state.
`void`	`postConfig()` Used by the OLCUT configuration system, and should not be called by external code.
`String`	`toString()`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Field Detail
  - weightExtractor
```
@Config(description="Extractor for the example weight.")
protected FieldExtractor<Float> weightExtractor
```
  - responseProcessor
```
@Config(mandatory=true,
        description="Processor which extracts the response.")
protected ResponseProcessor<T extends Output<T>> responseProcessor
```
  - fieldProcessorMap
```
protected Map<String,FieldProcessor> fieldProcessorMap
```
  - regexMappingProcessors
```
@Config(description="A map from a regex to field processors to apply to fields matching the regex.")
protected Map<String,FieldProcessor> regexMappingProcessors
```
  - replaceNewlinesWithSpaces
```
@Config(description="Replace newlines with spaces in values before passing them to field processors.")
protected boolean replaceNewlinesWithSpaces
```
  - configured
```
protected boolean configured
```
- Constructor Detail
  - RowProcessor
```
public RowProcessor(ResponseProcessor<T> responseProcessor,
                    Map<String,FieldProcessor> fieldProcessorMap)
```
    Constructs a RowProcessor using the supplied responseProcessor to extract the response variable, and the supplied fieldProcessorMap to control which fields are parsed and how they are parsed.
    This processor does not generate any additional metadata for the examples, nor does it set the weight value on generated examples.
    
    Parameters:
    
    responseProcessor - The response processor to use.
    
    fieldProcessorMap - The keys are the field names and the values are the field processors to apply to those fields.
  - RowProcessor
```
public RowProcessor(ResponseProcessor<T> responseProcessor,
                    Map<String,FieldProcessor> fieldProcessorMap,
                    Set<FeatureProcessor> featureProcessors)
```
    Constructs a RowProcessor using the supplied responseProcessor to extract the response variable, and the supplied fieldProcessorMap to control which fields are parsed and how they are parsed.
    After extraction the features are then processed using the supplied set of feature processors. These processors can be used to insert conjunction features which are triggered when multiple features appear, or to filter out unnecessary features.
    This processor does not generate any additional metadata for the examples, nor does it set the weight value on generated examples.
    
    Parameters:
    
    responseProcessor - The response processor to use.
    
    fieldProcessorMap - The keys are the field names and the values are the field processors to apply to those fields.
    
    featureProcessors - The feature processors to run on each extracted feature list.
  - RowProcessor
```
public RowProcessor(List<FieldExtractor<?>> metadataExtractors,
                    ResponseProcessor<T> responseProcessor,
                    Map<String,FieldProcessor> fieldProcessorMap)
```
    Constructs a RowProcessor using the supplied responseProcessor to extract the response variable, and the supplied fieldProcessorMap to control which fields are parsed and how they are parsed.
    Additionally this processor can extract and populate metadata fields on the generated examples (e.g., the row number, date stamps).
    
    Parameters:
    
    metadataExtractors - The metadata extractors to run per example. If two metadata extractors emit the same metadata name then the constructor throws a PropertyException.
    
    responseProcessor - The response processor to use.
    
    fieldProcessorMap - The keys are the field names and the values are the field processors to apply to those fields.
  - RowProcessor
```
public RowProcessor(List<FieldExtractor<?>> metadataExtractors,
                    FieldExtractor<Float> weightExtractor,
                    ResponseProcessor<T> responseProcessor,
                    Map<String,FieldProcessor> fieldProcessorMap,
                    Set<FeatureProcessor> featureProcessors)
```
    Constructs a RowProcessor using the supplied responseProcessor to extract the response variable, and the supplied fieldProcessorMap to control which fields are parsed and how they are parsed.
    After extraction the features are then processed using the supplied set of feature processors. These processors can be used to insert conjunction features which are triggered when multiple features appear, or to filter out unnecessary features.
    Additionally this processor can extract a weight from each row and insert it into the example, along with more general metadata fields (e.g., the row number, date stamps). The weightExtractor can be null, and if so the weights are left unset.
    
    Parameters:
    
    metadataExtractors - The metadata extractors to run per example. If two metadata extractors emit the same metadata name then the constructor throws a PropertyException.
    
    weightExtractor - The weight extractor, if null the weights are left unset at their default.
    
    responseProcessor - The response processor to use.
    
    fieldProcessorMap - The keys are the field names and the values are the field processors to apply to those fields.
    
    featureProcessors - The feature processors to run on each extracted feature list.
  - RowProcessor
```
public RowProcessor(List<FieldExtractor<?>> metadataExtractors,
                    FieldExtractor<Float> weightExtractor,
                    ResponseProcessor<T> responseProcessor,
                    Map<String,FieldProcessor> fieldProcessorMap,
                    Map<String,FieldProcessor> regexMappingProcessors,
                    Set<FeatureProcessor> featureProcessors)
```
    Constructs a RowProcessor using the supplied responseProcessor to extract the response variable, and the supplied fieldProcessorMap to control which fields are parsed and how they are parsed.
    In addition this processor can instantiate field processors which match the regexes supplied in the regexMappingProcessors. If a regex matches a field which already has a fieldProcessor assigned to it, it throws an IllegalArgumentException.
    After extraction the features are then processed using the supplied set of feature processors. These processors can be used to insert conjunction features which are triggered when multiple features appear, or to filter out unnecessary features.
    Additionally this processor can extract a weight from each row and insert it into the example, along with more general metadata fields (e.g., the row number, date stamps). The weightExtractor can be null, and if so the weights are left unset.
    
    Parameters:
    
    metadataExtractors - The metadata extractors to run per example. If two metadata extractors emit the same metadata name then the constructor throws a PropertyException.
    
    weightExtractor - The weight extractor, if null the weights are left unset at their default.
    
    responseProcessor - The response processor to use.
    
    fieldProcessorMap - The keys are the field names and the values are the field processors to apply to those fields.
    
    regexMappingProcessors - A set of field processors which can be instantiated if the regexes match the field names.
    
    featureProcessors - The feature processors to run on each extracted feature list.
  - RowProcessor
```
public RowProcessor(List<FieldExtractor<?>> metadataExtractors,
                    FieldExtractor<Float> weightExtractor,
                    ResponseProcessor<T> responseProcessor,
                    Map<String,FieldProcessor> fieldProcessorMap,
                    Map<String,FieldProcessor> regexMappingProcessors,
                    Set<FeatureProcessor> featureProcessors,
                    boolean replaceNewlinesWithSpaces)
```
    Constructs a RowProcessor using the supplied responseProcessor to extract the response variable, and the supplied fieldProcessorMap to control which fields are parsed and how they are parsed.
    In addition this processor can instantiate field processors which match the regexes supplied in the regexMappingProcessors. If a regex matches a field which already has a fieldProcessor assigned to it, it throws an IllegalArgumentException.
    After extraction the features are then processed using the supplied set of feature processors. These processors can be used to insert conjunction features which are triggered when multiple features appear, or to filter out unnecessary features.
    Additionally this processor can extract a weight from each row and insert it into the example, along with more general metadata fields (e.g., the row number, date stamps). The weightExtractor can be null, and if so the weights are left unset.
    
    Parameters:
    
    metadataExtractors - The metadata extractors to run per example. If two metadata extractors emit the same metadata name then the constructor throws a PropertyException.
    
    weightExtractor - The weight extractor, if null the weights are left unset at their default.
    
    responseProcessor - The response processor to use.
    
    fieldProcessorMap - The keys are the field names and the values are the field processors to apply to those fields.
    
    regexMappingProcessors - A set of field processors which can be instantiated if the regexes match the field names.
    
    featureProcessors - The feature processors to run on each extracted feature list.
    
    replaceNewlinesWithSpaces - Replace newlines with spaces in values before passing them to field processors.
  - RowProcessor
```
protected RowProcessor()
```
    For olcut.
- Method Detail
  - postConfig
```
public void postConfig()
```
    Used by the OLCUT configuration system, and should not be called by external code.
    
    Specified by:
    
    postConfig in interface com.oracle.labs.mlrg.olcut.config.Configurable
  - getResponseProcessor
```
public ResponseProcessor<T> getResponseProcessor()
```
    Returns the response processor this RowProcessor uses.
    
    Returns:
    
    The response processor.
  - getFieldProcessors
```
public Map<String,FieldProcessor> getFieldProcessors()
```
    Returns the map of FieldProcessors this RowProcessor uses.
    
    Returns:
    
    The field processors.
  - getFeatureProcessors
```
public Set<FeatureProcessor> getFeatureProcessors()
```
    Returns the set of FeatureProcessors this RowProcessor uses.
    
    Returns:
    
    The feature processors.
  - generateExample
```
public Optional<Example<T>> generateExample(ColumnarIterator.Row row,
                                            boolean outputRequired)
```
    Generate an Example from the supplied row. Returns an empty Optional if there are no features, or the response is required but it was not found. The latter case is used at training time.
    
    Parameters:
    
    row - The row to process.
    
    outputRequired - If an Output must be found in the row to return an Example.
    
    Returns:
    
    An Optional containing an Example if the row was valid, an empty Optional otherwise.
  - generateExample
```
public Optional<Example<T>> generateExample(Map<String,String> row,
                                            boolean outputRequired)
```
    Generate an Example from the supplied row. Returns an empty Optional if there are no features, or the response is required but it was not found.
    Supplies -1 as the example index, used in cases where the index isn't meaningful.
    
    Parameters:
    
    row - The row to process.
    
    outputRequired - If an Output must be found in the row to return an Example.
    
    Returns:
    
    An Optional containing an Example if the row was valid, an empty Optional otherwise.
  - generateExample
```
public Optional<Example<T>> generateExample(long idx,
                                            Map<String,String> row,
                                            boolean outputRequired)
```
    Generate an Example from the supplied row. Returns an empty Optional if there are no features, or the response is required but it was not found. The latter case is used at training time.
    
    Parameters:
    
    idx - The index for use in the example metadata if desired.
    
    row - The row to process.
    
    outputRequired - If an Output must be found in the row to return an Example.
    
    Returns:
    
    An Optional containing an Example if the row was valid, an empty Optional otherwise.
  - generateMetadata
```
public Map<String,Object> generateMetadata(ColumnarIterator.Row row)
```
    Generates the example metadata from the supplied row and index.
    
    Parameters:
    
    row - The row to process.
    
    Returns:
    
    A (possibly empty) map containing the metadata.
  - generateFeatures
```
public List<ColumnarFeature> generateFeatures(Map<String,String> row)
```
    Generates the features from the supplied row.
    
    Parameters:
    
    row - The row to process.
    
    Returns:
    
    A (possibly empty) list of ColumnarFeatures.
  - getColumnNames
```
public Set<String> getColumnNames()
```
    The set of column names this will use for the feature processing.
    
    Returns:
    
    The set of column names it processes.
  - getDescription
```
public String getDescription()
```
    Returns a description of the row processor and it's fields.
    
    Returns:
    
    A String description of the RowProcessor.
  - toString
```
public String toString()
```
    Overrides:
    
    toString in class Object
  - getMetadataTypes
```
public Map<String,Class<?>> getMetadataTypes()
```
    Returns the metadata keys and value types that are extracted by this RowProcessor.
    
    Returns:
    
    The metadata keys and value types.
  - isConfigured
```
public boolean isConfigured()
```
    Returns true if the regexes have been expanded into field processors.
    
    Returns:
    
    True if the RowProcessor has seen the set of input fields.
  - expandRegexMapping
```
public void expandRegexMapping(Model<T> model)
```
    Uses similar logic to TransformationMap.validateTransformations(org.tribuo.FeatureMap) to check the regexes against the ImmutableFeatureMap contained in the supplied Model. Throws an IllegalArgumentException if any regexes overlap with themselves, or with the currently defined set of fieldProcessorMap.
    
    Parameters:
    
    model - The model to use to expand the regexes.
  - expandRegexMapping
```
public void expandRegexMapping(ImmutableFeatureMap featureMap)
```
    Uses similar logic to TransformationMap.validateTransformations(org.tribuo.FeatureMap) to check the regexes against the supplied feature map. Throws an IllegalArgumentException if any regexes overlap with themselves, or with the currently defined set of fieldProcessorMap.
    
    Parameters:
    
    featureMap - The feature map to use to expand the regexes.
  - expandRegexMapping
```
public void expandRegexMapping(Collection<String> fieldNames)
```
    Uses similar logic to TransformationMap.validateTransformations(org.tribuo.FeatureMap) to check the regexes against the supplied list of field names. Throws an IllegalArgumentException if any regexes overlap with themselves, or with the currently defined set of fieldProcessorMap or if there are unmatched regexes after processing.
    
    Parameters:
    
    fieldNames - The list of field names.
  - partialExpandRegexMapping
```
protected Set<String> partialExpandRegexMapping(Collection<String> fieldNames)
```
    Caveat Implementor! This method contains the logic of expandRegexMapping(org.tribuo.Model<T>) without any of the checks that ensure the RowProcessor is in a valid state. This can be used in a subclass to expand a regex mapping several times for a single instance of RowProcessor. The caller is responsible for ensuring that fieldNames are not duplicated within or between calls.
    
    Parameters:
    
    fieldNames - The list of field names - should contain only previously unseen field names.
    
    Returns:
    
    the set of regexes that were matched by fieldNames.
  - copy
```
@Deprecated
public RowProcessor<T> copy()
```
    Deprecated.
    In a future release this API will change, in the meantime this is the correct way to get a row processor with clean state.
    When using regexMappingProcessors, RowProcessor is stateful in a way that can sometimes make it fail the second time it is used. Concretely:
    RowProcessor rp; Dataset ds1 = new MutableDataset(new CSVDataSource(csvfile1, rp)); Dataset ds2 = new MutableDataset(new CSVDataSource(csvfile2, rp)); // this may fail due to state in rp
    This method returns a RowProcessor with clean state and the same configuration as this row processor.
    Returns:
    
    a RowProcessor instance with clean state and the same configuration as this row processor.
  - getProvenance
```
public com.oracle.labs.mlrg.olcut.provenance.ConfiguredObjectProvenance getProvenance()
```
    Specified by:
    
    getProvenance in interface com.oracle.labs.mlrg.olcut.provenance.Provenancable<com.oracle.labs.mlrg.olcut.provenance.ConfiguredObjectProvenance>

Class RowProcessor<T extends Output<T>>

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

weightExtractor

responseProcessor

fieldProcessorMap

regexMappingProcessors

replaceNewlinesWithSpaces

configured

Constructor Detail

RowProcessor

RowProcessor

RowProcessor

RowProcessor

RowProcessor

RowProcessor

RowProcessor

Method Detail

postConfig

getResponseProcessor

getFieldProcessors

getFeatureProcessors

generateExample

generateExample

generateExample

generateMetadata

generateFeatures

getColumnNames

getDescription

toString

getMetadataTypes

isConfigured

expandRegexMapping

expandRegexMapping

expandRegexMapping

partialExpandRegexMapping

copy

getProvenance