Class AdaGrad

java.lang.Object
org.tribuo.math.optimisers.AdaGrad
All Implemented Interfaces:
com.oracle.labs.mlrg.olcut.config.Configurable, com.oracle.labs.mlrg.olcut.provenance.Provenancable<com.oracle.labs.mlrg.olcut.provenance.ConfiguredObjectProvenance>, StochasticGradientOptimiser
public class AdaGrad extends Object implements StochasticGradientOptimiser
An implementation of the AdaGrad gradient optimiser.

Creates one copy of the parameters to store learning rates.

See: 

 Duchi, J., Hazan, E., and Singer, Y.
 "Adaptive Subgradient Methods for Online Learning and Stochastic Optimization"
 Journal of Machine Learning Research, 2012, 2121-2159.

  • Constructor Summary

    Constructors
    Constructor
    Description
    AdaGrad(double initialLearningRate)
    Creates an AdaGrad optimiser using the specified initial learning rate.
    AdaGrad(double initialLearningRate, double epsilon)
    Creates an AdaGrad optimiser using the specified learning rate and epsilon.
    AdaGrad(double initialLearningRate, double epsilon, double initialValue)
    Creates an AdaGrad optimiser using the specified learning rate, epsilon and initial accumulator value.

  • Method Summary

    Modifier and Type
    Method
    Description
    AdaGrad
    copy()
    Copies a gradient optimiser with it's configuration.
    com.oracle.labs.mlrg.olcut.provenance.ConfiguredObjectProvenance
    getProvenance()
     
    void
    initialise(Parameters parameters)
    Initialises the gradient optimiser.
    void
    reset()
    Resets the optimiser so it's ready to optimise a new Parameters.
    Tensor[]
    step(Tensor[] updates, double weight)
    Take a Tensor array of gradients and transform them according to the current weight and learning rates.
    String
    toString()
     

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

    Methods inherited from interface com.oracle.labs.mlrg.olcut.config.Configurable

    postConfig

    Methods inherited from interface org.tribuo.math.StochasticGradientOptimiser

    finalise

  • Constructor Details

    • AdaGrad

      public AdaGrad(double initialLearningRate, double epsilon, double initialValue)
      Creates an AdaGrad optimiser using the specified learning rate, epsilon and initial accumulator value.
      Parameters:
      initialLearningRate - The learning rate.
      epsilon - The epsilon value for stabilising the gradient inversion.
      initialValue - The initial value for the gradient accumulator.

    • AdaGrad

      public AdaGrad(double initialLearningRate, double epsilon)
      Creates an AdaGrad optimiser using the specified learning rate and epsilon.

      Sets the initial value for the accumulator to zero.

      Parameters:
      initialLearningRate - The learning rate.
      epsilon - The epsilon value for stabilising the gradient inversion.

    • AdaGrad

      public AdaGrad(double initialLearningRate)
      Creates an AdaGrad optimiser using the specified initial learning rate.

      Sets epsilon to 1e-6, and the initial accumulator value to zero.

      Parameters:
      initialLearningRate - The learning rate.

  • Method Details

    • initialise

      public void initialise(Parameters parameters)
      Description copied from interface: StochasticGradientOptimiser
      Initialises the gradient optimiser.

      Configures any learning rate parameters.

      Specified by:
      initialise in interface StochasticGradientOptimiser
      Parameters:
      parameters - The parameters to optimise.

    • step

      public Tensor[] step(Tensor[] updates, double weight)
      Description copied from interface: StochasticGradientOptimiser
      Take a Tensor array of gradients and transform them according to the current weight and learning rates.

      Can return the same Tensor array or a new one.

      Specified by:
      step in interface StochasticGradientOptimiser
      Parameters:
      updates - An array of gradients.
      weight - The weight for the current gradients.
      Returns:
      A Tensor array of gradients.

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • reset

      public void reset()
      Description copied from interface: StochasticGradientOptimiser
      Resets the optimiser so it's ready to optimise a new Parameters.
      Specified by:
      reset in interface StochasticGradientOptimiser

    • copy

      public AdaGrad copy()
      Description copied from interface: StochasticGradientOptimiser
      Copies a gradient optimiser with it's configuration. Usually calls the copy constructor.
      Specified by:
      copy in interface StochasticGradientOptimiser
      Returns:
      A gradient optimiser with the same configuration, but independent state.

    • getProvenance

      public com.oracle.labs.mlrg.olcut.provenance.ConfiguredObjectProvenance getProvenance()
      Specified by:
      getProvenance in interface com.oracle.labs.mlrg.olcut.provenance.Provenancable<com.oracle.labs.mlrg.olcut.provenance.ConfiguredObjectProvenance>