Package org.tribuo.math.optimisers
Class AdaGradRDA
java.lang.Object
org.tribuo.math.optimisers.AdaGradRDA
- All Implemented Interfaces:
com.oracle.labs.mlrg.olcut.config.Configurable
,com.oracle.labs.mlrg.olcut.provenance.Provenancable<com.oracle.labs.mlrg.olcut.provenance.ConfiguredObjectProvenance>
,StochasticGradientOptimiser
An implementation of the AdaGrad gradient optimiser with regularized dual averaging.
This gradient optimiser rewrites all the Tensor
s in the Parameters
with AdaGradRDA.AdaGradRDATensor
. This means it keeps a different value in the Tensor
to the one produced when you call get(), so it can correctly apply regularisation to the parameters.
When finalise()
is called it rewrites the Parameters
with standard dense Tensor
s.
Follows the implementation in Factorie.
See:
Duchi, J., Hazan, E., and Singer, Y. "Adaptive Subgradient Methods for Online Learning and Stochastic Optimization" Journal of Machine Learning Research, 2012, 2121-2159.
-
Constructor Summary
ConstructorDescriptionAdaGradRDA
(double initialLearningRate, double epsilon) Creates an AdaGradRDA optimiser with the specified parameter values.AdaGradRDA
(double initialLearningRate, double epsilon, double l1, double l2, int numExamples) Creates an AdaGradRDA optimiser with the specified parameter values. -
Method Summary
Modifier and TypeMethodDescriptioncopy()
Copies a gradient optimiser with it's configuration.void
finalise()
Finalises the gradient optimisation, setting the parameters to their correct values.com.oracle.labs.mlrg.olcut.provenance.ConfiguredObjectProvenance
void
initialise
(Parameters parameters) Initialises the gradient optimiser.void
reset()
Resets the optimiser so it's ready to optimise a newParameters
.Tensor[]
Take aTensor
array of gradients and transform them according to the current weight and learning rates.toString()
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
Methods inherited from interface com.oracle.labs.mlrg.olcut.config.Configurable
postConfig
-
Constructor Details
-
AdaGradRDA
public AdaGradRDA(double initialLearningRate, double epsilon, double l1, double l2, int numExamples) Creates an AdaGradRDA optimiser with the specified parameter values.- Parameters:
initialLearningRate
- The learning rate.epsilon
- The epsilon value for stabilising the gradient inversion.l1
- The l1 penalty.l2
- The l2 penalty.numExamples
- The number of examples to scale the l1 and l2 penalties.
-
AdaGradRDA
public AdaGradRDA(double initialLearningRate, double epsilon) Creates an AdaGradRDA optimiser with the specified parameter values.Sets the regularisation parameters to zero.
- Parameters:
initialLearningRate
- The learning rate.epsilon
- The epsilon value for stabilising the gradient inversion.
-
-
Method Details
-
initialise
Description copied from interface:StochasticGradientOptimiser
Initialises the gradient optimiser.Configures any learning rate parameters.
- Specified by:
initialise
in interfaceStochasticGradientOptimiser
- Parameters:
parameters
- The parameters to optimise.
-
step
Description copied from interface:StochasticGradientOptimiser
Take aTensor
array of gradients and transform them according to the current weight and learning rates.Can return the same
Tensor
array or a new one.- Specified by:
step
in interfaceStochasticGradientOptimiser
- Parameters:
updates
- An array of gradients.weight
- The weight for the current gradients.- Returns:
- A
Tensor
array of gradients.
-
finalise
public void finalise()Description copied from interface:StochasticGradientOptimiser
Finalises the gradient optimisation, setting the parameters to their correct values. Used forParameterAveraging
amongst others.- Specified by:
finalise
in interfaceStochasticGradientOptimiser
-
toString
-
reset
public void reset()Description copied from interface:StochasticGradientOptimiser
Resets the optimiser so it's ready to optimise a newParameters
.- Specified by:
reset
in interfaceStochasticGradientOptimiser
-
copy
Description copied from interface:StochasticGradientOptimiser
Copies a gradient optimiser with it's configuration. Usually calls the copy constructor.- Specified by:
copy
in interfaceStochasticGradientOptimiser
- Returns:
- A gradient optimiser with the same configuration, but independent state.
-
getProvenance
public com.oracle.labs.mlrg.olcut.provenance.ConfiguredObjectProvenance getProvenance()- Specified by:
getProvenance
in interfacecom.oracle.labs.mlrg.olcut.provenance.Provenancable<com.oracle.labs.mlrg.olcut.provenance.ConfiguredObjectProvenance>
-