org.tribuo.math.optimisers

Class Adam

java.lang.Object

org.tribuo.math.optimisers.Adam

All Implemented Interfaces:

com.oracle.labs.mlrg.olcut.config.Configurable, com.oracle.labs.mlrg.olcut.provenance.Provenancable<com.oracle.labs.mlrg.olcut.provenance.ConfiguredObjectProvenance>, StochasticGradientOptimiser

public class Adam
extends Object
implements StochasticGradientOptimiser

An implementation of the Adam gradient optimiser.

Creates two copies of the parameters to store learning rates.

See:

 Kingma, D., and Ba, J.
 "Adam: A Method for Stochastic Optimization"
 arXiv preprint arXiv:1412.6980, 2014.
 

Constructor Summary

Constructors

Constructor and Description
Adam() Sets initialLearningRate to 0.001, betaOne to 0.9, betaTwo to 0.999, epsilon to 1e-6.
Adam(double initialLearningRate, double epsilon) Sets betaOne to 0.9 and betaTwo to 0.999
Adam(double initialLearningRate, double betaOne, double betaTwo, double epsilon) It's highly recommended not to modify these parameters, use one of the other constructors.

Method Summary

Modifier and Type	Method and Description
Adam	copy() Copies a gradient optimiser with it's configuration.
com.oracle.labs.mlrg.olcut.provenance.ConfiguredObjectProvenance	getProvenance()
void	initialise(Parameters parameters) Initialises the gradient optimiser.
void	reset() Resets the optimiser so it's ready to optimise a new Parameters.
Tensor[]	step(Tensor[] updates, double weight) Take a Tensor array of gradients and transform them according to the current weight and learning rates.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.tribuo.math.StochasticGradientOptimiser

finalise

Methods inherited from interface com.oracle.labs.mlrg.olcut.config.Configurable

postConfig

Constructor Detail

Adam

public Adam(double initialLearningRate,
            double betaOne,
            double betaTwo,
            double epsilon)

It's highly recommended not to modify these parameters, use one of the other constructors.

Parameters:

initialLearningRate - The initial learning rate.

betaOne - The value of beta-one.

betaTwo - The value of beta-two.

epsilon - The epsilon value.

Adam

public Adam(double initialLearningRate,
            double epsilon)

Sets betaOne to 0.9 and betaTwo to 0.999

Parameters:

initialLearningRate - The initial learning rate.

epsilon - The epsilon value.

Adam

public Adam()

Sets initialLearningRate to 0.001, betaOne to 0.9, betaTwo to 0.999, epsilon to 1e-6. These are the parameters from the Adam paper.

Method Detail

initialise

public void initialise(Parameters parameters)

Description copied from interface: StochasticGradientOptimiser

Initialises the gradient optimiser.

Configures any learning rate parameters.

Specified by:

initialise in interface StochasticGradientOptimiser

Parameters:

parameters - The parameters to optimise.

step

public Tensor[] step(Tensor[] updates,
                     double weight)

Description copied from interface: StochasticGradientOptimiser

Take a Tensor array of gradients and transform them according to the current weight and learning rates.

Can return the same Tensor array or a new one.

Specified by:

step in interface StochasticGradientOptimiser

Parameters:

updates - An array of gradients.

weight - The weight for the current gradients.

Returns:

A Tensor array of gradients.

toString

public String toString()

Overrides:

toString in class Object

reset

public void reset()

Description copied from interface: StochasticGradientOptimiser

Resets the optimiser so it's ready to optimise a new Parameters.

Specified by:

reset in interface StochasticGradientOptimiser

copy

public Adam copy()

Description copied from interface: StochasticGradientOptimiser

Copies a gradient optimiser with it's configuration. Usually calls the copy constructor.

Specified by:

copy in interface StochasticGradientOptimiser

Returns:

A gradient optimiser with the same configuration, but independent state.

getProvenance

public com.oracle.labs.mlrg.olcut.provenance.ConfiguredObjectProvenance getProvenance()

Specified by:

getProvenance in interface com.oracle.labs.mlrg.olcut.provenance.Provenancable<com.oracle.labs.mlrg.olcut.provenance.ConfiguredObjectProvenance>