PLearn::AdaptGradientOptimizer Class Reference

#include <AdaptGradientOptimizer.h>

Inheritance diagram for PLearn::AdaptGradientOptimizer:

[legend]Collaboration diagram for PLearn::AdaptGradientOptimizer:

[legend]List of all members.


Public Member Functions
	AdaptGradientOptimizer (real the_start_learning_rate=0.01, real the_decrease_constant=0, real the_min_learning_rate=0.001, real the_max_learning_rate=0.02, int the_learning_rate_adaptation=0, real the_adapt_coeff1=0, real the_adapt_coeff2=0, int n_updates=1, const string &filename="", int every_iterations=1)
	AdaptGradientOptimizer (VarArray the_params, Var the_cost, real the_start_learning_rate=0.01, real the_decrease_constant=0, real the_min_learning_rate=0.001, real the_max_learning_rate=0.02, int the_learning_rate_adaptation=0, real the_adapt_coeff1=0, real the_adapt_coeff2=0, int n_updates=1, const string &filename="", int every_iterations=1)
	AdaptGradientOptimizer (VarArray the_params, Var the_cost, VarArray update_for_measure, real the_start_learning_rate=0.01, real the_decrease_constant=0, real the_min_learning_rate=0.001, real the_max_learning_rate=0.02, int the_learning_rate_adaptation=0, real the_adapt_coeff1=0, real the_adapt_coeff2=0, int n_updates=1, const string &filename="", int every_iterations=1)
	PLEARN_DECLARE_OBJECT (AdaptGradientOptimizer)
virtual void	makeDeepCopyFromShallowCopy (CopiesMap &copies)
virtual void	build ()
	Should call simply inherited::build(), then this class's build_().
virtual real	optimize ()
	sub-classes should define this, which is the main method
virtual bool	optimizeN (VecStatsCollector &stats_coll)
	sub-classes should define this, which is the new main method
Public Attributes
real	learning_rate
	gradient descent specific parameters (directly modifiable by the user)
real	start_learning_rate
	initial learning rate
real	min_learning_rate
	min value for learning_rate when adapting
real	max_learning_rate
	max value for learning_rate when adapting
int	learning_rate_adaptation
	Learning rate adaptation kind : 0 : none 1 : basic 2 : ALAP1 3 : variance.
real	adapt_coeff1
	a coefficient for learning rate adaptation
real	adapt_coeff2
	a coefficient for learning rate adaptation
real	decrease_constant
int	mini_batch
int	adapt_every
	after how many updates we adapt learning rate
Static Protected Member Functions
void	declareOptions (OptionList &ol)
	redefine this in subclasses: call declareOption(...) for each option, and then call inherited::declareOptions(options) ( see the declareOption function further down)
Private Types
typedef Optimizer	inherited
Private Member Functions
void	build_ ()
void	adaptLearningRateBasic (Vec old_params, Vec new_evol)
void	adaptLearningRateALAP1 (Vec old_gradient, Vec new_gradient)
void	adaptLearningRateVariance ()
Private Attributes
bool	stochastic_hack
Vec	learning_rates
Vec	gradient
Vec	tmp_storage
Vec	old_evol
Array< Mat >	oldgradientlocations
Vec	store_var_grad
Vec	store_grad
Vec	store_quad_grad
int	count_updates

Detailed Description

CLASS ADAPTGRADIENTOPTIMIZER

A (possibly stochastic) gradient optimizer using various learning rate adaptation methods.

Definition at line 64 of file AdaptGradientOptimizer.h.

Member Typedef Documentation

typedef Optimizer PLearn::AdaptGradientOptimizer::inherited [private]

Reimplemented from PLearn::Optimizer.
Definition at line 66 of file AdaptGradientOptimizer.h.
Referenced by AdaptGradientOptimizer().

Constructor & Destructor Documentation

PLearn::AdaptGradientOptimizer::AdaptGradientOptimizer ( real the_start_learning_rate = 0.01,

real the_decrease_constant = 0,

real the_min_learning_rate = 0.001,

real the_max_learning_rate = 0.02,

int the_learning_rate_adaptation = 0,

real the_adapt_coeff1 = 0,

real the_adapt_coeff2 = 0,

int n_updates = 1,

const string & filename = "",

int every_iterations = 1

)

Definition at line 51 of file AdaptGradientOptimizer.cc.
References inherited.

PLearn::AdaptGradientOptimizer::AdaptGradientOptimizer ( VarArray the_params,

Var the_cost,

real the_start_learning_rate = 0.01,

real the_decrease_constant = 0,

real the_min_learning_rate = 0.001,

real the_max_learning_rate = 0.02,

int the_learning_rate_adaptation = 0,

real the_adapt_coeff1 = 0,

real the_adapt_coeff2 = 0,

int n_updates = 1,

const string & filename = "",

int every_iterations = 1

)

Definition at line 71 of file AdaptGradientOptimizer.cc.

PLearn::AdaptGradientOptimizer::AdaptGradientOptimizer ( VarArray the_params,

Var the_cost,

VarArray update_for_measure,

real the_start_learning_rate = 0.01,

real the_decrease_constant = 0,

real the_min_learning_rate = 0.001,

real the_max_learning_rate = 0.02,

int the_learning_rate_adaptation = 0,

real the_adapt_coeff1 = 0,

real the_adapt_coeff2 = 0,

int n_updates = 1,

const string & filename = "",

int every_iterations = 1

)

Definition at line 90 of file AdaptGradientOptimizer.cc.

Member Function Documentation

void PLearn::AdaptGradientOptimizer::adaptLearningRateALAP1 ( Vec old_gradient,

Vec new_gradient

) [private]

Definition at line 207 of file AdaptGradientOptimizer.cc.
References adapt_coeff1, learning_rate, max_learning_rate, min_learning_rate, and PLearn::VarArray::nelems().
Referenced by optimizeN().

void PLearn::AdaptGradientOptimizer::adaptLearningRateBasic ( Vec old_params,

Vec new_evol

) [private]

Definition at line 227 of file AdaptGradientOptimizer.cc.
References adapt_coeff1, adapt_coeff2, PLearn::TVec< Var >::data(), k, learning_rates, max_learning_rate, min_learning_rate, and PLearn::TVec< Var >::size().
Referenced by optimizeN().

void PLearn::AdaptGradientOptimizer::adaptLearningRateVariance ( ) [private]

Definition at line 278 of file AdaptGradientOptimizer.cc.
References adapt_coeff1, PLearn::TVec< T >::clear(), count_updates, learning_rates, max_learning_rate, min_learning_rate, PLearn::VarArray::nelems(), store_grad, store_quad_grad, and store_var_grad.
Referenced by optimizeN().

virtual void PLearn::AdaptGradientOptimizer::build ( ) [inline, virtual]

Should call simply inherited::build(), then this class's build_().
This method should be callable again at later times, after modifying some option fields to change the "architecture" of the object.
Reimplemented from PLearn::Optimizer.
Definition at line 142 of file AdaptGradientOptimizer.h.
References build_().

void PLearn::AdaptGradientOptimizer::build_ ( ) [private]

This method should be redefined in subclasses and do the actual building of the object according to previously set option fields. Constructors can just set option fields, and then call build_. This method is NOT virtual, and will typically be called only from three places: a constructor, the public virtual build() method, and possibly the public virtual read method (which calls its parent's read). build_() can assume that it's parent's build_ has already been called.
Reimplemented from PLearn::Optimizer.
Definition at line 163 of file AdaptGradientOptimizer.cc.
References PLearn::TVec< T >::clear(), PLearn::VarArray::clearGradient(), PLearn::VarArray::copyTo(), count_updates, PLearn::TVec< T >::fill(), gradient, learning_rate, learning_rate_adaptation, learning_rates, PLearn::VarArray::nelems(), PLearn::SumOfVariable::nsamples, old_evol, oldgradientlocations, PLearn::TVec< Mat >::resize(), PLearn::TVec< T >::resize(), PLearn::TVec< Var >::size(), start_learning_rate, stochastic_hack, store_grad, store_quad_grad, store_var_grad, and tmp_storage.
Referenced by build().

void PLearn::AdaptGradientOptimizer::declareOptions ( OptionList & ol ) [static, protected]

redefine this in subclasses: call declareOption(...) for each option, and then call inherited::declareOptions(options) ( see the declareOption function further down)
ex: static void declareOptions(OptionList& ol) { declareOption(ol, "inputsize", &MyObject::inputsize_, OptionBase::buildoption, "the size of the input\n it must be provided"); declareOption(ol, "weights", &MyObject::weights, OptionBase::learntoption, "the learnt model weights"); inherited::declareOptions(ol); }
Reimplemented from PLearn::Optimizer.
Definition at line 112 of file AdaptGradientOptimizer.cc.
References PLearn::declareOption(), and PLearn::OptionList.

virtual void PLearn::AdaptGradientOptimizer::makeDeepCopyFromShallowCopy ( CopiesMap & copies ) [inline, virtual]

Does the necessary operations to transform a shallow copy (this) into a deep copy by deep-copying all the members that need to be. Typical implementation:
void CLASS_OF_THIS::makeDeepCopyFromShallowCopy(CopiesMap& copies) { SUPERCLASS_OF_THIS::makeDeepCopyFromShallowCopy(copies); member_ptr = member_ptr->deepCopy(copies); member_smartptr = member_smartptr->deepCopy(copies); member_mat.makeDeepCopyFromShallowCopy(copies); member_vec.makeDeepCopyFromShallowCopy(copies); ... }
Reimplemented from PLearn::Object.
Definition at line 140 of file AdaptGradientOptimizer.h.
References PLearn::CopiesMap, and makeDeepCopyFromShallowCopy().
Referenced by makeDeepCopyFromShallowCopy().

real PLearn::AdaptGradientOptimizer::optimize ( ) [virtual]

sub-classes should define this, which is the main method

Implements PLearn::Optimizer.
Definition at line 312 of file AdaptGradientOptimizer.cc.
References PLERROR.

bool PLearn::AdaptGradientOptimizer::optimizeN ( VecStatsCollector & stats_coll ) [virtual]

sub-classes should define this, which is the new main method

Implements PLearn::Optimizer.
Definition at line 321 of file AdaptGradientOptimizer.cc.
References adapt_every, adaptLearningRateALAP1(), adaptLearningRateBasic(), adaptLearningRateVariance(), PLearn::VarArray::clearGradient(), PLearn::VarArray::copyGradientTo(), PLearn::VarArray::copyTo(), count_updates, decrease_constant, PLearn::endl(), PLearn::VarArray::fbprop(), gradient, learning_rate, learning_rate_adaptation, learning_rates, PLearn::VarArray::nelems(), old_evol, oldgradientlocations, PLearn::TVec< Var >::size(), start_learning_rate, stochastic_hack, store_grad, store_quad_grad, tmp_storage, PLearn::VecStatsCollector::update(), PLearn::VarArray::update(), and PLearn::VarArray::updateAndClear().

PLearn::AdaptGradientOptimizer::PLEARN_DECLARE_OBJECT ( AdaptGradientOptimizer )

Member Data Documentation

real PLearn::AdaptGradientOptimizer::adapt_coeff1

a coefficient for learning rate adaptation

Definition at line 84 of file AdaptGradientOptimizer.h.
Referenced by adaptLearningRateALAP1(), adaptLearningRateBasic(), and adaptLearningRateVariance().

real PLearn::AdaptGradientOptimizer::adapt_coeff2

a coefficient for learning rate adaptation

Definition at line 85 of file AdaptGradientOptimizer.h.
Referenced by adaptLearningRateBasic().

int PLearn::AdaptGradientOptimizer::adapt_every

after how many updates we adapt learning rate

Definition at line 88 of file AdaptGradientOptimizer.h.
Referenced by optimizeN().

int PLearn::AdaptGradientOptimizer::count_updates [private]

Definition at line 103 of file AdaptGradientOptimizer.h.
Referenced by adaptLearningRateVariance(), build_(), and optimizeN().

real PLearn::AdaptGradientOptimizer::decrease_constant

Definition at line 86 of file AdaptGradientOptimizer.h.
Referenced by optimizeN().

Vec PLearn::AdaptGradientOptimizer::gradient [private]

Definition at line 94 of file AdaptGradientOptimizer.h.
Referenced by build_(), and optimizeN().

real PLearn::AdaptGradientOptimizer::learning_rate

gradient descent specific parameters (directly modifiable by the user)

Definition at line 72 of file AdaptGradientOptimizer.h.
Referenced by adaptLearningRateALAP1(), build_(), and optimizeN().

int PLearn::AdaptGradientOptimizer::learning_rate_adaptation

Learning rate adaptation kind : 0 : none 1 : basic 2 : ALAP1 3 : variance.

Definition at line 83 of file AdaptGradientOptimizer.h.
Referenced by build_(), and optimizeN().

Vec PLearn::AdaptGradientOptimizer::learning_rates [private]

Definition at line 93 of file AdaptGradientOptimizer.h.
Referenced by adaptLearningRateBasic(), adaptLearningRateVariance(), build_(), and optimizeN().

real PLearn::AdaptGradientOptimizer::max_learning_rate

max value for learning_rate when adapting

Definition at line 77 of file AdaptGradientOptimizer.h.
Referenced by adaptLearningRateALAP1(), adaptLearningRateBasic(), and adaptLearningRateVariance().

real PLearn::AdaptGradientOptimizer::min_learning_rate

min value for learning_rate when adapting

Definition at line 76 of file AdaptGradientOptimizer.h.
Referenced by adaptLearningRateALAP1(), adaptLearningRateBasic(), and adaptLearningRateVariance().

int PLearn::AdaptGradientOptimizer::mini_batch

Definition at line 87 of file AdaptGradientOptimizer.h.

Vec PLearn::AdaptGradientOptimizer::old_evol [private]

Definition at line 98 of file AdaptGradientOptimizer.h.
Referenced by build_(), and optimizeN().

Array<Mat> PLearn::AdaptGradientOptimizer::oldgradientlocations [private]

Definition at line 99 of file AdaptGradientOptimizer.h.
Referenced by build_(), and optimizeN().

real PLearn::AdaptGradientOptimizer::start_learning_rate

initial learning rate

Definition at line 75 of file AdaptGradientOptimizer.h.
Referenced by build_(), and optimizeN().

bool PLearn::AdaptGradientOptimizer::stochastic_hack [private]

Definition at line 92 of file AdaptGradientOptimizer.h.
Referenced by build_(), and optimizeN().

Vec PLearn::AdaptGradientOptimizer::store_grad [private]

Definition at line 101 of file AdaptGradientOptimizer.h.
Referenced by adaptLearningRateVariance(), build_(), and optimizeN().

Vec PLearn::AdaptGradientOptimizer::store_quad_grad [private]

Definition at line 102 of file AdaptGradientOptimizer.h.
Referenced by adaptLearningRateVariance(), build_(), and optimizeN().

Vec PLearn::AdaptGradientOptimizer::store_var_grad [private]

Definition at line 100 of file AdaptGradientOptimizer.h.
Referenced by adaptLearningRateVariance(), and build_().

Vec PLearn::AdaptGradientOptimizer::tmp_storage [private]

Definition at line 95 of file AdaptGradientOptimizer.h.
Referenced by build_(), and optimizeN().

The documentation for this class was generated from the following files:

Generated on Tue Aug 17 16:24:10 2004 for PLearn by

1.3.7

PLearn::AdaptGradientOptimizer Class Reference

Public Member Functions

Public Attributes

Static Protected Member Functions

Private Types

Private Member Functions

Private Attributes

Detailed Description

Member Typedef Documentation

Constructor & Destructor Documentation

Member Function Documentation

Member Data Documentation