mxnet.initializer¶
Weight initializer.
Classes
|
Initialize weight for upsampling layers. |
|
Initializes the weights to a given value. |
|
Initialize parameters for fused rnn layers. |
Descriptor for the initialization pattern. |
|
|
The base class of an initializer. |
|
Initialize all biases of an LSTMCell to 0.0 except for the forget gate whose bias is set to custom value. |
|
Initializes variables by loading data from file or dict. |
|
Initialize the weight according to a MSRA paper. |
|
Initialize parameters using multiple initializers. |
|
Initializes weights with random values sampled from a normal distribution with a mean of zero and standard deviation of sigma. |
|
Initializes weights to one. |
|
Initialize weight as orthogonal matrix. |
|
Initializes weights with random values uniformly sampled from a given range. |
|
Returns an initializer performing “Xavier” initialization for weights. |
|
Initializes weights to zero. |
Functions
|
Registers a custom initializer. |
-
class
mxnet.initializer.
Bilinear
[source]¶ Bases:
mxnet.initializer.Initializer
Initialize weight for upsampling layers.
-
class
mxnet.initializer.
Constant
(value)[source]¶ Bases:
mxnet.initializer.Initializer
Initializes the weights to a given value. The value passed in can be a scalar or a NDarray that matches the shape of the parameter to be set.
- Parameters
value (float, NDArray) – Value to set.
Methods
dumps
()Saves the initializer to string
-
dumps
()[source]¶ Saves the initializer to string
- Returns
JSON formatted string that describes the initializer.
- Return type
str
Examples
>>> # Create initializer and retrieve its parameters ... >>> init = mx.init.Normal(0.5) >>> init.dumps() '["normal", {"sigma": 0.5}]' >>> init = mx.init.Xavier(factor_type="in", magnitude=2.34) >>> init.dumps() '["xavier", {"rnd_type": "uniform", "magnitude": 2.34, "factor_type": "in"}]'
-
class
mxnet.initializer.
FusedRNN
(init, num_hidden, num_layers, mode, bidirectional=False, forget_bias=1.0)[source]¶ Bases:
mxnet.initializer.Initializer
Initialize parameters for fused rnn layers.
- Parameters
init (Initializer) – initializer applied to unpacked weights. Fall back to global initializer if None.
num_hidden (int) – should be the same with arguments passed to FusedRNNCell.
num_layers (int) – should be the same with arguments passed to FusedRNNCell.
mode (str) – should be the same with arguments passed to FusedRNNCell.
bidirectional (bool) – should be the same with arguments passed to FusedRNNCell.
forget_bias (float) – should be the same with arguments passed to FusedRNNCell.
-
class
mxnet.initializer.
InitDesc
[source]¶ Bases:
str
Descriptor for the initialization pattern.
- Parameters
name (str) – Name of variable.
attrs (dict of str to str) – Attributes of this variable taken from
Symbol.attr_dict
.global_init (Initializer) – Global initializer to fallback to.
-
class
mxnet.initializer.
Initializer
(**kwargs)[source]¶ Bases:
object
The base class of an initializer.
Methods
dumps
()Saves the initializer to string
set_verbosity
([verbose, print_func])Switch on/off verbose mode
-
dumps
()[source]¶ Saves the initializer to string
- Returns
JSON formatted string that describes the initializer.
- Return type
str
Examples
>>> # Create initializer and retrieve its parameters ... >>> init = mx.init.Normal(0.5) >>> init.dumps() '["normal", {"sigma": 0.5}]' >>> init = mx.init.Xavier(factor_type="in", magnitude=2.34) >>> init.dumps() '["xavier", {"rnd_type": "uniform", "magnitude": 2.34, "factor_type": "in"}]'
-
set_verbosity
(verbose=False, print_func=None)[source]¶ Switch on/off verbose mode
- Parameters
verbose (bool) – switch on/off verbose mode
print_func (function) – A function that computes statistics of initialized arrays. Takes an NDArray and returns an str. Defaults to mean absolute value str((abs(x)/size(x)).asscalar()).
-
-
class
mxnet.initializer.
LSTMBias
(forget_bias=1.0)[source]¶ Bases:
mxnet.initializer.Initializer
Initialize all biases of an LSTMCell to 0.0 except for the forget gate whose bias is set to custom value.
- Parameters
forget_bias (float, default 1.0) – bias for the forget gate. Jozefowicz et al. 2015 recommends setting this to 1.0.
-
class
mxnet.initializer.
Load
(param, default_init=None, verbose=False)[source]¶ Bases:
object
Initializes variables by loading data from file or dict.
Note Load will drop
arg:
oraux:
from name and initialize the variables that match with the prefix dropped.- Parameters
param (str or dict of str->`NDArray`) – Parameter file or dict mapping name to NDArray.
default_init (Initializer) – Default initializer when name is not found in param.
verbose (bool) – Flag for enabling logging of source when initializing.
-
class
mxnet.initializer.
MSRAPrelu
(factor_type='avg', slope=0.25)[source]¶ Bases:
mxnet.initializer.Xavier
Initialize the weight according to a MSRA paper.
This initializer implements Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification, available at https://arxiv.org/abs/1502.01852.
This initializer is proposed for initialization related to ReLu activation, it makes some changes on top of Xavier method.
- Parameters
factor_type (str, optional) – Can be
'avg'
,'in'
, or'out'
.slope (float, optional) – initial slope of any PReLU (or similar) nonlinearities.
-
class
mxnet.initializer.
Mixed
(patterns, initializers)[source]¶ Bases:
object
Initialize parameters using multiple initializers.
- Parameters
patterns (list of str) – List of regular expressions matching parameter names.
initializers (list of Initializer) – List of initializers corresponding to patterns.
Example
>>> # Given 'module', an instance of 'mxnet.module.Module', initialize biases to zero ... # and every other parameter to random values with uniform distribution. ... >>> init = mx.initializer.Mixed(['bias', '.*'], [mx.init.Zero(), mx.init.Uniform(0.1)]) >>> module.init_params(init) >>> >>> for dictionary in module.get_params(): ... for key in dictionary: ... print(key) ... print(dictionary[key].asnumpy()) ... fullyconnected1_weight [[ 0.0097627 0.01856892 0.04303787]] fullyconnected1_bias [ 0.]
-
class
mxnet.initializer.
Normal
(sigma=0.01)[source]¶ Bases:
mxnet.initializer.Initializer
Initializes weights with random values sampled from a normal distribution with a mean of zero and standard deviation of sigma.
- Parameters
sigma (float, optional) – Standard deviation of the normal distribution. Default standard deviation is 0.01.
Example
>>> # Given 'module', an instance of 'mxnet.module.Module', initialize weights >>> # to random values sampled from a normal distribution. ... >>> init = mx.init.Normal(0.5) >>> module.init_params(init) >>> for dictionary in module.get_params(): ... for key in dictionary: ... print(key) ... print(dictionary[key].asnumpy()) ... fullyconnected0_weight [[-0.3214761 -0.12660924 0.53789419]]
-
class
mxnet.initializer.
One
[source]¶ Bases:
mxnet.initializer.Initializer
Initializes weights to one.
Example
>>> # Given 'module', an instance of 'mxnet.module.Module', initialize weights to one. ... >>> init = mx.initializer.One() >>> module.init_params(init) >>> for dictionary in module.get_params(): ... for key in dictionary: ... print(key) ... print(dictionary[key].asnumpy()) ... fullyconnected0_weight [[ 1. 1. 1.]]
-
class
mxnet.initializer.
Orthogonal
(scale=1.414, rand_type='uniform')[source]¶ Bases:
mxnet.initializer.Initializer
Initialize weight as orthogonal matrix.
This initializer implements Exact solutions to the nonlinear dynamics of learning in deep linear neural networks, available at https://arxiv.org/abs/1312.6120.
- Parameters
scale (float optional) – Scaling factor of weight.
rand_type (string optional) – Use “uniform” or “normal” random number to initialize weight.
-
class
mxnet.initializer.
Uniform
(scale=0.07)[source]¶ Bases:
mxnet.initializer.Initializer
Initializes weights with random values uniformly sampled from a given range.
- Parameters
scale (float, optional) – The bound on the range of the generated random values. Values are generated from the range [-scale, scale]. Default scale is 0.07.
Example
>>> # Given 'module', an instance of 'mxnet.module.Module', initialize weights >>> # to random values uniformly sampled between -0.1 and 0.1. ... >>> init = mx.init.Uniform(0.1) >>> module.init_params(init) >>> for dictionary in module.get_params(): ... for key in dictionary: ... print(key) ... print(dictionary[key].asnumpy()) ... fullyconnected0_weight [[ 0.01360891 -0.02144304 0.08511933]]
-
class
mxnet.initializer.
Xavier
(rnd_type='uniform', factor_type='avg', magnitude=3)[source]¶ Bases:
mxnet.initializer.Initializer
Returns an initializer performing “Xavier” initialization for weights.
This initializer is designed to keep the scale of gradients roughly the same in all layers.
By default, rnd_type is
'uniform'
and factor_type is'avg'
, the initializer fills the weights with random numbers in the range of \([-c, c]\), where \(c = \sqrt{\frac{3.}{0.5 * (n_{in} + n_{out})}}\). \(n_{in}\) is the number of neurons feeding into weights, and \(n_{out}\) is the number of neurons the result is fed to.If rnd_type is
'uniform'
and factor_type is'in'
, the \(c = \sqrt{\frac{3.}{n_{in}}}\). Similarly when factor_type is'out'
, the \(c = \sqrt{\frac{3.}{n_{out}}}\).If rnd_type is
'gaussian'
and factor_type is'avg'
, the initializer fills the weights with numbers from normal distribution with a standard deviation of \(\sqrt{\frac{3.}{0.5 * (n_{in} + n_{out})}}\).- Parameters
rnd_type (str, optional) – Random generator type, can be
'gaussian'
or'uniform'
.factor_type (str, optional) – Can be
'avg'
,'in'
, or'out'
.magnitude (float, optional) – Scale of random number.
-
class
mxnet.initializer.
Zero
[source]¶ Bases:
mxnet.initializer.Initializer
Initializes weights to zero.
Example
>>> # Given 'module', an instance of 'mxnet.module.Module', initialize weights to zero. ... >>> init = mx.initializer.Zero() >>> module.init_params(init) >>> for dictionary in module.get_params(): ... for key in dictionary: ... print(key) ... print(dictionary[key].asnumpy()) ... fullyconnected0_weight [[ 0. 0. 0.]]
-
mxnet.initializer.
register
(klass)[source]¶ Registers a custom initializer.
Custom initializers can be created by extending mx.init.Initializer and implementing the required functions like _init_weight and _init_bias. The created initializer must be registered using mx.init.register before it can be called by name.
- Parameters
klass (class) – A subclass of mx.init.Initializer that needs to be registered as a custom initializer.
Example
>>> # Create and register a custom initializer that ... # initializes weights to 0.1 and biases to 1. ... >>> @mx.init.register ... @alias('myinit') ... class CustomInit(mx.init.Initializer): ... def __init__(self): ... super(CustomInit, self).__init__() ... def _init_weight(self, _, arr): ... arr[:] = 0.1 ... def _init_bias(self, _, arr): ... arr[:] = 1 ... >>> # Module is an instance of 'mxnet.module.Module' ... >>> module.init_params("custominit") >>> # module.init_params("myinit") >>> # module.init_params(CustomInit())