mxnet.kvstore.BytePS

class BytePS[source]

Bases: mxnet.kvstore.base.KVStoreBase

BytePS backend for MXNet KVStore interface.

__init__()[source]

Initializes a new KVStore.

Methods

__init__()

Initializes a new KVStore.

broadcast(key, value, out[, priority])

Broadcast the value NDArray at rank 0 to all ranks’ out.

is_capable(capability)

Queries if the KVStore type supports certain capability, such as optimizer algorithm, gradient compression, sparsity, etc.

load_optimizer_states(fname)

Not Implement yet.

pushpull(key, value[, out, priority])

Performs push and pull a single value from the store.

register(klass)

Registers a new KVStore.

save_optimizer_states(fname[, dump_optimizer])

Not Implement yet.

set_optimizer(optimizer)

Not Implement yet.

Attributes

OPTIMIZER

kv_registry

local_rank

Returns the local rank of this worker on the node.

num_workers

Returns the number of worker nodes.

rank

Returns the rank of this worker node.

type

Returns the type of this kvstore.
broadcast(key, value, out, priority=0)[source]

Broadcast the value NDArray at rank 0 to all ranks’ out. If out is None, the result is stored in value.

Parameters

  • key (str, or int) – The keys.

  • value (NDArray, or list of NDArray) – Values corresponding to the key.

  • out (NDArray, or lise of NDArray) – Values corresponding to the keys.

Examples

>>> # broadcast a single key-value pair
>>> shape = (2,3)
>>> kv = mx.kv.create('byteps')
>>> a = mx.nd.zeros(shape)
>>> kv.broadcast('3', mx.nd.ones(shape)*2, out=a)
>>> print a.asnumpy()
[[ 2.  2.  2.]
[ 2.  2.  2.]]
static is_capable(capability)[source]

Queries if the KVStore type supports certain capability, such as optimizer algorithm, gradient compression, sparsity, etc. As byteps server does not store weight, this function will return false for any capabilities.

Parameters

capability (str) – The capability to query

Returns

result – Whether the capability is supported or not.

Return type

bool

load_optimizer_states(fname)[source]

Not Implement yet.

Parameters

fname (str) – Path to input states file.

property local_rank

Returns the local rank of this worker on the node.

Returns

rank – The local rank of this node, which is in range [0, num_workers_on_current_node())

Return type

int

property num_workers

Returns the number of worker nodes.

Returns

size – The number of worker nodes.

Return type

int

pushpull(key, value, out=None, priority=0)[source]

Performs push and pull a single value from the store. This function is coalesced form of push and pull operations. value is pushed to the kvstore server for the specified keys and the aggregated values are pulled from the server to out. If out is not specified the pulled values are written to value.

Parameters

  • key (str, or int) – The key.

  • value (NDArray, or list of NDArray) – Values corresponding to the key.

  • out (NDArray, or list of NDArray) – Values corresponding to the key.

  • priority (int, optional) – The priority of the operation. Higher priority operations are likely to be executed before other actions.

Examples

>>> # pushpull a single key-value pair
>>> kv.pushpull('3', mx.nd.ones(shape)*8, out=a)
>>> print a.asnumpy()
[[ 8.  8.  8.]
[ 8.  8.  8.]]
property rank

Returns the rank of this worker node.

Returns

rank – The rank of this node, which is in range [0, num_workers())

Return type

int

save_optimizer_states(fname, dump_optimizer=False)[source]

Not Implement yet.

Parameters

  • fname (str) – Path to the output states file.

  • dump_optimizer (bool, default False) – Whether to also save the optimizer itself. This would also save optimizer information such as learning rate and weight decay schedules.

set_optimizer(optimizer)[source]

Not Implement yet.

Parameters

optimizer (KVStoreBase) – The new optimizer for the store

property type

Returns the type of this kvstore.

Returns

type – the string type

Return type

str