参数管理
========
在选择了架构并设置了超参数后，我们就进入了训练阶段。
此时，我们的目标是找到使损失函数最小化的模型参数值。
经过训练后，我们将需要使用这些参数来做出未来的预测。
此外，有时我们希望提取参数，以便在其他环境中复用它们，
将模型保存下来，以便它可以在其他软件中执行，
或者为了获得科学的理解而进行检查。
之前的介绍中，我们只依靠深度学习框架来完成训练的工作，
而忽略了操作参数的具体细节。 本节，我们将介绍以下内容：
- 访问参数，用于调试、诊断和可视化；
- 参数初始化；
- 在不同模型组件间共享参数。
我们首先看一下具有单隐藏层的多层感知机。
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
from mindspore import nn, ops
from mindspore.common.initializer import Uniform, initializer
net = nn.SequentialCell([nn.Dense(4, 8), nn.ReLU(), nn.Dense(8, 1)])
X = ops.rand((2, 4))
net(X)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
Tensor(shape=[2, 1], dtype=Float32, value=
[[-1.31420806e-01],
[-4.23702121e-01]])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
import torch
from torch import nn
net = nn.Sequential(nn.Linear(4, 8), nn.ReLU(), nn.Linear(8, 1))
X = torch.rand(size=(2, 4))
net(X)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor([[-0.0323],
[ 0.0018]], grad_fn=
)
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
print(net[2].parameters_dict())
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
OrderedDict([('2.weight', Parameter (name=2.weight, shape=(1, 8), dtype=Float32, requires_grad=True)), ('2.bias', Parameter (name=2.bias, shape=(1,), dtype=Float32, requires_grad=True))])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
print(net[2].state_dict())
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
OrderedDict([('weight', tensor([[ 0.1540, -0.1524, 0.2029, 0.0950, -0.1503, 0.3407, -0.1616, 0.0538]])), ('bias', tensor([-0.2031]))])
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
print(type(net[2].bias))
print(net[2].bias)
print(net[2].bias.value())
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
Parameter (name=2.bias, shape=(1,), dtype=Float32, requires_grad=True)
[-0.09661776]
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
print(type(net[2].bias))
print(net[2].bias)
print(net[2].bias.data)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
Parameter containing:
tensor([-0.2031], requires_grad=True)
tensor([-0.2031])
参数是复合的对象，包含值、梯度和额外信息。
这就是我们需要显式参数值的原因。
除了值之外，我们还可以访问每个参数的梯度。
在上面这个网络中，由于我们还没有调用反向传播，所以参数的梯度处于初始状态。
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
net[2].weight.grad == None
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
True
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
print(*[(name, param.shape) for name, param in net[0].parameters_dict().items()])
print(*[(name, param.shape) for name, param in net.parameters_dict().items()])
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
('0.weight', (8, 4)) ('0.bias', (8,))
('0.weight', (8, 4)) ('0.bias', (8,)) ('2.weight', (1, 8)) ('2.bias', (1,))
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
print(*[(name, param.shape) for name, param in net[0].named_parameters()])
print(*[(name, param.shape) for name, param in net.named_parameters()])
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
('weight', torch.Size([8, 4])) ('bias', torch.Size([8]))
('0.weight', torch.Size([8, 4])) ('0.bias', torch.Size([8])) ('2.weight', torch.Size([1, 8])) ('2.bias', torch.Size([1]))
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
net.parameters_dict()['2.bias'].value()
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
Tensor(shape=[1], dtype=Float32, value= [-9.66177583e-02])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
net.state_dict()['2.bias'].data
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor([-0.2031])
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
def block1():
return nn.SequentialCell([nn.Dense(4, 8), nn.ReLU(),
nn.Dense(8, 4), nn.ReLU()])
def block2():
net = nn.SequentialCell()
for i in range(4):
# 在这里嵌套
net.append(block1())
return net
rgnet = nn.SequentialCell([block2(), nn.Dense(4, 1)])
rgnet(X)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
Tensor(shape=[2, 1], dtype=Float32, value=
[[ 5.45102693e-02],
[ 5.45102693e-02]])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
def block1():
return nn.Sequential(nn.Linear(4, 8), nn.ReLU(),
nn.Linear(8, 4), nn.ReLU())
def block2():
net = nn.Sequential()
for i in range(4):
# 在这里嵌套
net.add_module(f'block {i}', block1())
return net
rgnet = nn.Sequential(block2(), nn.Linear(4, 1))
rgnet(X)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor([[-0.0096],
[-0.0096]], grad_fn=
)
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
print(rgnet)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
SequentialCell(
(0): SequentialCell(
(0): SequentialCell(
(0): Dense(input_channels=4, output_channels=8, has_bias=True)
(1): ReLU()
(2): Dense(input_channels=8, output_channels=4, has_bias=True)
(3): ReLU()
)
(1): SequentialCell(
(0): Dense(input_channels=4, output_channels=8, has_bias=True)
(1): ReLU()
(2): Dense(input_channels=8, output_channels=4, has_bias=True)
(3): ReLU()
)
(2): SequentialCell(
(0): Dense(input_channels=4, output_channels=8, has_bias=True)
(1): ReLU()
(2): Dense(input_channels=8, output_channels=4, has_bias=True)
(3): ReLU()
)
(3): SequentialCell(
(0): Dense(input_channels=4, output_channels=8, has_bias=True)
(1): ReLU()
(2): Dense(input_channels=8, output_channels=4, has_bias=True)
(3): ReLU()
)
)
(1): Dense(input_channels=4, output_channels=1, has_bias=True)
)
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
print(rgnet)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
Sequential(
(0): Sequential(
(block 0): Sequential(
(0): Linear(in_features=4, out_features=8, bias=True)
(1): ReLU()
(2): Linear(in_features=8, out_features=4, bias=True)
(3): ReLU()
)
(block 1): Sequential(
(0): Linear(in_features=4, out_features=8, bias=True)
(1): ReLU()
(2): Linear(in_features=8, out_features=4, bias=True)
(3): ReLU()
)
(block 2): Sequential(
(0): Linear(in_features=4, out_features=8, bias=True)
(1): ReLU()
(2): Linear(in_features=8, out_features=4, bias=True)
(3): ReLU()
)
(block 3): Sequential(
(0): Linear(in_features=4, out_features=8, bias=True)
(1): ReLU()
(2): Linear(in_features=8, out_features=4, bias=True)
(3): ReLU()
)
)
(1): Linear(in_features=4, out_features=1, bias=True)
)
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
rgnet[0][1][0].bias.value()
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
Tensor(shape=[8], dtype=Float32, value= [-1.41463935e-01, 3.31256509e-01, 4.59664226e-01, 4.15232331e-02, 3.21486354e-01, -1.22822206e-02, 6.90865889e-02, -4.68575448e-01])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
rgnet[0][1][0].bias.data
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor([-0.1838, 0.4298, -0.0177, 0.0446, -0.4949, -0.4320, 0.0924, 0.0711])
.. raw:: html
.. raw:: html
.. raw:: html
默认情况下，MindSpore会使用Normal初始化权重矩阵， 偏置参数设置为0。
MindSpore的\ ``common.initializer``\ 模块中提供了各种初始化方法。
.. raw:: html
.. raw:: html
默认情况下，PyTorch会根据一个范围均匀地初始化权重和偏置矩阵，
这个范围是根据输入和输出维度计算出的。
PyTorch的\ ``nn.init``\ 模块提供了多种预置初始化方法。
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
def init_normal(m):
if type(m) == nn.Dense:
m.weight.set_data(initializer('normal', m.weight.shape, m.weight.dtype))
m.bias.set_data(initializer('zeros', m.bias.shape, m.bias.dtype))
net.apply(init_normal)
net[0].weight.data[0], net[0].bias.data[0]
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
(Tensor(shape=[4], dtype=Float32, value= [-7.03959377e-04, 1.28101204e-02, -1.53673217e-02, -1.59187196e-03]),
Tensor(shape=[], dtype=Float32, value= 0))
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
def init_normal(m):
if type(m) == nn.Linear:
nn.init.normal_(m.weight, mean=0, std=0.01)
nn.init.zeros_(m.bias)
net.apply(init_normal)
net[0].weight.data[0], net[0].bias.data[0]
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
(tensor([ 0.0077, -0.0071, 0.0050, -0.0177]), tensor(0.))
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
def init_constant(m):
if type(m) == nn.Dense:
m.weight.set_data(initializer('one', m.weight.shape, m.weight.dtype))
m.bias.set_data(initializer('zeros', m.bias.shape, m.bias.dtype))
net.apply(init_constant)
net[0].weight.data[0], net[0].bias.data[0]
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
(Tensor(shape=[4], dtype=Float32, value= [ 1.00000000e+00, 1.00000000e+00, 1.00000000e+00, 1.00000000e+00]),
Tensor(shape=[], dtype=Float32, value= 0))
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
def init_constant(m):
if type(m) == nn.Linear:
nn.init.constant_(m.weight, 1)
nn.init.zeros_(m.bias)
net.apply(init_constant)
net[0].weight.data[0], net[0].bias.data[0]
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
(tensor([1., 1., 1., 1.]), tensor(0.))
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
def init_xavier(m):
if type(m) == nn.Dense:
m.weight.set_data(initializer('xavier_uniform', m.weight.shape, m.weight.dtype))
def init_42(m):
if type(m) == nn.Dense:
m.weight.set_data(initializer(42, m.weight.shape, m.weight.dtype))
net[0].apply(init_xavier)
net[2].apply(init_42)
print(net[0].weight.data[0])
print(net[2].weight.data.value())
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
[ 0.67527854 0.31270772 0.20053129 -0.29991123]
[[42. 42. 42. 42. 42. 42. 42. 42.]]
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
def init_xavier(m):
if type(m) == nn.Linear:
nn.init.xavier_uniform_(m.weight)
def init_42(m):
if type(m) == nn.Linear:
nn.init.constant_(m.weight, 42)
net[0].apply(init_xavier)
net[2].apply(init_42)
print(net[0].weight.data[0])
print(net[2].weight.data)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor([-0.6666, 0.0822, 0.5257, 0.5681])
tensor([[42., 42., 42., 42., 42., 42., 42., 42.]])
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
def my_init(m):
if type(m) == nn.Dense:
print("Init", *[(name, param.shape)
for name, param in m.parameters_dict().items()][0])
m.weight.set_data(initializer(Uniform(scale=10), m.weight.shape, m.weight.dtype))
m.weight.data[:] *= (m.weight.data.abs() >= 5)
net.apply(my_init)
net[0].weight[:2]
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
Init 0.weight (8, 4)
Init 2.weight (1, 8)
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
Tensor(shape=[2, 4], dtype=Float32, value=
[[-0.00000000e+00, -0.00000000e+00, 0.00000000e+00, 0.00000000e+00],
[-0.00000000e+00, -0.00000000e+00, -0.00000000e+00, 5.32515860e+00]])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
def my_init(m):
if type(m) == nn.Linear:
print("Init", *[(name, param.shape)
for name, param in m.named_parameters()][0])
nn.init.uniform_(m.weight, -10, 10)
m.weight.data *= m.weight.data.abs() >= 5
net.apply(my_init)
net[0].weight[:2]
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
Init weight torch.Size([8, 4])
Init weight torch.Size([1, 8])
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor([[-8.9009, 0.0000, -7.2894, 9.6437],
[ 0.0000, -6.8565, 7.6186, 8.8137]], grad_fn=)
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
net[0].weight.data[:] += 1
net[0].weight.data[0, 0] = 42
net[0].weight.data[0]
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
Tensor(shape=[4], dtype=Float32, value= [ 4.20000000e+01, 1.00000000e+00, 1.00000000e+00, 1.00000000e+00])
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
net[0].weight.data[:] += 1
net[0].weight.data[0, 0] = 42
net[0].weight.data[0]
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor([42.0000, 1.0000, -6.2894, 10.6437])
.. raw:: html
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
# 我们需要给共享层一个名称，以便可以引用它的参数
# 我们需要给共享层一个名称，以便可以引用它的参数
shared = nn.Dense(8, 8)
net = nn.SequentialCell([nn.Dense(4, 8), nn.ReLU(),
shared, nn.ReLU(),
shared, nn.ReLU(),
nn.Dense(8, 1)])
net(X)
# 检查参数是否相同
print(net[2].weight.data[0] == net[4].weight.data[0])
net[2].weight.data[0, 0] = 100
# 确保它们实际上是同一个对象，而不只是有相同的值
print(net[2].weight.data[0] == net[4].weight.data[0])
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
[ True True True True True True True True]
[ True True True True True True True True]
.. raw:: html
.. raw:: html
.. raw:: latex
\diilbookstyleinputcell
.. code:: python
# 我们需要给共享层一个名称，以便可以引用它的参数
shared = nn.Linear(8, 8)
net = nn.Sequential(nn.Linear(4, 8), nn.ReLU(),
shared, nn.ReLU(),
shared, nn.ReLU(),
nn.Linear(8, 1))
net(X)
# 检查参数是否相同
print(net[2].weight.data[0] == net[4].weight.data[0])
net[2].weight.data[0, 0] = 100
# 确保它们实际上是同一个对象，而不只是有相同的值
print(net[2].weight.data[0] == net[4].weight.data[0])
.. raw:: latex
\diilbookstyleoutputcell
.. parsed-literal::
:class: output
tensor([True, True, True, True, True, True, True, True])
tensor([True, True, True, True, True, True, True, True])
.. raw:: html
.. raw:: html
.. raw:: html
`讨论 `__
.. raw:: html
.. raw:: html
`讨论 `__
.. raw:: html
.. raw:: html