神經網路的梯度下降演算法：梯度矩陣的鏈式法則（便於向量化代碼實現）

04-30

$f({f H}_{n imes m}): mathbb{R}^{n imes m} mapsto mathbb{R}$ 是可微函數，記 ${f D}=frac{partial f}{partial {f H}}$ ，若 ${f H}={f X}_{n imes d}{f W}_{d imes m}+{f B}_{n imes m}$ ，則有：

(1) $frac{partial f}{partial{f X}}={f D}{f W}^T$

(2) $frac{partial f}{partial {f W}}={f X}^T{f D}$

(3) $frac{partial f}{partial {f B}}={f D}$

推論：若又有 ${f B}={f 1}_{n imes 1} cdot {f b}_{1 imes m}$ ，則可由(1)(2)得 $frac{partial f}{partial {f b}}={f 1}_{1 imes n}{f D}=sum(D, axis=0)$

證明：

$h_{ik}=sumlimits_{t=1}^d x_{it}w_{tk} Rightarrow frac{partial h_{ik}}{partial x_{ij}}=w_{jk}$

$frac{partial f}{partial x_{ij}}=sumlimits_{k=1}^{m}frac{partial f}{partial h_{ik}} cdot frac{partial h_{ik}}{partial x_{ij}}=sumlimits_{k=1}^{m} d_{ik} w_{jk}$

所以 $frac{partial f}{partial {f X}}={f D}{f W}^T$

$h_{kj}=sumlimits_{t=1}^d x_{kt}w_{tj} Rightarrow frac{partial h_{kj}}{partial w_{ij}}=x_{ki}$

$frac{partial f}{partial w_{ij}}=sumlimits_{k=1}^nfrac{partial f}{partial h_{kj}} cdot frac{partial h_{kj}}{partial w_{ij}}=sumlimits_{k=1}^n d_{kj}x_{ki}$

所以 $frac{partial f}{partial {f W}}={f X}^T{f D}$

$frac{partial f}{partial b_{ij}}=frac{partial f}{partial h_{ij}} cdot frac{partial h_{ij}}{partial b_{ij}}=d_{ij}$

所以 $frac{partial f}{partial {f B}}={f D}$