# 激活函数

（重定向自激励函数

## 函數

Softsign函數[1][2]   ${\displaystyle f(x)={\frac {x}{1+|x|}}}$  ${\displaystyle f'(x)={\frac {1}{(1+|x|)^{2}}}}$  ${\displaystyle (-1,1)}$  ${\displaystyle C^{1}}$

with ${\displaystyle \lambda =1.0507}$  and ${\displaystyle \alpha =1.67326}$

${\displaystyle f'(\alpha ,x)=\lambda {\begin{cases}\alpha (e^{x})&{\text{for }}x<0\\1&{\text{for }}x\geq 0\end{cases}}}$  ${\displaystyle (-\lambda \alpha ,\infty )}$  ${\displaystyle C^{0}}$
S 型線性整流激活函數(SReLU)[8] ${\displaystyle f_{t_{l},a_{l},t_{r},a_{r}}(x)={\begin{cases}t_{l}+a_{l}(x-t_{l})&{\text{for }}x\leq t_{l}\\x&{\text{for }}t_{l}
${\displaystyle t_{l},a_{l},t_{r},a_{r}}$  are parameters.
${\displaystyle f'_{t_{l},a_{l},t_{r},a_{r}}(x)={\begin{cases}a_{l}&{\text{for }}x\leq t_{l}\\1&{\text{for }}t_{l}  ${\displaystyle (-\infty ,\infty )}$  ${\displaystyle C^{0}}$

SoftPlus函數[10]   ${\displaystyle f(x)=\ln(1+e^{x})}$  ${\displaystyle f'(x)={\frac {1}{1+e^{-x}}}}$  ${\displaystyle (0,\infty )}$  ${\displaystyle C^{\infty }}$

Sigmoid-weighted linear unit (SiLU)[11] (也被稱為Swish[12]) ${\displaystyle f(x)=x\cdot \sigma (x)}$ [4] ${\displaystyle f'(x)=f(x)+\sigma (x)(1-f(x))}$ [5] ${\displaystyle [\approx -0.28,\infty )}$  ${\displaystyle C^{\infty }}$
SoftExponential函數[13]   ${\displaystyle f(\alpha ,x)={\begin{cases}-{\frac {\ln(1-\alpha (x+\alpha ))}{\alpha }}&{\text{for }}\alpha <0\\x&{\text{for }}\alpha =0\\{\frac {e^{\alpha x}-1}{\alpha }}+\alpha &{\text{for }}\alpha >0\end{cases}}}$  ${\displaystyle f'(\alpha ,x)={\begin{cases}{\frac {1}{1-\alpha (\alpha +x)}}&{\text{for }}\alpha <0\\e^{\alpha x}&{\text{for }}\alpha \geq 0\end{cases}}}$  ${\displaystyle (-\infty ,\infty )}$  ${\displaystyle C^{\infty }}$  Yes iff ${\displaystyle \alpha =0}$

Sinc函數   ${\displaystyle f(x)={\begin{cases}1&{\text{for }}x=0\\{\frac {\sin(x)}{x}}&{\text{for }}x\neq 0\end{cases}}}$  ${\displaystyle f'(x)={\begin{cases}0&{\text{for }}x=0\\{\frac {\cos(x)}{x}}-{\frac {\sin(x)}{x^{2}}}&{\text{for }}x\neq 0\end{cases}}}$  ${\displaystyle [\approx -.217234,1]}$  ${\displaystyle C^{\infty }}$

^ 此處H單位階躍函數
^ α是在訓練時間從均勻分佈中抽取的隨機變量，並且在測試時間固定為分佈的期望值
^ ^ ^ 此處${\displaystyle \sigma }$ 邏輯函數

Softmax函數 ${\displaystyle f_{i}({\vec {x}})={\frac {e^{x_{i}}}{\sum _{j=1}^{J}e^{x_{j}}}}}$     for i = 1, …, J ${\displaystyle {\frac {\partial f_{i}({\vec {x}})}{\partial x_{j}}}=f_{i}({\vec {x}})(\delta _{ij}-f_{j}({\vec {x}}))}$ [6] ${\displaystyle (0,1)}$  ${\displaystyle C^{\infty }}$
Maxout函數[14] ${\displaystyle f({\vec {x}})=\max _{i}x_{i}}$  ${\displaystyle {\frac {\partial f}{\partial x_{j}}}={\begin{cases}1&{\text{for }}j={\underset {i}{\operatorname {argmax} }}\,x_{i}\\0&{\text{for }}j\neq {\underset {i}{\operatorname {argmax} }}\,x_{i}\end{cases}}}$  ${\displaystyle (-\infty ,\infty )}$  ${\displaystyle C^{0}}$

^ 此處δ克羅內克δ函數

## 參考資料

1. ^ Bergstra, James; Desjardins, Guillaume; Lamblin, Pascal; Bengio, Yoshua. Quadratic polynomials learn better image features". Technical Report 1337. Département d’Informatique et de Recherche Opérationnelle, Université de Montréal. 2009. （原始内容存档于2018-09-25）.
2. ^ Glorot, Xavier; Bengio, Yoshua, Understanding the difficulty of training deep feedforward neural networks (PDF), International Conference on Artificial Intelligence and Statistics (AISTATS’10), Society for Artificial Intelligence and Statistics, 2010, （原始内容存档 (PDF)于2017-04-01）
3. Carlile, Brad; Delamarter, Guy; Kinney, Paul; Marti, Akiko; Whitney, Brian. Improving Deep Learning by Inverse Square Root Linear Units (ISRLUs). 2017-11-09. [cs.LG].
4. ^ He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing; Sun, Jian. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. 2015-02-06. [cs.CV].
5. ^ Xu, Bing; Wang, Naiyan; Chen, Tianqi; Li, Mu. Empirical Evaluation of Rectified Activations in Convolutional Network. 2015-05-04. [cs.LG].
6. ^ Clevert, Djork-Arné; Unterthiner, Thomas; Hochreiter, Sepp. Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). 2015-11-23. [cs.LG].
7. ^ Klambauer, Günter; Unterthiner, Thomas; Mayr, Andreas; Hochreiter, Sepp. Self-Normalizing Neural Networks. 2017-06-08. [cs.LG].
8. ^ Jin, Xiaojie; Xu, Chunyan; Feng, Jiashi; Wei, Yunchao; Xiong, Junjun; Yan, Shuicheng. Deep Learning with S-shaped Rectified Linear Activation Units. 2015-12-22. [cs.CV].
9. ^ Forest Agostinelli; Matthew Hoffman; Peter Sadowski; Pierre Baldi. Learning Activation Functions to Improve Deep Neural Networks. 21 Dec 2014. [cs.NE].
10. ^ Glorot, Xavier; Bordes, Antoine; Bengio, Yoshua. Deep sparse rectifier neural networks (PDF). International Conference on Artificial Intelligence and Statistics. 2011. （原始内容存档 (PDF)于2018-06-19）.
11. ^ Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning. [2018-06-13]. （原始内容存档于2018-06-13）.
12. ^ Searching for Activation Functions. [2018-06-13]. （原始内容存档于2018-06-13）.
13. ^ Godfrey, Luke B.; Gashler, Michael S. A continuum among logarithmic, linear, and exponential functions, and its potential to improve generalization in neural networks. 7th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management: KDIR. 2016-02-03, 1602: 481–486. Bibcode:2016arXiv160201321G. .
14. ^ Goodfellow, Ian J.; Warde-Farley, David; Mirza, Mehdi; Courville, Aaron; Bengio, Yoshua. Maxout Networks. JMLR WCP. 2013-02-18, 28 (3): 1319–1327. Bibcode:2013arXiv1302.4389G. .