o Qeع@sddlmZddlmZddlmZddlmZddl m Z gZ Gdd d e Z Gd d d e Z Gd d d e ZGddde ZGddde ZGddde ZGddde ZGddde ZGddde ZGddde ZGddde ZGddde ZGd d!d!e ZGd"d#d#e ZGd$d%d%e ZGd&d'd'e ZGd(d)d)e ZGd*d+d+e ZGd,d-d-e ZGd.d/d/e ZGd0d1d1e Z Gd2d3d3e Z!Gd4d5d5e Z"Gd6d7d7e Z#Gd8d9d9e Z$Gd:d;d;e Z%Gdd?d?e Z'd@S)A) ParamAttr)Constant)get_default_dtype) functional)Layerc2eZdZdZd fdd ZddZdd ZZS) CELUa CELU Activation. .. math:: CELU(x) = max(0, x) + min(0, \alpha * (e^{x/\alpha}-1)) Parameters: alpha (float, optional): The 'alpha' value of the CELU formulation. Default is 1.0. name (str, optional): Name for the operation (optional, default is None). For more information, please refer to :ref:`api_guide_Name`. Shape: - input: Tensor with any shape. - output: Tensor with the same shape as input. Examples: .. code-block:: python import paddle x = paddle.to_tensor([[-1. ,6.], [1., 15.6]]) m = paddle.nn.CELU(0.2) out = m(x) # [[-0.19865242, 6. ], # [ 1. , 15.60000038]] ?Nctt|||_||_dSN)superr __init___alpha_nameselfalphaname __class__JD:\Projects\ConvertPro\env\Lib\site-packages\paddle/nn/layer/activation.pyr7 z CELU.__init__cCt||j|jSr )FZcelurrrxrrrforward<z CELU.forwardcC$|jr d|jnd}d|j|SN , name={}z alpha={}{}rformatrrZname_strrrr extra_repr?zCELU.extra_reprr N__name__ __module__ __qualname____doc__rrr( __classcell__rrrrr s r cr ) ELUa ELU Activation. .. math:: ELU(x)= \left\{ \begin{array}{lcl} x,& &\text{if } \ x > 0 \\ alpha * (e^{x} - 1),& &\text{if } \ x <= 0 \end{array} \right. Parameters: alpha (float, optional): The 'alpha' value of the ELU formulation. Default is 1.0. name (str, optional): Name for the operation (optional, default is None). For more information, please refer to :ref:`api_guide_Name`. Shape: - input: Tensor with any shape. - output: Tensor with the same shape as input. Examples: .. code-block:: python import paddle x = paddle.to_tensor([[-1. ,6.], [1., 15.6]]) m = paddle.nn.ELU(0.2) out = m(x) # [[-0.12642411 6. ] # [ 1. 15.6 ]] r Ncr r )rr1rrrrrrrrgrz ELU.__init__cCrr )rZelurrrrrrrlr z ELU.forwardcCr!r"r%r'rrrr(or)zELU.extra_reprr*r+rrrrr1D "r1cr ) GELUa GELU Activation. If approximate is True .. math:: GELU(x) = 0.5 * x * (1 + tanh(\sqrt{\frac{2}{\pi}} * (x + 0.044715x^{3}))) else .. math:: GELU(x) = 0.5 * x * (1 + erf(\frac{x}{\sqrt{2}})) Parameters: approximate (bool, optional): Wether to enable approximation. Default is False. name (str, optional): Name for the operation (optional, default is None). For more information, please refer to :ref:`api_guide_Name`. Shape: - input: Tensor with any shape. - output: Tensor with the same shape as input. Examples: .. code-block:: python import paddle x = paddle.to_tensor([[-1, 0.5],[1, 1.5]]) m = paddle.nn.GELU() out = m(x) # [-0.158655 0.345731 0.841345 1.39979] m = paddle.nn.GELU(True) out = m(x) # [-0.158808 0.345714 0.841192 1.39957] FNcr r )rr3r _approximater)rZ approximaterrrrrrz GELU.__init__cCrr )rZgelur4rrrrrrr z GELU.forwardcCr!)Nr#r$zapproximate={}{})rr&r4r'rrrr(r)zGELU.extra_repr)FNr+rrrrr3ts &r3cr ) Hardshrinkap Hardshrink Activation .. math:: hardshrink(x)= \left\{ \begin{array}{rcl} x, & & if \ x > threshold \\ x, & & if \ x < -threshold \\ 0, & & if \ others \end{array} \right. Parameters: threshold (float, optional): The value of threshold for hardthrink. Default is 0.5 name (str, optional): Name for the operation (optional, default is None). For more information, please refer to :ref:`api_guide_Name`. Shape: - input: Tensor with any shape. - output: Tensor with the same shape as input. Examples: .. code-block:: python import paddle x = paddle.to_tensor([-1, 0.3, 2.5]) m = paddle.nn.Hardshrink() out = m(x) # [-1., 0., 2.5] ?Ncr r )rr5r _thresholdrr thresholdrrrrrrzHardshrink.__init__cCrr )rZ hardshrinkr7rrrrrrr zHardshrink.forwardcCr!Nr#r$zthreshold={}{}rr&r7r'rrrr(r)zHardshrink.extra_reprr6Nr+rrrrr5r2r5c2eZdZdZd fdd ZddZddZZS) Hardswisha- Hardswish activation. Create a callable object of `Hardswish`. Hardswish is proposed in MobileNetV3, and performs better in computational stability and efficiency compared to swish function. For more details please refer to: https://arxiv.org/pdf/1905.02244.pdf .. math:: Hardswish(x)= \left\{ \begin{array}{cll} 0 &, & \text{if } x \leq -3 \\ x &, & \text{if } x \geq 3 \\ \frac{x(x+3)}{6} &, & \text{otherwise} \end{array} \right. Parameters: name (str, optional): Name for the operation (optional, default is None). For more information, please refer to :ref:`api_guide_Name`. Shape: - input: Tensor with any shape. - output: Tensor with the same shape as input. Examples: .. code-block:: python import paddle x = paddle.to_tensor([-4., 5., 1.]) m = paddle.nn.Hardswish() out = m(x) # [0., 5., 0.666667] Nctt|||_dSr )rr>rrrrrrrr zHardswish.__init__cCt||jSr )rZ hardswishrrrrrrzHardswish.forwardcC|jr d|j}|Sd}|SNzname={}r$rr&r'rrrr(zHardswish.extra_reprr r+rrrrr>s %r>cr=) Tanha Tanh Activation. .. math:: Tanh(x) = \frac{e^{x} - e^{-x}}{e^{x} + e^{-x}} Parameters: name (str, optional): Name for the operation (optional, default is None). For more information, please refer to :ref:`api_guide_Name`. Shape: - input: Tensor with any shape. - output: Tensor with the same shape as input. Examples: .. code-block:: python import paddle x = paddle.to_tensor([-0.4, -0.2, 0.1, 0.3]) m = paddle.nn.Tanh() out = m(x) print(out) # Tensor(shape=[4], dtype=float32, place=Place(gpu:0), stop_gradient=True, # [-0.37994894, -0.19737533, 0.09966800, 0.29131261]) Ncr?r )rrHrrr@rrrr'rAz Tanh.__init__cCrBr )rtanhrrrrrr+rCz Tanh.forwardcCrDrErFr'rrrr(.rGzTanh.extra_reprr r+rrrrrH  rHc2eZdZdZd fdd ZddZd d ZZS) Hardtanha Hardtanh Activation. Create a callable object of `Hardtanh`. .. math:: Hardtanh(x)= \left\{ \begin{array}{cll} max,& & \text{if } x > max \\ min,& & \text{if } x < min \\ x,& & \text{otherwise} \end{array} \right. Parameters: min (float, optional): The value of min for Hardtanh. Default is -1. max (float, optional): The value of max for Hardtanh. Default is 1. name (str, optional): Name for the operation (optional, default is None). For more information, please refer to :ref:`api_guide_Name`. Shape: - input: Tensor with any shape. - output: Tensor with the same shape as input. Examples: .. code-block:: python import paddle x = paddle.to_tensor([-1.5, 0.3, 2.5]) m = paddle.nn.Hardtanh() out = m(x) # [-1., 0.3, 1.] r Nc$tt|||_||_||_dSr )rrLr_min_maxr)rminmaxrrrrrW zHardtanh.__init__cCt||j|j|jSr )rZhardtanhrOrPrrrrrr]zHardtanh.forwardcC(|jr d|jnd}d|j|j|S)Nr#r$zmin={}, max={}{})rr&rOrPr'rrrr(`zHardtanh.extra_repr)rMr Nr+rrrrrL3s #rLcs<eZdZdZ     d fdd Zdd Zd d ZZS) PReLUa PReLU Activation. .. math:: PReLU(x) = max(0, x) + weight * min(0, x) Parameters: num_parameters (int, optional): Number of `weight` to learn. The supported values are: 1 - a single parameter `alpha` is used for all input channels; Number of channels - a separate `alpha` is used for each input channel. Default is 1. init (float, optional): Init value of learnable `weight`. Default is 0.25. weight_attr(ParamAttr, optional): The parameter attribute for the learnable `weight`. Default is None. For more information, please refer to :ref:`api_paddle_ParamAttr`. name (str, optional): Name for the operation (optional, default is None). For more information, please refer to :ref:`api_guide_Name`. data_format(str, optional): Data format that specifies the layout of input. It may be "NC", "NCL", "NCHW", "NCDHW", "NLC", "NHWC" or "NDHWC". Default: "NCHW". Shape: - input: Tensor with any shape. Default dtype is float32. - output: Tensor with the same shape as input. Examples: .. code-block:: python import paddle paddle.set_default_dtype("float64") data = paddle.to_tensor([[[[-2.0, 3.0, -4.0, 5.0], [ 3.0, -4.0, 5.0, -6.0], [-7.0, -8.0, 8.0, 9.0]], [[ 1.0, -2.0, -3.0, 4.0], [-5.0, 6.0, 7.0, -8.0], [ 6.0, 7.0, 8.0, 9.0]]]]) m = paddle.nn.PReLU(1, 0.25) out = m(data) print(out) # [[[[-0.5 , 3. , -1. , 5. ], # [ 3. , -1. , 5. , -1.5 ], # [-1.75, -2. , 8. , 9. ]], # [[ 1. , -0.5 , -0.75, 4. ], # [-1.25, 6. , 7. , -2. ], # [ 6. , 7. , 8. , 9. ]]]] ?NNCHWcsTtt|||_||_||_||_||_|j|j|jgt dt |jd|_ dS)NF)attrshapedtypeZis_biasZdefault_initializer) rrXr_num_parameters_initZ _weight_attrr _data_formatZcreate_parameterrr_weight)rZnum_parametersinitZ weight_attr data_formatrrrrrs zPReLU.__init__cCstj||j|jdS)N)rd)rZprelurbrarrrrrsz PReLU.forwardcC0|jr d|jnd}d|j|j|j|j|S)Nr#r$z6num_parameters={}, data_format={}, init={}, dtype={}{})rr&r_rar`_dtyper'rrrr(szPReLU.extra_repr)rYrZNr[Nr+rrrrrXes2rXcrK) RReLUa RReLU activation layer. Applies the randomized leaky rectified liner unit function to improve generalization performance, as described in the paper: `Empirical Evaluation of Rectified Activations in Convolutional Network `_ During training, randomly samples the negative slope for activation values as described below: .. math:: RReLU(x)= \left\{ \begin{array}{rcl} x, & & if \ x >= 0 \\ a * x, & & otherwise \\ \end{array} \right. where :math:`x` is the input tensor, :math:`a` is randomly sampled from uniform distribution in range (:math:`lower`, :math:`upper`), In the test phase, the negative slope will take the average value of :math:`lower` and :math:`upper`: .. math:: RReLU(x)= \left\{ \begin{array}{rcl} x, & & if \ x >= 0 \\ (lower + upper) * 0.5 * x, & & otherwise \\ \end{array} \right. where :math:`x` is the input tensor, :math:`lower` and :math:`upper` are the bounds of uniform distribution. Parameters: lower (float, optional): The lower bound of uniform distribution. Default: 0.125. upper (float, optional): The upper bound of uniform distribution. Default: 0.333. name (str, optional): Name for the operation (optional, default is None). For more information, please refer to :ref:`api_guide_Name`. Shape: - input: Tensor with any shape. Default dtype is float32. - output: Tensor with the same shape as input. Examples: .. code-block:: python import paddle input_tensor = paddle.to_tensor([[[[-2.0, 3.0, -4.0, 5.0], [ 3.0, -4.0, 5.0, -6.0], [-7.0, -8.0, 8.0, 9.0]], [[ 1.0, -2.0, -3.0, 4.0], [-5.0, 6.0, 7.0, -8.0], [ 6.0, 7.0, 8.0, 9.0]]]], dtype='float32') rrelu_layer = paddle.nn.RReLU(0.1, 0.3) out = rrelu_layer(input_tensor) print(out) #[[[[-0.20000899 3. -0.88108218 5. ] # [ 3. -0.55175185 5. -1.07761011] # [-1.06806871 -1.98962009 8. 9. ]] # [[ 1. -0.52382672 -0.65515128 4. ] # [-1.37663394 6. 7. -2.34657836] # [ 6. 7. 8. 9. ]]]] ?UUUUUU?NcrNr )rrgr_lower_upperr)rlowerupperrrrrrrSzRReLU.__init__cCstj||j|j|jdS)N)rlrmtraining)rZrrelurjrkrnrrrrrsz RReLU.forwardcCre)Nr#r$z+lower={}, upper={}, training={}, dtype={}{})rr&rjrkrnrfr'rrrr( szRReLU.extra_repr)rhriNr+rrrrrgs Frgcr=) ReLUaA ReLU Activation. .. math:: ReLU(x) = max(x, 0) Parameters: name (str, optional): Name for the operation (optional, default is None). For more information, please refer to :ref:`api_guide_Name`. Shape: - input: Tensor with any shape. - output: Tensor with the same shape as input. Examples: .. code-block:: python import paddle x = paddle.to_tensor([-2., 0., 1.]) m = paddle.nn.ReLU() out = m(x) print(out) # [0., 0., 1.] Ncr?r )rrorrr@rrrr0rAz ReLU.__init__cCrBr )rZrelurrrrrr4rCz ReLU.forwardcCrDrErFr'rrrr(7rGzReLU.extra_reprr r+rrrrro rocr=) ReLU6aK ReLU6 Activation .. math:: ReLU6(x) = min(max(0,x), 6) Parameters: name (str, optional): Name for the operation (optional, default is None). For more information, please refer to :ref:`api_guide_Name`. Shape: - input: Tensor with any shape. - output: Tensor with the same shape as input. Examples: .. code-block:: python import paddle x = paddle.to_tensor([-1., 0.3, 6.5]) m = paddle.nn.ReLU6() out = m(x) print(out) # [0, 0.3, 6] Ncr?r )rrqrrr@rrrrXrAzReLU6.__init__cCrBr )rZrelu6rrrrrr\rCz ReLU6.forwardcCrDrErFr'rrrr(_rGzReLU6.extra_reprr r+rrrrrq<rprqcs8eZdZdZ   d fdd ZddZd d ZZS) SELUa9 SELU Activation .. math:: SELU(x)= scale * \left\{ \begin{array}{lcl} x,& &\text{if } \ x > 0 \\ alpha * e^{x} - alpha,& &\text{if } \ x <= 0 \end{array} \right. Parameters: scale (float, optional): The value of scale(must be greater than 1.0) for SELU. Default is 1.0507009873554804934193349852946 alpha (float, optional): The value of alpha(must be no less than zero) for SELU. Default is 1.6732632423543772848170429916717 name (str, optional): Name for the operation (optional, default is None). For more information, please refer to :ref:`api_guide_Name`. Shape: - input: Tensor with any shape. - output: Tensor with the same shape as input. Examples: .. code-block:: python import paddle x = paddle.to_tensor([[0.0, 1.0],[2.0, 3.0]]) m = paddle.nn.SELU() out = m(x) print(out) # [[0, 1.050701],[2.101402, 3.152103]] 2֫?,x?NcrNr )rrrr_scalerr)rscalerrrrrrs z SELU.__init__cCrTr )rZselururrrrrrrrUz SELU.forwardcCrV)Nr#r$zscale={:.16f}, alpha={:.16f}{})rr&rurr'rrrr( zSELU.extra_repr)rsrtNr+rrrrrrds% rrcr ) LeakyReLUa Leaky ReLU Activation. Create a callable object of `LeakyReLU` to calculate the `LeakyReLU` of input `x`. .. math:: LeakyReLU(x)= \left\{ \begin{array}{rcl} x, & & if \ x >= 0 \\ negative\_slope * x, & & otherwise \\ \end{array} \right. Parameters: negative_slope (float, optional): Slope of the activation function at :math:`x < 0` . Default is 0.01. name (str, optional): Name for the operation (optional, default is None). For more information, please refer to :ref:`api_guide_Name`. Shape: - input: Tensor with any shape. - output: Tensor with the same shape as input. Examples: .. code-block:: python import paddle m = paddle.nn.LeakyReLU() x = paddle.to_tensor([-2.0, 0, 1]) out = m(x) # [-0.02, 0., 1.] {Gz?Ncr r )rrxr_negative_sloper)rZnegative_sloperrrrrrzLeakyReLU.__init__cCrr )rZ leaky_relurzrrrrrrr zLeakyReLU.forwardcCr!)Nr#r$znegative_slope={}{})rr&rzr'rrrr(r)zLeakyReLU.extra_repr)ryNr+rrrrrxs #rxcr=) Sigmoida this interface is used to construct a callable object of the ``Sigmoid`` class. This layer calcluate the `sigmoid` of input x. .. math:: sigmoid(x) = \frac{1}{1 + e^{-x}} Parameters: name (str, optional): For details, please refer to :ref:`api_guide_Name`. Generally, no setting is required. Default: None. Shape: x: N-D tensor, available dtype is float16, float32, float64. Returns: A callable object of Sigmoid. Examples: .. code-block:: python import paddle m = paddle.nn.Sigmoid() x = paddle.to_tensor([1.0, 2.0, 3.0, 4.0]) out = m(x) # [0.7310586, 0.880797, 0.95257413, 0.98201376] Ncr?r )rr{rrr@rrrrrAzSigmoid.__init__cCrBr )rZsigmoidrrrrrrrCzSigmoid.forwardcCrDrErr&r'rrrr(rGzSigmoid.extra_reprr r+rrrrr{rpr{cr=) Hardsigmoida ``Hardsigmoid`` Activiation Layers, Construct a callable object of the ``Hardsigmoid`` class. This layer calcluate the `hardsigmoid` of input x. A 3-part piecewise linear approximation of sigmoid(https://arxiv.org/abs/1603.00391), which is much faster than sigmoid. .. math:: Hardsigmoid(x)= \left\{ \begin{array}{rcl} 0, & & \text{if } \ x \leq -3 \\ 1, & & \text{if } \ x \geq 3 \\ x/6 + 1/2, & & \text{otherwise} \end{array} \right. Parameters: name (str, optional): Name for the operation (optional, default is None). For more information, please refer to :ref:`api_guide_Name`. Shape: x: N-D tensor, available dtype is float32, float64. Returns: A callable object of Hardsigmoid. Examples: .. code-block:: python import paddle m = paddle.nn.Hardsigmoid() x = paddle.to_tensor([-4., 5., 1.]) out = m(x) # [0., 1, 0.666667] Ncr?r )rr}rrr@rrrrrAzHardsigmoid.__init__cCstj||jdS)N)r)rZ hardsigmoidrrrrrr!szHardsigmoid.forwardcCrDrEr|r'rrrr($rGzHardsigmoid.extra_reprr r+rrrrr}s &r}crK) Softplusa Softplus Activation .. math:: softplus(x)=\begin{cases} \frac{1}{\beta} * \log(1 + e^{\beta * x}),&x\leqslant\frac{\varepsilon}{\beta};\\ x,&x>\frac{\varepsilon}{\beta}. \end{cases} Parameters: beta (float, optional): The value of :math:`\beta` for Softplus. Default is 1 threshold (float, optional): The value of :math:`\varepsilon` for Softplus. Default is 20 name (str, optional): For details, please refer to :ref:`api_guide_Name`. Generally, no setting is required. Default: None. Shape: - input: Tensor with any shape. - output: Tensor with the same shape as input. Examples: .. code-block:: python import paddle x = paddle.to_tensor([-0.4, -0.2, 0.1, 0.3], dtype='float32') m = paddle.nn.Softplus() out = m(x) # [0.513015, 0.598139, 0.744397, 0.854355] rYNcrNr )rr~r_betar7r)rbetar9rrrrrFrSzSoftplus.__init__cCrTr )rZsoftplusrr7rrrrrrLrUzSoftplus.forwardcCrV)Nr#r$zbeta={}, threshold={}{})rr&rr7r'rrrr(OrwzSoftplus.extra_repr)rYrNr+rrrrr~)s r~cr ) SoftshrinkaT Softshrink Activation .. math:: Softshrink(x)= \left\{ \begin{array}{rcl} x - threshold,& & \text{if } x > threshold \\ x + threshold,& & \text{if } x < -threshold \\ 0,& & \text{otherwise} \end{array} \right. Parameters: threshold (float, optional): The value of threshold(must be no less than zero) for softplus. Default is 0.5 name (str, optional): Name for the operation (optional, default is None). For more information, please refer to :ref:`api_guide_Name`. Shape: - input: Tensor with any shape. - output: Tensor with the same shape as input. Examples: .. code-block:: python import paddle x = paddle.to_tensor([-0.9, -0.2, 0.1, 0.8]) m = paddle.nn.Softshrink() out = m(x) print(out) # Tensor(shape=[4], dtype=float32, place=Place(gpu:0), stop_gradient=True, # [-0.39999998, 0. , 0. , 0.30000001]) r6Ncr r )rrrr7rr8rrrr|rzSoftshrink.__init__cCrr )rZ softshrinkr7rrrrrrr zSoftshrink.forwardcCr!r:r;r'rrrr(r)zSoftshrink.extra_reprr<r+rrrrrVs %rcr=) Softsigna Softsign Activation .. math:: Softsign(x) = \frac{x}{1 + |x|} Parameters: name (str, optional): Name for the operation (optional, default is None). For more information, please refer to :ref:`api_guide_Name`. Shape: - input: Tensor with any shape. - output: Tensor with the same shape as input. Examples: .. code-block:: python import paddle x = paddle.to_tensor([-0.4, -0.2, 0.1, 0.3]) m = paddle.nn.Softsign() out = m(x) print(out) # Tensor(shape=[4], dtype=float32, place=Place(gpu:0), stop_gradient=True, # [-0.28571430, -0.16666666, 0.09090909, 0.23076925]) Ncr?r )rrrrr@rrrrrAzSoftsign.__init__cCrBr )rZsoftsignrrrrrrrCzSoftsign.forwardcCrDrErFr'rrrr(rGzSoftsign.extra_reprr r+rrrrrrJrcr=) Swisha Swish Activation. .. math:: Swish(x) = \frac{x}{1 + e^{-x}} Parameters: name (str, optional): Name for the operation (optional, default is None). For more information, please refer to :ref:`api_guide_Name`. Shape: - input: Tensor with any shape. - output: Tensor with the same shape as input. Examples: .. code-block:: python import paddle x = paddle.to_tensor([-2., 0., 1.]) m = paddle.nn.Swish() out = m(x) print(out) # Tensor(shape=[3], dtype=float32, place=Place(gpu:0), stop_gradient=True, # [-0.23840584, 0. , 0.73105854]) Ncr?r )rrrrr@rrrrrAzSwish.__init__cCrBr )rZswishrrrrrrrCz Swish.forwardcCrDrErFr'rrrr(rGzSwish.extra_reprr r+rrrrrrJrcr=) Misha Mish Activation. .. math:: softplus(x) = \begin{cases} x, \text{if } x > \text{threshold} \\ \ln(1 + e^{x}), \text{otherwise} \end{cases} Mish(x) = x * \tanh(softplus(x)) Parameters: name (str, optional): Name for the operation (optional, default is None). For more information, please refer to :ref:`api_guide_Name`. Shape: - input: Tensor with any shape. - output: Tensor with the same shape as input. Examples: .. code-block:: python import paddle x = paddle.to_tensor([-5., 0., 5.]) m = paddle.nn.Mish() out = m(x) # [-0.03357624, 0., 4.99955208] Ncr?r )rrrrr@rrrrrAz Mish.__init__cCrBr )rZmishrrrrrrrCz Mish.forwardcCrDrErFr'rrrr(rGzMish.extra_reprr r+rrrrrs  rcr=) Tanhshrinka Tanhshrink Activation .. math:: Tanhshrink(x) = x - tanh(x) Parameters: name (str, optional): Name for the operation (optional, default is None). For more information, please refer to :ref:`api_guide_Name`. Shape: - input: Tensor with any shape. - output: Tensor with the same shape as input. Examples: .. code-block:: python import paddle x = paddle.to_tensor([-0.4, -0.2, 0.1, 0.3]) m = paddle.nn.Tanhshrink() out = m(x) print(out) # Tensor(shape=[4], dtype=float32, place=Place(gpu:0), stop_gradient=True, # [-0.02005106, -0.00262468, 0.00033200, 0.00868741]) Ncr?r )rrrrr@rrrr%rAzTanhshrink.__init__cCrBr )rZ tanhshrinkrrrrrr)rCzTanhshrink.forwardcCrDrErFr'rrrr(,rGzTanhshrink.extra_reprr r+rrrrrrJrcr ) ThresholdedReLUa Thresholded ReLU Activation .. math:: ThresholdedReLU(x) = \left\{ \begin{array}{rl} x,& \text{if } \ x > threshold \\ 0,& \text{otherwise} \end{array} \right. Parameters: threshold (float, optional): The value of threshold for ThresholdedReLU. Default is 1.0 name (str, optional): Name for the operation (optional, default is None). For more information, please refer to :ref:`api_guide_Name`. Shape: - input: Tensor with any shape. - output: Tensor with the same shape as input. Examples: .. code-block:: python import paddle x = paddle.to_tensor([2., 0., 1.]) m = paddle.nn.ThresholdedReLU() out = m(x) print(out) # Tensor(shape=[3], dtype=float32, place=Place(gpu:0), stop_gradient=True, # [2., 0., 0.]) r Ncr r )rrrr7rr8rrrrVrzThresholdedReLU.__init__cCrr )rZthresholded_relur7rrrrrr[r zThresholdedReLU.forwardcCr!r:r;r'rrrr(^r)zThresholdedReLU.extra_reprr*r+rrrrr1s $rcr=) Siluai Silu Activation .. math:: silu(x) = \frac{x}{1 + \mathrm{e}^{-x}} Where :math:`x` is the input Tensor. Parameters: name (str, optional): For details, please refer to :ref:`api_guide_Name`. Generally, no setting is required. Default: None. Shape: - input: Tensor with any shape. - output: Tensor with the same shape as input. Examples: .. code-block:: python import paddle x = paddle.to_tensor([1.0, 2.0, 3.0, 4.0]) m = paddle.nn.Silu() out = m(x) # [ 0.731059, 1.761594, 2.857722, 3.928055 ] Ncr?r )rrrrr@rrrr~rAz Silu.__init__cCrBr )rZsilurrrrrrrCz Silu.forwardcCrDrErFr'rrrr(rGzSilu.extra_reprr r+rrrrrc rcr=) LogSigmoida LogSigmoid Activation. .. math:: LogSigmoid(x) = log \frac{1}{1 + e^{-x}} Parameters: x (Tensor): The input Tensor with data type float32, or float64. name (str, optional): Name for the operation (optional, default is None). For more information, please refer to :ref:`api_guide_Name`. Shape: - input: Tensor with any shape. - output: Tensor with the same shape as input. Examples: .. code-block:: python import paddle x = paddle.to_tensor([1.0, 2.0, 3.0, 4.0]) m = paddle.nn.LogSigmoid() out = m(x) # [-0.313262 -0.126928 -0.0485874 -0.0181499] Ncr?r )rrrrr@rrrrrAzLogSigmoid.__init__cCrBr )rZ log_sigmoidrrrrrrrCzLogSigmoid.forwardcCrDrErFr'rrrr(rGzLogSigmoid.extra_reprr r+rrrrrrrcr ) Softmaxa Softmax Activation. This operator implements the softmax layer. The calculation process is as follows: 1. The dimension :attr:`axis` of ``x`` will be permuted to the last. 2. Then ``x`` will be logically flattened to a 2-D matrix. The matrix's second dimension(row length) is the same as the dimension :attr:`axis` of ``x``, and the first dimension(column length) is the product of all other dimensions of ``x``. For each row of the matrix, the softmax operator squashes the K-dimensional(K is the width of the matrix, which is also the size of ``x``'s dimension :attr:`axis`) vector of arbitrary real values to a K-dimensional vector of real values in the range [0, 1] that add up to 1. 3. After the softmax operation is completed, the inverse operations of steps 1 and 2 are performed to restore the two-dimensional matrix to the same dimension as the ``x`` . It computes the exponential of the given dimension and the sum of exponential values of all the other dimensions in the K-dimensional vector input. Then the ratio of the exponential of the given dimension and the sum of exponential values of all the other dimensions is the output of the softmax operator. For each row :math:`i` and each column :math:`j` in the matrix, we have: .. math:: Softmax[i, j] = \frac{\exp(x[i, j])}{\sum_j(exp(x[i, j])} Example: .. code-block:: text Case 1: Input: x.shape = [2, 3, 4] x.data = [[[2.0, 3.0, 4.0, 5.0], [3.0, 4.0, 5.0, 6.0], [7.0, 8.0, 8.0, 9.0]], [[1.0, 2.0, 3.0, 4.0], [5.0, 6.0, 7.0, 8.0], [6.0, 7.0, 8.0, 9.0]]] Attrs: axis = -1 Output: out.shape = [2, 3, 4] out.data = [[[0.0320586 , 0.08714432, 0.23688282, 0.64391426], [0.0320586 , 0.08714432, 0.23688282, 0.64391426], [0.07232949, 0.19661193, 0.19661193, 0.53444665]], [[0.0320586 , 0.08714432, 0.23688282, 0.64391426], [0.0320586 , 0.08714432, 0.23688282, 0.64391426], [0.0320586 , 0.08714432, 0.23688282, 0.64391426]]] Case 2: Input: x.shape = [2, 3, 4] x.data = [[[2.0, 3.0, 4.0, 5.0], [3.0, 4.0, 5.0, 6.0], [7.0, 8.0, 8.0, 9.0]], [[1.0, 2.0, 3.0, 4.0], [5.0, 6.0, 7.0, 8.0], [6.0, 7.0, 8.0, 9.0]]] Attrs: axis = 1 Output: out.shape = [2, 3, 4] out.data = [[[0.00657326, 0.00657326, 0.01714783, 0.01714783], [0.01786798, 0.01786798, 0.04661262, 0.04661262], [0.97555875, 0.97555875, 0.93623955, 0.93623955]], [[0.00490169, 0.00490169, 0.00490169, 0.00490169], [0.26762315, 0.26762315, 0.26762315, 0.26762315], [0.72747516, 0.72747516, 0.72747516, 0.72747516]]] Parameters: axis (int, optional): The axis along which to perform log_softmax calculations. It should be in range [-D, D), where D is the dimensions of ``x`` . If ``axis`` < 0, it works the same way as :math:`axis + D` . Default is -1. name (str, optional): Name for the operation (optional, default is None). For more information, please refer to :ref:`api_guide_Name`. Shape: - input: Tensor with any shape. - output: Tensor with the same shape as input. Examples: .. code-block:: python import paddle x = paddle.to_tensor([[[2.0, 3.0, 4.0, 5.0], [3.0, 4.0, 5.0, 6.0], [7.0, 8.0, 8.0, 9.0]], [[1.0, 2.0, 3.0, 4.0], [5.0, 6.0, 7.0, 8.0], [6.0, 7.0, 8.0, 9.0]]], dtype='float32') m = paddle.nn.Softmax() out = m(x) # [[[0.0320586 , 0.08714432, 0.23688282, 0.64391426], # [0.0320586 , 0.08714432, 0.23688282, 0.64391426], # [0.07232949, 0.19661193, 0.19661193, 0.53444665]], # [[0.0320586 , 0.08714432, 0.23688282, 0.64391426], # [0.0320586 , 0.08714432, 0.23688282, 0.64391426], # [0.0320586 , 0.08714432, 0.23688282, 0.64391426]]] Ncs$tt|||_d|_||_dSr )rrr_axisrfrraxisrrrrr rSzSoftmax.__init__cCrTr )rsoftmaxrrfrrrrrr&rUzSoftmax.forwardcCr!Nr#r$z axis={}{}rr&rr'rrrr()r)zSoftmax.extra_reprrNr+rrrrrs nrcr ) LogSoftmaxa This operator implements the log_softmax layer. The calculation process is as follows: .. math:: \begin{array} {rcl} Out[i, j] &= &log(softmax(x)) \\ &= &log(\frac{\exp(X[i, j])}{\sum_j(\exp(X[i, j])}) \end{array} Parameters: axis (int, optional): The axis along which to perform log_softmax calculations. It should be in range [-D, D), where D is the dimensions of the input Tensor . If ``axis`` < 0, it works the same way as :math:`axis + D` . Default is -1. name (str, optional): Name for the operation (optional, default is None). For more information, please refer to :ref:`api_guide_Name`. Shape: - input: Tensor with any shape. - output: Tensor with the same shape as input. Examples: .. code-block:: python import paddle x = [[[-2.0, 3.0, -4.0, 5.0], [3.0, -4.0, 5.0, -6.0], [-7.0, -8.0, 8.0, 9.0]], [[1.0, -2.0, -3.0, 4.0], [-5.0, 6.0, 7.0, -8.0], [6.0, 7.0, 8.0, 9.0]]] m = paddle.nn.LogSoftmax() x = paddle.to_tensor(x) out = m(x) # [[[ -7.1278396 -2.1278396 -9.127839 -0.12783948] # [ -2.1270514 -9.127051 -0.12705144 -11.127051 ] # [-16.313261 -17.313261 -1.3132617 -0.31326184]] # [[ -3.0518122 -6.051812 -7.051812 -0.051812 ] # [-12.313267 -1.3132664 -0.3132665 -15.313267 ] # [ -3.4401896 -2.4401896 -1.4401896 -0.44018966]]] rNcr r )rrrrrrrrrr[rzLogSoftmax.__init__cCrBr )rZ log_softmaxrrrrrr`rCzLogSoftmax.forwardcCr!rrr'rrrr(cr)zLogSoftmax.extra_reprrr+rrrrr.s ,rcr ) Maxouta> Maxout Activation. Create a callable object of `Maxout`. Assumed the input shape is (N, Ci, H, W). The output shape is (N, Co, H, W). Then Co = Ci/groups and the operator formula is as follows: .. math:: \begin{array}{l} &out_{si+j} = \max_{k} x_{gsi + sk + j} \\ &g = groups \\ &s = \frac{input.size}{num\_channels} \\ &0 \le i < \frac{num\_channels}{groups} \\ &0 \le j < s \\ &0 \le k < groups \end{array} Parameters: groups (int, optional): The groups number of maxout. `groups` specifies the index of channel dimension where maxout will be performed. This must be a factor of number of features. Default is 1. axis (int, optional): The axis along which to perform maxout calculations. It should be 1 when data format is NCHW, be -1 or 3 when data format is NHWC. If ``axis`` < 0, it works the same way as :math:`axis + D` , where D is the dimensions of ``x`` . Default is 1. name (str, optional): Name for the operation (optional, default is None). For more information, please refer to :ref:`api_guide_Name`. Shape: - input: :math:`(N, C_{in}, H_{in}, W_{in})` - output: :math:`(N, C_{out}, H_{out}, W_{out})` Examples: .. code-block:: python import paddle x = paddle.rand([1, 2, 3, 4]) # [[[[0.5002636 0.22272532 0.17402348 0.2874594 ] # [0.95313174 0.6228939 0.7129065 0.7087491 ] # [0.02879342 0.88725346 0.61093384 0.38833922]] # [[0.5231306 0.03807496 0.91661984 0.15602879] # [0.666127 0.616567 0.30741522 0.24044901] # [0.7142536 0.7351477 0.31588817 0.23782359]]]] m = paddle.nn.Maxout(groups=2) out = m(x) # [[[[0.5231306 0.22272532 0.91661984 0.2874594 ] # [0.95313174 0.6228939 0.7129065 0.7087491 ] # [0.7142536 0.88725346 0.61093384 0.38833922]]]] rYNcrNr )rrr_groupsrr)rgroupsrrrrrrrSzMaxout.__init__cCrTr )rZmaxoutrrrrrrrrrUzMaxout.forwardcCrV)Nr#r$zgroups={}, axis={}{})rr&rrr'rrrr(rWzMaxout.extra_repr)rYNr+rrrrrhs 4rcr=) Softmax2Dap Softmax2D Activation. Given a Tensor with shape (B, C, H, W) or (C, H, W), it will apply Softmax to each location (C, h_i, w_j). The sum of result in each location (C, H_i, W_j) will be one. Shape: - Input: :math:`(B, C, H, W)` or :math:`(C, H, W)` - Output: :math:`(B, C, H, W)` or :math:`(C, H, W)`(same as input) Return: A Tensor of the same shape and dtype as input with value in range [0, 1]. Examples: .. code-block:: python import paddle x = paddle.rand([1, 2, 3, 4]) # [[[[0.42496058 0.1172187 0.14664008 0.8151267 ] # [0.24430142 0.42052492 0.60372984 0.79307914] # [0.4539401 0.90458065 0.10235776 0.62009853]] # [[0.11731581 0.16053623 0.05667042 0.91876775] # [0.9413854 0.30770817 0.6788164 0.9543593 ] # [0.4145064 0.75909156 0.11598814 0.73599935]]]] m = paddle.nn.Softmax2D() out = m(x) # [[[[0.5763103 0.48917228 0.5224772 0.4741129 ] # [0.3324591 0.5281743 0.48123717 0.45976716] # [0.5098571 0.5363083 0.49659243 0.4710572 ]] # [[0.42368975 0.51082766 0.47752273 0.5258871 ] # [0.66754097 0.47182566 0.5187628 0.5402329 ] # [0.49014282 0.46369177 0.50340754 0.5289428 ]]]] Ncstt|d|_||_dSr )rrrrfrr@rrrrrzSoftmax2D.__init__cCs:|jdks|jdksJd|jtj|d|j|jdS)Nrz=Softmax2D requires a 3D or 4D tensor as input. Received: {}D.)rr^r)ndimr&rrrfrrrrrrs zSoftmax2D.forwardcCrDrErFr'rrrr(rGzSoftmax2D.extra_reprr r+rrrrrs $rN)(Z frameworkrZ initializerrZpaddle.frameworkrr$rrZ paddle.nnr__all__r r1r3r5r>rHrLrXrgrorqrrrxr{r}r~rrrrrrrrrrrrrrrrsD     *0402)2VY((91(3-3))-)2''}:C