o Qe@sddlZddlmZddlmZddlmZmZmZddlm Z ddlm Z dd l m Z m Z dd lmZdd lmZmZmZddlZddlZddlZdd lmZdd lmZddlmZmZddlmZddlmZddl m!Z!m"Z"gZ#GdddeZ$Gddde$Z%Gddde$Z&Gddde$Z'GdddeZ(GdddeZ)GdddeZ*Gd d!d!e*Z+Gd"d#d#e*Z,Gd$d%d%e*Z-Gd&d'd'e*Z.Gd(d)d)eZ/dS)*N) BatchNorm) SpectralNorm)get_default_dtypeset_default_dtype_non_static_mode)Constant) ParamAttr)check_variable_and_dtype check_type) dygraph_utils) batch_norm layer_norm instance_norm)no_grad) functional)_C_ops _legacy_C_ops)Layer)in_dynamic_mode)in_dygraph_mode_in_legacy_dygraphcsFeZdZdZ      dfdd Zdd Zd d Zd d ZZS)_InstanceNormBasez This class is based class for InstanceNorm1D, 2d, 3d. See InstaceNorm1D, InstanceNorm2D or InstanceNorm3D for more details. h㈵>?NNCHWcstt||dks|dkr||ksJd||_||_||_||_|dkrI|dkrI|j|j|gtddd|_ |j|j|gtddd|_ dSd|_ d|_ dS)NFzOweight_attr and bias_attr must be set to Fasle at the same time in InstanceNorm?attrshapedefault_initializeris_biasT) superr__init___epsilon _weight_attr _bias_attr _num_featurescreate_parameterr scalebias)self num_featuresepsilonmomentum weight_attr bias_attr data_formatname __class__DD:\Projects\ConvertPro\env\Lib\site-packages\paddle/nn/layer/norm.pyr%@s0    z_InstanceNormBase.__init__cCtd)NzInstanceNorm Base errorNotImplementedErrorr-inputr7r7r8_check_input_dimfz"_InstanceNormBase._check_input_dimcCs ||t||j|j|jdS)N)weightr,Zeps)r>rr+r,r&r<r7r7r8forwardis z_InstanceNormBase.forwardcCd|j|jS)Nznum_features={}, epsilon={})formatr)r&r-r7r7r8 extra_reprpz_InstanceNormBase.extra_repr)rrNNrN) __name__ __module__ __qualname____doc__r%r>rArE __classcell__r7r7r5r8r9s &rc@eZdZdZddZdS)InstanceNorm1Da Create a callable object of `InstanceNorm1D`. Applies Instance Normalization over a 3D input (a mini-batch of 1D inputs with additional channel dimension) as described in the paper Instance Normalization: The Missing Ingredient for Fast Stylization . DataLayout: NCL `[batch, in_channels, length]` :math:`input` is the input features over a mini-batch. .. math:: \mu_{\beta} &\gets \frac{1}{HW} \sum_{i=1}^{HW} x_i \qquad &//\ \ mean\ of\ one\ feature\ map\ in\ mini-batch \\ \sigma_{\beta}^{2} &\gets \frac{1}{HW} \sum_{i=1}^{HW}(x_i - \ \mu_{\beta})^2 \qquad &//\ variance\ of\ one\ feature\ map\ in\ mini-batch \\ \hat{x_i} &\gets \frac{x_i - \mu_\beta} {\sqrt{\ \sigma_{\beta}^{2} + \epsilon}} \qquad &//\ normalize \\ y_i &\gets \gamma \hat{x_i} + \beta \qquad &//\ scale\ and\ shift Where `H` means height of feature map, `W` means width of feature map. Parameters: num_features(int): Indicate the number of channels of the input ``Tensor``. epsilon(float, optional): A value added to the denominator for numerical stability. Default is 1e-5. momentum(float, optional): The value used for the moving_mean and moving_var computation. Default: 0.9. weight_attr(ParamAttr|bool, optional): The parameter attribute for Parameter `scale` of instance_norm. If it is set to None or one attribute of ParamAttr, instance_norm will create ParamAttr as weight_attr, the name of scale can be set in ParamAttr. If the Initializer of the weight_attr is not set, the parameter is initialized one. If it is set to False, will not create weight_attr. Default: None. bias_attr(ParamAttr|bool, optional): The parameter attribute for the bias of instance_norm. If it is set to None or one attribute of ParamAttr, instance_norm will create ParamAttr as bias_attr, the name of bias can be set in ParamAttr. If the Initializer of the bias_attr is not set, the bias is initialized zero. If it is set to False, will not create bias_attr. Default: None. data_format(str, optional): Specify the input data format, may be "NC", "NCL". Default "NCL". name(str, optional): Name for the InstanceNorm, default is None. For more information, please refer to :ref:`api_guide_Name`.. Shape: - x: 2-D or 3-D tensor with shape: (batch, num_features) or (batch, num_features, length). - output: 3-D tensor with same shape as input x. Returns: None. Examples: .. code-block:: python import paddle x = paddle.rand((2, 2, 3)) instance_norm = paddle.nn.InstanceNorm1D(2) instance_norm_out = instance_norm(x) print(instance_norm_out) cC8t|jdkrt|jdkrtdt|jdSdSNrrz'expected 2D or 3D input (got {}D input)lenr ValueErrorrCr<r7r7r8r>zInstanceNorm1D._check_input_dimNrGrHrIrJr>r7r7r7r8rMvs zInstanceNorm2D._check_input_dimNrTr7r7r7r8rU ;rUc@rL)InstanceNorm3Da Create a callable object of `InstanceNorm3D`. Applies Instance Normalization over a 5D input (a mini-batch of 3D inputs with additional channel dimension) as described in the paper Instance Normalization: The Missing Ingredient for Fast Stylization . DataLayout: NCHW `[batch, in_channels, D, in_height, in_width]` :math:`input` is the input features over a mini-batch. .. math:: \mu_{\beta} &\gets \frac{1}{HW} \sum_{i=1}^{HW} x_i \qquad &//\ \ mean\ of\ one\ feature\ map\ in\ mini-batch \\ \sigma_{\beta}^{2} &\gets \frac{1}{HW} \sum_{i=1}^{HW}(x_i - \ \mu_{\beta})^2 \qquad &//\ variance\ of\ one\ feature\ map\ in\ mini-batch \\ \hat{x_i} &\gets \frac{x_i - \mu_\beta} {\sqrt{\ \sigma_{\beta}^{2} + \epsilon}} \qquad &//\ normalize \\ y_i &\gets \gamma \hat{x_i} + \beta \qquad &//\ scale\ and\ shift Where `H` means height of feature map, `W` means width of feature map. Parameters: num_features(int): Indicate the number of channels of the input ``Tensor``. epsilon(float, optional): A value added to the denominator for numerical stability. Default is 1e-5. momentum(float, optional): The value used for the moving_mean and moving_var computation. Default: 0.9. weight_attr(ParamAttr|bool, optional): The parameter attribute for Parameter `scale` of instance_norm. If it is set to None or one attribute of ParamAttr, instance_norm will create ParamAttr as weight_attr, the name of scale can be set in ParamAttr. If the Initializer of the weight_attr is not set, the parameter is initialized one. If it is set to False, will not create weight_attr. Default: None. bias_attr(ParamAttr|bool, optional): The parameter attribute for the bias of instance_norm. If it is set to None or one attribute of ParamAttr, instance_norm will create ParamAttr as bias_attr, the name of bias can be set in ParamAttr. If the Initializer of the bias_attr is not set, the bias is initialized zero. If it is set to False, will not create bias_attr. Default: None. data_format(str, optional): Specify the input data format, could be "NCDHW". Default: NCDHW. name(str, optional): Name for the InstanceNorm, default is None. For more information, please refer to :ref:`api_guide_Name`.. Shape: - x: 5-D tensor with shape: (batch, num_features, dims, height, weight). - output: 5-D tensor with same shape as input x. Returns: None. Examples: .. code-block:: python import paddle x = paddle.rand((2, 2, 2, 2, 3)) instance_norm = paddle.nn.InstanceNorm3D(2) instance_norm_out = instance_norm(x) print(instance_norm_out.numpy) cCrVNz!expected 5D input (got {}D input)rPr<r7r7r8r>;rYzInstanceNorm3D._check_input_dimNrTr7r7r7r8r[rZr[cs<eZdZdZ     d fdd ZddZd d ZZS) GroupNormaf This interface is used to construct a callable object of the ``GroupNorm`` class. For more details, refer to code examples. It implements the function of the Group Normalization Layer. Refer to `Group Normalization `_ . Parameters: num_groups(int): The number of groups that divided from channels. num_channels(int): The number of channels of input. epsilon(float, optional): The small value added to the variance to prevent division by zero. Default: 1e-05. weight_attr(ParamAttr|bool, optional): The parameter attribute for the learnable scale :math:`g`. If it is set to False, no scale will be added to the output units. If it is set to None, the bias is initialized one. Default: None. bias_attr(ParamAttr|bool, optional): The parameter attribute for the learnable bias :math:`b`. If it is set to False, no bias will be added to the output units. If it is set to None, the bias is initialized zero. Default: None. data_format(str, optional): Specify the input data format. Only NCHW is supported. Default: NCHW. name(str, optional): Name for the GroupNorm, default is None. For more information, please refer to :ref:`api_guide_Name`.. Shape: - x: Tensor with shape: (batch, num_features, *). - output: The same shape as input x. Returns: None Examples: .. code-block:: python import paddle x = paddle.arange(48, dtype="float32").reshape((2, 6, 2, 2)) group_norm = paddle.nn.GroupNorm(num_channels=6, num_groups=6) group_norm_out = group_norm(x) print(group_norm_out) rNrc stt|||_||_||_||_||_|dkr td||jg}|dkr8|j d|t dd|_ d|j _ n|j |j|t dd|_ |jdkoN|jj dk|j _ |dkrg|j d|t ddd|_d|j_ dS|j |j|dd |_|jdko{|jj dk|j_ dS) Nrzunsupported data layout:Frrr r!Tr#rrr r")r$r^r%r'r(r& _num_channels _num_groupsrRr*r r@ stop_gradient learning_rater,) r-Z num_groupsZ num_channelsr/r1r2r3r4 param_shaper5r7r8r%jsF        zGroupNorm.__init__c Cs|jj|jdd}|jj|jdd}tr*t||j|j|j|j d}t j |ddSt rHt ||j|j||d|jd|j \}}}t j |ddSd|i}|jdurV|j|d<|jdur`|j|d <|jj|jd }|jjd ||||d |j|j d d|j|dS)NTdtypercr)Zactr/groupsXBiasScale)rg group_norm)YMeanVariance)r/rhtypeinputsoutputsattrs)_helper"create_variable_for_type_inferencergrrrlr@r,r&rbr Z_append_activation_in_dygraphrr append_opZappend_activation)r-r=mean_out variance_outZpre_act_rrZgroup_norm_outr7r7r8rAsf         zGroupNorm.forwardcCsd|j|j|jS)Nz*num_groups={}, num_channels={}, epsilon={})rCrbrar&rDr7r7r8rEs zGroupNorm.extra_repr)rNNrNrGrHrIrJr%rArErKr7r7r5r8r^Bs+5>r^cs:eZdZdZ    d fdd ZddZdd ZZS) LayerNormal Construct a callable object of the ``LayerNorm`` class. For more details, refer to code examples. It implements the function of the Layer Normalization Layer and can be applied to mini-batch input data. Refer to `Layer Normalization `_ The formula is as follows: .. math:: \mu & = \frac{1}{H}\sum_{i=1}^{H} x_i \sigma & = \sqrt{\frac{1}{H}\sum_{i=1}^{H}{(x_i - \mu)^2} + \epsilon} y & = f(\frac{g}{\sigma}(x - \mu) + b) - :math:`x`: the vector representation of the summed inputs to the neurons in that layer. - :math:`H`: the number of hidden units in a layers - :math:`\epsilon`: the small value added to the variance to prevent division by zero. - :math:`g`: the trainable scale parameter. - :math:`b`: the trainable bias parameter. Parameters: normalized_shape(int|list|tuple): Input shape from an expected input of size :math:`[*, normalized_shape[0], normalized_shape[1], ..., normalized_shape[-1]]`. If it is a single integer, this module will normalize over the last dimension which is expected to be of that specific size. epsilon(float, optional): The small value added to the variance to prevent division by zero. Default: 1e-05. weight_attr(ParamAttr|bool, optional): The parameter attribute for the learnable gain :math:`g`. If False, weight is None. If is None, a default :code:`ParamAttr` would be added as scale. The :attr:`param_attr` is initialized as 1 if it is added. Default: None. bias_attr(ParamAttr|bool, optional): The parameter attribute for the learnable bias :math:`b`. If is False, bias is None. If is None, a default :code:`ParamAttr` would be added as bias. The :attr:`bias_attr` is initialized as 0 if it is added. Default: None. name(str, optional): Name for the LayerNorm, default is None. For more information, please refer to :ref:`api_guide_Name`.. Shape: - x: 2-D, 3-D, 4-D or 5-D tensor. - output: same shape as input x. Returns: None Examples: .. code-block:: python import paddle x = paddle.rand((2, 2, 2, 3)) layer_norm = paddle.nn.LayerNorm(x.shape[1:]) layer_norm_out = layer_norm(x) print(layer_norm_out) rNcstt|t|tjr|g}t||_||_||_ ||_ t |jg}|dur-d|_ n |j|j |tdd|_ |durBd|_dS|j|j |dd|_dS)NFrr_Tr`)r$r|r% isinstancenumbersIntegrallist_normalized_shaper&r'r(npprodr@r*r r,)r-normalized_shaper/r1r2r4rer5r7r8r%s(    zLayerNorm.__init__cCst||j|j|j|jdS)N)rr@r,r/)rrr@r,r&r<r7r7r8rA?szLayerNorm.forwardcCrB)Nznormalized_shape={}, epsilon={})rCrr&rDr7r7r8rEHrFzLayerNorm.extra_repr)rNNNr{r7r7r5r8r|s<" r|csPeZdZdZ       dfdd Zdd Zd d Zd d ZddZZ S)_BatchNormBasez BatchNorm base . rrNrc stt|||_||_||_||_tdkrd|_nt|_|g} |dkr9|j d| |jt dd|_ d|j _ n|j |j| |jt dd|_ |jdkoQ|jj dk|j _ |dkrk|j d| |jt ddd|_d|j_ n|j |j| |jdd |_|jdko|jj dk|j_ d} d} |dur|d } |d } |j |jt| t dddd | d |_d|j_ |j |jt| t dddd | d |_d|j_ ||_d|_||_||_d|_||_dS)Nfloat16float32Fr)rr rgr!Tr#)rr rgr!r")rr rgr"_mean _variance)r4 initializerZ trainableZdo_model_average)rgrr )r$rr%r)r'r(_use_global_statsr_dtyper*r r@rcrdr,r rr _data_formatZ _in_place _momentumr&Z_fuse_with_relu_name) r-r.r0r/r1r2r3use_global_statsr4reZmoving_mean_nameZmoving_variance_namer5r7r8r%Ss         z_BatchNormBase.__init__cCr9)NzBatchNorm Base errorr:r<r7r7r8r>r?z_BatchNormBase._check_input_dimcCr9)Nz BatchNorm Base data format errorr:r<r7r7r8_check_data_formatr?z!_BatchNormBase._check_data_formatc CsT||j|||jrtdt||j|j|j |j |j|j |j |j|j d S)Nzrwarningswarnrrrr@r,rr&rr<r7r7r8rAs$  z_BatchNormBase.forwardcCsLd|j|j|j}|jdkr|d|j7}|jdur$|d|j7}|S)Nz(num_features={}, momentum={}, epsilon={}r, data_format={} , name={})rCr)rr&rrr-Zmain_strr7r7r8rEs   z_BatchNormBase.extra_repr)rrNNrNN) rGrHrIrJr%r>rrArErKr7r7r5r8rNsgrc@eZdZdZ       d fdd Zdd Zd d ZZS) BatchNorm1Da Applies Batch Normalization over a 2D or 3D input (a mini-batch of 1D inputswith additional channel dimension) as described in the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift . When use_global_stats = False, the :math:`\mu_{\beta}` and :math:`\sigma_{\beta}^{2}` are the statistics of one mini-batch. Calculated as follows: .. math:: \mu_{\beta} &\gets \frac{1}{m} \sum_{i=1}^{m} x_i \qquad &//\ \ mini-batch\ mean \\ \sigma_{\beta}^{2} &\gets \frac{1}{m} \sum_{i=1}^{m}(x_i - \ \mu_{\beta})^2 \qquad &//\ mini-batch\ variance \\ When use_global_stats = True, the :math:`\mu_{\beta}` and :math:`\sigma_{\beta}^{2}` are not the statistics of one mini-batch. They are global or running statistics (moving_mean and moving_variance). It usually got from the pre-trained model. Calculated as follows: .. math:: moving\_mean = moving\_mean * momentum + \mu_{\beta} * (1. - momentum) \quad &// global \ mean \\ moving\_variance = moving\_variance * momentum + \sigma_{\beta}^{2} * (1. - momentum) \quad &// global \ variance \\ The normalization function formula is as follows: .. math:: \hat{x_i} &\gets \frac{x_i - \mu_\beta} {\sqrt{\sigma_{\beta}^{2} + \epsilon}} \qquad &//\ normalize \\ y_i &\gets \gamma \hat{x_i} + \beta \qquad &//\ scale\ and\ shift - :math:`\epsilon` : add a smaller value to the variance to prevent division by zero - :math:`\gamma` : trainable proportional parameter - :math:`\beta` : trainable deviation parameter Parameters: num_features(int): Indicate the number of channels of the input ``Tensor``. epsilon(float, optional): The small value added to the variance to prevent division by zero. Default: 1e-5. momentum(float, optional): The value used for the moving_mean and moving_var computation. Default: 0.9. weight_attr(ParamAttr|bool, optional): The parameter attribute for Parameter `scale` of batch_norm. If it is set to None or one attribute of ParamAttr, batch_norm will create ParamAttr as weight_attr. If it is set to Fasle, the weight is not learnable. If the Initializer of the weight_attr is not set, the parameter is initialized with ones. Default: None. bias_attr(ParamAttr|bool, optional): The parameter attribute for the bias of batch_norm. If it is set to None or one attribute of ParamAttr, batch_norm will create ParamAttr as bias_attr. If it is set to Fasle, the weight is not learnable. If the Initializer of the bias_attr is not set, the bias is initialized zero. Default: None. data_format(str, optional): Specify the input data format, may be "NC", "NCL" or "NLC". Default "NCL". use_global_stats(bool|None, optional): Whether to use global mean and variance. If set to False, use the statistics of one mini-batch, if set to True, use the global statistics, if set to None, use global statistics in the test phase and use the statistics of one mini-batch in the training phase. Default: None. name(str, optional): Name for the BatchNorm, default is None. For more information, please refer to :ref:`api_guide_Name`.. Shape: - x: 2-D or 3-D tensor with shape: (batch, num_features) or (batch, num_features, length) when data_format is "NC" or "NCL", (batch, length, num_features) when data_format is "NLC". - output: 3-D tensor with same shape as input x. Returns: None. Examples: .. code-block:: python import paddle x = paddle.rand((2, 1, 3)) batch_norm = paddle.nn.BatchNorm1D(1) batch_norm_out = batch_norm(x) print(batch_norm_out) rrNNCLc "tt|||||||||dSN)r$rr% r-r.r0r/r1r2r3rr4r5r7r8r%+ zBatchNorm1D.__init__cCsD|dks |dks |dkrd|_dS|dks|dkrd|_dStd)NrNCrNHWCNLCz4expected NC , NCL, NLC or None for data_format inputrrRr<r7r7r8rAs  zBatchNorm1D._check_data_formatcCrNrOrPr<r7r7r8r>KrSzBatchNorm1D._check_input_dim)rrNNrNNrGrHrIrJr%rr>rKr7r7r5r8rsJ rc@s eZdZdZddZddZdS) BatchNorm2Da Applies Batch Normalization over a 4D input (a mini-batch of 2D inputswith additional channel dimension) as described in the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift . When use_global_stats = False, the :math:`\mu_{\beta}` and :math:`\sigma_{\beta}^{2}` are the statistics of one mini-batch. Calculated as follows: .. math:: \mu_{\beta} &\gets \frac{1}{m} \sum_{i=1}^{m} x_i \qquad &// \ mini-batch\ mean \\ \sigma_{\beta}^{2} &\gets \frac{1}{m} \sum_{i=1}^{m}(x_i - \mu_{\beta})^2 \qquad &//\ mini-batch\ variance \\ When use_global_stats = True, the :math:`\mu_{\beta}` and :math:`\sigma_{\beta}^{2}` are not the statistics of one mini-batch. They are global or running statistics (moving_mean and moving_variance). It usually got from the pre-trained model. Calculated as follows: .. math:: moving\_mean = moving\_mean * momentum + \mu_{\beta} * (1. - momentum) \quad &// global \ mean \\ moving\_variance = moving\_variance * momentum + \sigma_{\beta}^{2} * (1. - momentum) \quad &// global \ variance \\ The normalization function formula is as follows: .. math:: \hat{x_i} &\gets \frac{x_i - \mu_\beta} {\sqrt{\sigma_{\beta}^{2} + \epsilon}} \qquad &//\ normalize \\ y_i &\gets \gamma \hat{x_i} + \beta \qquad &//\ scale\ and\ shift - :math:`\epsilon` : add a smaller value to the variance to prevent division by zero - :math:`\gamma` : trainable proportional parameter - :math:`\beta` : trainable deviation parameter Parameters: num_features(int): Indicate the number of channels of the input ``Tensor``. epsilon(float, optional): The small value added to the variance to prevent division by zero. Default: 1e-5. momentum(float, optional): The value used for the moving_mean and moving_var computation. Default: 0.9. weight_attr(ParamAttr|bool, optional): The parameter attribute for Parameter `scale` of batch_norm. If it is set to None or one attribute of ParamAttr, batch_norm will create ParamAttr as weight_attr. If it is set to Fasle, the weight is not learnable. If the Initializer of the weight_attr is not set, the parameter is initialized with ones. Default: None. bias_attr(ParamAttr|bool, optional): The parameter attribute for the bias of batch_norm. If it is set to None or one attribute of ParamAttr, batch_norm will create ParamAttr as bias_attr. If it is set to Fasle, the weight is not learnable. If the Initializer of the bias_attr is not set, the bias is initialized zero. Default: None. data_format(str, optional): Specify the input data format, the data format can be "NCHW" or "NHWC". Default: NCHW. use_global_stats(bool|None, optional): Whether to use global mean and variance. If set to False, use the statistics of one mini-batch, if set to True, use the global statistics, if set to None, use global statistics in the test phase and use the statistics of one mini-batch in the training phase. Default: None. name(str, optional): Name for the BatchNorm, default is None. For more information, please refer to :ref:`api_guide_Name`.. Shape: - x: 4-D tensor with shape: (batch, num_features, height, weight) when data_format is "NCHW", or (batch, height, weight, num_features) when data_format is "NHWC". - output: 4-D tensor with same shape as input x. Returns: None Examples: .. code-block:: python import paddle x = paddle.rand((2, 1, 2, 3)) batch_norm = paddle.nn.BatchNorm2D(1) batch_norm_out = batch_norm(x) print(batch_norm_out) cCs,|dkr ||_dS|dkr||_dStd)Nrrz+expected NCHW or NHWC for data_format inputrr<r7r7r8rs   zBatchNorm2D._check_data_formatcCrVrWrPr<r7r7r8r>rYzBatchNorm2D._check_input_dimN)rGrHrIrJrr>r7r7r7r8rTsF rcr) BatchNorm3Da Applies Batch Normalization over a 5D input (a mini-batch of 3D inputswith additional channel dimension) as described in the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift . When use_global_stats = False, the :math:`\mu_{\beta}` and :math:`\sigma_{\beta}^{2}` are the statistics of one mini-batch. Calculated as follows: .. math:: \mu_{\beta} &\gets \frac{1}{m} \sum_{i=1}^{m} x_i \qquad &//\ \ mini-batch\ mean \\ \sigma_{\beta}^{2} &\gets \frac{1}{m} \sum_{i=1}^{m}(x_i - \ \mu_{\beta})^2 \qquad &//\ mini-batch\ variance \\ When use_global_stats = True, the :math:`\\mu_{\\beta}` and :math:`\\sigma_{\\beta}^{2}` are not the statistics of one mini-batch. They are global or running statistics (moving_mean and moving_variance). It usually got from the pre-trained model. Calculated as follows: .. math:: moving\_mean = moving\_mean * momentum + \mu_{\beta} * (1. - momentum) \quad &// global \ mean \\ moving\_variance = moving\_variance * momentum + \sigma_{\beta}^{2} * (1. - momentum) \quad &// global \ variance \\ The normalization function formula is as follows: .. math:: \hat{x_i} &\gets \frac{x_i - \mu_\beta} {\sqrt{\sigma_{\beta}^{2} + \epsilon}} \qquad &//\ normalize \\ y_i &\gets \gamma \hat{x_i} + \beta \qquad &//\ scale\ and\ shift - :math:`\epsilon` : add a smaller value to the variance to prevent division by zero - :math:`\gamma` : trainable proportional parameter - :math:`\beta` : trainable deviation parameter Parameters: num_features(int): Indicate the number of channels of the input ``Tensor``. epsilon(float, optional): The small value added to the variance to prevent division by zero. Default: 1e-5. momentum(float, optional): The value used for the moving_mean and moving_var computation. Default: 0.9. weight_attr(ParamAttr|bool, optional): The parameter attribute for Parameter `scale` of batch_norm. If it is set to None or one attribute of ParamAttr, batch_norm will create ParamAttr as weight_attr. If it is set to Fasle, the weight is not learnable. If the Initializer of the weight_attr is not set, the parameter is initialized with ones. Default: None. bias_attr(ParamAttr|bool, optional): The parameter attribute for the bias of batch_norm. If it is set to None or one attribute of ParamAttr, batch_norm will create ParamAttr as bias_attr. If it is set to Fasle, the weight is not learnable. If the Initializer of the bias_attr is not set, the bias is initialized zero. Default: None. data_format(str, optional): Specify the input data format, the data format can be "NCDHW" or "NDHWC. Default: NCDHW. use_global_stats(bool|None, optional): Whether to use global mean and variance. If set to False, use the statistics of one mini-batch, if set to True, use the global statistics, if set to None, use global statistics in the test phase and use the statistics of one mini-batch in the training phase. Default: None. name(str, optional): Name for the BatchNorm, default is None. For more information, please refer to :ref:`api_guide_Name`.. Shape: - x: 5-D tensor with shape: (batch, num_features, dims, height, weight) when data_format is "NCDHW", or (batch, dims, height, weight, num_features) when data_format is "NDHWC". - output: 5-D tensor with same shape as input x. Returns: None Examples: .. code-block:: python import paddle x = paddle.rand((2, 1, 2, 2, 3)) batch_norm = paddle.nn.BatchNorm3D(1) batch_norm_out = batch_norm(x) print(batch_norm_out) rrNNCDHWc rr)r$rr%rr5r7r8r%rzBatchNorm3D.__init__cCs<|dks|dkr d|_dS|dks|dkrd|_dStd)NrrrNDHWCz3expected NCDHW, NDHWC or None for data_format inputrr<r7r7r8rs  zBatchNorm3D._check_data_formatcCrVr\rPr<r7r7r8r>rYzBatchNorm3D._check_input_dim)rrNNrNNrr7r7r5r8rsI rcsJeZdZdZ      dfdd Zdd Zd d Zed d ZZ S) SyncBatchNorma This interface is used to construct a callable object of the ``SyncBatchNorm`` class. It implements the function of the Cross-GPU Synchronized Batch Normalization Layer, and can be used as a normalizer function for other operations, such as conv2d and fully connected operations. The data is normalized by the mean and variance of the channel based on whole mini-batch , which including data in all gpus. Refer to `Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift `_ for more details. When model in training mode, the :math:`\\mu_{\\beta}` and :math:`\\sigma_{\\beta}^{2}` are the statistics of whole mini-batch data in all gpus. Calculated as follows: .. math:: \mu_{\beta} &\gets \frac{1}{m} \sum_{i=1}^{m} x_i \qquad &//\ \ mini-batch\ mean \\ \sigma_{\beta}^{2} &\gets \frac{1}{m} \sum_{i=1}^{m}(x_i - \ \mu_{\beta})^2 \qquad &//\ mini-batch\ variance \\ - :math:`x` : whole mini-batch data in all gpus - :math:`m` : the size of the whole mini-batch data When model in evaluation mode, the :math:`\\mu_{\\beta}` and :math:`\sigma_{\beta}^{2}` are global statistics (moving_mean and moving_variance, which usually got from the pre-trained model). Global statistics calculated as follows: .. math:: moving\_mean = moving\_mean * momentum + \mu_{\beta} * (1. - momentum) \quad &// global \ mean \\ moving\_variance = moving\_variance * momentum + \sigma_{\beta}^{2} * (1. - momentum) \quad &// global \ variance \\ The formula of normalization is as follows: .. math:: \hat{x_i} &\gets \frac{x_i - \mu_\beta} {\sqrt{\ \sigma_{\beta}^{2} + \epsilon}} \qquad &//\ normalize \\ y_i &\gets \gamma \hat{x_i} + \beta \qquad &//\ scale\ and\ shift - :math:`\epsilon` : add a smaller value to the variance to prevent division by zero - :math:`\gamma` : trainable scale parameter vector - :math:`\beta` : trainable shift parameter vector Note: If you want to use container to pack your model and has ``SyncBatchNorm`` in the evaluation phase, please use ``nn.LayerList`` or ``nn.Sequential`` instead of ``list`` to pack the model. Parameters: num_features(int): Indicate the number of channels of the input ``Tensor``. epsilon(float, optional): The small value added to the variance to prevent division by zero. Default: 1e-5. momentum(float, optional): The value used for the moving_mean and moving_var computation. Default: 0.9. weight_attr(ParamAttr|bool, optional): The parameter attribute for Parameter `scale` of this layer. If it is set to None or one attribute of ParamAttr, this layerr will create ParamAttr as param_attr. If the Initializer of the param_attr is not set, the parameter is initialized with ones. If it is set to False, this layer will not have trainable scale parameter. Default: None. bias_attr(ParamAttr|bool, optional): The parameter attribute for the bias of this layer. If it is set to None or one attribute of ParamAttr, this layer will create ParamAttr as bias_attr. If the Initializer of the bias_attr is not set, the bias is initialized zero. If it is set to False, this layer will not have trainable bias parameter. Default: None. Shapes: input: Tensor that the dimension from 2 to 5. output: Tensor with the same shape as input. Examples: .. code-block:: python # required: gpu import paddle import paddle.nn as nn x = paddle.to_tensor([[[[0.3, 0.4], [0.3, 0.07]], [[0.83, 0.37], [0.18, 0.93]]]]).astype('float32') if paddle.is_compiled_with_cuda(): sync_batch_norm = nn.SyncBatchNorm(2) hidden1 = sync_batch_norm(x) print(hidden1) # Tensor(shape=[1, 2, 2, 2], dtype=float32, place=Place(gpu:0), stop_gradient=False, # [[[[ 0.26824948, 1.09363246], # [ 0.26824948, -1.63013160]], # [[ 0.80956620, -0.66528702], # [-1.27446556, 1.13018656]]]]) rrNrc s"tt|||||||d|dSr)r$rr%)r-r.r0r/r1r2r3r4r5r7r8r%ts zSyncBatchNorm.__init__cCs0|jdvr d|_dS|jdvrd|_dStd)N)rrrrr)rrrrzMexpected 'NCDHW', 'NDHWC', 'NCL', 'NLC', 'NC', 'NCHW', 'NHWC' for data_formatrrDr7r7r8rs    z SyncBatchNorm._check_data_formatc Cs~||j}|j}tr.t||j|j|j|j|j|j |j |j ddd \}}}}}}|St rcd|jd|j d|j d|j ddddddd df}t j||j|j|j|j||g|R\}}}}}}|St|d gd d |j|j |j |j ddddd }|g|jg|jg|jg|jgd}|jj|jdd}|jj|jdd} |j|j}|g|g|g|g| gd} |jjd|| |d|S)NFr0r/is_test data_layout use_mkldnnfuse_with_relurtrainable_statisticsr=)rrZfloat64r)r0r/rrrrrr)rirkrjrnroTrf)rmZMeanOutZ VarianceOutZ SavedMeanZ SavedVariancesync_batch_normrp)rrrrrZsync_batch_norm_r@r,rr&rrrrrr rurvrrw) r-xrxryZsync_batch_norm_outrzrtrrZ saved_meanZsaved_variancersr7r7r8rAs   zSyncBatchNorm.forwardcCs|}t|trw|jdkr t|jts |jjdkr |jjd|j_|jdkr9t|jts9|jjdkr9|jjd|j_t|j|j|j |j|j|j |j }|jdkro|jdkrot |j |_ |j|_Wdn1sjwY|j|_|j|_|D] \}}||||q{~|S)a Helper function to convert :class: `paddle.nn.BatchNorm*d` layers in the model to :class: `paddle.nn.SyncBatchNorm` layers. Parameters: layer(paddle.nn.Layer): model containing one or more `BatchNorm*d` layers. Returns: The original model with converted SyncBatchNorm layers. If BatchNorm*d layer in the model, use SyncBatchNorm layer instead. Examples: .. code-block:: python import paddle import paddle.nn as nn model = nn.Sequential(nn.Conv2D(3, 5, 3), nn.BatchNorm2D(5)) sync_model = nn.SyncBatchNorm.convert_sync_batchnorm(model) NZ_syncF)r}rr'boolr4r(rr)rr&rrrr@r,rrZnamed_childrenZ add_sublayerconvert_sync_batchnorm)clslayerZ layer_outputr4Zsublayerr7r7r8rsD          z$SyncBatchNorm.convert_sync_batchnorm)rrNNrN) rGrHrIrJr%rrA classmethodrrKr7r7r5r8rs^ hrcs<eZdZdZ     d fdd Zd d Zd d ZZS)LocalResponseNorma Local Response Normalization performs a type of "lateral inhibition" by normalizing over local input regions. For more information, please refer to `ImageNet Classification with Deep Convolutional Neural Networks `_ See more details in :ref:`api_paddle_nn_functional_local_response_norm` . Parameters: size (int): The number of channels to sum over. alpha (float, optional): The scaling parameter, positive. Default:1e-4 beta (float, optional): The exponent, positive. Default:0.75 k (float, optional): An offset, positive. Default: 1.0 data_format (str, optional): Specify the data format of the input, and the data format of the output will be consistent with that of the input. An optional string from: If input is 3-D Tensor, the string could be `"NCL"` or `"NLC"` . When it is `"NCL"`, the data is stored in the order of: `[batch_size, input_channels, feature_length]`. If input is 4-D Tensor, the string could be `"NCHW"`, `"NHWC"`. When it is `"NCHW"`, the data is stored in the order of: `[batch_size, input_channels, input_height, input_width]`. If input is 5-D Tensor, the string could be `"NCDHW"`, `"NDHWC"` . When it is `"NCDHW"`, the data is stored in the order of: `[batch_size, input_channels, input_depth, input_height, input_width]`. name (str, optional): Name for the operation (optional, default is None). For more information, please refer to :ref:`api_guide_Name`. Shape: - input: 3-D/4-D/5-D tensor. - output: 3-D/4-D/5-D tensor, the same shape as input. Examples: .. code-block:: python import paddle x = paddle.rand(shape=(3, 3, 112, 112), dtype="float32") m = paddle.nn.LocalResponseNorm(size=5) y = m(x) print(y.shape) # [3, 3, 112, 112] -C6??rrNcs6tt|||_||_||_||_||_||_dSr) r$rr%sizealphabetakr3r4)r-rrrrr3r4r5r7r8r%_s  zLocalResponseNorm.__init__c Cs&t||j|j|j|j|j|j}|Sr)FZlocal_response_normrrrrr3r4)r-r=outr7r7r8rAps zLocalResponseNorm.forwardcCsPd|j|j|j|j}|jdkr|d|j7}|jdur&|d|j7}|S)Nz size={}, alpha={}, beta={}, k={}rrr)rCrrrrr3r4rr7r7r8rE|s  zLocalResponseNorm.extra_repr)rrrrNr{r7r7r5r8r8s) r)0sixZ fluid.dygraphrrZ frameworkrrrrr r Zfluid.data_feederr r Zfluidr rrrrnumpyrr~rrrZpaddlerrrrZpaddle.fluid.frameworkrr__all__rrMrUr[r^r|rrrrrrr7r7r7r8sF         =FCC"kqVn"