o Qe]@sddlZddlmZddlmZmZddlmZddlm Z ddl m Z ddlm Z dd lm Z ddlZdd lmZmZdd lmZdd lmZmZmZmZgZd!ddZ      d"ddZ d#ddZ         d$ddZ d%dd ZdS)&N)check_variable_and_dtype check_type) LayerHelper)create_parameter)Constant) ParamAttr) dygraph_utils)_C_ops _legacy_C_ops)in_dynamic_mode)core_non_static_modein_dygraph_mode_in_legacy_dygraph-q=c Csrtr"tjjj|g|jd}t|t|||dd}|t ||St rGtjjj|g|jd}t |d|dt|ddd| }|t ||St |dttfd t |dtd t|d gd d t|jd krv|d krv|dkrvtd||t|d|d}tdit}|j|jd}|jdd|id|i|d|jj|jd}tjd g||jd}tj|t |||dS)a` Normalize ``x`` along dimension ``axis`` using :math:`L_p` norm. This layer computes .. math:: y = \frac{x}{ \max\left( \lvert \lvert x \rvert \rvert_p, epsilon\right) } .. math:: \lvert \lvert x \rvert \rvert_p = \left( \sum_i {\lvert x_i \rvert^p} \right)^{1/p} where, :math:`\sum_i{\lvert x_i \rvert^p}` is calculated along the ``axis`` dimension. Parameters: x (Tensor): The input tensor could be N-D tensor, and the input data type could be float32 or float64. p (float|int, optional): The exponent value in the norm formulation. Default: 2. axis (int, optional): The axis on which to apply normalization. If `axis < 0`, the dimension to normalization is `x.ndim + axis`. -1 is the last dimension. epsilon (float, optional): Small float added to denominator to avoid dividing by zero. Default is 1e-12. name (str, optional): Name for the operation (optional, default is None). For more information, please refer to :ref:`api_guide_Name`. Returns: Tensor, the output has the same shape and data type with ``x``. Examples: .. code-block:: python import paddle import paddle.nn.functional as F paddle.disable_static() x = paddle.arange(6, dtype="float32").reshape([2,3]) y = F.normalize(x) print(y) # Tensor(shape=[2, 3], dtype=float32, place=Place(gpu:0), stop_gradient=True, # [[0. , 0.44721359, 0.89442718], # [0.42426404, 0.56568539, 0.70710671]]) y = F.normalize(x, p=1.5) print(y) # Tensor(shape=[2, 3], dtype=float32, place=Place(gpu:0), stop_gradient=True, # [[0. , 0.40862012, 0.81724024], # [0.35684016, 0.47578689, 0.59473360]]) y = F.normalize(x, axis=0) print(y) # Tensor(shape=[2, 3], dtype=float32, place=Place(gpu:0), stop_gradient=True, # [[0. , 0.24253564, 0.37139067], # [1. , 0.97014254, 0.92847669]]) )dtypeTFaxisporderkeepdimepsilonp normalizexfloat16float32float64rrzCAxis must be 0 or -1 when x is a 1-D tensor, but received axis = {})rrrrp_normXZOuttypeinputsoutputsattrs)shapeZ fill_valuernameN)r!)rfluidZdygraphbaseZ to_variablerr r!floatmaximumrr Zelementwise_maxrintrlenr( ValueErrorformatrlocals"create_variable_for_type_inference append_opblockZ create_varpaddlefulldivide) rrrrr*epsoutr'helperr=ID:\Projects\ConvertPro\env\Lib\site-packages\paddle/nn/functional/norm.pyr%sT3  rF?h㈵>NCHWc Cst|jdks Jd|} |} gd} || vrtd||ddkr&dnd}| d kr2| } d }n| }trVt||||||||| | |d \}}}}}}tj|d d St rd |d |d| d|dd dd d| d|f}t j|||||d | | g|R\}}}}}}tj|d d St |dgdd||| |d d | |d}|g|g|g|g|gd}t d!it }|jdkr|jnd}|j|dd}|j|dd}||j}|g|g|g|g|gd}|s|r|j|jdd}|g|d<|jd|||d ||S)"a Applies Batch Normalization as described in the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift . nn.functional.batch_norm is uesd for nn.BatchNorm1D, nn.BatchNorm2D, nn.BatchNorm3D. Please use above API for BatchNorm. Parameters: x(Tesnor): input value. It's data type should be float32, float64. running_mean(Tensor): running mean. running_var(Tensor): running variance. weight(Tensor): The weight tensor of batch_norm, can not be None. bias(Tensor): The bias tensor of batch_norm can not be None. epsilon(float, optional): The small value added to the variance to prevent division by zero. Default: 1e-5. momentum(float, optional): The value used for the moving_mean and moving_var computation. Default: 0.9. training(bool, optional): True means train mode which compute by batch data and track global mean and var during train period. False means inference mode which compute by global mean and var which calculated by train period. Default False. data_format(str, optional): Specify the input data format, may be "NC", "NCL", "NCHW", "NCDHW", "NLC", "NHWC" or "NDHWC". Default "NCHW". use_global_stats(bool|None, optional): Whether to use global mean and variance. If set to False, use the statistics of one mini-batch, if set to True, use the global statistics, if set to None, use global statistics in the test phase and use the statistics of one mini-batch in the training phase. Default: None. name(str, optional): Name for the BatchNorm, default is None. For more information, please refer to :ref:`api_guide_Name`.. Returns: None Examples: .. code-block:: python import paddle x = paddle.arange(12, dtype="float32").reshape([2, 1, 2, 3]) print(x) # Tensor(shape=[2, 1, 2, 3], dtype=float32, place=Place(gpu:0), stop_gradient=True, # [[[[0. , 1. , 2. ], # [3. , 4. , 5. ]]], # [[[6. , 7. , 8. ], # [9. , 10., 11.]]]]) running_mean = paddle.to_tensor([0], dtype="float32") running_variance = paddle.to_tensor([1], dtype="float32") weight = paddle.to_tensor([2], dtype="float32") bias = paddle.to_tensor([1], dtype="float32") batch_norm_out = paddle.nn.functional.batch_norm(x, running_mean, running_variance, weight, bias) print(batch_norm_out) # Tensor(shape=[2, 1, 2, 3], dtype=float32, place=Place(gpu:0), stop_gradient=True, # [[[[1. , 2.99998999 , 4.99997997 ], # [6.99996948 , 8.99995995 , 10.99994946]]], # [[[12.99993896, 14.99992943, 16.99991989], # [18.99990845, 20.99989891, 22.99988937]]]]) rzinput dim must be larger than 1)ZNCNCLrANCDHWNLCNHWCNDHWCz^data_format must be one of 'NC', 'NCL', 'NCHW', 'NCDHW', 'NLC', 'NHWC', 'NDHWC' but receive {}rCrArENFZactmomentumris_test data_layout use_mkldnnfuse_with_reluuse_global_statstrainable_statisticsinputrZ BatchNorm)rIrrJrKrLrMrNrO)r"ScaleBiasMeanVariance batch_normrrTrZ stop_gradient)YZMeanOutZ VarianceOut SavedMean SavedVarianceZ ReserveSpacer#)rU)r0r(r1r2rr rUr _append_activation_in_dygraphrr rrr3rr4r5append_activation)r running_mean running_varweightbiasZtrainingrIr data_formatrNr*mean_out variance_outZtrue_data_formatrOZbatch_norm_out_r'r%r<Z param_dtype saved_meansaved_variancer&Z reserve_spacer=r=r>rUs?       rUc Cst|j}t|}t|tjr|g}nt|trt|}n t|ts&tdt|}||} ||ks:|| d|krRt|} td| d| dddt|t rit ||||| d\} } } t j | ddStrt |||d |d | \} } } t j | ddSt|d gd d t} |g| d<|r|g| d<|r|g| d<|| d}tdit}|j}|j|dd}|j|dd}||}|jd| |||d|| dd||S)aq see more detail in paddle.nn.LayerNorm Parameters: x(Tensor): Input Tensor. It's data type should be float32, float64. normalized_shape(int|list|tuple): Input shape from an expected input of size :math:`[*, normalized_shape[0], normalized_shape[1], ..., normalized_shape[-1]]`. If it is a single integer, this module will normalize over the last dimension which is expected to be of that specific size. epsilon(float, optional): The small value added to the variance to prevent division by zero. Default: 1e-05. weight(Tensor, optional): The weight tensor of batch_norm. Default: None. bias(Tensor, optional): The bias tensor of batch_norm. Default: None. name(str, optional): Name for the LayerNorm, default is None. For more information, please refer to :ref:`api_guide_Name`.. Returns: None Examples: .. code-block:: python import paddle x = paddle.rand((2, 2, 2, 3)) layer_norm_out = paddle.nn.functional.layer_norm(x, x.shape[1:]) print(layer_norm_out) z@`normalized_shape` should be int, list of ints or tuple of ints.NzGiven normalized_shape is z , expected input with shape [*, rz, but got input shape FrHrbegin_norm_axisrPrZ LayerNormr"rQrR)rrf layer_normTrV)rWrSrTr#)rg)listr(r0 isinstancenumbersIntegraltupler1strrr rgr rZrr rdictrr3rr4r5r[)rZnormalized_shaper^r_rr*Z input_shapeZ input_ndimZnormalized_ndimrfZstr_normalized_shapeZpre_actrcr%r'r<rrarbZlayer_norm_outr=r=r>rgNs                rgTc  Cstr t||||} | Str"t|||d|d|d| \} } } | St|dddgd|||d} |r?|r?|g|g|gd } nd |gi} tdit}|j|j d d }|j|j d d }||j }|g|g|gd}|j d | || d|S)a See more detail in nn.layer.InstanceNorm2D. Parameters: x(Tensor): Input Tensor. It's data type should be float32, float64. running_mean(Tensor, optional): running mean. Default None. running_var(Tensor, optional): running variance. Default None. weight(Tensor, optional): The weight tensor of instance_norm. Default: None. bias(Tensor, optional): The bias tensor of instance_norm. Default: None. eps(float, optional): A value added to the denominator for numerical stability. Default is 1e-5. momentum(float, optional): The value used for the moving_mean and moving_var computation. Default: 0.9. use_input_stats(bool, optional): Default True. data_format(str, optional): Specify the input data format, may be "NC", "NCL", "NCHW" or "NCDHW". Defalut "NCHW". name(str, optional): Name for the InstanceNorm, default is None. For more information, please refer to :ref:`api_guide_Name`.. Returns: None. Examples: .. code-block:: python import paddle x = paddle.rand((2, 2, 2, 3)) instance_norm_out = paddle.nn.functional.instance_norm(x) print(instance_norm_out) rrIr`rPrrZ InstanceNorm)rrIr`)r"rQrRr" instance_normTrV)rWrXrYr#N)ro) rr rorr rrr3r4rr5)rr\r]r^r_Zuse_input_statsrIr:r`r*r;rcr'r%r<rdreZinstance_norm_outr&r=r=r>rosJ*     ro-C6???c Csvts t|ddgd|dvrtd||j}t|}|dkr(td|t|D]\} } | dks@| dkr@td | | q,|d d krId nd } ddlm} | dd|dd} t j t ||dd}| sdd|d|ddg}|df}|dd|d|dt | |d|dg}dddd|d|ddg}|ddf}n:|d|ddddg}d|f}|dd|dt | |d|d |d g}|d|ddddddg}dd|f}|dkrt j jj||d}t j jj||dd}t j|dd}n&t j||d}t j jj||dd}t j jj||dd}t t j|dd|}t j|||d}t ||}t j|||d}|S)a| Local Response Normalization performs a type of "lateral inhibition" by normalizing over local input regions. For more information, please refer to `ImageNet Classification with Deep Convolutional Neural Networks `_ The formula is as follows: .. math:: Output(i, x, y) = Input(i, x, y) / \left(k + \alpha \sum\limits^{\min(C-1, i + size/2)}_{j = \max(0, i - size/2)}(Input(j, x, y))^2\right)^{\beta} In the above equation: - :math:`size` : The number of channels to sum over. - :math:`k` : The offset (avoid being divided by 0). - :math:`\\alpha` : The scaling parameter. - :math:`\\beta` : The exponent parameter. Args: x (Tensor): The input 3-D/4-D/5-D tensor. The data type is float32. size (int): The number of channels to sum over. alpha (float, optional): The scaling parameter, positive. Default:1e-4 beta (float, optional): The exponent, positive. Default:0.75 k (float, optional): An offset, positive. Default: 1.0 data_format (str, optional): Specify the data format of the input, and the data format of the output will be consistent with that of the input. An optional string from: If x is 3-D Tensor, the string could be `"NCL"` or `"NLC"` . When it is `"NCL"`, the data is stored in the order of: `[batch_size, input_channels, feature_length]`. If x is 4-D Tensor, the string could be `"NCHW"`, `"NHWC"`. When it is `"NCHW"`, the data is stored in the order of: `[batch_size, input_channels, input_height, input_width]`. If x is 5-D Tensor, the string could be `"NCDHW"`, `"NDHWC"` . When it is `"NCDHW"`, the data is stored in the order of: `[batch_size, input_channels, input_depth, input_height, input_width]`. name (str, optional): Name for the operation (optional, default is None). For more information, please refer to :ref:`api_guide_Name`. Returns: A tensor storing the transformation result with the same shape and data type as input. Examples: .. code-block:: python import paddle x = paddle.rand(shape=(3, 3, 112, 112), dtype="float32") y = paddle.nn.functional.local_response_norm(x, size=5) print(y.shape) # [3, 3, 112, 112] rrlocal_response_norm)rBrDrArErCrFzPdata_format should be in one of [NCL, NCHW, NCDHW, NLC, NHWC, NDHWC], but got {}rzAExpected 3D or higher dimensionality input, but got {} dimensionsrzRExpected every dim's size to be larger than 0, but the size of the {}-th dim is {}r rGTF)reducecSs||S)Nr=)ryr=r=r>lsz%local_response_norm..rN)rr)pad)Z kernel_sizeZstride)r(rC)rwr`)scaler_r))r rr1r2r(r0 enumerate functoolsrtr7Z unsqueezemultiplyr/nnZ functionalrwZ avg_pool2dZsqueezeZreshapeZ avg_pool3drxpowr9)rsizealphabetakr`r*sizesdimiszZ channel_lastrtZ sum_sizesdivZ pad4d_shapeZ pool2d_shapeZ reshape_shapeZ pad5d_shapeZ pool3d_shaperesr=r=r>rss4    rs)rrrN)Fr?r@rANN)NNr@N) NNNNTr?r@rAN)rprqrrrAN)r7Z paddle.fluidr+Zfluid.data_feederrrZfluid.layer_helperrZ frameworkrZ initializerrr r rjr r r Zpaddle.fluid.frameworkrrrr__all__rrUrgrorsr=r=r=r>sH        i H w Z