o Ne4U@sddlmZddlZddlmZddlmZddlmZddlm Z ddlm Z d d lmZdd l m Z mZmZmZmZmZmZmZmZmZdd lmZmZmZmZdd lmZddlmZm Z m!Z!ddlm"Z"d dl#m$Z$ddlmZmZddl%Z&ddl'Z'ddl(Z(ddl)Z)ddl*mm+Z+ddlm,Z,m-Z-gdZ.Gdddej/Z0Gdddej/Z1Gdddej/Z2Gdddej/Z3Gdddej/Z4Gdddej/Z5Gd d!d!ej/Z6Gd"d#d#ej/Z7Gd$d%d%ej/Z8Gd&d'd'ej/Z9Gd(d)d)ej/Z:Gd*d+d+ej/Z;Gd,d-d-ej/ZGd2d3d3ej/Z?Gd4d5d5ej/Z@Gd6d7d7ej/ZAGd8d9d9ej/ZBGd:d;d;ej/ZCGd)print_functionN)reduce)core)utils)nn) dygraph_utils)layers) Variable_non_static_mode OpProtoHolder Parameter_dygraph_tracer_varbase_creatordefault_main_program _global_flagsin_dygraph_mode_in_legacy_dygraph) convert_dtypecheck_variable_and_dtype check_type check_dtype) ParamAttr)NormalConstantNumpyArrayInitializer) unique_name)LayerObjectHelper)rr)_C_ops _legacy_C_ops)Conv2DConv3DPool2DLinear BatchNormDropout EmbeddingGRUUnit InstanceNorm LayerNormNCEPReluBilinearTensorProductConv2DTransposeConv3DTranspose GroupNorm SpectralNormTreeConvFlattenc<eZdZdZ         d fdd Zd d ZZS) r!a8 This interface is used to construct a callable object of the ``Conv2D`` class. For more details, refer to code examples. The convolution2D layer calculates the output based on the input, filter and strides, paddings, dilations, groups parameters. Input and Output are in NCHW format, where N is batch size, C is the number of the feature map, H is the height of the feature map, and W is the width of the feature map. Filter's shape is [MCHW] , where M is the number of output feature map, C is the number of input feature map, H is the height of the filter, and W is the width of the filter. If the groups is greater than 1, C will equal the number of input feature map divided by the groups. Please refer to UFLDL's `convolution `_ for more details. If bias attribution and activation type are provided, bias is added to the output of the convolution, and the corresponding activation function is applied to the final result. For each input :math:`X`, the equation is: .. math:: Out = \\sigma (W \\ast X + b) Where: * :math:`X`: Input value, a ``Tensor`` with NCHW format. * :math:`W`: Filter value, a ``Tensor`` with shape [MCHW] . * :math:`\\ast`: Convolution operation. * :math:`b`: Bias value, a 2-D ``Tensor`` with shape [M, 1]. * :math:`\\sigma`: Activation function. * :math:`Out`: Output value, the shape of :math:`Out` and :math:`X` may be different. Example: - Input: Input shape: :math:`(N, C_{in}, H_{in}, W_{in})` Filter shape: :math:`(C_{out}, C_{in}, H_f, W_f)` - Output: Output shape: :math:`(N, C_{out}, H_{out}, W_{out})` Where .. math:: H_{out}&= \\frac{(H_{in} + 2 * paddings[0] - (dilations[0] * (H_f - 1) + 1))}{strides[0]} + 1 \\\\ W_{out}&= \\frac{(W_{in} + 2 * paddings[1] - (dilations[1] * (W_f - 1) + 1))}{strides[1]} + 1 Parameters: num_channels(int): The number of channels in the input image. num_filters(int): The number of filter. It is as same as the output feature map. filter_size (int or tuple): The filter size. If filter_size is a tuple, it must contain two integers, (filter_size_H, filter_size_W). Otherwise, the filter will be a square. stride (int or tuple, optional): The stride size. If stride is a tuple, it must contain two integers, (stride_H, stride_W). Otherwise, the stride_H = stride_W = stride. Default: 1. padding (int or tuple, optional): The padding size. If padding is a tuple, it must contain two integers, (padding_H, padding_W). Otherwise, the padding_H = padding_W = padding. Default: 0. dilation (int or tuple, optional): The dilation size. If dilation is a tuple, it must contain two integers, (dilation_H, dilation_W). Otherwise, the dilation_H = dilation_W = dilation. Default: 1. groups (int, optional): The groups number of the Conv2D Layer. According to grouped convolution in Alex Krizhevsky's Deep CNN paper: when group=2, the first half of the filters is only connected to the first half of the input channels, while the second half of the filters is only connected to the second half of the input channels. Default: 1. param_attr (ParamAttr, optional): The parameter attribute for learnable weights(Parameter) of conv2d. If it is set to None or one attribute of ParamAttr, conv2d will create ParamAttr as param_attr. If the Initializer of the param_attr is not set, the parameter is initialized with :math:`Normal(0.0, std)`, and the :math:`std` is :math:`(\\frac{2.0 }{filter\_elem\_num})^{0.5}`. Default: None. bias_attr (ParamAttr or bool, optional): The attribute for the bias of conv2d. If it is set to False, no bias will be added to the output units. If it is set to None or one attribute of ParamAttr, conv2d will create ParamAttr as bias_attr. If the Initializer of the bias_attr is not set, the bias is initialized zero. Default: None. use_cudnn (bool, optional): Use cudnn kernel or not, it is valid only when the cudnn library is installed. Default: True. act (str, optional): Activation type, if it is set to None, activation is not appended. Default: None. dtype (str, optional): Data type, it can be "float32" or "float64". Default: "float32". Attribute: **weight** (Parameter): the learnable weights of filter of this layer. **bias** (Parameter or None): the learnable bias of this layer. Returns: None Raises: ValueError: if ``use_cudnn`` is not a bool value. Examples: .. code-block:: python from paddle.fluid.dygraph.base import to_variable import paddle.fluid as fluid from paddle.fluid.dygraph import Conv2D import numpy as np data = np.random.uniform(-1, 1, [10, 3, 32, 32]).astype('float32') with fluid.dygraph.guard(): conv2d = Conv2D(3, 2, 3) data = to_variable(data) conv = conv2d(data) r rNTfloat32c s|dusJdtttrtjddrd} |_|_ t |dd_ t |dd_ t |dd_| _t| tsGtd| _td __|_|_| _| _jj krv|jd krvjsvjsvd _nd _trjj krjjkrd _nd _|_j durj} njj d krtd jj } t jddj| g}fdd}jj|j|d_jjjgjdd_dS)NF$param_attr should not be False here.ZFLAGS_conv2d_disable_cudnnrstridepaddingdilation!use_cudnn should be True or FalseFLAGS_use_mkldnnrdepthwise_conv2dconv2d)num_channels must be divisible by groups. filter_sizecs.ddj}d|d}td|dS)Nrr @? _num_channelsrZfilter_elem_numZstdr?selfGD:\Projects\ConvertPro\env\Lib\site-packages\paddle/fluid/dygraph/nn.py_get_default_param_initializers  z7Conv2D.__init__.._get_default_param_initializerattrshapedtypedefault_initializerTrLrMrNis_bias) superr!__init__rZis_compiled_with_cudapaddleZfluidZ get_flagsrD_groupsrconvert_to_list_stride_padding _dilation_act isinstancebool ValueError _use_cudnnr _use_mkldnn _filter_size _num_filters _param_attr _bias_attr_dtype_l_typeZis_compiled_with_npucreate_parameterweightbiasrG num_channels num_filtersr?r7r8r9groups param_attr bias_attr use_cudnnactrNZnum_filter_channels filter_shaperJ __class__rFrIrSsp         zConv2D.__init__c Cstr<|jdkrr?cs6dddj}d|d}td|dS)Nrr rr@rArBrCrErFrHrIrJs  z7Conv3D.__init__.._get_default_param_initializerrKTrP)rRr"rSrDrUrrVrWrXrYrZr^r`rarbrcrdr]rfrgrhrirrrFrIrSsB    zConv3D.__init__c Cs|jj|jd}|jjd||jdd|i|j|j|j|jr |jnd|j ddd|j durL|jj|jd}|jjd |g|j gd d |gid didn|}|jj ||j d S)NrZconv3dr}rr Frrrrrrwr) rrrdrrgrWrXrYrUr^rhrrZrGrrrrHrHrIrs: zConv3D.forwardrrrHrHrrrIr"9sr7r"cs<eZdZdZ         d fdd Zd d ZZS) r/a **Convlution3D transpose layer** The convolution3D transpose layer calculates the output based on the input, filter, and dilations, strides, paddings. Input(Input) and output(Output) are in NCDHW format. Where N is batch size, C is the number of channels, D is the depth of the feature, H is the height of the feature, and W is the width of the feature. Parameters(dilations, strides, paddings) are two elements. These two elements represent height and width, respectively. The details of convolution transpose layer, please refer to the following explanation and references `therein `_. If bias attribution and activation type are provided, bias is added to the output of the convolution, and the corresponding activation function is applied to the final result. For each input :math:`X`, the equation is: .. math:: Out = \sigma (W \\ast X + b) In the above equation: * :math:`X`: Input value, a tensor with NCDHW format. * :math:`W`: Filter value, a tensor with MCDHW format. * :math:`\\ast`: Convolution operation. * :math:`b`: Bias value, a 2-D tensor with shape [M, 1]. * :math:`\\sigma`: Activation function. * :math:`Out`: Output value, the shape of :math:`Out` and :math:`X` may be different. Example: - Input: Input shape: :math:`(N, C_{in}, D_{in}, H_{in}, W_{in})` Filter shape: :math:`(C_{in}, C_{out}, D_f, H_f, W_f)` - Output: Output shape: :math:`(N, C_{out}, D_{out}, H_{out}, W_{out})` Where .. math:: D^\prime_{out} &= (D_{in} - 1) * strides[0] - 2 * paddings[0] + dilations[0] * (D_f - 1) + 1 \\\\ H^\prime_{out} &= (H_{in} - 1) * strides[1] - 2 * paddings[1] + dilations[1] * (H_f - 1) + 1 \\\\ W^\prime_{out} &= (W_{in} - 1) * strides[2] - 2 * paddings[2] + dilations[2] * (W_f - 1) + 1 \\\\ D_{out} &\in [ D^\prime_{out}, D^\prime_{out} + strides[0] ] \\\\ H_{out} &\in [ H^\prime_{out}, H^\prime_{out} + strides[1] ] \\\\ **Note**: The conv3d_transpose can be seen as the backward of the conv3d. For conv3d, when stride > 1, conv3d maps multiple input shape to the same output shape, so for conv3d_transpose, when stride > 1, input shape maps multiple output shape. If output_size is None, :math:`H_{out} = H^\prime_{out}, :math:`H_{out} = \ H^\prime_{out}, W_{out} = W^\prime_{out}`; else, the :math:`D_{out}` of the output size must between :math:`D^\prime_{out}` and :math:`D^\prime_{out} + strides[0]`, the :math:`H_{out}` of the output size must between :math:`H^\prime_{out}` and :math:`H^\prime_{out} + strides[1]`, and the :math:`W_{out}` of the output size must between :math:`W^\prime_{out}` and :math:`W^\prime_{out} + strides[2]`, conv3d_transpose can compute the kernel size automatically. Parameters: num_channels(int): The number of channels in the input image. num_filters(int): The number of the filter. It is as same as the output image channel. filter_size(int|tuple): The filter size. If filter_size is a tuple, it must contain three integers, (filter_size_D, filter_size_H, filter_size_W). Otherwise, the filter will be a square. padding(int|tuple, optional): The padding size. The padding argument effectively adds `dilation * (kernel - 1)` amount of zero-padding on both sides of input. If `padding` is a string, either 'VALID' or 'SAME' supported, which is the padding algorithm. If `padding` is a tuple or list, it could be in three forms: `[pad_depth, pad_height, pad_width]` or `[pad_depth_front, pad_depth_back, pad_height_top, pad_height_bottom, pad_width_left, pad_width_right]`, and when `data_format` is `'NCDHW'`, `padding` can be in the form `[[0,0], [0,0], [pad_depth_front, pad_depth_back], [pad_height_top, pad_height_bottom], [pad_width_left, pad_width_right]]`. when `data_format` is `'NDHWC'`, `padding` can be in the form `[[0,0], [pad_depth_front, pad_depth_back], [pad_height_top, pad_height_bottom], [pad_width_left, pad_width_right], [0,0]]`. The default value is 0. stride(int|tuple, optional): The stride size. It means the stride in transposed convolution. If stride is a tuple, it must contain three integers, (stride_depth, stride_height, stride_width). Otherwise, stride_depth = stride_height = stride_width = stride. The default value is 1. dilation(int|tuple, optional): The dilation size. If dilation is a tuple, it must contain three integers, (dilation_D, dilation_H, dilation_W). Otherwise, the dilation_D = dilation_H = dilation_W = dilation. The default value is 1. groups(int, optional): The groups number of the Conv3D transpose layer. Inspired by grouped convolution in Alex Krizhevsky's Deep CNN paper, in which when group=2, the first half of the filters is only connected to the first half of the input channels, while the second half of the filters is only connected to the second half of the input channels. The default value is 1. param_attr (ParamAttr, optional): The parameter attribute for learnable parameters/weights of conv3d_transpose. If it is set to None or one attribute of ParamAttr, conv3d_transpose will create ParamAttr as param_attr. If the Initializer of the param_attr is not set, the parameter is initialized with Xavier. The default value is None. bias_attr (ParamAttr|bool, optional): The parameter attribute for the bias of conv3d_transpose. If it is set to False, no bias will be added to the output units. If it is set to None or one attribute of ParamAttr, conv3d_transpose will create ParamAttr as bias_attr. If the Initializer of the bias_attr is not set, the bias is initialized zero. The default value is None. use_cudnn(bool, optional): Use cudnn kernel or not, it is valid only when the cudnn library is installed. The default value is True. act (str, optional): Activation type, if it is set to None, activation is not appended. The default value is None. name(str, optional): The default value is None. Normally there is no need for user to set this property. For more information, please refer to :ref:`api_guide_Name`. Attribute: **weight** (Parameter): the learnable weights of filters of this layer. **bias** (Parameter): the learnable bias of this layer. Returns: None. Raises: ValueError: If the shapes of input, filter_size, stride, padding and groups mismatch. Examples: .. code-block:: python import paddle.fluid as fluid import numpy with fluid.dygraph.guard(): data = numpy.random.random((5, 3, 12, 32, 32)).astype('float32') conv3dTranspose = fluid.dygraph.nn.Conv3DTranspose( num_channels=3, num_filters=12, filter_size=12, use_cudnn=False) ret = conv3dTranspose(fluid.dygraph.base.to_variable(data)) rr NTr5c stt|t| tstd|dusJdt|dd|_t|dd|_ t|dd|_ ||_ ||_ ||_ |dur?dn||_||_| |_| |_| |_| |_t|j dd |_ |j |j|jg|j } |j|j| |j d |_|j|j|jg|jd d |_dS) Nr:Fz3param_attr should not be False in conv3d_transpose.rr8r7r9r zconv3d_transpose.filter_sizerNrMrLTrP)rRr/rSr[r\r]rrVrXrWrYrbrDr`rUrar^rcrZrdrfrgrh)rGrjrkr?r8r7r9rlrmrnrorprNrqrrrHrIrSs>  zConv3DTranspose.__init__c Cs|jj|jd}|jjd|g|jgdd|i|j|j|j|jr"|jnd|j dd|j rK|jj|jd}|jjd|g|j gd d |gid didn|}|jj ||j d S) NrZconv3d_transposer}rr )rzr{r|rlrorrrrrwr)rrrdrrgrWrXrYrUr^rcrhrrZrrHrHrIrs8zConv3DTranspose.forward) rr r NNNTNr5rrHrHrrrIr/s+r/cs<eZdZdZ         d fd d Zd d ZZS)r#a6 This interface is used to construct a callable object of the ``Pool2D`` class. For more details, refer to code examples. The pooling2d operation calculates the output based on the input, pool_type and pool_size, pool_stride, pool_padding parameters.Input and output are in NCHW format, where N is batch size, C is the number of feature map, H is the height of the feature map, and W is the width of the feature map. Parameters(ksize, strides, paddings) are two elements. These two elements represent height and width, respectively. The input(X) size and output(Out) size may be different. Example: - Input: Input shape: :math:`(N, C, H_{in}, W_{in})` - Output: Output shape: :math:`(N, C, H_{out}, W_{out})` If ``ceil_mode`` = False: .. math:: H_{out} = \\frac{(H_{in} - ksize[0] + 2 * paddings[0])}{strides[0]} + 1 \\\\ W_{out} = \\frac{(W_{in} - ksize[1] + 2 * paddings[1])}{strides[1]} + 1 If ``ceil_mode`` = True: .. math:: H_{out} = \\frac{(H_{in} - ksize[0] + 2 * paddings[0] + strides[0] - 1)}{strides[0]} + 1 \\\\ W_{out} = \\frac{(W_{in} - ksize[1] + 2 * paddings[1] + strides[1] - 1)}{strides[1]} + 1 If ``exclusive`` = False: .. math:: hstart &= i * strides[0] - paddings[0] \\\\ hend &= hstart + ksize[0] \\\\ wstart &= j * strides[1] - paddings[1] \\\\ wend &= wstart + ksize[1] \\\\ Output(i ,j) &= \\frac{sum(Input[hstart:hend, wstart:wend])}{ksize[0] * ksize[1]} If ``exclusive`` = True: .. math:: hstart &= max(0, i * strides[0] - paddings[0])\\\\ hend &= min(H, hstart + ksize[0]) \\\\ wstart &= max(0, j * strides[1] - paddings[1]) \\\\ wend & = min(W, wstart + ksize[1]) \\\\ Output(i ,j) & = \\frac{sum(Input[hstart:hend, wstart:wend])}{(hend - hstart) * (wend - wstart)} Parameters: pool_size (int or list or tuple, optional): The pool kernel size. If pool kernel size is a tuple or list, it must contain two integers, (pool_size_Height, pool_size_Width). Otherwise, the pool kernel size will be a square of an int. Default: -1. pool_type(str, optional) : The pooling type, can be "max" for max-pooling and "avg" for average-pooling. Default: max. pool_stride (int or list or tuple, optional): The pool stride size. If pool stride size is a tuple or list, it must contain two integers, (pool_stride_Height, pool_stride_Width). Otherwise, the pool stride size will be a square of an int. Default: 1. pool_padding (int or list or tuple, optional): The padding size for pooling operation. If ``pool_padding`` is a tuple, it must contain two integers, (pool_padding_on_Height, pool_padding_on_Width). Otherwise, the padding size for pooling operation will be a square of an int. Default: 0. global_pooling (bool, optional): Whether to use the global pooling. If global_pooling = true, kernel size and paddings will be ignored. Default: False. use_cudnn (bool, optional): Only used in cudnn kernel, need install cudnn. Default: True. ceil_mode (bool, optional): Whether to use the ceil function to calculate output height and width. False is the default. If it is set to False, the floor function will be used. Default: False. exclusive (bool, optional): Whether to exclude padding points in average pooling mode. Default: True. data_format (string): The data format of the input and output data. An optional string from: `"NCHW"`, `"NHWC"`. The default is `"NCHW"`. When it is `"NCHW"`, the data is stored in the order of: ``[batch_size, input_channels, input_height, input_width]``. When it is `"NHWC"`, the data is stored in the order of: ``[batch_size, input_height, input_width, input_channels]`` Returns: None Raises: ValueError: If ``pool_type`` is not "max" nor "avg". ValueError: If ``global_pooling`` is False and ``pool_size`` is -1. ValueError: If ``use_cudnn`` is not a bool value. ValueError: If ``data_format`` is not "NCHW" nor "NHWC". Examples: .. code-block:: python import paddle.fluid as fluid from paddle.fluid.dygraph.base import to_variable import numpy as np with fluid.dygraph.guard(): data = numpy.random.random((3, 32, 32, 5)).astype('float32') pool2d = fluid.dygraph.Pool2D(pool_size=2, pool_type='max', pool_stride=1, global_pooling=False) pool2d_res = pool2d(to_variable(data)) rvmaxr rFTruc s| } |}|dvrtdt||dur#|dkr#tdt|t|ts,tdtd|_| dvr>td t| tt | ||_ t |d d |_t |d d |_t |d d |_||_||_||_||_| |_d|_dS)N)rZavgz7Unknown pool_type: '%s'. It can only be 'max' or 'avg'.FrvzeWhen the global_pooling is False, pool_size must be passed and be a valid value. Received pool_size: r:r;)ruZNHWCzMAttr(data_format) should be 'NCHW' or 'NHWC'. Received Attr(data_format): %s.r pool_size pool_padding pool_stridepool2d)upperlowerr]strr[r\rr_rRr#rS _pool_typerrV _pool_size _pool_padding _pool_stride_global_poolingr^ _ceil_mode _exclusive _data_formatre) rGrZ pool_typerrglobal_poolingro ceil_mode exclusive data_formatrrrHrIrSIsF    zPool2D.__init__cCstrK|js"tr"t||j|j|j|j|j |j |j |j dd|j Sd|j d|jd|j d|jd|jd|j d |jd |jd |j d |j f}tj|g|RSt|d gdd|j |j|j |j|j|j |j|j|j |j d }d|gi}|j|j}|jj|jd|id|i|d|S)NFrt pooling_typeksizerrzr{rorryrrr)int8uint8rr5rr#) rrrrzr{rorryrrrrr)r r_rrrrrrrrrrrr^r rrrrdrre)rGrrrZpool_outrHrHrIrwsN        zPool2D.forward) rvrr rFTFTrurrHrHrrrIr#sj.r#c2eZdZdZ    dfdd ZddZZS) r$a Fully-connected linear transformation layer: .. math:: Out = Act({XW + b}) where :math:`X` is the input Tensor, :math:`W` and :math:`b` are weight and bias respectively. Linear layer takes only one ``Tensor`` input. The Linear layer multiplies input tensor with weight matrix and produces an output Tensor of shape [N, *, `output_dim`], where N is batch size and `*` means any number of additional dimensions. If ``bias_attr`` is not None, a bias variable will be created and added to the output. Finally, if ``act`` is not None, it will be applied to the output as well. Parameters: input_dim(int): The number of input units in this layer. output_dim(int): The number of output units in this layer. param_attr(ParamAttr or list of ParamAttr, optional): The parameter attribute for learnable weights(Parameter) of this layer. Default: None. bias_attr(ParamAttr or list of ParamAttr, optional): The attribute for the bias of this layer. If it is set to False, no bias will be added to the output units. If it is set to None, the bias is initialized zero. Default: None. act(str, optional): Activation to be applied to the output of this layer. Default: None. dtype(str, optional): Dtype used for weight, it can be "float32" or "float64". Default: "float32". Attributes: **weight** (Parameter): the learnable weights of this layer. **bias** (Parameter or None): the learnable bias of this layer. Returns: None Examples: .. code-block:: python from paddle.fluid.dygraph.base import to_variable import paddle.fluid as fluid from paddle.fluid.dygraph import Linear import numpy as np data = np.random.uniform(-1, 1, [30, 10, 32]).astype('float32') with fluid.dygraph.guard(): linear = Linear(32, 64) data = to_variable(data) res = linear(data) # [30, 10, 64] Nr5csXtt|||_||_|j||g||dd|_|j|g||dd|_td|_ dS)NF)rMrLrNrQTr;) rRr$rSrZrdrfrgrhrr_)rGZ input_dim output_dimrmrnrprNrrrHrIrSs zLinear.__init__c Cs*tr4t|jd}t||j|ddddddd|j tj||j t |j d|jd}tj ||j |jd St|d gd d ddd|jd }|g|jgd}|j|j}|jjd|d|i|d|j dur|jj|jd}|jjd|g|j gdd|git |j d|jddn|}|jj||j dS)Nr transpose_XF transpose_Yalphar ryrrxrrr$)rrrryrmatmulrrrr)r rrNr rrgr_rrrhlenrMrrZrrrrdrr)rGrrrrrtmppre_activationrHrHrIrsZ       zLinear.forwardNNNr5rrHrHrrrIr$s6r$cs2eZdZdZ    d fdd ZddZZS) r)a This interface is used to construct a callable object of the ``InstanceNorm`` class. For more details, refer to code examples. Can be used as a normalizer function for convolution or fully_connected operations. The required data format for this layer is one of the following: DataLayout: NCHW `[batch, in_channels, in_height, in_width]` Refer to `Instance Normalization: The Missing Ingredient for Fast Stylization `_ for more details. :math:`input` is the input features over a mini-batch. .. math:: \\mu_{\\beta} &\\gets \\frac{1}{HW} \\sum_{i=1}^{HW} x_i \\qquad &//\\ \\ mean\ of\ one\ feature\ map\ in\ mini-batch \\\\ \\sigma_{\\beta}^{2} &\\gets \\frac{1}{HW} \\sum_{i=1}^{HW}(x_i - \\ \\mu_{\\beta})^2 \\qquad &//\ variance\ of\ one\ feature\ map\ in\ mini-batch \\\\ \\hat{x_i} &\\gets \\frac{x_i - \\mu_\\beta} {\\sqrt{\\ \\sigma_{\\beta}^{2} + \\epsilon}} \\qquad &//\ normalize \\\\ y_i &\\gets \\gamma \\hat{x_i} + \\beta \\qquad &//\ scale\ and\ shift Note: `H` means height of feature map, `W` means width of feature map. Parameters: num_channels(int): Indicate the number of channels of the input ``Tensor``. epsilon(float, optional): A value added to the denominator for numerical stability. Default is 1e-5. param_attr(ParamAttr|bool, optional): The parameter attribute for Parameter `scale` of instance_norm. If it is set to None or one attribute of ParamAttr, instance_norm will create ParamAttr as param_attr, the name of scale can be set in ParamAttr. If the Initializer of the param_attr is not set, the parameter is initialized one. If it is set to False, will not create param_attr. Default: None. bias_attr(ParamAttr|bool, optional): The parameter attribute for the bias of instance_norm. If it is set to None or one attribute of ParamAttr, instance_norm will create ParamAttr as bias_attr, the name of bias can be set in ParamAttr. If the Initializer of the bias_attr is not set, the bias is initialized zero. If it is set to False, will not create bias_attr. Default: None. dtype(str, optional): Indicate the data type of the input ``Tensor``, which can be float32 or float64. Default: float32. Returns: None. Examples: .. code-block:: python import paddle.fluid as fluid from paddle.fluid.dygraph.base import to_variable import numpy as np import paddle # x's shape is [1, 3, 1, 2] x = np.array([[[[1.0, 8.0]], [[10.0, 5.0]], [[4.0, 6.0]]]]).astype('float32') with fluid.dygraph.guard(): x = to_variable(x) instanceNorm = paddle.nn.InstanceNorm(3) ret = instanceNorm(x) # ret's shape is [1, 3, 1, 2]; value is [-1 1 0.999999 -0.999999 -0.999995 0.999995] print(ret) h㈵>Nr5cstt||dks|dkr||ksJd||_||_||_||_|dkrM|dkrM|j|j|g|jtddd|_ |j|j|g|jtddd|_ dSd|_ d|_ dS)NFzNparam_attr and bias_attr must be set to Fasle at the same time in InstanceNorm?)rLrMrNrOrQrBT) rRr)rS_epsilonrbrcrdrfrscalerh)rGrjepsilonrmrnrNrrrHrIrSas.  zInstanceNorm.__init__c Cstrt||j|j|j}|Str$t||j|jd|j\}}}|St|dddgdd|ji}|jrD|jrD|g|jg|jgd}nd|gi}|j j |j dd }|j j |j dd }|j |j }|g|g|gd } |j j d || |d |S) Nrrr5rr))rScaleBiasrTrN stop_gradient)r SavedMean SavedVariance instance_normr) rrrrrhrrr rrrrdr) rGrr_rr saved_meansaved_varianceZinstance_norm_outrrHrHrIrsJ      zInstanceNorm.forward)rNNr5rrHrHrrrIr)sEr)csFeZdZdZ              d fd d Zd d ZZS)r%aX This interface is used to construct a callable object of the ``BatchNorm`` class. For more details, refer to code examples. It implements the function of the Batch Normalization Layer and can be used as a normalizer function for conv2d and fully connected operations. The data is normalized by the mean and variance of the channel based on the current batch data. Refer to `Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift `_ for more details. When use_global_stats = False, the :math:`\mu_{\beta}` and :math:`\sigma_{\beta}^{2}` are the statistics of one mini-batch. Calculated as follows: .. math:: \mu_{\beta} &\gets \frac{1}{m} \sum_{i=1}^{m} x_i \qquad & //\ mini-batch\ mean \\ \sigma_{\beta}^{2} &\gets \frac{1}{m} \sum_{i=1}^{m}(x_i - \mu_{\beta})^2 \qquad & //\ mini-batch\ variance \\ - :math:`x` : mini-batch data - :math:`m` : the size of the mini-batch data When use_global_stats = True, the :math:`\\mu_{\\beta}` and :math:`\\sigma_{\\beta}^{2}` are not the statistics of one mini-batch. They are global or running statistics (moving_mean and moving_variance). It usually got from the pre-trained model. Calculated as follows: .. math:: moving\_mean = moving\_mean * momentum + \mu_{\beta} * (1. - momentum) \quad &// global mean \\ moving\_variance = moving\_variance * momentum + \sigma_{\beta}^{2} * (1. - momentum) \quad &// global variance \\ The normalization function formula is as follows: .. math:: \hat{x_i} &\gets \frac{x_i - \mu_\beta} {\sqrt{\ \sigma_{\beta}^{2} + \epsilon}} \qquad &//\ normalize \\ y_i &\gets \gamma \hat{x_i} + \beta \qquad &//\ scale\ and\ shift - :math:`\epsilon` : add a smaller value to the variance to prevent division by zero - :math:`\gamma` : trainable proportional parameter - :math:`\beta` : trainable deviation parameter Parameters: num_channels(int): Indicate the number of channels of the input ``Tensor``. act(str, optional): Activation to be applied to the output of batch normalization. Default: None. is_test (bool, optional): A flag indicating whether it is in test phrase or not. This flag only has effect on static graph mode. For dygraph mode, please use ``eval()``. Default: False. momentum(float, optional): The value used for the moving_mean and moving_var computation. Default: 0.9. epsilon(float, optional): The small value added to the variance to prevent division by zero. Default: 1e-5. param_attr(ParamAttr, optional): The parameter attribute for Parameter `scale` of batch_norm. If it is set to None or one attribute of ParamAttr, batch_norm will create ParamAttr as param_attr. If the Initializer of the param_attr is not set, the parameter is initialized with Xavier. Default: None. bias_attr(ParamAttr, optional): The parameter attribute for the bias of batch_norm. If it is set to None or one attribute of ParamAttr, batch_norm will create ParamAttr as bias_attr. If the Initializer of the bias_attr is not set, the bias is initialized zero. Default: None. dtype(str, optional): Indicate the data type of the input ``Tensor``, which can be float32 or float64. Default: float32. data_layout(str, optional): Specify the input data format, the data format can be "NCHW" or "NHWC". Default: NCHW. in_place(bool, optional): Make the input and output of batch norm reuse memory. Default: False. moving_mean_name(str, optional): The name of moving_mean which store the global Mean. Default: None. moving_variance_name(str, optional): The name of the moving_variance which store the global Variance. Default: None. do_model_average_for_mean_and_var(bool, optional): Whether parameter mean and variance should do model average when model average is enabled. Default: True. use_global_stats(bool, optional): Whether to use global mean and variance. In inference or test mode, set use_global_stats to true or is_test to true, and the behavior is equivalent. In train mode, when setting use_global_stats True, the global mean and variance are also used during train period. Default: False. trainable_statistics(bool, optional): Whether to calculate mean and var in eval mode. In eval mode, when setting trainable_statistics True, mean and variance will be calculated by current batch statistics. Default: False. Returns: None Examples: .. code-block:: python import paddle.fluid as fluid from paddle.fluid.dygraph.base import to_variable import numpy as np x = np.random.random(size=(3, 10, 3, 7)).astype('float32') with fluid.dygraph.guard(): x = to_variable(x) batch_norm = fluid.BatchNorm(10) hidden1 = batch_norm(x) NF?rr5ruTcs@tt|||_||_||_td|_|dusJd|dkr&d|_n||_|g}|j |j||jt dd|_ |oA|jj dk|j _ |j |j||jd d |_|oW|jj dk|j_ |j t| t dd| d ||jd |_d |j_ |j t| t dd| d ||jd |_d |j_ | |_| |_||_||_||_d|_||_||_dS) Nr;Fz,bias_attr should not be False in batch_norm.rr5rrKrBTrP)name initializerZ trainableZdo_model_averagerLrMrN)rRr%rSrbrcrZrr_rdrfrrgZ learning_raterrhr_mean _variance _in_place _data_layout _momentumr_is_test_fuse_with_relu_use_global_stats_trainable_statistics)rGrjrpis_testmomentumrrmrnrN data_layoutZin_placeZmoving_mean_nameZmoving_variance_nameZ!do_model_average_for_mean_and_varuse_global_statstrainable_statistics param_shaperrrHrIrS sd  zBatchNorm.__init__cCs|j}|j}trytr7t||j|j|j|j|j|j |j |j |j |j d \}}}}}} tj||j|jdStrod|jd|j d|j d|j d|jd|jd |j d |j f} tj||j|j|j|jd||g| R\}} } } } } tj||j|jdSt|d gd d |j|j |j|j d|j|j |j d} |g|jg|jg|jg|jgd} |jj|jdd} |jj|jdd} |jj|j|dd}|jr|n|j|j}|g|g|g| g| gd}|dur|g|d<|jjd| || d|j||jS)NF)rpryrrrrryfuse_with_relurrrrr%)rrrrryrrr)rrrMeanVarianceTr)rZMeanOutZ VarianceOutrrZ ReserveSpace batch_normr)rrr rrrrgrhrrrtrainingrrrrrZr_rrr rrrrrd input_dtyperrr)rGrmean_out variance_outZbatch_norm_outt1t2t3Zt4rrrrrZ reserve_spacerrHrHrIrRs          zBatchNorm.forward)NFrrNNr5ruFNNTFFrrHrHrrrIr%s$cGr%c2eZdZdZ    d fdd Zdd ZZS) r&aA This interface is used to construct a callable object of the ``Dropout`` class. For more details, refer to code examples. Drop or keep each element of input independently. Dropout is a regularization technique for reducing overfitting by preventing neuron co-adaption during training. The dropout operator randomly sets (according to the given dropout probability) the outputs of some units to zero, while others are remain unchanged. Dropout layer can be removed for efficiency concern. Parameters: p (float, optional): Probability of setting units to zero. Default: 0.5 seed (int, optional): A Python integer used to create random seeds. If this parameter is set to None, a random seed is used. NOTE: If an integer seed is given, always the same output units will be dropped. DO NOT use a fixed seed in training. Default: None. dropout_implementation(string, optional): ['downgrade_in_infer'(default)|'upscale_in_train'] 1. downgrade_in_infer(default), downgrade the outcome at inference - train: out = input * mask - inference: out = input * (1.0 - p) (mask is a tensor same shape with input, value is 0 or 1 ratio of 0 is dropout_prob) 2. upscale_in_train, upscale the outcome at training time - train: out = input * mask / ( 1.0 - p ) - inference: out = input (mask is a tensor same shape with input, value is 0 or 1 ratio of 0 is p) is_test (bool, optional): A flag indicating whether it is in test phrase or not. This flag only has effect on static graph mode. For dygraph mode, please use ``eval()``. Default: False. Returns: None Examples: .. code-block:: python import paddle.fluid as fluid from paddle.fluid.dygraph.base import to_variable import numpy as np x = np.random.random(size=(3, 10, 3, 7)).astype('float32') with fluid.dygraph.guard(): x = to_variable(x) m = fluid.dygraph.Dropout(p=0.5) droped_train = m(x) # switch to eval mode m.eval() droped_eval = m(x) rANdowngrade_in_inferFcstt|t|ttfsJdd|krdks$JdJd||_|dus4t|ts4Jd||_|dvs?Jd||_||_ dS)Nzp argument should be a numberrr z!p argument should between 0 and 1z)seed argument should be None or a integer)rZupscale_in_trainzTdropout_implementation argument should be 'downgrade_in_infer' or 'upscale_in_train') rRr&rSr[floatint _dropout_prob_seed_dropout_implementationr)rGpseeddropout_implementationrrrrHrIrSs$   zDropout.__init__cCs|jdkr|St}|jdus|jdkr|jdkr|j|_|jtr&|j n|j|jdu|jdur4|jnd|jd}trQt| d}t j |g|R\}}|S|j j |jd}|j j tjjjdd}|j jdd|gi|g|gd |d |S) Nr)Z dropout_probrZfix_seedrrrHrTrdropoutr)rMaskr)rrrZ random_seedr rrrsumitemsr rrrrNrZVarDescZVarTypeZUINT8r)rGrprogrrmaskrHrHrIrs6   zDropout.forward)rANrFrrHrHrrrIr&s<r&cs4eZdZdZ     d fdd ZddZZS) r'a :alias_main: paddle.nn.Embedding :alias: paddle.nn.Embedding,paddle.nn.layer.Embedding,paddle.nn.layer.common.Embedding :old_api: paddle.fluid.dygraph.Embedding **Embedding Layer** This interface is used to construct a callable object of the ``Embedding`` class. For specific usage, refer to code examples. It implements the function of the Embedding Layer. This layer is used to lookup embeddings vector of ids provided by :attr:`input` . It automatically constructs a 2D embedding matrix based on the input :attr:`size` (vocab_size, emb_size) and :attr:`dtype` . The shape of output Tensor is generated by appending an emb_size dimension to the last dimension of the input Tensor shape. **Note:** The id in :attr:`input` must satisfy :math:`0 =< id < size[0]` , otherwise the program will throw an exception and exit. .. code-block:: text Case 1: input is a Tensor. padding_idx = -1 input.data = [[1, 3], [2, 4], [4, 127] input.shape = [3, 2] Given size = [128, 16] output is a Tensor: out.shape = [3, 2, 16] out.data = [[[0.129435295, 0.244512452, ..., 0.436322452], [0.345421456, 0.524563927, ..., 0.144534654]], [[0.345249859, 0.124939536, ..., 0.194353745], [0.945345345, 0.435394634, ..., 0.435345365]], [[0.945345345, 0.435394634, ..., 0.435345365], [0.0, 0.0, ..., 0.0 ]]] # padding data The input padding_idx is less than 0, it is automatically converted to padding_idx = -1 + 128 = 127 It will pad all-zero data when ids is 127. Parameters: size(tuple|list): The shape of the look up table parameter. It should have two elements which indicate the size of the dictionary of embeddings and the size of each embedding vector respectively. is_sparse(bool): The flag indicating whether to use sparse update. This parameter only affects the performance of the backwards gradient update. It is recommended to set True because sparse update is faster. But some optimizer does not support sparse update, such as :ref:`api_fluid_optimizer_AdadeltaOptimizer` , :ref:`api_fluid_optimizer_AdamaxOptimizer` , :ref:`api_fluid_optimizer_DecayedAdagradOptimizer` , :ref:`api_fluid_optimizer_FtrlOptimizer` , :ref:`api_fluid_optimizer_LambOptimizer` and :ref:`api_fluid_optimizer_LarsMomentumOptimizer` . In these case, is_sparse must be False. Default: False. is_distributed(bool): Whether to store the embedding matrix in a distributed manner. Only used in multi-machine distributed CPU training. Default: False. padding_idx(int|long|None): padding_idx needs to be in the interval [-vocab_size, vocab_size). If :math:`padding\_idx < 0`, the :math:`padding\_idx` will automatically be converted to :math:`vocab\_size + padding\_idx` . It will output all-zero padding data whenever lookup encounters :math:`padding\_idx` in id. And the padding data will not be updated while training. If set None, it makes no effect to output. Default: None. param_attr(ParamAttr): To specify the weight parameter property. Default: None, which means the default weight parameter property is used. See usage for details in :ref:`api_fluid_ParamAttr` . In addition, user-defined or pre-trained word vectors can be loaded with the :attr:`param_attr` parameter. The local word vector needs to be transformed into numpy format, and the shape of local word vector should be consistent with :attr:`size` . Then :ref:`api_fluid_initializer_NumpyArrayInitializer` is used to load custom or pre-trained word vectors. See code example 2 for details. dtype(np.dtype|core.VarDesc.VarType|str): It refers to the data type of output Tensor. It must be "float32" or "float64". Default: "float32". Attribute: **weight** (Parameter): the learnable weights of this layer. Returns: Variable: Embedding Tensor or LoDTensor mapped by input. The data type is the same as :attr:`dtype` . Examples: .. code-block:: python import paddle.fluid as fluid import paddle.fluid.dygraph.base as base import numpy as np # example 1 inp_word = np.array([[2, 3, 5], [4, 2, 1]]).astype('int64') inp_word.shape # [2, 3] dict_size = 20 with fluid.dygraph.guard(): emb = fluid.dygraph.Embedding( size=[dict_size, 32], param_attr='emb.w', is_sparse=False) static_rlt3 = emb(base.to_variable(inp_word)) static_rlt3.shape # [2, 3, 32] # example 2: load custom or pre-trained word vectors weight_data = np.random.random(size=(128, 100)) # word vectors with numpy format w_param_attrs = fluid.ParamAttr( name="emb_weight", learning_rate=0.5, initializer=fluid.initializer.NumpyArrayInitializer(weight_data), trainable=True) with fluid.dygraph.guard(): emb = fluid.dygraph.Embedding( size=[128, 100], param_attr= w_param_attrs, is_sparse=False) static_rlt3 = emb(base.to_variable(inp_word)) FNr5cstt|||_||_||_|durdn |dkr|n|d||_||_||_|jo/|j |_ |j r@|jdur>|jdus@J|j |j|j|jdd|_ dS)NrvrTFrP) rRr'rS_size _is_sparse_is_distributed _padding_idxrbrd_remote_prefetchrfrg)rGsize is_sparseis_distributed padding_idxrmrNrrrHrIrSs"  zEmbedding.__init__c Cstrt|j|d|jd|jd|jd|j St|dgdd|j|j|j|jd}|j |j }|j j d ||jd d |i|d |S) Nr rremote_prefetchrr)rrZint16int32int64r')r rrrlookup_table_v2)ZIdsWrr) r r rrgrr r r rrrrdr)rGrrrrHrHrIrs0  zEmbedding.forward)FFNNr5rrHrHrrrIr'smr'cs8eZdZdZ       d fdd Zdd ZZS) r*a :alias_main: paddle.nn.LayerNorm :alias: paddle.nn.LayerNorm,paddle.nn.layer.LayerNorm,paddle.nn.layer.norm.LayerNorm :old_api: paddle.fluid.dygraph.LayerNorm This interface is used to construct a callable object of the ``LayerNorm`` class. For more details, refer to code examples. It implements the function of the Layer Normalization Layer and can be applied to mini-batch input data. Refer to `Layer Normalization `_ The formula is as follows: .. math:: \\mu & = \\frac{1}{H}\\sum_{i=1}^{H} x_i \\sigma & = \\sqrt{\\frac{1}{H}\sum_{i=1}^{H}{(x_i - \\mu)^2} + \\epsilon} y & = f(\\frac{g}{\\sigma}(x - \\mu) + b) - :math:`x`: the vector representation of the summed inputs to the neurons in that layer. - :math:`H`: the number of hidden units in a layers - :math:`\\epsilon`: the small value added to the variance to prevent division by zero. - :math:`g`: the trainable scale parameter. - :math:`b`: the trainable bias parameter. Parameters: normalized_shape(int or list or tuple): Input shape from an expected input of size :math:`[*, normalized_shape[0], normalized_shape[1], ..., normalized_shape[-1]]`. If it is a single integer, this module will normalize over the last dimension which is expected to be of that specific size. scale(bool, optional): Whether to learn the adaptive gain :math:`g` after normalization. Default: True. shift(bool, optional): Whether to learn the adaptive bias :math:`b` after normalization. Default: True. epsilon(float, optional): The small value added to the variance to prevent division by zero. Default: 1e-05. param_attr(ParamAttr, optional): The parameter attribute for the learnable gain :math:`g`. If :attr:`scale` is False, :attr:`param_attr` is omitted. If :attr:`scale` is True and :attr:`param_attr` is None, a default :code:`ParamAttr` would be added as scale. The :attr:`param_attr` is initialized as 1 if it is added. Default: None. bias_attr(ParamAttr, optional): The parameter attribute for the learnable bias :math:`b`. If :attr:`shift` is False, :attr:`bias_attr` is omitted. If :attr:`shift` is True and :attr:`param_attr` is None, a default :code:`ParamAttr` would be added as bias. The :attr:`bias_attr` is initialized as 0 if it is added. Default: None. act(str, optional): Activation to be applied to the output of layer normalization. Default: None. dtype (str, optional): Data type, it can be "float32" or "float64". Default: "float32". Returns: None Examples: .. code-block:: python import paddle.fluid as fluid from paddle.fluid.dygraph.base import to_variable import numpy x = numpy.random.random((3, 32, 32)).astype('float32') with fluid.dygraph.guard(): x = to_variable(x) layerNorm = fluid.LayerNorm([32, 32]) ret = layerNorm(x) TrNr5c stt|t|tjr|g}t||_||_||_ ||_ ||_ ||_ ||_ ||_t|jg} |jrC|j|j | |jtdd|_n |j rKtdd|_|j rf|j dusXJ|j|j | |jdd|_dS|j rntdd|_dS)NrrKz0param_attr are only available with scale is TrueFTrPz/bias_attr are only available with shift is True)rRr*rSr[numbersIntegrallist_normalized_shape_scale_shiftrrbrcrZrdnpprodrfrrgloggingwarnrh) rGZnormalized_shapershiftrrmrnrprNrrrrHrIrSs@       zLayerNorm.__init__c Cst|j}t|}t|j}|||_||ks!||jd|jkr:t|j}td|d|dddt|trstrYt ||j |j |j |jd\}}}tj||jdSt ||j |j d|j d|j\}}}tj||jdSt|d d d gd t}|g|d <|jr|j g|d<|jr|j g|d<|j |jd} |jj|jdd} |jj|jdd} |j|j} |jjd|| | | d|j |jdd|jj| |jdS)NzGiven normalized_shape is z , expected input with shape [*, r z, but got input shape Frrbegin_norm_axisrr5rr*rrr)rr Tr layer_normrrrr)rrMrrZ_begin_norm_axisrr]r rrr!rgrhrrrrZr rdictrrrrrdrr) rGr input_shapeZ input_ndimZnormalized_ndimZstr_normalized_shaperrrrrrZlayer_norm_outrHrHrIr*s              zLayerNorm.forward)TTrNNNr5rrHrHrrrIr*sH,r*cs6eZdZdZ      d fdd Zd d ZZS) r(at **GRU unit layer** It creates a callable object from GRUUnit class. If origin_mode is True, then the equation of a gru step is from paper `Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation `_ .. math:: u_t & = actGate(xu_{t} + W_u h_{t-1} + b_u) r_t & = actGate(xr_{t} + W_r h_{t-1} + b_r) m_t & = actNode(xm_t + W_c dot(r_t, h_{t-1}) + b_m) h_t & = dot(u_t, h_{t-1}) + dot((1-u_t), m_t) If origin_mode is False, then the equation of a gru step is from paper `Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling `_ .. math:: u_t & = actGate(xu_{t} + W_u h_{t-1} + b_u) r_t & = actGate(xr_{t} + W_r h_{t-1} + b_r) m_t & = actNode(xm_t + W_c dot(r_t, h_{t-1}) + b_m) h_t & = dot((1-u_t), h_{t-1}) + dot(u_t, m_t) The inputs of gru unit includes :math:`z_t`, :math:`h_{t-1}`. In terms of the equation above, the :math:`z_t` is split into 3 parts - :math:`xu_t`, :math:`xr_t` and :math:`xm_t`. This means that in order to implement a full GRU unit operator for an input, a fully connected layer has to be applied, such that :math:`z_t = W_{fc}x_t`. The terms :math:`u_t` and :math:`r_t` represent the update and reset gates of the GRU cell. Unlike LSTM, GRU has one lesser gate. However, there is an intermediate candidate hidden output, which is denoted by :math:`m_t`. This layer has three outputs :math:`h_t`, :math:`dot(r_t, h_{t-1})` and concatenation of :math:`u_t`, :math:`r_t` and :math:`m_t`. Parameters: size (int): The input dimension value. param_attr(ParamAttr, optional): The parameter attribute for the learnable hidden-hidden weight matrix. **Note**: 1. The shape of the weight matrix is :math:`[T, 3*D]`, where D is the hidden size. 2. All elements in the weight matrix can be divided into two parts. The first part are weights of the update gate and reset gate with shape :math:`[D, 2*D]`, and the second part are weights for candidate hidden state with shape :math:`[D, D]`. If it is set to None or one attribute of ParamAttr, gru_unit will create ParamAttr as param_attr. If the Initializer of the param_attr is not set, the parameter is initialized with Xavier. The default value is None. bias_attr (ParamAttr|bool, optional): The parameter attribute for the bias of GRU.Note that the bias with :math:`[1, 3*D]` concatenates the bias in the update gate, reset gate and candidate calculations. If it is set to False, no bias will be applied to the update gate, reset gate and candidate calculations. If it is set to None or one attribute of ParamAttr, gru_unit will create ParamAttr as bias_attr. If the Initializer of the bias_attr is not set, the bias is initialized zero. The default value is None. activation (str): The activation type for cell (actNode). The default value is 'tanh'. gate_activation (str): The activation type for gates (actGate). The default value is 'sigmoid'. dtype(str): The dtype of the layers. The data type can be set as 'float32', 'float64'. The default value is 'float32'. Attribute: **weight** (Parameter): the learnable weights of this layer. **bias** (Parameter): the learnable bias of this layer. Returns: tuple: The hidden value, reset-hidden value and gate values. The hidden value is a 2-D tensor with shape :math:`[T, D]` . The reset-hidden value is a 2-D tensor with shape :math:`[T, D]` . The gate value is a 2-D tensor with shape :math:`[T, 3*D]`. Examples: .. code-block:: python import paddle.fluid as fluid import paddle.fluid.dygraph.base as base import numpy lod = [[2, 4, 3]] D = 5 T = sum(lod[0]) input = numpy.random.rand(T, 3 * D).astype('float32') hidden_input = numpy.random.rand(T, D).astype('float32') with fluid.dygraph.guard(): x = numpy.random.random((3, 32, 32)).astype('float32') gru = fluid.dygraph.GRUUnit(size=D * 3) dy_ret = gru( base.to_variable(input), base.to_variable(hidden_input)) NtanhsigmoidFr5c stt|||_tddddd}|||_|||_||_|d}|j||d|g|d|_ dd|g} | |_ |j|| |dd|_ dS) Nrr rr)identityr&r%ZrelurTrP) rRr(rSrcr# activationgate_activationrdrfrgZ _bias_sizerh) rGr rmrnr(r)Z origin_moderNZactivation_dict bias_sizerrrHrIrSs.     zGRUUnit.__init__c Cstrt|||j|jd|jd|j\}}}|||fSt|dddgdt|dddgd|g|g|jgd}|jdurB|jg|d <|j |j }|j |j }|j |j }|jj d ||||d |j|jd d |||fS)Nr(r)rr5rr(hidden)r~Z HiddenPrevWeightrgru_unit)ZGateZResetHiddenPrevZHidden)r(r)r) r r r-rgrhr(r)rrrrdr)rGrr+ZgateZreset_hidden_preZupdated_hiddenrrHrHrIrsH       zGRUUnit.forward)NNr%r&Fr5rrHrHrrrIr(ksn"r(cs>eZdZdZ         d fdd Zd d d ZZS) r+a This interface is used to construct a callable object of the ``NCE`` class. For more details, refer to code examples. It implements the function of the ``NCE`` loss function. By default this function uses a uniform distribution for sampling, and it compute and return the noise-contrastive estimation training loss. See `Noise-contrastive estimation: A new estimation principle for unnormalized statistical models `_ . Parameters: num_total_classes (int): Total number of classes in all samples. dim (int): Dimension of input (possibly embedding dim). param_attr (ParamAttr, optional): The parameter attribute for learnable weights(Parameter) of nce. If it is set to None or one attribute of ParamAttr, nce will create ParamAttr as param_attr. If the Initializer of the param_attr is not set, the parameter is initialized with Xavier. Default: None. bias_attr (ParamAttr or bool, optional): The attribute for the bias of nce. If it is set to False, no bias will be added to the output units. If it is set to None or one attribute of ParamAttr, nce will create ParamAttr as bias_attr. If the Initializer of the bias_attr is not set, the bias is initialized zero. Default: None. num_neg_samples (int, optional): The number of negative classes. The default value is 10. sampler (str, optional): The sampler used to sample class from negative classes. It can be 'uniform', 'log_uniform' or 'custom_dist'. default: 'uniform'. custom_dist (float[], optional): A float[] with size=num_total_classes. It is used when sampler is set to 'custom_dist'. custom_dist[i] is the probability of i-th class to be sampled. Default: None. seed (int, optional): The seed used in sampler. Default: 0. is_sparse(bool, optional): The flag indicating whether to use sparse update. If is_sparse is True, the weight@GRAD and bias@GRAD will be changed to SelectedRows. Default: False. dtype (str, optional): Data type, it can be "float32" or "float64". Default: "float32". Attribute: **weight** (Parameter): the learnable weights of this layer. **bias** (Parameter or None): the learnable bias of this layer. Returns: None Examples: .. code-block:: python import numpy as np import paddle.fluid as fluid window_size = 5 dict_size = 20 label_word = int(window_size // 2) + 1 inp_word = np.array([[1], [2], [3], [4], [5]]).astype('int64') nid_freq_arr = np.random.dirichlet(np.ones(20) * 1000).astype('float32') with fluid.dygraph.guard(): words = [] for i in range(window_size): words.append(fluid.dygraph.base.to_variable(inp_word[i])) emb = fluid.Embedding( size=[dict_size, 32], param_attr='emb.w', is_sparse=False) embs3 = [] for i in range(window_size): if i == label_word: continue emb_rlt = emb(words[i]) embs3.append(emb_rlt) embs3 = fluid.layers.concat(input=embs3, axis=1) nce = fluid.NCE( num_total_classes=dict_size, dim=embs3.shape[1], num_neg_samples=2, sampler="custom_dist", custom_dist=nid_freq_arr.tolist(), seed=1, param_attr='nce.w', bias_attr='nce.b') wl = fluid.layers.unsqueeze(words[label_word], axes=[0]) nce_loss3 = nce(embs3, wl) NuniformrFr5c stt|_|_|_| _t_|dur|ngjd<|dkr*d}n|dkr2d}n |dkr9|dus=Jt |} dg| } dg| }g}g}t | D],}||| }|ddkri| ||fqSd|dkrw| ||fqS|| |<d||<qSt |rt |r| d}| d}|d}|d}|d| |d<|||d<|d|dd}|ddkr| ||fnd|dkr| ||fn|| |<d||<t |rt |st |r| d}d| |d<d||d<t |r | d}d| |d<d||d<fd d }|t |d jd <|t |d jd<|t | d jd<d}ntd|durEd}nt|}|_| }tdt||| || |d_jjj|gdjd_jrjjjdgdjd_jjd<jjd<dS)N SampleWeightr.rZ log_uniformr custom_distrrvcs&jt|j|jt|d}d|_|S)NrKT)rfrrMrNrr)Z numpy_arrayretrGrHrI_init_by_numpy_arraysz*NCE.__init__.._init_by_numpy_arrayr5CustomDistProbsrCustomDistAliasCustomDistAliasProbsrzUnsupported sampler type. zWWith sparse mode, if your models has only small parameter prefetch may cause speed down)num_total_classesnum_neg_samplesrsamplerr rF)rLrMrQrNTrr,)rRr+rSrbrcZ_num_total_classesrdr#_inputsrrangeappendpoprarrayZastype Exceptionr_num_neg_samplesprint_attrsrfrgrh)rGr8dim sample_weightrmrnr9r:r0rr rNZcustom_dist_lenZ alias_probs_Zalias_ZbigsZlittlesiZ normal_probbiglittleZbig_idxZbig_probZbig_leftr3rrrr2rIrSws                              z NCE.__init__c CsrtrKd|jdd|jdd|jdd|jdd|jdd|jdf }tj|||j|j|jd|jd|jd |jd g|R\}}}||jd St|d d dgdt|ddgdt |dt t dfdt |t snJt |t suJ||jd<||jd<|dur|ng|jd<|j j|jd}|j j|jd}|j j|jd}|j jd|j|||d|jd||jd S)Nr8r9rr:r rr/r4r5r6r rr5rr+labelrrEr~Labelrnce)ZCostZ SampleLogitsZ SampleLabelsr)r rCr rKrgrhr;rArrr rr[rrrNr) rGrrIrErZcostrZ sample_logitsZ sample_labelsrHrHrIrsX     z NCE.forward) NNNNr.NrFr5NrrHrHrrrIr+ sYxr+cr) r,ao This interface is used to construct a callable object of the ``PRelu`` class. For more details, refer to code examples. It implements three activation methods of the ``PRelu`` activation function. Equation: .. math:: y = \max(0, x) + \\alpha * \min(0, x) Parameters: mode (str): The mode for weight sharing. It supports all, channel and element. all: all elements share same weight channel:elements in a channel share same weight element:each element has a weight channel (int, optional): The number of channels. This argument is required when mode is "channel". Default: None. input_shape (list or tuple, optional): The shape of input. This argument is required when mode is "element". Default: None. param_attr(ParamAttr, optional): The parameter attribute for the learnable weight (alpha). Default: None. dtype (str, optional): Data type, it can be "float32" or "float64". Default: "float32". Attribute: **weight** (Parameter): the learnable weights of this layer. Returns: None Examples: .. code-block:: python import paddle.fluid as fluid from paddle.fluid.dygraph.base import to_variable import numpy as np inp_np = np.ones([5, 200, 100, 100]).astype('float32') with fluid.dygraph.guard(): inp_np = to_variable(inp_np) prelu0 = fluid.PRelu( mode='all', param_attr=fluid.ParamAttr(initializer=fluid.initializer.Constant(1.0))) dy_rlt0 = prelu0(inp_np) prelu1 = fluid.PRelu( mode='channel', channel=200, param_attr=fluid.ParamAttr(initializer=fluid.initializer.Constant(1.0))) dy_rlt1 = prelu1(inp_np) prelu2 = fluid.PRelu( mode='element', input_shape=inp_np.shape, param_attr=fluid.ParamAttr(initializer=fluid.initializer.Constant(1.0))) dy_rlt2 = prelu2(inp_np) Nr5cstt|jdd||_||_||_|dkrdg|_n5|dkr0t|ts(Jdd|ddg|_n |dkrLt|t t fs?Jddgt |dd|_nt d |j |j|jd d t d d |_dS)Nprelu) name_scopeallr channelz4channel argument is required when mode is 'channel'.elementz8input_shape argument is required when mode is 'element'.z,mode should be one of all, channel, element.r5Fr)rLrMrNrQrO)rRr,rS_moderbrdZ _alpha_shaper[rrtupler]rfrrg)rGmoderPr$rmrNrrrHrIrSW s:  zPRelu.__init__cCsbtr t||jd|jSt|ddgd|j|j}|jj d||jdd|jid|id |S) Nrurr5r,rM)rAlpharTr)rrrr) rrrMrgrRrrrrdrrGrrrHrHrIr{ sz PRelu.forwardrrrHrHrrrIr, s=$r,csBeZdZdZ     d fdd Zedddd d d ZZS) r-a **Add Bilinear Tensor Product Layer** This layer performs bilinear tensor product on two inputs. For example: .. math:: out_{i} = x * W_{i} * {y^\mathrm{T}}, i=0,1,...,size-1 In this formula: - :math:`x`: the first input contains M elements, shape is [batch_size, M]. - :math:`y`: the second input contains N elements, shape is [batch_size, N]. - :math:`W_{i}`: the i-th learned weight, shape is [M, N] - :math:`out_{i}`: the i-th element of out, shape is [batch_size, size]. - :math:`y^\mathrm{T}`: the transpose of :math:`y`. Parameters: input1_dim (int): The dimension of each first input. input2_dim (int): The dimension of each second input. output_dim (int): The dimension of output of this layer. name (str, optional): The default value is None. Normally there is no need for user to set this property. For more information, please refer to :ref:`api_guide_Name`. Default: None. act (str, optional): Activation to be applied to the output of this layer. The default value is None. param_attr (ParamAttr, optional): The parameter attribute for the learnable w, parameters/weights of this layer. The default value is None. bias_attr (ParamAttr, optional): The parameter attribute for the bias of this layer. If it is set to False, no bias will be added to the output units. If it is set to None, the bias is initialized zero. The default value is None. dtype (str, optional): Data type, it can be "float32" or "float64". Default: "float32". Attribute: **weight** (Parameter): the learnable weights of this layer. **bias** (Parameter): the learnable bias of this layer. Returns: Tensor: A 2-D Tensor of shape [batch_size, size]. Examples: .. code-block:: python import paddle import numpy layer1 = numpy.random.random((5, 5)).astype('float32') layer2 = numpy.random.random((5, 4)).astype('float32') bilinearTensorProduct = paddle.nn.BilinearTensorProduct( input1_dim=5, input2_dim=4, output_dim=1000) ret = bilinearTensorProduct(paddle.to_tensor(layer1), paddle.to_tensor(layer2)) Nr5c stt|||_||_||_||_||_||_||_ t |_ ||_ |j |j|jg} |j |j| |j dd|_d|j g} |j |j| |j dd|_dS)NFrPr T)rRr-rSrbrcrZ_nameZ _input1_dimZ _input2_dimZ _output_dimr#r;rdrfrgrh) rGZ input1_dimZ input2_dimrrrprmrnrNrr*rrrHrIrS s,   zBilinearTensorProduct.__init__z2.0.0zpaddle.nn.Bilinearz1New name and new args in Bilinear, easier to use.)ZsinceZ update_toreasoncCst|dddgdt|dddgd|||jd|_|jdur%|j|jd<|jdur=|jjd||jg|j d d }n |jj|j d d }|jj d |jd |id|jj ||j dS)Nxr5rr-y)rrr,r.FrrN persistable)rNr]Zbilinear_tensor_productrrrrr) rrgr;rhrWrcreate_variablejoinZ full_namerdrrrZ)rGrYrZrrHrHrIr s0        zBilinearTensorProduct.forward)NNNNr5)rrrrrS deprecatedrrrHrHrrrIr- s:r-cs>eZdZdZ          d fdd Zd d ZZS) r.a This interface is used to construct a callable object of the ``Conv2DTranspose`` class. For more details, refer to code examples. The convolution2D transpose layer calculates the output based on the input, filter, and dilations, strides, paddings. Input and output are in NCHW format. Where N is batch size, C is the number of feature map, H is the height of the feature map, and W is the width of the feature map. Filter's shape is [MCHW] , where M is the number of input feature map, C is the number of output feature map, H is the height of the filter, and W is the width of the filter. If the groups is greater than 1, C will equal the number of input feature map divided by the groups. If bias attribution and activation type are provided, bias is added to the output of the convolution, and the corresponding activation function is applied to the final result. The details of convolution transpose layer, please refer to the following explanation and references `conv2dtranspose `_ . For each input :math:`X`, the equation is: .. math:: Out = \sigma (W \\ast X + b) Where: * :math:`X`: Input value, a ``Tensor`` with NCHW format. * :math:`W`: Filter value, a ``Tensor`` with shape [MCHW] . * :math:`\\ast`: Convolution operation. * :math:`b`: Bias value, a 2-D ``Tensor`` with shape [M, 1]. * :math:`\\sigma`: Activation function. * :math:`Out`: Output value, the shape of :math:`Out` and :math:`X` may be different. Example: - Input: Input shape: :math:`(N, C_{in}, H_{in}, W_{in})` Filter shape: :math:`(C_{in}, C_{out}, H_f, W_f)` - Output: Output shape: :math:`(N, C_{out}, H_{out}, W_{out})` Where .. math:: H^\prime_{out} &= (H_{in} - 1) * strides[0] - 2 * paddings[0] + dilations[0] * (H_f - 1) + 1 \\\\ W^\prime_{out} &= (W_{in} - 1) * strides[1] - 2 * paddings[1] + dilations[1] * (W_f - 1) + 1 \\\\ H_{out} &\in [ H^\prime_{out}, H^\prime_{out} + strides[0] ) \\\\ W_{out} &\in [ W^\prime_{out}, W^\prime_{out} + strides[1] ) Parameters: num_channels(int): The number of channels in the input image. num_filters(int): The number of the filter. It is as same as the output feature map. filter_size(int or tuple): The filter size. If filter_size is a tuple, it must contain two integers, (filter_size_H, filter_size_W). Otherwise, the filter will be a square. output_size(int or tuple, optional): The output image size. If output size is a tuple, it must contain two integers, (image_H, image_W). None if use filter_size, padding, and stride to calculate output_size. if output_size and filter_size are specified at the same time, They should follow the formula above. Default: None. padding(int or tuple, optional): The padding size. If padding is a tuple, it must contain two integers, (padding_H, padding_W). Otherwise, the padding_H = padding_W = padding. Default: 0. stride(int or tuple, optional): The stride size. If stride is a tuple, it must contain two integers, (stride_H, stride_W). Otherwise, the stride_H = stride_W = stride. Default: 1. dilation(int or tuple, optional): The dilation size. If dilation is a tuple, it must contain two integers, (dilation_H, dilation_W). Otherwise, the dilation_H = dilation_W = dilation. Default: 1. groups(int, optional): The groups number of the Conv2D transpose layer. Inspired by grouped convolution in Alex Krizhevsky's Deep CNN paper, in which when group=2, the first half of the filters is only connected to the first half of the input channels, while the second half of the filters is only connected to the second half of the input channels. Default: 1. param_attr (ParamAttr, optional): The parameter attribute for learnable weights(Parameter) of conv2d_transpose. If it is set to None or one attribute of ParamAttr, conv2d_transpose will create ParamAttr as param_attr. If the Initializer of the param_attr is not set, the parameter is initialized with Xavier. Default: None. bias_attr (ParamAttr or bool, optional): The attribute for the bias of conv2d_transpose. If it is set to False, no bias will be added to the output units. If it is set to None or one attribute of ParamAttr, conv2d_transpose will create ParamAttr as bias_attr. If the Initializer of the bias_attr is not set, the bias is initialized zero. Default: None. use_cudnn(bool, optional): Use cudnn kernel or not, it is valid only when the cudnn library is installed. Default: True. act (str, optional): Activation type, if it is set to None, activation is not appended. Default: None. dtype (str, optional): Data type, it can be "float32" or "float64". Default: "float32". Attribute: **weight** (Parameter): the learnable weights of filters of this layer. **bias** (Parameter or None): the learnable bias of this layer. Returns: None Examples: .. code-block:: python import paddle.fluid as fluid import numpy as np with fluid.dygraph.guard(): data = np.random.random((3, 32, 32, 5)).astype('float32') conv2DTranspose = fluid.dygraph.nn.Conv2DTranspose( num_channels=32, num_filters=2, filter_size=3) ret = conv2DTranspose(fluid.dygraph.base.to_variable(data)) Nrr Tr5csFtt|| dusJd| |_| |_| |_||_||_||_| |_ ||_ ||_ ||_ ||_ ||_| |_|j|jkrI|j|jkrI|j sId|_nd|_t|j dd|_ t|j dd|_ t|j dd|_ t|j dd |_ |jduryg|_not|jtrt|jrt|j|_n[t|jdd |_nQt|jtrt|jdd |_nAt|jtrt|jjd d d gd t|jjdkr|jjddks|jjddkr|jjddkr|j|jg|_ntdtdt|j dd|_ |jdurdn|j|_|j|j|jg|j }|j|j||jd|_|j|j|jg|jdd|_ dS)NFz3param_attr should not be False in conv2d_transpose.Zdepthwise_conv2d_transposeZconv2d_transposerr8r7r9zconv2d_transpose.filter_size output_sizerrr.r rz-output_size must contain one or two integers.z+output_size should be list or int or TensorrTrP)!rRr.rSrbrcrZrUrDrar^rXrWrYr` _output_sizerd_op_typerrVr[rZ _contain_varZ_convert_to_tensor_listrr rrNrrMr]rfrgrh)rGrjrkr?rbr8r7r9rlrmrnrorprNrqrrrHrIrSr s          zConv2DTranspose.__init__cCs0tr3tt|j}|||jd|jd|jd|jd|jd|j d|j }|}t ||j d}t j||jdSt|d gd d |g|jgd }|j|j|j|j|j |j d }|jj|jd}|jj|j|d|i|d|j dur|jj|jd}|jjd|g|j gdd|giddidn|}|jj||jd}|S)Nrbrzr{r|rlror rrrr.r})rbrzr{r|rlrorrrrrrrw)r getattrr rdrgrcrWrXrYrUr^rrrhrrZrrrrNrrdr)rGroprrrrrrHrHrIr s`      zConv2DTranspose.forward) Nrr r NNNTNr5rrHrHrrrIr. sySr.cs>eZdZdZ      d fdd ZddZd d ZZS) SequenceConva: This function creates the op for sequence_conv, using the inputs and other convolutional configurations for the filters and stride as given in the input parameters to the function. Parameters: name_scope(str): The name of this class. num_filters (int): number of filters. filter_size (int): the filter size (H and W). Default: 3. filter_stride (int): stride of the filter. Default: 1. padding (bool|None): if True, add paddings. Default: None bias_attr (ParamAttr|bool|None): The parameter attribute for the bias of sequence_conv. If it is set to False, no bias will be added to the output units. If it is set to None or one attribute of ParamAttr, sequence_conv will create ParamAttr as bias_attr. If the Initializer of the bias_attr is not set, the bias is initialized zero. Default: None. param_attr (ParamAttr|None): The parameter attribute for learnable parameters/weights of sequence_conv. If it is set to None or one attribute of ParamAttr, sequence_conv will create ParamAttr as param_attr. If the Initializer of the param_attr is not set, the parameter is initialized with Xavier. Default: None. act (str): Activation type, if it is set to None, activation is not appended. Default: None. Attributes: weight (Parameter): the learnable weights of filters of this layer. bias (Parameter|None): the learnable bias of this layer. Returns: Variable: output of sequence_conv rr Nc sLtrJdtt||||_||_||_||_||_||_ ||_ dS)Nz8SequenceConv is not supported by dynamic graph mode yet!) r rRrgrSrar`_filter_striderXrcrbrZ) rGrNrkr?Z filter_strider8rnrmrprrrHrIrS s  zSequenceConv.__init__cCsZ|j||_|j|jd|jg}|j|j||jd|_|j|j |jg|jdd|_ dS)Nr rTrP) rrrdr`rMrarfrbrgrcrhrGrrqrHrHrI _build_once, s zSequenceConv._build_oncecCs|j|j}|jjd|g|jgdd|i|jt|jd |jdd|jdurH|jj|jd}|jjd|g|jgd d|gid d idn|}|jj ||j d S) NZ sequence_convrrrr)Z contextStrideZ contextStartZ contextLengthrrrrrwr r) rrrdrrgrhrr`rhrrZrrHrHrIr8 s0 zSequenceConv.forward)rr NNNNrrrrrSrjrrrHrHrrrIrg s" rgcs6eZdZdZ  d fdd ZddZddZZS) RowConva ***Row-convolution operator*** The row convolution is called lookahead convolution. This operator was introduced in the following paper for DeepSpeech2: http://www.cs.cmu.edu/~dyogatam/papers/wang+etal.iclrworkshop2016.pdf The main motivation is that a bidirectional RNN, useful in DeepSpeech like speech models, learns representation for a sequence by performing a forward and a backward pass through the entire sequence. However, unlike unidirectional RNNs, bidirectional RNNs are challenging to deploy in an online and low-latency setting. The lookahead convolution incorporates information from future subsequences in a computationally efficient manner to improve unidirectional recurrent neural networks. The row convolution operator is different from the 1D sequence convolution, and is computed as follows: Given an input sequence X of length t and input dimension D, and a filter (W) of size context * D. More details about row_conv please refer to the design document https://github.com/PaddlePaddle/Paddle/issues/2228#issuecomment-303903645 . Parameters: name_scope(str): The name of this class. future_context_size (int): Future context size. Please note, the shape of convolution kernel is [future_context_size + 1, D]. param_attr (ParamAttr): Attributes of parameters, including name, initializer etc. Default: None. act (str): Non-linear activation to be applied to output variable. Default: None. Attributes: weight (Parameter): the learnable weights of this layer. Returns: the output(Out) is a LodTensor, which supports variable time-length input sequences. The underlying tensor in this LodTensor is a matrix with shape T x N, i.e., the same shape as X. Examples: .. code-block:: python import paddle.fluid as fluid import numpy with fluid.dygraph.guard(): x = numpy.random.random((16)).astype('float32') rowConv = fluid.dygraph.nn.RowConv( 'RowConv', future_context_size=2) ret = rowConv(fluid.dygraph.base.to_variable(x)) Ncs4trJdtt||||_||_||_dS)Nz3RowConv is not supported by dynamic graph mode yet!)r rRrmrSrZrb_future_context_size)rGrNZfuture_context_sizermrprrrHrIrS s zRowConv.__init__cCs>|j||_|jd|jdg}|j|j||jdd|_dS)Nr FrP)rrrdrnrMrfrbrgrirHrHrIrj s zRowConv._build_oncecCsD|j|j}|jjd|g|jgdd|gid|jj||jdS)NZrow_convrkrr^r)rrrdrrgrrZrVrHrHrIr szRowConv.forward)NNrlrHrHrrrIrmV s2 rmcs6eZdZdZ      d fdd Zdd ZZS) r0a :alias_main: paddle.nn.GroupNorm :alias: paddle.nn.GroupNorm,paddle.nn.layer.GroupNorm,paddle.nn.layer.norm.GroupNorm :old_api: paddle.fluid.dygraph.GroupNorm This interface is used to construct a callable object of the ``GroupNorm`` class. For more details, refer to code examples. It implements the function of the Group Normalization Layer. Refer to `Group Normalization `_ . Parameters: channels(int): The number of channels of input. groups(int): The number of groups that divided from channels. epsilon(float, optional): The small value added to the variance to prevent division by zero. Default: 1e-05. param_attr(ParamAttr, optional): The parameter attribute for the learnable scale :math:`g`. If it is set to False, no scale will be added to the output units. If it is set to None, the bias is initialized one. Default: None. bias_attr(ParamAttr, optional): The parameter attribute for the learnable bias :math:`b`. If it is set to False, no bias will be added to the output units. If it is set to None, the bias is initialized zero. Default: None. act(str, optional): Activation to be applied to the output of group normalization. Default: None. data_layout(str, optional): Specify the input data format. Only NCHW is supported. Default: NCHW. Returns: None Examples: .. code-block:: python import paddle.fluid as fluid import numpy as np with fluid.dygraph.guard(): x = np.random.random((8, 32, 32)).astype('float32') groupNorm = fluid.dygraph.nn.GroupNorm(channels=32, groups=4) ret = groupNorm(fluid.dygraph.base.to_variable(x)) rNrur5c stt|||_||_||_||_||_||_||_ |dkr&t d||jg} |j |jp0d| |j t dd|_ |j |jp@d| |j dd|_dS)Nruzunsupported data layout:FrrKTrP)rRr0rSrbrcrZ _channelsrUrZrdr]rfrrgrh) rGZchannelsrlrrmrnrprrNrrrrHrIrS s*     zGroupNorm.__init__c Cs|jj|jdd}|jj|jdd}tr*t||j|j|j|j d}t ||j St rNd|jd|j f}tj||j|j||g|R\}}}t ||j Sd|i}|jdur\|j|d<|jdurf|j|d<|jj|jd }|jjd ||||d |j|j d d |j||j S)NTrrurrlrrrr group_normr")rrlr)rrrdrrrorgrhrrUrrrZrr rr) rGrrrrrrrZgroup_norm_outrHrHrIr sL      zGroupNorm.forward)rNNNrur5rrHrHrrrIr0 s+ r0cr) r1a~ This interface is used to construct a callable object of the ``SpectralNorm`` class. For more details, refer to code examples. It implements the function of the Spectral Normalization Layer. This layer calculates the spectral normalization value of weight parameters of fc, conv1d, conv2d, conv3d layers which should be 2-D, 3-D, 4-D, 5-D Parameters. Calculations are showed as follows. Step 1: Generate vector U in shape of [H], and V in shape of [W]. While H is the :attr:`dim` th dimension of the input weights, and W is the product result of remaining dimensions. Step 2: :attr:`power_iters` should be a positive integer, do following calculations with U and V for :attr:`power_iters` rounds. .. math:: \mathbf{v} := \frac{\mathbf{W}^{T} \mathbf{u}}{\|\mathbf{W}^{T} \mathbf{u}\|_2} \mathbf{u} := \frac{\mathbf{W}^{T} \mathbf{v}}{\|\mathbf{W}^{T} \mathbf{v}\|_2} Step 3: Calculate :math:`\sigma(\mathbf{W})` and normalize weight values. .. math:: \sigma(\mathbf{W}) = \mathbf{u}^{T} \mathbf{W} \mathbf{v} \mathbf{W} = \frac{\mathbf{W}}{\sigma(\mathbf{W})} Refer to `Spectral Normalization `_ . Parameters: weight_shape(list or tuple): The shape of weight parameter. dim(int, optional): The index of dimension which should be permuted to the first before reshaping Input(Weight) to matrix, it should be set as 0 if Input(Weight) is the weight of fc layer, and should be set as 1 if Input(Weight) is the weight of conv layer. Default: 0. power_iters(int, optional): The number of power iterations to calculate spectral norm. Default: 1. eps(float, optional): The epsilon for numerical stability in calculating norms. Default: 1e-12. name (str, optional): The default value is None. Normally there is no need for user to set this property. For more information, please refer to :ref:`api_guide_Name` . dtype (str, optional): Data type, it can be "float32" or "float64". Default: "float32". Returns: None Examples: .. code-block:: python import paddle x = paddle.rand((2,8,32,32)) spectral_norm = paddle.nn.SpectralNorm(x.shape, dim=1, power_iters=2) spectral_norm_out = spectral_norm(x) print(spectral_norm_out.shape) # [2, 8, 32, 32] rr -q=r5cstt|||_||_||_||_t||_t |jdks$Jd|t |jks2Jd ||j|j}t |j|}|j t|g|jtddd|_d|j_|j t|g|jtddd|_d|j_dS)Nrz5Any dimension of `weight_shape` cannot be equal to 0.zUThe input `dim` should be less than the length of `weight_shape`, but received dim={}rBrrKT)rRr1rS _power_iters_eps_dimrdrZ _weight_shaperrrformatrfrrweight_urweight_v)rGZ weight_shaperD power_itersepsrNhwrrrHrIrST s>   zSpectralNorm.__init__cCs~trt||j|j|j|j|jSt|dddgd||j|jd}|j |j }|j j d|d|i|j|j|jdd |S) Nrgr5rr1)r,UV spectral_normr)rDrwrxr) rrr}rurvrsrqrrrrrrdr)rGrgrrrHrHrIrx s&   zSpectralNorm.forward)rr rpr5rrHrHrrrIr1 s<$r1cs8eZdZdZ       d fdd Zd d ZZS) r2a This interface is used to construct a callable object of the ``TreeConv`` class. For more details, refer to code examples. Tree-Based Convolution is a kind of convolution based on tree structure. Tree-Based Convolution is a part of Tree-Based Convolution Neural Network(TBCNN), which is used to classify tree structures, such as Abstract Syntax Tree. Tree-Based Convolution proposed a kind of data structure called continuous binary tree, which regards multiway tree as binary tree. The paper of Tree-Based Convolution Operator is here: `tree-based convolution `_ . Parameters: feature_size(int): last dimension of nodes_vector. output_size(int): output feature width. num_filters(int, optional): number of filters, Default: 1. max_depth(int, optional): max depth of filters, Default: 2. act(str, optional): activation function, Default: tanh. param_attr(ParamAttr, optional): the parameter attribute for the filters, Default: None. bias_attr(ParamAttr, optional): the parameter attribute for the bias of this layer, Default: None. name(str, optional): The default value is None. Normally there is no need for user to set this property. For more information, please refer to :ref:`api_guide_Name` . dtype (str, optional): Data type, it can be "float32" or "float64". Default: "float32". Attribute: **weight** (Parameter): the learnable weights of filters of this layer. **bias** (Parameter or None): the learnable bias of this layer. Returns: None Examples: .. code-block:: python import paddle.fluid as fluid import numpy with fluid.dygraph.guard(): nodes_vector = numpy.random.random((1, 10, 5)).astype('float32') edge_set = numpy.random.random((1, 9, 2)).astype('int32') treeConv = fluid.dygraph.nn.TreeConv( feature_size=5, output_size=6, num_filters=1, max_depth=2) ret = treeConv(fluid.dygraph.base.to_variable(nodes_vector), fluid.dygraph.base.to_variable(edge_set)) r rr%Nr5c stt|||_||_||_||_||_||_||_ ||_ | |_ |jd|j|jg} |j r<|j |j |jg|j dd|_ |j |j | |j dd|_dS)NrTrPF)rRr2rSrWZ _feature_sizercrZ _max_depthrarcrbrdrfrhrg) rGZ feature_sizerbrk max_depthrprmrnrrNZw_shaperrrHrIrS s,  zTreeConv.__init__cCst|dtdt|dtd|jr|j|j|jdd}n|jj|jd}|jjd|||jdd |id |j id |j rY|jj|jd}|jjd |g|j gd d |giddid n|}|jj ||j dS)N nodes_vectorr2edge_setFr\rZ tree_conv)Z NodesVectorZEdgeSetrrrrrrrwr r)rr rWr_rdrrrrgr~rcrhrrZ)rGrrrrrHrHrIr s@ zTreeConv.forward)r rr%NNNr5rrHrHrrrIr2 s/r2cs*eZdZdZdfdd ZddZZS) r3a This interface is used to construct a callable object of the ``FLatten`` class. For more details, refer to code examples. It implements flatten a contiguous range of dims into a tensor. Parameters: start_axis(int): first dim to flatten (default = 1) stop_axis(int): last dim to flatten (default = -1). Returns: None Examples: .. code-block:: python import paddle import numpy as np inp_np = np.ones([5, 2, 3, 4]).astype('float32') inp_np = paddle.to_tensor(inp_np) flatten = paddle.nn.Flatten(start_axis=1, stop_axis=2) flatten_res = flatten(inp_np) r rvcstt|||_||_dSrL)rRr3rS start_axis stop_axis)rGrrrrrHrIrS s zFlatten.__init__cCstjjj||j|jd}|S)N)rr)rTZtensorZ manipulationflattenrrrVrHrHrIr s zFlatten.forward)r rvrrHrHrrrIr3 sr3)E __future__rrTZ six.movesrrr rrrrZ frameworkr r r rrrrrrrZ data_feederrrrrrmrrrrrrZlayer_object_helperrnumpyrrrosZpaddle.utils.deprecatedrarr __all__ZLayerr!r"r/r#r$r)r%r&r'r*r(r+r,r-r.rgrmr0r1r2r3rHrHrHrIsn       0    K]Ez |q#56|pq}^Otvo