o Me@sddlmZddlZddlZddlmZddlmZddlmZmZm Z m Z m Z ddl m Z ddlmZddlZdd lmZdd lmZdd lmZmZmZdd lmZmZddlZgd ZdadaGdddeZGdddeZGdddeZ GdddeZ!GdddeZ"GdddeZ#GdddeZ$GdddeZ%GdddeZ&d(d d!Z'd"d#Z(d$d%Z)d(d&d'Z*eZ+e Z,e!Z-e"Z.e#Z/e$Z0e%Z1dS)))print_functionN) framework)core)_non_static_modein_dygraph_mode_in_legacy_dygraphdefault_main_program_current_expected_place)lazy_init_helper) program_guard)VarDesc) unique_name)check_variable_and_dtype check_type check_dtype)_C_ops _legacy_C_ops)ConstantUniformNormalTruncatedNormalXavierBilinearMSRAConstantInitializerUniformInitializerNormalInitializerTruncatedNormalInitializerXavierInitializerBilinearInitializerMSRAInitializerNumpyArrayInitializerset_global_initializerc@sFeZdZdZddZdddZdddZdd d Zd d Zd dZ dS) InitializeraBase class for variable initializers Defines the common interface of variable initializers. They add operations to the init program that are used to initialize variables. Users should not use this class directly, but need to use one of its implementations. cCsdSNselfr&r&HD:\Projects\ConvertPro\env\Lib\site-packages\paddle/fluid/initializer.py__init__3szInitializer.__init__NcCs tjs |||S|||Sr%)r stateforward _lazy_initr(paramblockr&r&r)__call__6s  zInitializer.__call__cCst)zCAdd corresponding initialization operations to the network )NotImplementedErrorr.r&r&r)r,<szInitializer.forwardcCs>tsJdd}|t|j||t||j||_|S)z+ Apply lazy initialization cSsD|jd|d}t|||WddS1swYdS)NT)r0)Z_to_static_varr )r,r/r0Znew_varr&r&r)init_op_creatorGs "z/Initializer._lazy_init..init_op_creator)rZ set_init_func functoolspartialr,Z_init_op_creator)r(r/r0r3r&r&r)r-As zInitializer._lazy_initcCs|dur t}|Sr%)r Z global_block)r(r0r&r&r) _check_blockTs zInitializer._check_blockcCs|j}|r t|dkrd}}||fSt|dkr#|d}}||fSt|dkr5|d}|d}||fSt|dd}|d|}|d|}||fS)aCompute the fan_in and the fan_out for layers This method computes the fan_in and the fan_out for neural network layers, if not specified. It is not possible to perfectly estimate fan_in and fan_out. This method will estimate it correctly for matrix multiply and convolutions. Args: var: variable for which fan_in and fan_out have to be computed Returns: tuple of two integers (fan_in, fan_out) rrN)shapelennpprod)r(varr8fan_infan_outZreceptive_field_sizer&r&r) _compute_fansZs     zInitializer._compute_fansr%) __name__ __module__ __qualname____doc__r*r1r,r-r6r?r&r&r&r)r$*s    r$cs,eZdZdZd fdd Zd ddZZS) raImplements the constant initializer Args: value (float32): constant value to initialize the variable Examples: .. code-block:: python import paddle import paddle.fluid as fluid paddle.enable_static() x = fluid.data(name="data", shape=[8, 32, 32], dtype="float32") fc = fluid.layers.fc( input=x, size=10, param_attr=fluid.initializer.Constant(value=2.0)) Fcs*|dusJtt|||_||_dSr%)superrr*_value _force_cpu)r(value force_cpu __class__r&r)r*s  zConstantInitializer.__init__Nc Cs||}t|tjst|tjsJt|tjsJtr:t}|jr(t }t ||j tt|j|j|dStr[t|dt|jd|jdt|jdtt|jd|j dS|jdd|i|j t|jt|jtt|j|jd d d }||_|S) aKInitialize the input tensor with constant. Args: var(Tensor): Tensor that needs to be initialized. block(Block, optional): The block in which initialization ops should be added. Used in static graph only, default None. Returns: The initialization op NrHrIdtype str_valuer8 fill_constantOut)r8rLrHrMrITtypeoutputsattrs stop_gradient)r6 isinstancerVariableZEagerParamBaseBlockrr rGrZCPUPlacerZfull_r8strfloatrFrLrrrNint append_opop)r(r<r0placer\r&r&r)r,sB     zConstantInitializer.forward)rDFr%r@rArBrCr*r, __classcell__r&r&rJr)r~srcs8eZdZdZ      d fdd Zd dd ZZS) raImplements the random uniform distribution initializer Args: low (float): lower boundary of the uniform distribution high (float): upper boundary of the uniform distribution seed (int): random seed diag_num (int): the number of diagonal elements to initialize. If set to 0, diagonal initialization will be not performed. diag_step (int): Step size between two diagonal elements, which is generally the width of the square matrix. diag_val (float): the value of the diagonal element to be initialized, default 1.0. It takes effect only if the diag_num is greater than 0. Examples: .. code-block:: python import paddle.fluid as fluid x = fluid.data(name='x', shape=[None, 1], dtype='float32') fc = fluid.layers.fc(input=x, size=10, param_attr=fluid.initializer.Uniform(low=-0.5, high=0.5)) ?rcs|dusJ|dus J||ksJ|dusJ|dusJ|dus$J|dus*J|dks2|dkr<|dkr:|dks`_ by Xavier Glorot and Yoshua Bengio. This initializer is designed to keep the scale of the gradients approximately same in all the layers. In case of Uniform distribution, the range is [-x, x], where .. math:: x = \sqrt{\\frac{6.0}{fan\_in + fan\_out}} In case of Normal distribution, the mean is 0 and the standard deviation is .. math:: \sqrt{\\frac{2.0}{fan\_in + fan\_out}} Args: uniform (bool,default True): whether to use uniform ,if False use normal distribution fan_in (float,default None): fan_in for Xavier initialization. If None, it is inferred from the variable. fan_out (float,default None): fan_out for Xavier initialization. If None, it is inferred from the variable. seed (int): random seed Note: It is recommended to set fan_in and fan_out to None for most cases. Examples: .. code-block:: python import paddle.fluid as fluid queries = fluid.data(name='x', shape=[None,1], dtype='float32') fc = fluid.layers.fc( input=queries, size=10, param_attr=fluid.initializer.Xavier(uniform=False)) TNrcsB|dusJ|dus Jtt|||_||_||_||_dSr%)rErr*_uniform_fan_in_fan_outre)r(uniformr=r>rkrJr&r)r*0s   zXavierInitializer.__init__c Cs||}t|tjs Jt|dgdd||\}}|jdur$|n|j}|jdur.|n|j}|jdkr;|j j |_|j t j jksL|j t j jkrh|jsht j j}|jtdd|jdg|j|t j jdd }n|j }|}tr|jrtd t||} trt|j|| | |jt }nCt!rt"d |jd | d | d|jd| }n.tdt||} trt } t#|jd| |j|| }nt"#d |jd|ddd| d|j }|j t j jks|j t j jkr |js trt$||j } nt!rt"$|d|j d|j } | %|dS|%|dS|jr5td t||} |j&did|i|j|| | |jddd} ntdt||} |j&dd|i|j|j d| |jddd} |j t j jksh|j t j jkrz|jsz|j&dd|id|i|j |j dd | |_'| S)!aXInitialize the input tensor with Xavier initialization. Args: var(Tensor): Tensor that needs to be initialized. block(Block, optional): The block in which initialization ops should be added. Used in static graph only, default None. Returns: The initialization op rOroZ xavier_initNrrrrsFrtg@r8rvrwrkrL@rDrrrxryrqr8rLrvrwrkTrzrrrPr|r}r~r)(r6rUrrWrr?rrrerrrLr rrrrrrrrrrur8rrmathsqrtrYrrrqr rrrr|rr[r\)r(r<r0f_inf_outr=r>ryrlimitrr]rr\r&r&r)r,9s               zXavierInitializer.forward)TNNrr%r^r&r&rJr)rs, rcs6eZdZdZ     d fdd Zd dd ZZS) r!aImplements the MSRA initializer a.k.a. Kaiming Initializer This class implements the weight initialization from the paper `Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification `_ by Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun. This is a robust initialization method that particularly considers the rectifier nonlinearities. In case of Uniform distribution, the range is [-x, x], where .. math:: x = gain \times \sqrt{\frac{3}{fan\_in}} In case of Normal distribution, the mean is 0 and the standard deviation is .. math:: \frac{gain}{\sqrt{{fan\_in}}} Args: uniform (bool, optional): whether to use uniform or normal distribution fan_in (float32|None, optional): fan_in (in_features) of trainable Tensor, If None, it will be infered automaticly. If you don't want to use in_features of the Tensor, you can set the value of 'fan_in' smartly by yourself. default is None. seed (int32, optional): random seed. negative_slope (float, optional): negative_slope (only used with leaky_relu). default is 0.0. nonlinearity(str, optional): the non-linear function. default is relu. Note: It is recommended to set fan_in to None for most cases. Examples: .. code-block:: python import paddle import paddle.fluid as fluid paddle.enable_static() x = fluid.data(name="data", shape=[8, 32, 32], dtype="float32") fc = fluid.layers.fc(input=x, size=10, param_attr=fluid.initializer.MSRA(uniform=False)) TNrrelucsH|dusJ|dus Jtt|||_||_||_||_||_dS)z(Constructor for MSRAInitializer N)rEr!r*rrre_negative_slope _nonlinearity)r(rr=rkZnegative_slope nonlinearityrJr&r)r*s   zMSRAInitializer.__init__c Cs ||}t|tjs Jt|tjsJ||\}}|jdur#|n|j}|jdkr0|jj |_|j t j j ksA|j t j jkr]|js]t j j}|jtdd|jdg|j|t j jdd}n|j }|}tr|jrt|j|j}|tdt|} trt !|j|| | |jt"}nIt#!d |jd | d | d |jd t$| }n5t|j|j}|tt|} trt"} t %|jd| |j|| }nt#%d |jd t$|ddd| d |j }|j t j j ks|j t j jkr |js trt &||j } nt'rt#&|d|j d|j } | (|dS|(|dS|jr@t|j|j}|tdt|} |j)did|i|jt$|| | |jddd} n%t|j|j}|tt|} |j)dd|i|jt$|d| |jddd} |j t j j ksy|j t j jkr|js|j)dd|id|i|j |j dd| |_*| S)aVInitialize the input tensor with MSRA initialization. Args: var(Tensor): Tensor that needs to be initialized. block(Block, optional): The block in which initialization ops should be added. Used in static graph only, default None. Returns: The initialization op NrrrZ masra_initrsFrtg@r8rvrwrkrLrDrrrxryrqrOrTrzrrrPr|r}r~r)+r6rUrrVrWr?rrerrrLr rrrrrrrrrrur8rrcalculate_gainrrrrrYrrrqr rrZrr|rrr[r\)r(r<r0rrr=ryrZgainrrr]rr\r&r&r)r,s              zMSRAInitializer.forward)TNrrrr%r^r&r&rJr)r!s+r!c*eZdZdZfddZdddZZS)r a This initializer can be used in transposed convolution operator to act as upsampling. Users can upsample a feature map with shape of (B, C, H, W) by any integer factor. The usage is: Examples: .. code-block:: python import math import paddle import paddle.nn as nn from paddle.regularizer import L2Decay factor = 2 C = 2 B = 8 H = W = 32 w_attr = paddle.ParamAttr(learning_rate=0., regularizer=L2Decay(0.), initializer=nn.initializer.Bilinear()) data = paddle.rand([B, 3, H, W], dtype='float32') conv_up = nn.Conv2DTranspose(3, out_channels=C, kernel_size=2 * factor - factor % 2, padding=int( math.ceil((factor - 1) / 2.)), stride=factor, weight_attr=w_attr, bias_attr=False) x = conv_up(data) Where, `out_channels=C` and `groups=C` means this is channel-wise transposed convolution. The filter shape will be (C, 1, K, K) where K is `kernel_size`, This initializer will set a (K, K) interpolation kernel for every channel of the filter identically. The resulting shape of the output feature map will be (B, C, factor * H, factor * W). Note that the learning rate and the weight decay are set to 0 in order to keep coefficient values of bilinear interpolation unchanged during training. cstt|dS)z-Constructor for BilinearInitializer. N)rEr r*r'rJr&r)r*szBilinearInitializer.__init__Nc Cs||}t|tjstdt|tjstd|j}t|dkr&td|d|dkr2tdtj t |jdd }|d}t |d }d|d |dd |}t t |D]"}||} |||} d t | ||d t | ||||<q]t||}|jtjjtjjtjjfvrtjj} |jtd d |jdg|j| tjjdd} n|j} |} | tjjkrd} dd|jD}ntd|jt |dkrtdtr4trt !| t"|| |t#nt$rt%&| dt"|d| | ||jtjjtjjtjjfvr-trt '| |j}nt$r&t%'| d| jd|j}|(|dS| (|dS|j)dd| gid| dt"|| |id}|jtjjtjjtjjfvrh|j)dd | id|i| j|jd!d"||_*|S)#aZInitialize the input tensor with Bilinear initialization. Args: var(Tensor): Tensor that needs to be initialized. block(Block, optional): The block in which initialization ops should be added. Used in static graph only, default None. Returns: The initialization op zvar must be framework.Variable.zblock must be framework.Block.zthe length of shape must be 4.r7z#shape[2] must be equal to shape[3].rp)rLrrrrZ bilinear_initrsFrt fp32_valuescSg|]}t|qSr&rY.0vr&r&r) z/BilinearInitializer.forward..Unsupported dtype %sizThe size of input is too big. r8rLrxryN assign_valuerO)rQrRrSr|r}r~r)+r6rUrrV ValueErrorrWr8r9r:Zzerosr;ceilrangeabsZreshaperLr rrrZFP64rrrrrrurflat TypeErrorrrr assign_value_listr rrrr|rr[r\)r(r<r0r8weightsizefcixyryr value_namevaluesrr\r&r&r)r,s    .          zBilinearInitializer.forwardr%r^r&r&rJr)r as +r cr)r"a%Init an parameter with an numpy array This op initialize the variable by numpy array. Args: value (numpy): numpy array to initialize the variable Returns: A Tensor variable initialized by numpy. Examples: .. code-block:: python import paddle.fluid as fluid import numpy x = fluid.data(name="x", shape=[2, 1], dtype='float32') fc = fluid.layers.fc(input=x, size=10, param_attr=fluid.initializer.NumpyArrayInitializer(numpy.array([1,2]))) cs0ddl}t||js Jtt|||_dSrb)numpyrUZndarrayrEr"r*rF)r(rHrrJr&r)r*s zNumpyArrayInitializer.__init__Nc Cs||}t|tjs Jt|tjsJ|jtjjtjj fvrBtjj }|j d}|j tdd|jdg|j|tjjdd}n|}|j}|j }|tjj kr[d}dd |jD}n|tjjkrld }d d |jD}ntd |j j|j jd kr}tdtrtrt|t|j j||tntrt |dt|j jd||||jtjjtjj fvrtrt!||j}ntrt!|d|jd|j}|"|dS|"|dS|j#dd|id|dt|j j||idd} |jtjjtjj fvr |j#dd|id|i|j|jdd| |_$| S)aNInitialize the input tensor with Numpy array. Args: var(Tensor): Tensor that needs to be initialized. block(Block, optional): The block in which initialization ops should be added. Used in static graph only, default None. Returns: The initialization op rprrZnumpy_array_initrsFrtrcSrr&rrr&r&r)r9rz1NumpyArrayInitializer.forward..Z int32_valuescSrr&)rZrr&r&r)r<rri@zXThe size of input is too big. Please consider saving it to file and 'load_op' to load itr8rLrxryNrrOTrPr|r}r~r)%r6rUrrVrWrLr rrrrrFZastyperrrrrur8rrZINT32rrrrrrrr rrrr|rr[r\) r(r<r0ryZnp_valuerrrrr\r&r&r)r,s            zNumpyArrayInitializer.forwardr%r^r&r&rJr)r"s r"cCs8t|dttdfd|at|dttdfd|adS)a This API is used to set up global model parameter initializer in framework. After this API is invoked, the global initializer will takes effect in subsequent code. The model parameters include ``weight`` and ``bias`` . In the framework, they correspond to ``paddle.ParamAttr`` , which is inherited from ``paddle.Tensor`` , and is a persistable Variable. This API only takes effect for model parameters, not for variables created through apis such as :ref:`api_fluid_layers_create_global_var` , :ref:`api_fluid_layers_create_tensor`. If the initializer is also set up by ``param_attr`` or ``bias_attr`` when creating a network layer, the global initializer setting here will not take effect because it has a lower priority. If you want to cancel the global initializer in framework, please set global initializer to ``None`` . Args: weight_init (Initializer): set the global initializer for ``weight`` of model parameters. bias_init (Initializer, optional): set the global initializer for ``bias`` of model parameters. Default: None. Returns: None Examples: .. code-block:: python import paddle import paddle.nn as nn nn.initializer.set_global_initializer(nn.initializer.Uniform(), nn.initializer.Constant()) x_var = paddle.uniform((2, 4, 8, 8), dtype='float32', min=-1., max=1.) # The weight of conv1 is initialized by Uniform # The bias of conv1 is initialized by Constant conv1 = nn.Conv2D(4, 6, (3, 3)) y_var1 = conv1(x_var) # If set param_attr/bias_attr too, global initializer will not take effect # The weight of conv2 is initialized by Xavier # The bias of conv2 is initialized by Normal conv2 = nn.Conv2D(4, 6, (3, 3), weight_attr=nn.initializer.XavierUniform(), bias_attr=nn.initializer.Normal()) y_var2 = conv2(x_var) # Cancel the global initializer in framework, it will takes effect in subsequent code nn.initializer.set_global_initializer(None) weight_initNr# bias_init)rr$rQ_global_weight_initializer__global_bias_initializer_)rrr&r&r)r#ns2r#cCtSzP Return the global weight initializer, The user doesn't need to use it. )rr&r&r&r)_global_weight_initializerrcCrr)rr&r&r&r)_global_bias_initializerrrcCs|durd}nt|tttfsJt|}dddddddddtdtdd|ddd }||vr:||Std |) a Get the recommended ``gain`` value of some nonlinearity function. ``gain`` value can be used in some ``paddle.nn.initializer`` api to adjust the initialization value. Args: nonlinearity(str): name of nonlinearity activation function. If it is a linear function, such as: `linear/conv1d/conv2d/conv3d/conv1d_transpose/conv2d_transpose/conv3d_transpose` , 1.0 will be returned. param(bool|int|float, optional): optional parameter for somme nonlinearity function. Now, it only applies to 'leaky_relu'. Default: None, it will be calculated as 0.01 in the formula. Returns: A float value, which is the recommended gain for this nonlinearity function. Examples: .. code-block:: python import paddle gain = paddle.nn.initializer.calculate_gain('tanh') # 5.0 / 3 gain = paddle.nn.initializer.calculate_gain('leaky_relu', param=1.0) # 1.0 = math.sqrt(2.0 / (1+param^2)) initializer = paddle.nn.initializer.Orthogonal(gain) Ng{Gz?rg?rr7g?) ZsigmoidZlinearZconv1dZconv2dZconv3dZconv1d_transposeZconv2d_transposeZconv3d_transposetanhrZ leaky_reluZseluz/nonlinearity function {} is not suppported now.) rUboolrZrYrrkeysrformat)rr/Zrecommended_gainr&r&r)rs0 rr%)2 __future__rrr4rrrrrr r Z lazy_initr r rr:r rZ data_feederrrrZpaddlerr__all__rrobjectr$rrrrrr!r r"r#rrrrrrrrrrr&r&r&r)sR       TIPh-3 p= ;