o Me@sddlmZddlZddlZddlZddlZddlZddlmZddlm Z ddlm Z ddlm Z ddl m Zdd lmZdd l mZmZmZdd lmZdd l mZdd lmZmZgdZdaddZddZddZGdddeZGdddeZ ddZ!GdddeZ"Gddde"Z#Gd d!d!e"Z$da%d"d#Z&Gd$d%d%e"Z'e j(d,d&d'Z)d(d)Z*d*d+Z+e"Z,e#Z-e$Z.e'Z/dS)-)print_functionN)layers) framework)core) name_scope)base)check_variable_and_dtype)_non_static_modein_dygraph_mode_in_legacy_dygraph) LayerHelper)default_main_program)_C_ops _legacy_C_ops)set_gradient_clipErrorClipByValueClipGradByValueClipGradByNormClipGradByGlobalNormFcGsBt|dksJt|dkrt|dtsJt}|da|StS)Nrr)len isinstancebool'_clip_by_global_norm_using_mp_type_flagargs old_valuerAD:\Projects\ConvertPro\env\Lib\site-packages\paddle/fluid/clip.py"_clip_by_global_norm_using_mp_type*s rcCs:|jtjjjks|jtjjjkrtr|tjjjS|SN) dtyperVarDescVarTypeFP16BF16rastypeFP32xrrr_cast_to_mp_type_if_enabled6s r*cCst|}ts|jtjjjks|jtjjjkr$t |}t |}|St r,t |Str4t |Sd}t|dddg|t|fit}||j}d|i}d|i}|j|||d|S)z: This OP returns the squared L2 norm of a tensor. squared_l2_normr)float32float64XOuttypeinputsoutputs)r*rZis_compiled_with_xpur!r"r#r$r%rsquareZ reduce_sumr rr+r rr r localsZ"create_variable_for_type_inference append_op)r)r4 sum_squareZop_typehelperoutr2r3rrr_squared_l2_norm?s$      r:c@seZdZddZddZdS)BaseErrorClipAttrcCtr NotImplementedErrorselfrrr__str__]zBaseErrorClipAttr.__str__cCr<r r=)r@block grad_namerrr_append_clip_op`rBz!BaseErrorClipAttr._append_clip_opN)__name__ __module__ __qualname__rArErrrrr;[s r;c@s*eZdZdZd ddZddZddZdS) ra Clips tensor values to the range [min, max]. Given a tensor ``t`` (see Examples below), this operation clips its value \ to ``min`` and ``max`` inplace. - Any values less than min are set to min. - Any values greater than max are set to max. Args: max (float): The maximum value to clip by. min (float, optional): The minimum value to clip by. if not set by user, \ will be set to ``-max`` by framework. Examples: .. code-block:: python import paddle.fluid as fluid BATCH_SIZE = 128 CLIP_MAX = 2e-6 CLIP_MIN = -1e-6 prog = fluid.framework.Program() with fluid.program_guard(main_program=prog): image = fluid.layers.data( name='x', shape=[784], dtype='float32') hidden1 = fluid.layers.fc(input=image, size=128, act='relu') hidden2 = fluid.layers.fc(input=hidden1, size=64, act='relu') predict = fluid.layers.fc( input=hidden2, size=10, act='softmax') label = fluid.layers.data(name='y', shape=[1], dtype='int64') cost = fluid.layers.cross_entropy(input=predict, label=label) avg_cost = fluid.layers.mean(cost) prog_clip = prog.clone() prog_clip.block(0).var(hidden1.name)._set_error_clip( fluid.clip.ErrorClipByValue( max=CLIP_MAX, min=CLIP_MIN) NcCs0t|}|dur | }nt|}||_||_dSr )floatmaxminr@rJrKrrr__init__s  zErrorClipByValue.__init__cCd|j|jfS)NzByValue, min=%f, max=%frKrJr?rrrrAzErrorClipByValue.__str__cCsP|j}|d|d|g|d|g|d|j|d|jdS)Nclipr.r/rKrJ)descr6set_typeZ set_inputZ set_output _set_attrrKrJ)r@rCrDZ clip_op_descrrrrEs  z ErrorClipByValue._append_clip_opr )rFrGrH__doc__rMrArErrrrrds  & rcs||j|jd}fdd|DD]&}||}t|dd}|dus4t|ts4td|dur>| ||qdS)Nrcsg|]}|vr|qSrr).0nZ grad_to_varrr sz'error_clip_callback.. error_clipzIVariable's error_clip should be an instance of BaseErrorClipAttr or None.) rRopZop_sizeZoutput_arg_namesZ_var_recursivegetattrrr; TypeErrorrE)rCcontextZop_descZgrad_nZfwd_varrZrrXrerror_clip_callbacks  r_csReZdZfddZddZejddZddZd d Z d d Z d dZ Z S) ClipGradBasecstt|dSr )superr`rMr? __class__rrrMszClipGradBase.__init__cCr<r r=r?rrrrArBzClipGradBase.__str__cCtr r=r@ params_gradsrrr _dygraph_clipszClipGradBase._dygraph_clipcCrdr r=rerrr _static_clipzClipGradBase._static_clipcCsHtr ||S|D]\}}t|dddurtdnq ||S)Ngradient_clip_attrz'set_gradient_clip' will be ineffective, because you have set 'need_clip' in 'ParamAttr'. So, 'set_gradient_clip' is redundant and you can remove it.)rr rgr\warningswarnrh)r@rfpgrrr__call__s   zClipGradBase.__call__cCr<r r=r@r^paramgradrrr_process_contextrBzClipGradBase._process_contextcCr<r r=)r@rqrrrrr_create_operatorsrBzClipGradBase._create_operators) rFrGrHrMrAimperative_baseno_gradrgrhrorsrt __classcell__rrrbrr`s   r`csPeZdZdZdfdd ZddZejddZd d Z d d Z d dZ Z S)ra Limit the value of multi-dimensional Tensor :math:`X` to the range [min, max]. - Any values less than min are set to ``min``. - Any values greater than max are set to ``max``. The multi-dimensional Tensor :math:`X` is not passed from this class, but the gradients of all parameters set in ``optimizer``. If ``need_clip`` of specific param is ``False`` in its ``ParamAttr``, then the gradients of this param will not be clipped. Gradient clip will takes effect after being set in ``optimizer`` , see the document ``optimizer`` (for example: :ref:`api_paddle_optimizer_SGD`). Note: ``need_clip`` of ``ClipGradByValue`` HAS BEEN DEPRECATED since 2.0. Please use ``need_clip`` in ``ParamAttr`` to speficiy the clip scope. Args: max (float): The maximum value to clip by. min (float, optional): The minimum value to clip by. if not set by user, it will be set to ``-max`` automatically. In this case, ``max`` must be greater than 0. Examples: .. code-block:: python import paddle x = paddle.uniform([10, 10], min=-1.0, max=1.0, dtype='float32') linear = paddle.nn.Linear(in_features=10, out_features=10, weight_attr=paddle.ParamAttr(need_clip=True), bias_attr=paddle.ParamAttr(need_clip=False)) out = linear(x) loss = paddle.mean(out) loss.backward() clip = paddle.nn.ClipGradByValue(min=-1, max=1) sdg = paddle.optimizer.SGD(learning_rate=0.1, parameters=linear.parameters(), grad_clip=clip) sdg.step() Ncs@tt||dur|dksJ| }t||_t||_dS)Ng)rarrMrIrJrKrLrbrrrMs   zClipGradByValue.__init__cCrN)Nz(Clip Gradient By Value, min = %f, max=%frOr?rrrrArPzClipGradByValue.__str__cCsbg}|D]*\}}|dur qt|dddur|||fqtj||j|jd}|||fq|S)N need_clipTFr)rKrJ)r\appendpaddlerQrKrJr@rfparams_and_gradsrmrnnew_gradrrrrgs zClipGradByValue._dygraph_clipc Csg}t}tdT|D]I\}}|durq t|dddur&|||fq |jj||gtj ||j |j d}Wdn1sDwY|||f|j ||j <q Wdn1sawYt |||S)N gradient_cliprxTFry)dictrrr\rzrCprogram_optimized_guardrrQrKrJname_correct_clip_op_role_varr@rfr}param_new_grad_name_dictrmrnr~rrrrhs$   zClipGradByValue._static_clipcCdSr rrprrrrs&riz ClipGradByValue._process_contextcCstj||j|jd}||fS)Nry)rrQrKrJr@rqrrr~rrrrt)sz!ClipGradByValue._create_operatorsr rFrGrHrUrMrArurvrgrhrsrtrwrrrbrrs(  rcsNeZdZdZfddZddZejddZdd Z d d Z d d Z Z S)ra Limit the l2 norm of multi-dimensional Tensor :math:`X` to ``clip_norm`` . - If the l2 norm of :math:`X` is greater than ``clip_norm`` , :math:`X` will be compressed by a ratio. - If the l2 norm of :math:`X` is less than or equal to ``clip_norm`` , nothing will be done. The multidimensional Tensor :math:`X` is not passed from this class, but the gradients of all parameters set in ``optimizer``. If ``need_clip`` of specific param is ``False`` in its ``ParamAttr``, then the gradients of this param will not be clipped. Gradient clip will takes effect after being set in ``optimizer`` , see the document ``optimizer`` (for example: :ref:`api_paddle_optimizer_SGD`). The clipping formula is: .. math:: Out = \left\{ \begin{array}{ccl} X & & if (norm(X) \leq clip\_norm) \\ \frac{clip\_norm*X}{norm(X)} & & if (norm(X) > clip\_norm) \\ \end{array} \right. where :math:`norm(X)` represents the L2 norm of :math:`X`. .. math:: norm(X) = ( \sum_{i=1}^{n}|x\_i|^2)^{ \frac{1}{2}} Note: ``need_clip`` of ``ClipGradByNorm`` HAS BEEN DEPRECATED since 2.0. Please use ``need_clip`` in ``ParamAttr`` to speficiy the clip scope. Args: clip_norm(float): The maximum norm value. Examples: .. code-block:: python import paddle x = paddle.uniform([10, 10], min=-1.0, max=1.0, dtype='float32') linear = paddle.nn.Linear(in_features=10, out_features=10, weight_attr=paddle.ParamAttr(need_clip=True), bias_attr=paddle.ParamAttr(need_clip=False)) out = linear(x) loss = paddle.mean(out) loss.backward() clip = paddle.nn.ClipGradByNorm(clip_norm=1.0) sdg = paddle.optimizer.SGD(learning_rate=0.1, parameters=linear.parameters(), grad_clip=clip) sdg.step() cstt|t||_dSr )rarrMrI clip_norm)r@rrbrrrMfszClipGradByNorm.__init__cC d|jS)Nz#Gradient Clip By Norm, clip_norm=%frr?rrrrAj zClipGradByNorm.__str__cCs^g}|D](\}}|dur qt|dddur|||fqtj||jd}|||fq|S)NrxTFr)Zmax_norm)r\rzr clip_by_normrr|rrrrgms zClipGradByNorm._dygraph_clipc Csg}tdUt}|D]G\}}|durq t|dddur&|||fq |jj||gtj ||j d}Wdn1sBwY|j ||j <|||fq Wdn1s_wYt |||S)NrrxTFr) rrrr\rzrCrrrrrrrrrrrrhzs$    zClipGradByNorm._static_clipcCrr rrprrrrsrizClipGradByNorm._process_contextcCstj||jd}||fS)Nr)rrrrrrrrtsz ClipGradByNorm._create_operatorsrrrrbrr.s 7  rcGs>t|dkrtSt|dkrt|dtsJt}|da|S)Nrr)r&_allow_pure_fp16_global_norm_clip_flagrrrrrr!_allow_pure_fp16_global_norm_clips rcsTeZdZdZ  dfdd ZddZejdd Zd d Z d d Z ddZ Z S)ra Given a list of Tensor :math:`t\_list` , calculate the global norm for the elements of all tensors in :math:`t\_list` , and limit it to ``clip_norm`` . - If the global norm is greater than ``clip_norm`` , all elements of :math:`t\_list` will be compressed by a ratio. - If the global norm is less than or equal to ``clip_norm`` , nothing will be done. The list of Tensor :math:`t\_list` is not passed from this class, but the gradients of all parameters set in ``optimizer``. If ``need_clip`` of specific param is ``False`` in its ``ParamAttr``, then the gradients of this param will not be clipped. Gradient clip will takes effect after being set in ``optimizer`` , see the document ``optimizer`` (for example: :ref:`api_paddle_optimizer_SGD`). The clipping formula is: .. math:: t\_list[i] = t\_list[i] * \frac{clip\_norm}{\max(global\_norm, clip\_norm)} where: .. math:: global\_norm = \sqrt{\sum_{i=0}^{N-1}(l2norm(t\_list[i]))^2} Note: ``need_clip`` of ``ClipGradyGlobalNorm`` HAS BEEN DEPRECATED since 2.0. Please use ``need_clip`` in ``ParamAttr`` to speficiy the clip scope. Args: clip_norm (float): The maximum norm value. group_name (str, optional): The group name for this clip. Default value is ``default_group``. Examples: .. code-block:: python import paddle x = paddle.uniform([10, 10], min=-1.0, max=1.0, dtype='float32') linear = paddle.nn.Linear(in_features=10, out_features=10, weight_attr=paddle.ParamAttr(need_clip=True), bias_attr=paddle.ParamAttr(need_clip=False)) out = linear(x) loss = paddle.mean(out) loss.backward() clip = paddle.nn.ClipGradByGlobalNorm(clip_norm=1.0) sdg = paddle.optimizer.SGD(learning_rate=0.1, parameters=linear.parameters(), grad_clip=clip) sdg.step() default_groupFcs6tt|t||_||_t|tsJ||_dSr ) rarrMrIr group_namerrauto_skip_clip)r@rrrrbrrrMs   zClipGradByGlobalNorm.__init__cCr)Nz+Gradient Clip By GlobalNorm, global_norm=%frr?rrrrArzClipGradByGlobalNorm.__str__cCsg}g}g}g}|D]d\}}|durq t|dddurq |}tr/|r/t|}|}n|jtjj j krAt|}t |}t |} | j tjj jksU| j tjj jkr[|| q | j tjj jkri|| q || q t|t|t|dkr|St|dkrdnd} g} t|dkrt|} | | | t|dkrt|} | dkr| | n| | | t|dkrt|}| |t| } t| } tjdg| j |jd}d}|jsd}tj|tj| |d d }n| |krd}tj|| d }|D]F\}}|durqt|dddur!|||fq|rB|j |j kr1||j n|}t||}|||fq|||fq|S) NrxTFrr-r,rshaper!valuer)y)r\r Zis_selected_rowsrmerge_selected_rowsZ_get_tensor_from_selected_rowsr1rr"r# SELECTED_ROWSget_tensor_from_selected_rowsr:r!r$r%rzr'rr{Zadd_nr&sqrt fill_constantrrelementwise_divelementwise_maxelementwise_mul)r@rfr}sum_square_listsum_square_list_fp16sum_square_list_fp32rmrn merge_gradr7 sum_dtypeglobal_norm_varglobal_norm_var_fp16global_norm_var_fp32Zglobal_norm_var_fp64max_global_normrxclip_varZ clip_inputr~rrrrgs                        z"ClipGradByGlobalNorm._dygraph_clipc Csg}g}g}g}td|D]d\}}|durqt|dddur#q|}|jj||g?|jtjj j krAt |}t |}t|} | jtjj jkrS|| n| jtjj jkra|| n|| Wdn1spwYqt|t|t|dkr|WdS|jj||gt|dkrdnd} g} t|dkrt |} |s|sts| | | n| | t|dkrt |} | dkr| | n| | | t|dkrt |}| |t| dkrt | n| d} t j| d } t jdg| j|jd }t j|t j|| d d }Wdn 1s'wYt}|D]\}}|dur<q1t|dddurN|||fq1|jj||gNt|}|jtjj jkrs|jtjj jkrs|d n|}t}|j d ||dd|id||ur|j dd|id|i|j|jddWdn 1swY|j!||j!<|||fq1Wdn 1swYt"|||S)NrrxTFrr-r,rr(rrZfloat16rr.Yr/r0castr.)Zin_dtypeZ out_dtype)r1r2r3attrs)#rrr\rCrrr1rr"r#rrrrr:r!r$rzr'rsumsrr&rrrrrrr*rZ current_blockr6rr)r@rfr}rrrrmrnrr7rrrrZglobal_norm_var_other_dtyperZ scale_varrZnew_gZ scale_inputrCrrrrh8s               #       bz!ClipGradByGlobalNorm._static_clipcCs|j|vr#g||j<|j||jd<tjdg|j|jd||jd<n|j||jdks1td|}|jtjj j krEt |}t |}t |}||j|||_dS)NZ _clip_valuerr_clipz>All parameters' 'clip_norm' of a same group should be the same)rrrrr! ValueErrorr1rr"r#rrrr:rzr^)r@r^rqrrrZlocal_norm_varrrrrss"      z%ClipGradByGlobalNorm._process_contextcCs|jd}||jvr:tj|j|jd}tj|d}|j|jd}tj|tj||dd}|jdks5J||j|<|jj d||j|dd |id ||fS) NZ_scale)inputr(rr)rrrr/r0) rr^rrrrrrrCr6)r@rqrrZgroup_scale_nameZgroup_norm_varrZgroup_scale_varrrrrts(    z&ClipGradByGlobalNorm._create_operators)rFrrrrbrrs6  Sjrcstdt|tstddurtdjD]}d| vr2d| dvr2tdnq|dur>d }t dd |DrPfd d |D}t d d |Ds]td |D]}t ||_q_dS)a :api_attr: Static Graph Warning: This API must be used after building network, and before ``minimize`` , and it may be removed in future releases, so it is not recommended. It is recommended to set ``grad_clip`` when initializing the ``optimizer`` , this is a better method to clip gradient. There are three clipping strategies: :ref:`api_fluid_clip_GradientClipByGlobalNorm` , :ref:`api_fluid_clip_GradientClipByNorm` , :ref:`api_fluid_clip_GradientClipByValue` . To specify parameters that require gradient clip. Args: grad_clip (GradientClipBase, optional): Gradient cliping strategy, it's an instance of some derived class of ``GradientClipBase`` . There are three cliping strategies ( :ref:`api_fluid_clip_GradientClipByGlobalNorm` , :ref:`api_fluid_clip_GradientClipByNorm` , :ref:`api_fluid_clip_GradientClipByValue` ). Default value: None, and there is no gradient clipping. param_list (list(Variable), optional): Parameters that require gradient clip. It can be a list of parameter or a list of parameter's name. Default None, meaning that all parameters in the program will be included. program (Program, optional): The program where parameters are located. Default None, meaning that using :ref:`api_fluid_default_main_program` . Returns: None Examples: .. code-block:: python import paddle.fluid as fluid def network(): image = fluid.data(name='image', shape=[ None, 28], dtype='float32') param_attr1 = fluid.ParamAttr("fc1_param") fc1 = fluid.layers.fc(image, size=10, param_attr=param_attr1) param_attr2 = fluid.ParamAttr("fc2_param") fc2 = fluid.layers.fc(fc1, size=10, param_attr=param_attr2) loss = fluid.layers.reduce_mean(fc2) return loss # network 1: clip all parameter gradient with fluid.program_guard(fluid.Program(), fluid.Program()): loss = network() fluid.clip.set_gradient_clip( fluid.clip.GradientClipByGlobalNorm(clip_norm=2.0)) sgd = fluid.optimizer.SGD(learning_rate=1e-3) sgd.minimize(loss) # network 2: clip parameter gradient by name with fluid.program_guard(fluid.Program(), fluid.Program()): loss = network() fluid.clip.set_gradient_clip( fluid.clip.GradientClipByValue(min=-1.0, max=1.0), param_list=["fc1_param", "fc2_param"]) sgd = fluid.optimizer.SGD(learning_rate=1e-3) sgd.minimize(loss) # network 3: clip parameter gradient by value with fluid.program_guard(fluid.Program(), fluid.Program()): loss = network() param_var1 = fluid.default_main_program().global_block().var("fc1_param") param_var2 = fluid.default_main_program().global_block().var("fc2_param") fluid.clip.set_gradient_clip( fluid.clip.GradientClipByValue(min=-1.0, max=1.0), param_list=[param_var1, param_var2]) sgd = fluid.optimizer.SGD(learning_rate=1e-3) sgd.minimize(loss) # network 4: use 'set_gradient_clip' and 'optimize(grad_clip=clip)' together with fluid.program_guard(fluid.Program(), fluid.Program()): loss = network() clip1 = fluid.clip.GradientClipByValue(min=-1.0, max=1.0) clip2 = fluid.clip.GradientClipByNorm(clip_norm=1.0) # Set the gradient clipping strategy: clip1 fluid.clip.set_gradient_clip(clip1) # Set the gradient clipping strategy: clip2 sgd = fluid.optimizer.SGD(learning_rate=1e-3, grad_clip=clip2) sgd.minimize(loss) # 'set_gradient_clip' will not take effect when setting has a conflict, # and the gradient clipping strategy will be 'clip2' zCaution! 'set_gradient_clip' is not recommended and may be deprecated in future! We recommend a new strategy: set 'grad_clip' when initializing the 'optimizer'. This method can reduce the mistakes, please refer to documention of 'optimizer'.z<'clip' should be an instance of ClipGradBase's derived classNr op_namescopeZ optimizerz'minimize' has been invoked before, this will make 'set_gradient_clip' be ineffective! Please invoke 'set_gradient_clip' before 'minimize'.cs|] }t|tjVqdSr )rsix string_typesrVelemrrr Bz$set_gradient_clip..csg|] }d|qS)r)rCvarrrrrrYCsz%set_gradient_clip..csrr )rr ParameterrrrrrDrzK'param_list' should be a list of Parameter or basestring(parameter's name).)rkrlrr`r]rrrCopsZ all_attrsattrZall_parametersallcopydeepcopyrj)rQZ param_listrr[rqrrrrs6 Z rc Cst}|D]d\}}|durq|jj||gJtd5t|dd}|dur:|WdWdSt|tsCt d|j |||dWdn1sUwYWdn1sdwYqg}t}|D]M\}}|durzqq|jj||g3td|j ||d\}}|j ||j <| ||gWdn1swYWdn1swYqqt|||S)Nrrjz8clip attribute should be an instance of GradientClipBase)r^rqrr)rqrr)rrCrrrrr\rr`r]rsrtrrzr) Z param_gradsr^rmrnZ clip_attrresrrqr~rrrappend_gradient_clip_opsMsR  "     rc Csg}t|dkr dS|D]I\}}|durq |jj}||vrq |||jjjD]*}|drTd|dvrT|drT|dd}||vrT|||g}| d|q*q dS)NrrrZ op_role_var) rrCidxrzrZ global_blockrZhas_attrrrT) rfrZ block_id_listrqrrZblock_idr[ param_nameZ correct_p_grrrros0    r)NN)0 __future__rrrrk functoolsr{rrrrZdygraphrruZ data_feederr r r r Z layer_helperr rrr__all__rrr*r:objectr;rr_r`rrrrrZdygraph_not_supportrrrZGradientClipBaseZGradientClipByValueZGradientClipByNormZGradientClipByGlobalNormrrrrsP            <#[f 0 |"