o Qel@sddlZddlmZddlmZddlmZddlmZddlm Z dd l m Z m Z dd lm Z dd lmZdd lmZdd lmZddlmZddlmZddlmZmZddlZgZGdddeZdS)N) defaultdict) Optimizer) LRScheduler)core) framework)Variable Parameter) unique_name)layers) LayerHelper)GradientClipBase)base)Callable)_C_ops _legacy_C_opsc@seZdZdZdZdZdZdZ    d%d dZddZ ddZ ddZ ddZ ddZ ddZddZddZdd Zejejd!d"Zd#d$Zd S)&AdamWa The AdamW optimizer is implemented based on the AdamW Optimization in paper `DECOUPLED WEIGHT DECAY REGULARIZATION `_. it can resolves the problem of L2 regularization failure in the Adam optimizer. .. math:: t & = t + 1 moment\_1\_out & = {\beta}_1 * moment\_1 + (1 - {\beta}_1) * grad moemnt\_2\_out & = {\beta}_2 * moment\_2 + (1 - {\beta}_2) * grad * grad learning\_rate & = learning\_rate * \frac{\sqrt{1 - {\beta}_2^t}}{1 - {beta}_1^t} param\_out & = param - learning\_rate * (\frac{moment\_1}{\sqrt{moment\_2} + \epsilon} + \lambda * param) Args: learning_rate (float|LRScheduler, optional): The learning rate used to update ``Parameter``. It can be a float value or a LRScheduler. The default value is 0.001. parameters (list|tuple, optional): List/Tuple of ``Tensor`` names to update to minimize ``loss``. \ This parameter is required in dygraph mode. And you can specify different options for \ different parameter groups such as the learning rate, weight decay, etc, \ then the parameters are list of dict. Note that the learning_rate in paramter groups \ represents the scale of base learning_rate. \ The default value is None in static mode, at this time all parameters will be updated. beta1 (float|Tensor, optional): The exponential decay rate for the 1st moment estimates. It should be a float number or a Tensor with shape [1] and data type as float32. The default value is 0.9. beta2 (float|Tensor, optional): The exponential decay rate for the 2nd moment estimates. It should be a float number or a Tensor with shape [1] and data type as float32. The default value is 0.999. epsilon (float, optional): A small float value for numerical stability. The default value is 1e-08. weight_decay (float|Tensor, optional): The weight decay coefficient, it can be float or Tensor. The default value is 0.01. lr_ratio (function|None, optional): If it is not None, the learning rate will be updated with layerwise learning rate ratio. Otherwise, the learning rate is the original. Default: None. apply_decay_param_fun (function|None, optional): If it is not None, only tensors that makes apply_decay_param_fun(Tensor.name)==True will be updated with weight decay. It only works when we want to specify tensors. Default: None. grad_clip (GradientClipBase, optional): Gradient cliping strategy, it's an instance of some derived class of ``GradientClipBase`` . There are three cliping strategies ( :ref:`api_fluid_clip_GradientClipByGlobalNorm` , :ref:`api_fluid_clip_GradientClipByNorm` , :ref:`api_fluid_clip_GradientClipByValue` ). Default None, meaning there is no gradient clipping. lazy_mode (bool, optional): The official Adam algorithm has two moving-average accumulators. The accumulators are updated at every step. Every element of the two moving-average is updated in both dense mode and sparse mode. If the size of parameter is very large, then the update may be very slow. The lazy mode only update the element that has gradient in current mini-batch, so it will be much more faster. But this mode has different semantics with the original Adam algorithm and may lead to different result. The default value is False. multi_precision (bool, optional): Whether to use multi-precision during weight updating. Default is false. name (str, optional): Normally there is no need for user to set this property. For more information, please refer to :ref:`api_guide_Name`. The default value is None. **Notes**: **Currently, AdamW doesn't support sparse parameter optimization.** Examples: .. code-block:: python import paddle linear = paddle.nn.Linear(10, 10) inp = paddle.rand([10,10], dtype="float32") out = linear(inp) loss = paddle.mean(out) beta1 = paddle.to_tensor([0.9], dtype="float32") beta2 = paddle.to_tensor([0.99], dtype="float32") opt = paddle.optimizer.AdamW(learning_rate=0.1, parameters=linear.parameters(), beta1=beta1, beta2=beta2, weight_decay=0.01) out.backward() opt.step() opt.clear_grad() #Note that the learning_rate of linear_2 is 0.01. linear_1 = paddle.nn.Linear(10, 10) linear_2 = paddle.nn.Linear(10, 10) inp = paddle.uniform(shape=[10, 10], min=-0.1, max=0.1) out = linear_1(inp) out = linear_2(out) loss = paddle.mean(out) opt = paddle.optimizer.AdamW( learning_rate=0.1, parameters=[{ 'params': linear_1.parameters() }, { 'params': linear_2.parameters(), 'weight_decay': 0.001, 'learning_rate': 0.1, 'beta1': 0.8 }], weight_decay=0.01, beta1=0.9) out.backward() opt.step() opt.clear_grad() moment1moment2 beta1_pow_acc beta2_pow_accMbP??+?:0yE>N{Gz?Fc Cs|dusJ|dus J|dusJ|dusJd|kr%dks*tdtdd|kr7dks= 0.z'weight_decay should be float or Tensor.z/'lr_ratio' is unimplemented in CPU, XPU and NPUzt`parameters` argument given to the optimizer should be an iterable of paddle Tensors, but got argument type is `{}`.zv`parameters` argument should not get dict type, if parameter groups is needed, please set `parameters` as list of dictzNparameters argument given to the Optimizer should not be None in dygraph mode.z9learning rate should be float or LRScheduler, got %s herezE'grad_clip' should be an instance of GradientClipBase's derived classparamszYparams should be set in parameters if parameter groups are optimized in different optionscStSN)dictr!r!FD:\Projects\ConvertPro\env\Lib\site-packages\paddle/optimizer/adamw.pyz AdamW.__init__..adamw) weight_decaybeta1beta2epsilon lazy_mode grad_clip)6 ValueError isinstancefloatrr TypeErrorrrZis_compiled_with_cudaNotImplementedErrorpaddleZTensoreagerformattyper list_parameter_list_name_non_static_modeAttributeErrorrrZ_dtypedtypeZ_learning_rate_mapr _accumulatorshelperZ_opti_name_listZ_accumulators_holderZ_param_device_mapZ clear_gradZclear_gradientsZ_learning_rateset _params_name_apply_decay_param_fun _weight_decayZ _grad_clip _lr_ratio_beta1_beta2_epsilon _lazy_mode_multi_precision_master_weights _default_dict _param_groups_add_param_groupcopyZ_use_multi_tensorregularization_auxiliary_vars)self learning_rater'r(r) parametersr&lr_ratioZapply_decay_param_funr+r*multi_precisionname param_groupr!r!r"__init__s               zAdamW.__init__cCs||j|<dSrrM)rNkeyvalr!r!r"_set_auxiliary_varszAdamW._set_auxiliary_varcCs||jvr |j|SdSrrV)rNrWr!r!r"_get_auxiliary_vars  zAdamW._get_auxiliary_varcCs|d}t|tr|g|d<nt|trtdt||d<|jD] \}}|||q#t}|jD] }| t|dq4| t|dsMt d|dD] }| dd|j d<qQ|j|dS)z Add a param group to parameter_list. Args: param_group (dict): The group of Tensors to be optimzed with different optimization options. rz`optimizer parameters should be in ordered collections,but received set, please use list instead.z7some parameters appear in more than one parameter grouprO?N)r-r r=r/r5rHitems setdefaultrIupdate isdisjointr,getZ optimize_attrappend)rNrTrkvZ param_setgroupparamr!r!r"rJs,       zAdamW._add_param_groupcCs|j|jvr|j|j}|St|jtsJ|jd}t|}tj||j dddd}|jj }|j dd|gid|gi|j tjjjd d ||j|j<|S) NZ _fp32_masterrZfloat32T)rSshapevaluer:Z persistablecastXZOut)Zin_dtypeZ out_dtype)r4inputsoutputsattrs)rSrGr-r<r r generater Zcreate_global_varrfstartup_programZ global_block append_opr:rVarDescVarTypeFP32)rNrevarvar_nameblockr!r!r"_create_master_weightBs,      zAdamW._create_master_weightcCs~|jdur |jd|}|jo|jtjjjk}|r|j|jn|}|j}||j vs0||j |vr8t d |||j ||S)a Utility function to fetch an accumulator for a parameter Args: name: name of the accumulator param: parameter variable for which accumulator is to be fetched Returns: accumulator variable for the parameter N_z.Accumulator {} does not exist for parameter {}) r7rFr:rrprqFP16rGrSr; Exceptionr3)rNrSre find_masterZ target_param target_namer!r!r"_get_accumulatorZs$  zAdamW._get_accumulatorc Cs|j}|tjjjkrtjjj}|j|j||d|j|j||d|j|j ||t |j t r/dn|j dgtjjj dd|j|j||t |jt rIdn|jdgtjjj dddS)N)r:rrcpu)rSrer:Z fill_valuerfr4Zdevicer)r:rrprqrxrrZ_add_accumulator_moment1_acc_str_moment2_acc_str_beta1_pow_acc_strr-rBr Z LOD_TENSOR_beta2_pow_acc_strrC)rNpZ acc_dtyper!r!r"_add_moments_powsos.    zAdamW._add_moments_powscCst|tjsJt|tr||}|D]-}|jr,|jtjj j kr,| |}| |q|jtjj j kr<|js)rNr!r!r"__str__sz AdamW.__str__cCst|jdtsXg}|jD]@}|jrq |durM|}tr4t|dr3|r3|j dur3t dnt|drF| rF|j durFt d| ||fq |j dd|d}dS|jD]c}tdd}|d D]B}|jrmqg|dur|}trt|dr|r|j durt dnt|dr| r|j durt d|d  ||fqg|d d |D|j dd|dq[dS) a Execute the optimizer and update parameters once. Returns: None Examples: .. code-block:: python import paddle a = paddle.rand([2,13], dtype="float32") linear = paddle.nn.Linear(13, 5) # This can be any optimizer supported by dygraph. opt = paddle.optimizer.AdamW(learning_rate = 0.01, parameters = linear.parameters()) out = linear(a) out.backward() opt.step() opt.clear_grad() rNis_selected_rowszOAdamW don't support weight_decay with sparse parameters, please set it to None. _is_sparse)Zlossrn params_gradscSrr)r5r!r!r!r"r#Hr$zAdamW.step..rcSsi|] \}}|dkr||qS)rr!).0rbrcr!r!r" ^szAdamW.step..)r-r6r rZ _grad_ivarrrhasattrrrL RuntimeErrorrraZ_apply_optimizerIrr^r\)rNrreZgrad_varZ optimize_opsrTr!r!r"steps            z AdamW.stepcCsr|d|jd|_|d|jd|_|d|jd|_|d|jd|_|d|jd|_|d}|S)Nr'r(r)r*r&r)r`rHrBrCrDrEr@)rNrPr!r!r"rds zAdamW._update_param_group) rrrrNrNNNFFN)__name__ __module__ __qualname____doc__r~rrrrUrYrZrJrvr|rrrrimperative_baseZno_gradrZ dygraph_onlyrrr!r!r!r"r"s@o ~$x  Or)r collectionsrZ optimizerrrrZfluidrrZfluid.frameworkr r r r Zfluid.layer_helperr Z fluid.cliprZ fluid.dygraphrrcollections.abcrrrr1__all__rr!r!r!r"s"