o Meo@sDddlmZddlmZddlmZgZddZGdddeZdS) ) AmpScaler)OptimizerState) defaultdictcCs dtjiS)Nstate)rINITrrFD:\Projects\ConvertPro\env\Lib\site-packages\paddle/amp/grad_scaler.py_refresh_optimizer_states r cs eZdZdZ       d0fdd Zfd d Zfd d ZddZddZfddZ fddZ fddZ fddZ fddZ fddZfddZfd d!Zfd"d#Zfd$d%Zfd&d'Zfd(d)Zfd*d+Zfd,d-Zfd.d/ZZS)1 GradScaleraa GradScaler is used for Auto-Mixed-Precision training in dynamic graph mode. It controls the scaling of loss, helps avoiding numerical overflow. The object of this class has nineteen methods `scale()`, `unscale_()`, `minimize()`, `step()`, `update()` and `get`/`set` api of parameters. `scale()` is used to multiply the loss by a scale ratio. `unscale_()` is used to unscale the gradients of parameters, multiplies the gradients of parameters by 1/(scale ratio) `minimize()` is similar as `optimizer.minimize()`, performs parameters updating, and it will update the loss_scaling, it equal to `step()` + `update()`. `step()` is similar as `optimizer.step()`, which performs parameters updating. `update` is used to update the loss_scaling. Commonly, it is used together with `paddle.amp.auto_cast` to achieve Auto-Mixed-Precision in dynamic graph mode. Args: enable(bool, optional): Enable loss scaling or not. Default is True. init_loss_scaling (float, optional): The initial loss scaling factor. Default is 2**15. incr_ratio(float, optional): The multiplier to use when increasing the loss scaling. Default is 2.0. decr_ratio(float, optional): The less-than-one-multiplier to use when decreasing the loss scaling. Default is 0.5. incr_every_n_steps(int, optional): Increases loss scaling every n consecutive steps with finite gradients. Default is 1000. decr_every_n_nan_or_inf(int, optional): Decreases loss scaling every n accumulated steps with nan or inf gradients. Default is 2. use_dynamic_loss_scaling(bool, optional): Whether to use dynamic loss scaling. If False, fixed loss_scaling is used. If True, the loss scaling is updated dynamicly. Default is True. Returns: An GradScaler object. Examples: .. code-block:: python import paddle model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) scaler = paddle.amp.GradScaler(init_loss_scaling=1024) data = paddle.rand([10, 3, 32, 32]) with paddle.amp.auto_cast(): conv = model(data) loss = paddle.mean(conv) scaled = scaler.scale(loss) # scale the loss scaled.backward() # do backward scaler.minimize(optimizer, scaled) # update parameters optimizer.clear_grad() T@@?c s tt||||||||dS)N)superr __init__)selfenableZinit_loss_scalingZ incr_ratioZ decr_ratioZincr_every_n_stepsZdecr_every_n_nan_or_infZuse_dynamic_loss_scaling __class__rrrNs zGradScaler.__init__ctt||S)a7 Multiplies a Tensor by the scale factor and returns scaled outputs. If this instance of :class:`GradScaler` is not enabled, output are returned unmodified. Args: var (Tensor): The tensor to scale. Returns: The scaled tensor or original tensor. Examples: .. code-block:: python import paddle model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) scaler = paddle.amp.GradScaler(init_loss_scaling=1024) data = paddle.rand([10, 3, 32, 32]) with paddle.amp.auto_cast(): conv = model(data) loss = paddle.mean(conv) scaled = scaler.scale(loss) # scale the loss scaled.backward() # do backward scaler.minimize(optimizer, scaled) # update parameters optimizer.clear_grad() )rr scale)rvarrrrr[szGradScaler.scalecstt|j|g|Ri|S)a This function is similar as `optimizer.minimize()`, which performs parameters updating. If the scaled gradients of parameters contains NAN or INF, the parameters updating is skipped. Otherwise, if `unscale_()` has not been called, it first unscales the scaled gradients of parameters, then updates the parameters. Finally, the loss scaling ratio is updated. Args: optimizer(Optimizer): The optimizer used to update parameters. args: Arguments, which will be forward to `optimizer.minimize()`. kwargs: Keyword arguments, which will be forward to `optimizer.minimize()`. Examples: .. code-block:: python import paddle model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) scaler = paddle.amp.GradScaler(init_loss_scaling=1024) data = paddle.rand([10, 3, 32, 32]) with paddle.amp.auto_cast(): conv = model(data) loss = paddle.mean(conv) scaled = scaler.scale(loss) # scale the loss scaled.backward() # do backward scaler.minimize(optimizer, scaled) # update parameters optimizer.clear_grad() )rr minimize)r optimizerargskwargsrrrr{s"zGradScaler.minimizecCs|js|S|jt|}|dtjurtd|dtjur%|||j r,d|_ n|d|_ tj|d<|j sBt t |_dSdS)a This function is similar as `optimizer.step()`, which performs parameters updating. If the scaled gradients of parameters contains NAN or INF, the parameters updating is skipped. Otherwise, if `unscale_()` has not been called, it first unscales the scaled gradients of parameters, then updates the parameters. Args: optimizer(Optimizer): The optimizer used to update parameters. Examples: .. code-block:: python # required: gpu import paddle model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) scaler = paddle.amp.GradScaler(init_loss_scaling=1024) data = paddle.rand([10, 3, 32, 32]) with paddle.amp.auto_cast(): conv = model(data) loss = paddle.mean(conv) scaled = scaler.scale(loss) # scale the loss scaled.backward() # do backward scaler.step(optimizer) # update parameters scaler.update() # update the loss scaling ratio optimizer.clear_grad() rz7step() has already been called since the last update().TFN)_enablestep_optimizer_statesidrZSTEPPED RuntimeErrorr_unscaleZ _found_infZ_cache_founf_inf_use_dynamic_loss_scalingrr )rrZoptimizer_staterrrrs"  zGradScaler.stepcCs&|jsdS|jr|tt|_dS)a Updates the loss_scaling. Examples: .. code-block:: python # required: gpu import paddle model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) scaler = paddle.amp.GradScaler(init_loss_scaling=1024) data = paddle.rand([10, 3, 32, 32]) with paddle.amp.auto_cast(): conv = model(data) loss = paddle.mean(conv) scaled = scaler.scale(loss) # scale the loss scaled.backward() # do backward scaler.step(optimizer) # update parameters scaler.update() # update the loss scaling ratio optimizer.clear_grad() N)rr#_updaterr rrrrrupdates  zGradScaler.updatecr)a Unscale the gradients of parameters, multiplies the gradients of parameters by 1/(loss scaling ratio). If this instance of :class:`GradScaler` is not enabled, output are returned unmodified. Args: optimizer(Optimizer): The optimizer used to update parameters. Returns: The unscaled parameters or original parameters. Examples: .. code-block:: python # required: gpu import paddle model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) scaler = paddle.amp.GradScaler(init_loss_scaling=1024) data = paddle.rand([10, 3, 32, 32]) with paddle.amp.auto_cast(): conv = model(data) loss = paddle.mean(conv) scaled = scaler.scale(loss) # scale the loss scaled.backward() # do backward scaler.unscale_(optimizer) # unscale the parameter scaler.step(optimizer) scaler.update() optimizer.clear_grad() )rr r")rrrrrunscale_s zGradScaler.unscale_ctt|S)a7 Enable loss scaling or not. Returns: bool: enable loss scaling return True else return False. Examples: .. code-block:: python # required: gpu,xpu import paddle scaler = paddle.amp.GradScaler(enable=True, init_loss_scaling=1024, incr_ratio=2.0, decr_ratio=0.5, incr_every_n_steps=1000, decr_every_n_nan_or_inf=2, use_dynamic_loss_scaling=True) enable = scaler.is_enable() print(enable) # True )rr is_enabler%rrrr)zGradScaler.is_enablecr()a Whether to use dynamic loss scaling. Returns: bool: if fixed loss_scaling is used return False, if the loss scaling is updated dynamicly return true. Examples: .. code-block:: python # required: gpu,xpu import paddle scaler = paddle.amp.GradScaler(enable=True, init_loss_scaling=1024, incr_ratio=2.0, decr_ratio=0.5, incr_every_n_steps=1000, decr_every_n_nan_or_inf=2, use_dynamic_loss_scaling=True) use_dynamic_loss_scaling = scaler.is_use_dynamic_loss_scaling() print(use_dynamic_loss_scaling) # True )rr is_use_dynamic_loss_scalingr%rrrr+-r*z&GradScaler.is_use_dynamic_loss_scalingcr()aT Return the initial loss scaling factor. Reurns: float: the initial loss scaling factor. Examples: .. code-block:: python # required: gpu,xpu import paddle scaler = paddle.amp.GradScaler(enable=True, init_loss_scaling=1024, incr_ratio=2.0, decr_ratio=0.5, incr_every_n_steps=1000, decr_every_n_nan_or_inf=2, use_dynamic_loss_scaling=True) init_loss_scaling = scaler.get_init_loss_scaling() print(init_loss_scaling) # 1024 )rr get_init_loss_scalingr%rrrr,Er*z GradScaler.get_init_loss_scalingctt||dS)a. Set the initial loss scaling factor by `new_init_loss_scaling`. Args: new_init_loss_scaling(float): The new_init_loss_scaling used to update initial loss scaling factor. Examples: .. code-block:: python # required: gpu,xpu import paddle scaler = paddle.amp.GradScaler(enable=True, init_loss_scaling=1024, incr_ratio=2.0, decr_ratio=0.5, incr_every_n_steps=1000, decr_every_n_nan_or_inf=2, use_dynamic_loss_scaling=True) print(scaler.get_init_loss_scaling()) # 1024 new_init_loss_scaling = 1000 scaler.set_init_loss_scaling(new_init_loss_scaling) print(scaler.get_init_loss_scaling()) # 1000 N)rr set_init_loss_scaling)rZnew_init_loss_scalingrrrr.]z GradScaler.set_init_loss_scalingcr()al Return the multiplier to use when increasing the loss scaling. Reurns: float: the multiplier to use when increasing the loss scaling. Examples: .. code-block:: python # required: gpu,xpu import paddle scaler = paddle.amp.GradScaler(enable=True, init_loss_scaling=1024, incr_ratio=2.0, decr_ratio=0.5, incr_every_n_steps=1000, decr_every_n_nan_or_inf=2, use_dynamic_loss_scaling=True) incr_ratio = scaler.get_incr_ratio() print(incr_ratio) # 2.0 )rr get_incr_ratior%rrrr0wr*zGradScaler.get_incr_ratiocr-)a4 Set the multiplier to use when increasing the loss scaling by `new_incr_ratio`, `new_incr_ratio` should > 1.0. Args: new_incr_ratio(float): The new_incr_ratio used to update the multiplier to use when increasing the loss scaling. Examples: .. code-block:: python # required: gpu,xpu import paddle scaler = paddle.amp.GradScaler(enable=True, init_loss_scaling=1024, incr_ratio=2.0, decr_ratio=0.5, incr_every_n_steps=1000, decr_every_n_nan_or_inf=2, use_dynamic_loss_scaling=True) print(scaler.get_incr_ratio()) # 2.0 new_incr_ratio = 3.0 scaler.set_incr_ratio(new_incr_ratio) print(scaler.get_incr_ratio()) # 3.0 N)rr set_incr_ratio)rZnew_incr_ratiorrrr1r/zGradScaler.set_incr_ratiocr()a Get the less-than-one-multiplier to use when decreasing the loss scaling. Reurns: float: the less-than-one-multiplier to use when decreasing the loss scaling. Examples: .. code-block:: python # required: gpu,xpu import paddle scaler = paddle.amp.GradScaler(enable=True, init_loss_scaling=1024, incr_ratio=2.0, decr_ratio=0.5, incr_every_n_steps=1000, decr_every_n_nan_or_inf=2, use_dynamic_loss_scaling=True) decr_ratio = scaler.get_decr_ratio() print(decr_ratio) # 0.5 )rr get_decr_ratior%rrrr2r*zGradScaler.get_decr_ratiocr-)aP Set the less-than-one-multiplier to use when decreasing the loss scaling by `new_incr_ratio`, `new_decr_ratio` should < 1.0. Args: new_decr_ratio(float): The new_decr_ratio used to update the less-than-one-multiplier to use when decreasing the loss scaling. Examples: .. code-block:: python # required: gpu,xpu import paddle scaler = paddle.amp.GradScaler(enable=True, init_loss_scaling=1024, incr_ratio=2.0, decr_ratio=0.5, incr_every_n_steps=1000, decr_every_n_nan_or_inf=2, use_dynamic_loss_scaling=True) print(scaler.get_decr_ratio()) # 0.5 new_decr_ratio = 0.1 scaler.set_decr_ratio(new_decr_ratio) print(scaler.get_decr_ratio()) # 0.1 N)rr set_decr_ratio)rZnew_decr_ratiorrrr3r/zGradScaler.set_decr_ratiocr()a Return the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Reurns: int: the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Examples: .. code-block:: python # required: gpu,xpu import paddle scaler = paddle.amp.GradScaler(enable=True, init_loss_scaling=1024, incr_ratio=2.0, decr_ratio=0.5, incr_every_n_steps=1000, decr_every_n_nan_or_inf=2, use_dynamic_loss_scaling=True) incr_every_n_steps = scaler.get_incr_every_n_steps() print(incr_every_n_steps) # 1000 )rr get_incr_every_n_stepsr%rrrr4r*z!GradScaler.get_incr_every_n_stepscr-)a Set the num `n` by `new_incr_every_n_steps`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Args: new_incr_every_n_steps(int): The new_incr_every_n_steps used to update the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Examples: .. code-block:: python # required: gpu,xpu import paddle scaler = paddle.amp.GradScaler(enable=True, init_loss_scaling=1024, incr_ratio=2.0, decr_ratio=0.5, incr_every_n_steps=1000, decr_every_n_nan_or_inf=2, use_dynamic_loss_scaling=True) print(scaler.get_incr_every_n_steps()) # 1000 new_incr_every_n_steps = 2000 scaler.set_incr_every_n_steps(new_incr_every_n_steps) print(scaler.get_incr_every_n_steps()) # 2000 N)rr set_incr_every_n_steps)rZnew_incr_every_n_stepsrrrr5r/z!GradScaler.set_incr_every_n_stepscr()a Return the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Reurns: int: the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Examples: .. code-block:: python # required: gpu,xpu import paddle scaler = paddle.amp.GradScaler(enable=True, init_loss_scaling=1024, incr_ratio=2.0, decr_ratio=0.5, incr_every_n_steps=1000, decr_every_n_nan_or_inf=2, use_dynamic_loss_scaling=True) decr_every_n_nan_or_inf = scaler.get_decr_every_n_nan_or_inf() print(decr_every_n_nan_or_inf) # 2 )rr get_decr_every_n_nan_or_infr%rrrr6 r*z&GradScaler.get_decr_every_n_nan_or_infcr-)a Set the num `n` by `new_decr_every_n_nan_or_inf`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Args: new_decr_every_n_nan_or_inf(int): The new_decr_every_n_nan_or_inf used to update the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Examples: .. code-block:: python # required: gpu,xpu import paddle scaler = paddle.amp.GradScaler(enable=True, init_loss_scaling=1024, incr_ratio=2.0, decr_ratio=0.5, incr_every_n_steps=1000, decr_every_n_nan_or_inf=2, use_dynamic_loss_scaling=True) print(scaler.get_decr_every_n_nan_or_inf()) # 2 new_decr_every_n_nan_or_inf = 3 scaler.set_decr_every_n_nan_or_inf(new_decr_every_n_nan_or_inf) print(scaler.get_decr_every_n_nan_or_inf()) # 3 N)rr set_decr_every_n_nan_or_inf)rZnew_decr_every_n_nan_or_infrrrr7%s z&GradScaler.set_decr_every_n_nan_or_infcr()an Returns the state of the scaler as a `dict`, If this instance is not enabled, returns an empty dict. Reurns: A dict of scaler includes: scale (tensor): The loss scaling factor. incr_ratio(float): The multiplier to use when increasing the loss scaling. decr_ratio(float): The less-than-one-multiplier to use when decreasing the loss scaling. incr_every_n_steps(int): Increases loss scaling every n consecutive steps with finite gradients. decr_every_n_nan_or_inf(int): Decreases loss scaling every n accumulated steps with nan or inf gradients. incr_count(int): The number of recent consecutive unskipped steps. decr_count(int): The number of recent consecutive skipped steps. use_dynamic_loss_scaling(bool): Whether to use dynamic loss scaling. If False, fixed loss_scaling is used. If True, the loss scaling is updated dynamicly. Default is True. Examples: .. code-block:: python # required: gpu,xpu import paddle scaler = paddle.amp.GradScaler(enable=True, init_loss_scaling=1024, incr_ratio=2.0, decr_ratio=0.5, incr_every_n_steps=1000, decr_every_n_nan_or_inf=2, use_dynamic_loss_scaling=True) scaler_state = scaler.state_dict() )rr state_dictr%rrrr8@s zGradScaler.state_dictcr-)a Loads the scaler state. Args: state_dict(dict): scaler state. Should be an object returned from a call to `GradScaler.state_dict()`. Examples: .. code-block:: python # required: gpu,xpu import paddle scaler = paddle.amp.GradScaler(enable=True, init_loss_scaling=1024, incr_ratio=2.0, decr_ratio=0.5, incr_every_n_steps=1000, decr_every_n_nan_or_inf=2, use_dynamic_loss_scaling=True) scaler_state = scaler.state_dict() scaler.load_state_dict(scaler_state) N)rr load_state_dict)rr8rrrr9br/zGradScaler.load_state_dict)Tr r r rrT)__name__ __module__ __qualname____doc__rrrrr&r'r)r+r,r.r0r1r2r3r4r5r6r7r8r9 __classcell__rrrrr s:4 $5  "            "r N)Zpaddle.fluid.dygraph.amprr collectionsr__all__r r rrrrs