o Ne-@sddlmZddlZddlmZddlmZmZmZddlmZddl m Z m Z gdZ Gd d d e ZGd d d eZGd ddeZeZeZdS))print_functionN) framework)_non_static_mode_varbase_creatorin_dygraph_mode)core)_C_ops _legacy_C_ops)L1DecayL2DecayL1DecayRegularizerL2DecayRegularizerc@s(eZdZdZddZddZddZdS) WeightDecayRegularizeraBase class for weight decay regularizers Defines the common interface of weight-decay regularizers. Weight-decay regularizers are added only during the backward pass for faster regularization. They add operations to the network that correspond to gradient of the regularization function. Users should not use this class directly, but need to use one of its implementations cCsdSNselfrrHD:\Projects\ConvertPro\env\Lib\site-packages\paddle/fluid/regularizer.py__init__%szWeightDecayRegularizer.__init__cCt)zAAdd corresponding weight decay operations to the network NotImplementedError)rparamgradblockrrr__call__(zWeightDecayRegularizer.__call__cCr)zDebug string rrrrr__str__-rzWeightDecayRegularizer.__str__N)__name__ __module__ __qualname____doc__rrrrrrrrs   rc2eZdZdZd fdd ZddZddZZS) ra: Implement the L2 Weight Decay Regularization, which helps to prevent the model over-fitting. It can be set in :ref:`api_fluid_ParamAttr` or ``optimizer`` (such as :ref:`api_fluid_optimizer_SGDOptimizer` ). When set in ``ParamAttr`` , it only takes effect for trainable parameters in this layer. When set in ``optimizer`` , it takes effect for all trainable parameters. When set together, ``ParamAttr`` has higher priority than ``optimizer`` . In the implementation, the formula of L2 Weight Decay Regularization is as follows: .. math:: L2WeightDecay = reg\_coeff * parameter Args: regularization_coeff(float, optional): regularization coeff. Default:0.0 Examples: .. code-block:: python # Example1: set Regularizer in optimizer import paddle.fluid as fluid main_prog = fluid.Program() startup_prog = fluid.Program() with fluid.program_guard(main_prog, startup_prog): data = fluid.layers.data(name='image', shape=[3, 28, 28], dtype='float32') label = fluid.layers.data(name='label', shape=[1], dtype='int64') hidden = fluid.layers.fc(input=data, size=128, act='relu') prediction = fluid.layers.fc(input=hidden, size=10, act='softmax') loss = fluid.layers.cross_entropy(input=prediction, label=label) avg_loss = fluid.layers.mean(loss) optimizer = fluid.optimizer.Adagrad( learning_rate=1e-4, regularization=fluid.regularizer.L2Decay( regularization_coeff=0.1)) optimizer.minimize(avg_loss) # Example2: set Regularizer both in ParamAttr and optimizer import paddle.fluid as fluid l1 = fluid.regularizer.L1Decay(regularization_coeff=0.1) l2 = fluid.regularizer.L2Decay(regularization_coeff=0.1) x = fluid.layers.uniform_random([3,4]) # set L1 regularization in fluid.ParamAttr w_param = fluid.ParamAttr(regularizer=l1) hidden1 = fluid.layers.fc(x, 8, param_attr=w_param) # fc_0.w_0(L1), fc_0.b_0 hidden2 = fluid.layers.fc(hidden1, 16, param_attr=w_param) # fc_1.w_0(L1), fc_1.b_0 predict = fluid.layers.fc(hidden2, 32) # fc_3.w_0, fc_3.b_0 avg_loss = fluid.layers.mean(predict) # set L2 regularization in optimizer optimizer = fluid.optimizer.SGD(learning_rate=1e-4, regularization=l2) optimizer.minimize(avg_loss) # it will Print Message: # Regularization of [fc_0.w_0, fc_1.w_0] have been set by ParamAttr or WeightNormParamAttr already. # So, the Regularization of Optimizer will not take effect for these parameters! c$|dusJtt|||_dSr)superrr_regularization_coeffrZregularization_coeff __class__rrrs  zL2DecayRegularizer.__init__cCst|tjsJt|tjsJtr)tr!t||jddSt |d|jS|j |j |j |j d}|jdd|id|id|jid|S)aMAdd L2 weight decay ops to network Adds L2 weight decay ops. L2WeightDecay = reg_coeff * parameter Args: param: parameter variable for which regularization is applied block: block in which variable is to be created Returns: new variable for weight decay r$Tscaledtypeshape lod_levelXOuttypeinputsoutputsattrs) isinstancerVariableBlockrrr r,r'r create_varr.r/r0 append_op)rrrrdecayrrrrxs(  zL2DecayRegularizer.__call__cC d|jS)Nz L2Decay, regularization_coeff=%fr'rrrrr zL2DecayRegularizer.__str__r$rr r!r"rrr __classcell__rrr)rr3s ?$rcr#) r aL Implement the L1 Weight Decay Regularization, which encourages the weights to be sparse. It can be set in :ref:`api_fluid_ParamAttr` or ``optimizer`` (such as :ref:`api_fluid_optimizer_SGDOptimizer` ). When set in ``ParamAttr`` , it only takes effect for trainable parameters in this layer. When set in ``optimizer`` , it takes effect for all trainable parameters. When set together, ``ParamAttr`` has higher priority than ``optimizer`` . In the implementation, the formula of L1 Weight Decay Regularization is as follows: .. math:: L1WeightDecay = reg\_coeff * sign(parameter) Args: regularization_coeff(float, optional): regularization coeff. Default:0.0. Examples: .. code-block:: python # Example1: set Regularizer in optimizer import paddle.fluid as fluid main_prog = fluid.Program() startup_prog = fluid.Program() with fluid.program_guard(main_prog, startup_prog): data = fluid.layers.data(name='image', shape=[3, 28, 28], dtype='float32') label = fluid.layers.data(name='label', shape=[1], dtype='int64') hidden = fluid.layers.fc(input=data, size=128, act='relu') prediction = fluid.layers.fc(input=hidden, size=10, act='softmax') loss = fluid.layers.cross_entropy(input=prediction, label=label) avg_loss = fluid.layers.mean(loss) optimizer = fluid.optimizer.Adagrad( learning_rate=1e-4, regularization=fluid.regularizer.L1DecayRegularizer( regularization_coeff=0.1)) optimizer.minimize(avg_loss) # Example2: set Regularizer both in ParamAttr and optimizer import paddle.fluid as fluid l1 = fluid.regularizer.L1Decay(regularization_coeff=0.1) l2 = fluid.regularizer.L2Decay(regularization_coeff=0.1) x = fluid.layers.uniform_random([3,4]) # set L1 regularization in fluid.ParamAttr w_param = fluid.ParamAttr(regularizer=l1) hidden1 = fluid.layers.fc(x, 8, param_attr=w_param) # fc_0.w_0(L1), fc_0.b_0 hidden2 = fluid.layers.fc(hidden1, 16, param_attr=w_param) # fc_1.w_0(L1), fc_1.b_0 predict = fluid.layers.fc(hidden2, 32) # fc_3.w_0, fc_3.b_0 avg_loss = fluid.layers.mean(predict) # set L2 regularization in optimizer optimizer = fluid.optimizer.SGD(learning_rate=1e-4, regularization=l2) optimizer.minimize(avg_loss) # it will Print Message: # Regularization of [fc_0.w_0, fc_1.w_0] have been set by ParamAttr or WeightNormParamAttr already. # So, the Regularization of Optimizer will not take effect for these parameters! r$cr%r)r&r rr'r(r)rrrr+zL1DecayRegularizer.__init__cCst|tjsJt|tjsJtr'|j|j|jd}|j|j|jd}n|j|j|j|jd}|j|j|j|jd}t rNt |}t ||j ddS|jdd|id|id|jd d|id|id |j id |S) aSAdd L1 weight decay ops to network Adds L1 weight decay ops. L1WeightDecay = reg_coeff * sign(parameter) Args: param: parameter variable for which regularization is applied block: block in which variable is to be created Returns: new variable for weight decay )r.r/r-r$Tsignr1r2)r4r5r6r,r3)r8rr9r:rr;r.r/r0rr rDr,r'r<)rrrrrDr=rrrrs.  zL1DecayRegularizer.__call__cCr>)Nz L1Decay, regularization_coeff=%fr?rrrrrr@zL1DecayRegularizer.__str__rArBrrr)rr s ?)r ) __future__rloggingrrrrrZpaddler r __all__objectrrr r r rrrrs   m{