o Ne%M@shddlmZddlmZmZmZmZmZmZm Z m Z ddl Z ddgZ GdddeZ Gd ddeZdS) )Layer)sigmoidtanhconcat fill_constantmatmulelementwise_addelementwise_mulsplitNLSTMCellGRUCellcs8eZdZdZ       d fdd Zdd ZZS) r a LSTMCell implementation using basic operators. There are two LSTMCell version, the default one is compatible with CUDNN LSTM implementation. The algorithm can be described as the equations below. .. math:: i_t &= sigmoid(W_{ix}x_{t} + W_{ih}h_{t-1} + bx_i + bh_i) f_t &= sigmoid(W_{fx}x_{t} + W_{fh}h_{t-1} + bx_f + bh_f) o_t &= sigmoid(W_{ox}x_{t} + W_{oh}h_{t-1} + bx_o + bh_o) \\tilde{c_t} &= tanh(W_{cx}x_t + W_{ch}h_{t-1} + bx_c + bh_c) c_t &= f_t \\odot c_{t-1} + i_t \\odot \\tilde{c_t} h_t &= o_t \\odot tanh(c_t) The other LSTMCell version is compatible with the BasicLSTMUnit used in static graph. The algorithm can be described as the equations below. .. math:: i_t &= sigmoid(W_{ix}x_{t} + W_{ih}h_{t-1} + b_i) f_t &= sigmoid(W_{fx}x_{t} + W_{fh}h_{t-1} + b_f + forget_bias ) o_t &= sigmoid(W_{ox}x_{t} + W_{oh}h_{t-1} + b_o) \\tilde{c_t} &= tanh(W_{cx}x_t + W_{ch}h_{t-1} + b_c) c_t &= f_t \\odot c_{t-1} + i_t \\odot \\tilde{c_t} h_t &= o_t \\odot tanh(c_t) Args: hidden_size (integer): The hidden size used in the Cell. input_size (integer): The input size used in the Cell. param_attr(ParamAttr|None): The parameter attribute for the learnable weight matrix. Note: If it is set to None or one attribute of ParamAttr, LSTMCell will create ParamAttr as param_attr. If the Initializer of the param_attr is not set, the parameter is initialized with Xavier. Default: None. bias_attr (ParamAttr|None): The parameter attribute for the bias of LSTMCell. If it is set to None or one attribute of ParamAttr, LSTMCell will create ParamAttr as bias_attr. If the Initializer of the bias_attr is not set, the bias is initialized as zero. Default: None. gate_activation (function|None): The activation function for gates (actGate). Default: 'fluid.layers.sigmoid' activation (function|None): The activation function for cells (actNode). Default: 'fluid.layers.tanh' forget_bias(float|1.0): forget bias used when computing forget gate. This is not used in default LSTMCell implementation (CUDNN compatiable) use_cudnn_impl(bool|True): whether to use CUDNN compatible LSTMCell dtype(string): data type used in this cell Returns: None Examples: .. code-block:: python from paddle import fluid import paddle.fluid.core as core from paddle.fluid.dygraph import LSTMCell import numpy as np batch_size = 64 input_size = 128 hidden_size = 256 step_input_np = np.random.uniform(-0.1, 0.1, ( batch_size, input_size)).astype('float64') pre_hidden_np = np.random.uniform(-0.1, 0.1, ( batch_size, hidden_size)).astype('float64') pre_cell_np = np.random.uniform(-0.1, 0.1, ( batch_size, hidden_size)).astype('float64') if core.is_compiled_with_cuda(): place = core.CUDAPlace(0) else: place = core.CPUPlace() with fluid.dygraph.guard(place): cudnn_lstm = LSTMCell(hidden_size, input_size) step_input_var = fluid.dygraph.to_variable(step_input_np) pre_hidden_var = fluid.dygraph.to_variable(pre_hidden_np) pre_cell_var = fluid.dygraph.to_variable(pre_cell_np) new_hidden, new_cell = cudnn_lstm(step_input_var, pre_hidden_var, pre_cell_var) N?Tfloat64c stt|| ||_||_||_||_| |_|pt|_ |pt |_ ||_ |j r|jdurM|jj durMt|j} t|j} | j d7_ | j d7_ n|j} |j} |jdury|jj duryt|j} t|j} | j d7_ | j d7_ n|j} |j} |j| d|j|jg|jd|_|j| d|j|jg|jd|_|j| d|jg|jdd|_|j| d|jg|jdd|_dStd g| |d |_d |j_|j|j|j|jd|jg| d|_|j|jd|jg| dd|_dS) N _weight_ih _weight_hh_bias_ih_bias_hhattrshapedtypeTrrrZis_biasr)rvalueF)superr __init__ _hidden_size _input_size _param_attr _bias_attr_dtyper_gate_activationr _activation_use_cudnn_implnamecopydeepcopycreate_parameterrrrrr _forget_biasZ stop_gradient_weight_bias)self hidden_size input_size param_attr bias_attrgate_activation activationZ forget_biasuse_cudnn_implrweight_ih_param_attrweight_hh_param_attrbias_ih_param_attrbias_hh_param_attr __class__HD:\Projects\ConvertPro\env\Lib\site-packages\paddle/fluid/dygraph/rnn.pyrrsx            zLSTMCell.__init__cCsv|jrxt||jdd}t||j}t||jdd}t||j}t|ddd}t|ddd}t|d|d}||}t|d|d} || } t|d|d} | | } t|d |d } || } | ||| } | | | } | | fSt ||gd}t||j d }t||j }t|dd d\}}}}tt ||t||jt t|t|} | | ||} | | fS) NTy transpose_yr?rrZnum_or_sectionsdimr rxr>)r%rrr rrrr r#r$rr+r,r r*rr)r-input pre_hiddenZpre_celligateshgateschunked_igateschunked_hgatesZingateZ forgetgateZcellgateZoutgateZnew_cell new_hiddenconcat_input_hidden gate_inputijfor;r;r<forwardsB       zLSTMCell.forward)NNNNrTr__name__ __module__ __qualname____doc__rrT __classcell__r;r;r9r<r s^Ocs6eZdZdZ      d fdd ZddZZS) raY GRU implementation using basic operators. There are two GRUCell version, the default one is compatible with CUDNN GRU implementation. The algorithm can be described as the equations below. .. math:: u_t & = sigmoid(W_{ux} x_{t} + b_ux + W_{uh} h_{t-1} + b_uh) r_t & = sigmoid(W_{rx} x_{t} + b_rx + W_{rh} h_{t-1} + b_rh) \\tilde{h_{t}} & = tanh(W_{cx} x_{t} + b_cx + r_t \\odot (W_{ch} h_{t-1} + b_ch)) h_t & = u_t h_{t-1} + (1-u_t) \\tilde{h_{t}} The other LSTMCell version is compatible with the BasicGRUUnit used in static graph. The algorithm can be described as the equations below. .. math:: u_t & = sigmoid(W_{ux} x_{t} + W_{uh} h_{t-1} + b_u) r_t & = sigmoid(W_{rx} x_{t} + W_{rh} h_{t-1} + b_r) \\tilde{h_{t}} & = tanh(W_{cx} x_{t} + W_{ch} \\odot(r_t, h_{t-1}) + b_m) h_t & = u_t h_{t-1} + (1-u_t) \\tilde{h_{t}} Args: hidden_size (integer): The hidden size used in the Cell. input_size (integer): The input size used in the Cell. param_attr(ParamAttr|None): The parameter attribute for the learnable weight matrix. Note: If it is set to None or one attribute of ParamAttr, GRUCell will create ParamAttr as param_attr. If the Initializer of the param_attr is not set, the parameter is initialized with Xavier. Default: None. bias_attr (ParamAttr|None): The parameter attribute for the bias of GRUCell. If it is set to None or one attribute of ParamAttr, GRUCell will create ParamAttr as bias_attr. If the Initializer of the bias_attr is not set, the bias is initialized zero. Default: None. gate_activation (function|None): The activation function for gates (actGate). Default: 'fluid.layers.sigmoid' activation (function|None): The activation function for cell (actNode). Default: 'fluid.layers.tanh' use_cudnn_impl(bool|True): whether to use CUDNN compatible LSTMCell dtype(string): data type used in this cell Returns: None Examples: .. code-block:: python from paddle import fluid import paddle.fluid.core as core from paddle.fluid.dygraph import GRUCell import numpy as np batch_size = 64 input_size = 128 hidden_size = 256 step_input_np = np.random.uniform(-0.1, 0.1, ( batch_size, input_size)).astype('float64') pre_hidden_np = np.random.uniform(-0.1, 0.1, ( batch_size, hidden_size)).astype('float64') if core.is_compiled_with_cuda(): place = core.CUDAPlace(0) else: place = core.CPUPlace() with fluid.dygraph.guard(place): cudnn_gru = GRUCell(hidden_size, input_size) step_input_var = fluid.dygraph.to_variable(step_input_np) pre_hidden_var = fluid.dygraph.to_variable(pre_hidden_np) NTrc stt|||_||_||_||_||_|pt|_ |pt |_ ||_ |j r|jdurL|jj durLt|j} t|j} | j d7_ | j d7_ n|j} |j} |jdurx|jj durxt|j} t|j} | j d7_ | j d7_ n|j} |j} |j| d|j|jg|jd|_|j| d|j|jg|jd|_|j| d|jg|jdd|_|j| d|jg|jdd|_dS|jdur|jj durt|j} t|j}| j d 7_ |j d 7_ n|j} |j}|jdur|jj durt|j}t|j}|j d 7_ |j d 7_ n|j}|j}|j| |j|jd |jg|d|_|j||j|j|jg|d|_|j|d |jg|dd|_|j||jg|dd|_dS)NrrrrrCrTr _gate_weight_candidate_weight _gate_bias_candidate_biasr)rrrrrr r!r"rr#rr$r%r&r'r(r)rrrrr[r\r]r^)r-r.r/r0r1r2r3r4rr5r6r7r8Zgate_weight_param_attrZcandidate_weight_param_attrZgate_bias_param_attrZcandidate_bias_param_attrr9r;r<r;s                zGRUCell.__init__cCsZ|jret||jdd}t||j}t||jdd}t||j}t|ddd}t|ddd}t|d|d}||}t|d|d}||}||d} t|d| } | | } || || } | St ||gd} t| |j d } t| |j } || } t| ddd\}}||}tt ||gd|j }t||j}| |}||d||} | S) NTr=r@rCrrAr rrD)r%rrr rrrr r#r$rr[r]r\r^)r-rGrHrIrJrKrLZ reset_gateZ input_gate_tempZnew_gaterMrNrOruZr_hidden candidatecr;r;r<rTs:          zGRUCell.forward)NNNNTrrUr;r;r9r<rsPf)rZlayersrrrrrr r r r'__all__r rr;r;r;r<s (X