o Qe@sddlZddlZddlZddlZddlmZmZddlm Z ddl m Z ddl m Z dd lmZdd l mZmZdd lmZdd lmZgZd dZddZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeZdS)N)LinearDropout) LayerNorm) functional)tensorlayers)Layer LayerList) ParamAttr) convert_dtypecCst|ttfr:t||ksJd|g}|D] }t|tr/|r)|tdq|dq|t|q|St|trUg}|rNddt|D}|Sdg|}|Sg}t|}t|D]}t |}|j rt|j dt ||_ ||q`|S)a If `param_attr` is a list or tuple, convert every element in it to a ParamAttr instance. Otherwise, repeat `param_attr` `n` times to construct a list, and rename every one by appending a increasing index suffix to avoid having same names when `param_attr` contains a name. Parameters: param_attr (list|tuple|ParamAttr): A list, tuple or something can be converted to a ParamAttr instance by `ParamAttr._to_attr`. n (int): The times to repeat to construct a list when `param_attr` is not a list or tuple. Returns: list: A list composed of each including cell's `param_attr`. z9length of param_attr should be %d when it is a list/tupleNFcSsg|]}tdqSN)r_to_attr.0irKD:\Projects\ConvertPro\env\Lib\site-packages\paddle/nn/layer/transformer.py Bz/_convert_param_attr_to_list.._) isinstancelisttuplelenboolappendrrrangecopydeepcopynamestr)Z param_attrnZ param_attrsattrrZattr_irrr_convert_param_attr_to_list"s6         r'cCsT|dur(|j|kr(t|j}|dksd|vr"t||dd}|St||}|S)a* Convert the attention mask to the target dtype we expect. Parameters: attn_mask (Tensor, optional): A tensor used in multi-head attention to prevents attention to some unwanted positions, usually the paddings or the subsequent positions. It is a tensor with shape broadcasted to `[batch_size, n_head, sequence_length, sequence_length]`. When the data type is bool, the unwanted positions have `False` values and the others have `True` values. When the data type is int, the unwanted positions have 0 values and the others have 1 values. When the data type is float, the unwanted positions have `-INF` values and the others have 0 values. It can be None when nothing wanted or needed to be prevented attention to. Default None. dtype (VarType): The target type of `attn_mask` we expect. Returns: Tensor: A Tensor with shape same as input `attn_mask`, with data type `dtype`. Nrintg?geA)dtyperpaddlecast) attn_maskr)Zattn_mask_dtyperrr_convert_attention_maskPs  r-csxeZdZdZedddgZedddgZ      dfd d Zdd d Z d dZ defddZ dddZ Z S)MultiHeadAttentiona Attention mapps queries and a set of key-value pairs to outputs, and Multi-Head Attention performs multiple parallel attention to jointly attending to information from different representation subspaces. Please refer to `Attention Is All You Need `_ for more details. Parameters: embed_dim (int): The expected feature size in the input and output. num_heads (int): The number of heads in multi-head attention. dropout (float, optional): The dropout probability used on attention weights to drop some attention targets. 0 for no dropout. Default 0 kdim (int, optional): The feature size in key. If None, assumed equal to `embed_dim`. Default None. vdim (int, optional): The feature size in value. If None, assumed equal to `embed_dim`. Default None. need_weights (bool, optional): Indicate whether to return the attention weights. Default False. weight_attr(ParamAttr, optional): To specify the weight parameter property. Default: None, which means the default weight parameter property is used. See usage for details in :code:`ParamAttr` . bias_attr (ParamAttr|bool, optional): To specify the bias parameter property. Default: None, which means the default bias parameter property is used. If it is set to False, this layer will not have trainable bias parameter. See usage for details in :code:`ParamAttr` . Examples: .. code-block:: python import paddle # encoder input: [batch_size, sequence_length, d_model] query = paddle.rand((2, 4, 128)) # self attention mask: [batch_size, num_heads, query_len, query_len] attn_mask = paddle.rand((2, 2, 4, 4)) multi_head_attn = paddle.nn.MultiHeadAttention(128, 2) output = multi_head_attn(query, None, None, attn_mask=attn_mask) # [2, 4, 128] Cachekv StaticCacheNFc stt||dksJd||dksJd|||_|dur&|n||_|dur/|n||_||_||_||_ |||_ |j ||jksLJdt ||||d|_ t |j|||d|_ t |j|||d|_t ||||d|_dS)Nrz8Expected embed_dim to be greater than 0, but received {}z8Expected num_heads to be greater than 0, but received {}z(embed_dim must be divisible by num_heads bias_attr)superr.__init__format embed_dimkdimvdim num_headsdropout need_weightshead_dimrq_projk_projv_projout_proj) selfr9r<r=r:r;r> weight_attrr5 __class__rrr7sF    zMultiHeadAttention.__init__cCs||}tj|dd|j|jgd}tj|gdd}t||jr)|j|j }}n| ||\}}t||j rQtj |j|gdd}tj |j |gdd}| ||}|durZ|||fS||||fS)a Prapares linear projected queries, keys and values for usage of subsequnt multiple parallel attention. If `cache` is not None, using cached results to reduce redundant calculations. Parameters: query (Tensor): The queries for multi-head attention. It is a tensor with shape `[batch_size, query_length, embed_dim]`. The data type should be float32 or float64. key (Tensor): The keys for multi-head attention. It is a tensor with shape `[batch_size, key_length, kdim]`. The data type should be float32 or float64. If None, use `query` as `key`. value (Tensor): The values for multi-head attention. It is a tensor with shape `[batch_size, value_length, vdim]`. The data type should be float32 or float64. If None, use `query` as `value`. cache (MultiHeadAttention.Cache|MultiHeadAttention.StaticCache, optional): It is a namedtuple with `k` and `v` as fields, and stores tensors shaped `[batch_size, num_heads, length, embed_dim]` which are results of linear projection, reshape and transpose calculations in MultiHeadAttention. If is an instance of `Cache`, `k` and `v` fields reserve intermediate results of previous positions, which mostly used for decoder self attention. If it is an instance of `StaticCache`, `key` and `value` args would be ignored, `k` and `v` fields would be used as calculated results on `key` and `value`, which mostly used for decoder-encoder cross attention. It is only used for inference and should be None for training. Default None. Returns: tuple: A tuple including linear projected keys and values. These two \ tensors have shapes `[batch_size, n_head, sequence_length, d_key]` \ and `[batch_size, n_head, sequence_length, d_value]` separately, \ and their data types are same as inputs. rxshaperrrrrIpermr)ZaxisN) r@r reshaper<r? transposerr2r0r1 compute_kvr/concat)rDquerykeyvaluecacheqr0r1rrr _prepare_qkvs %   zMultiHeadAttention._prepare_qkvcCst||}||}tj|dd|j|jgd}tj|gdd}tj|dd|j|jgd}tj|gdd}||fS)a Applies linear projection on input keys and values, then splits heads (reshape and transpose) to get keys and values from different representation subspaces. The results are used as key-values pairs for subsequent multiple parallel attention. It is part of calculations in multi-head attention, and is provided as a method to pre-compute and prefetch these results, thus we can use them to construct cache for inference. Parameters: key (Tensor): The keys for multi-head attention. It is a tensor with shape `[batch_size, sequence_length, kdim]`. The data type should be float32 or float64. value (Tensor): The values for multi-head attention. It is a tensor with shape `[batch_size, sequence_length, vdim]`. The data type should be float32 or float64. Returns: tuple: A tuple including transformed keys and values. Their shapes \ both are `[batch_size, num_heads, sequence_length, embed_dim // num_heads]`, \ and their data types are same as inputs. rrHrKrL)rArBr rNr<r?rO)rDrSrTr0r1rrrrPs  zMultiHeadAttention.compute_kvcCs|tjkr|||\}}|||S|dur=tj|d|jd|jg|jdd}tj|d|jd|jg|jdd}|||S|||S)a Generates cache for `forward` usage in inference accroding to arguments. The generated cache is an instance of `MultiHeadAttention.Cache` or an instance of `MultiHeadAttention.StaticCache`. `Cache` or `StaticCache` is namedtuple with `k` and `v` as fields, and it stores tensors shaped `[batch_size, num_heads, length, embed_dim]` which are results of linear projection, reshape and transpose calculations in MultiHeadAttention. If the generated cache is an instance of `Cache`, `k` and `v` fields reserve intermediate result tensors of previous positions, and the tensors are incremental among decoding steps, which mostly are used for decoder decoder self attention. If the generated cache is an instance of `StaticCache`, `k` and `v` fields would be used as calculated result tensors on keys an values in `forward`, and the tensors keep unchanged among decoding steps, which are mostly used for decoder-encoder cross attention. The cache is generated as follows: 1. If `type` is `StaticCache`, apply `compute_kv(key, value)` and use the results to create an instance of `StaticCache`. 2. If `type` is `Cache` and `value` is None, generate empty tensors shaped `[batch_size, num_heads, 0, embed_dim // num_heads]` and use the results to create an instance of `Cache`, where `batch_size` is from the first dimension of `key`. 3. If `type` is `Cache` and `value` is not None, use `key`, `value` to create an instance of `Cache`. Parameters: key (Tensor): The keys for multi-head attention. It is a tensor with shape `[batch_size, key_length, kdim]`. The data type should be float32 or float64. If `value` is None, it is only for batch size and data type reference. value (Tensor, optional): The values for multi-head attention. It is a tensor with shape `[batch_size, value_length, vdim]`. The data type should be float32 or float64. If None, `key` is only for batch size reference. Default None. type (type): It should be `MultiHeadAttention.StaticCache` or `MultiHeadAttention.Cache` to indicate the cache type to generate. Returns: namedtuple: an instance of `Cache` or `StaticCache` accordingly. Nr)inputrJr)rT) r.r2rPr Zfill_constant_batch_size_liker<r?r)r/)rDrSrTtyper0r1rrr gen_caches$ 1   zMultiHeadAttention.gen_cachec Cs@|dur|n|}|dur|n|}|dur |||||\}}}n |||||\}}}}tj||jd|dd} |durGt|| j}| |} t| } |jrZtj| |j|j dd} t | |} t j | gdd} t j | d d | j d | j d gd } || } | g} |jr| | |dur| |t| d kr| St| S)ac Applies multi-head attention to map queries and a set of key-value pairs to outputs. Parameters: query (Tensor): The queries for multi-head attention. It is a tensor with shape `[batch_size, query_length, embed_dim]`. The data type should be float32 or float64. key (Tensor, optional): The keys for multi-head attention. It is a tensor with shape `[batch_size, key_length, kdim]`. The data type should be float32 or float64. If None, use `query` as `key`. Default None. value (Tensor, optional): The values for multi-head attention. It is a tensor with shape `[batch_size, value_length, vdim]`. The data type should be float32 or float64. If None, use `query` as `value`. Default None. attn_mask (Tensor, optional): A tensor used in multi-head attention to prevents attention to some unwanted positions, usually the paddings or the subsequent positions. It is a tensor with shape broadcasted to `[batch_size, n_head, sequence_length, sequence_length]`. When the data type is bool, the unwanted positions have `False` values and the others have `True` values. When the data type is int, the unwanted positions have 0 values and the others have 1 values. When the data type is float, the unwanted positions have `-INF` values and the others have 0 values. It can be None when nothing wanted or needed to be prevented attention to. Default None. cache (MultiHeadAttention.Cache|MultiHeadAttention.StaticCache, optional): It is a namedtuple with `k` and `v` as fields, and stores tensors shaped `[batch_size, num_heads, length, embed_dim]` which are results of linear projection, reshape and transpose calculations in MultiHeadAttention. If it is an instance of `Cache`, `k` and `v` fields reserve intermediate results of previous positions, which mostly used for decoder self attention. If it is an instance of `StaticCache`, `key` and `value` args would be ignored, `k` and `v` fields would be used as calculated results on `key` and `value`, which mostly used for decoder-encoder cross attention. It is only used for inference and should be None for training. Default None. Returns: Tensor|tuple: It is a tensor that has the same shape and data type \ as `query`, representing attention output. Or a tuple if \ `need_weights` is True or `cache` is not None. If `need_weights` \ is True, except for attention output, the tuple also includes \ the attention weights tensor shaped `[batch_size, num_heads, query_length, key_length]`. \ If `cache` is not None, the tuple then includes the new cache \ having the same type as `cache`, and if it is `StaticCache`, it \ is same as the input `cache`, if it is `Cache`, the new cache \ reserves tensors concatanating raw tensors with intermediate \ results of current query. NgT)rIyZ transpose_yupscale_in_train)trainingmoderK)rMrrrrHr)rWr*matmulr?r-r)FZsoftmaxr=r^r rOrNrJrCr>rrr) rDrRrSrTr,rUrVr0r1productweightsoutZoutsrrrforward`s:4   $   zMultiHeadAttention.forward)r3NNFNNr)NNNN)__name__ __module__ __qualname____doc__ collections namedtupler/r2r7rWrPr[re __classcell__rrrFrr.ms) +7 Dr.cBeZdZdZ       d fdd Zd dd Zd d ZZS)TransformerEncoderLayera TransformerEncoderLayer is composed of two sub-layers which are self (multi-head) attention and feedforward network. Before and after each sub-layer, pre-process and post-precess would be applied on the input and output accordingly. If `normalize_before` is True, pre-process is layer normalization and post-precess includes dropout, residual connection. Otherwise, no pre-process and post-precess includes dropout, residual connection, layer normalization. Parameters: d_model (int): The expected feature size in the input and output. nhead (int): The number of heads in multi-head attention(MHA). dim_feedforward (int): The hidden layer size in the feedforward network(FFN). dropout (float, optional): The dropout probability used in pre-process and post-precess of MHA and FFN sub-layer. Default 0.1 activation (str, optional): The activation function in the feedforward network. Default relu. attn_dropout (float, optional): The dropout probability used in MHA to drop some attention target. If None, use the value of `dropout`. Default None act_dropout (float, optional): The dropout probability used after FFN activition. If None, use the value of `dropout`. Default None normalize_before (bool, optional): Indicate whether to put layer normalization into preprocessing of MHA and FFN sub-layers. If True, pre-process is layer normalization and post-precess includes dropout, residual connection. Otherwise, no pre-process and post-precess includes dropout, residual connection, layer normalization. Default False weight_attr(ParamAttr|list|tuple, optional): To specify the weight parameter property. If it is a list/tuple, `weight_attr[0]` would be used as `weight_attr` for MHA, and `weight_attr[1]` would be used as `weight_attr` for linear in FFN. Otherwise, MHA and FFN both use it as `weight_attr` to create parameters. Default: None, which means the default weight parameter property is used. See usage for details in :code:`ParamAttr` . bias_attr (ParamAttr|list|tuple|bool, optional): To specify the bias parameter property. If it is a list/tuple, `bias_attr[0]` would be used as `bias_attr` for MHA, and `bias_attr[1]` would be used as `bias_attr` for linear in FFN. Otherwise, MHA and FFN both use it as `bias_attr` to create parameters. The `False` value means the corresponding layer would not have trainable bias parameter. See usage for details in :code:`ParamAttr` . Default: None, which means the default bias parameter property is used. Examples: .. code-block:: python import paddle from paddle.nn import TransformerEncoderLayer # encoder input: [batch_size, src_len, d_model] enc_input = paddle.rand((2, 4, 128)) # self attention mask: [batch_size, n_head, src_len, src_len] attn_mask = paddle.rand((2, 2, 4, 4)) encoder_layer = TransformerEncoderLayer(128, 2, 512) enc_output = encoder_layer(enc_input, attn_mask) # [2, 4, 128] 皙?reluNFc sJt|_|jd|jddtt||dks#Jd||dks.Jd||dks9Jd||dur?|n|}|durG|n|}||_t| d} t| d} t |||| d| dd|_ t ||| d | d d |_ t |d d |_t ||| d | d d |_t||_t||_t |d d |_t |d d |_tt||_dS) NrDrGr6Expected d_model to be greater than 0, but received {}4Expected nhead to be greater than 0, but received {}>Expected dim_feedforward to be greater than 0, but received {}rr=rEr5rr4r]r_)locals_configpopr6rnr7r8normalize_beforer'r. self_attnrlinear1rr=linear2rnorm1norm2dropout1dropout2getattrra activation rDd_modelnheaddim_feedforwardr=r attn_dropout act_dropoutryrEr5Z weight_attrsZ bias_attrsrFrrr7sR        z TransformerEncoderLayer.__init__c Cst||j}|}|jr||}|dur|||||}n ||||||\}}|||}|js7||}|}|jrA||}||| | |}|| |}|js^||}|durd|S||fS)a Applies a Transformer encoder layer on the input. Parameters: src (Tensor): The input of Transformer encoder layer. It is a tensor with shape `[batch_size, sequence_length, d_model]`. The data type should be float32 or float64. src_mask (Tensor, optional): A tensor used in multi-head attention to prevents attention to some unwanted positions, usually the paddings or the subsequent positions. It is a tensor with shape broadcasted to `[batch_size, n_head, sequence_length, sequence_length]`. When the data type is bool, the unwanted positions have `False` values and the others have `True` values. When the data type is int, the unwanted positions have 0 values and the others have 1 values. When the data type is float, the unwanted positions have `-INF` values and the others have 0 values. It can be None when nothing wanted or needed to be prevented attention to. Default None. cache (Tensor, optional): It is an instance of `MultiHeadAttention.Cache`. See `TransformerEncoderLayer.gen_cache` for more details. It is only used for inference and should be None for training. Default None. Returns: Tensor|tuple: It is a tensor that has the same shape and data type \ as `enc_input`, representing the output of Transformer encoder \ layer. Or a tuple if `cache` is not None, except for encoder \ layer output, the tuple includes the new cache which is same \ as input `cache` argument but `incremental_cache` has an \ incremental length. See `MultiHeadAttention.gen_cache` and \ `MultiHeadAttention.forward` for more details. N) r-r)ryr}rzrr~r|r=rr{r)rDsrcsrc_maskrUresidualincremental_cacherrrre)s(      zTransformerEncoderLayer.forwardcCs|jj||jjd}|S)a Generates cache for `forward` usage. The generated cache is an instance of `MultiHeadAttention.Cache`. Parameters: src (Tensor): The input of Transformer encoder. It is a tensor with shape `[batch_size, source_length, d_model]`. The data type should be float32 or float64. Returns: incremental_cache: It is an instance of `MultiHeadAttention.Cache` \ produced by `self_attn.gen_cache`, it reserves two tensors shaped `[batch_size, nhead, 0, d_model // nhead]`. See \ `MultiHeadAttention.gen_cache` and `MultiHeadAttention.forward` \ for more details. rZ)rzr[r/)rDrrrrrr[bsz!TransformerEncoderLayer.gen_cacherorpNNFNNNNrfrgrhrir7rer[rlrrrFrrns< 49rncs4eZdZdZd fdd Zd ddZddZZS) TransformerEncoderaN TransformerEncoder is a stack of N encoder layers. Parameters: encoder_layer (Layer): an instance of the `TransformerEncoderLayer`. It would be used as the first layer, and the other layers would be created according to the configurations of it. num_layers (int): The number of encoder layers to be stacked. norm (LayerNorm, optional): the layer normalization component. If provided, apply layer normalization on the output of last encoder layer. Examples: .. code-block:: python import paddle from paddle.nn import TransformerEncoderLayer, TransformerEncoder # encoder input: [batch_size, src_len, d_model] enc_input = paddle.rand((2, 4, 128)) # self attention mask: [batch_size, n_head, src_len, src_len] attn_mask = paddle.rand((2, 2, 4, 4)) encoder_layer = TransformerEncoderLayer(128, 2, 512) encoder = TransformerEncoder(encoder_layer, 2) enc_output = encoder(enc_input, attn_mask) # [2, 4, 128] Nc:tt|tfddt|D|_||_||_dS)Nc,g|]}|dkr n tdijqSrrrZrwr encoder_layerrrr z/TransformerEncoder.__init__..)r6rr7r r r num_layersnorm)rDrrrrFrrr7    zTransformerEncoder.__init__c Cst||j}|}g}t|jD]\}}|dur|||d}q|||||d\}}||q|jdur9||}|dur?|S||fS)a2 Applies a stack of N Transformer encoder layers on inputs. If `norm` is provided, also applies layer normalization on the output of last encoder layer. Parameters: src (Tensor): The input of Transformer encoder. It is a tensor with shape `[batch_size, sequence_length, d_model]`. The data type should be float32 or float64. src_mask (Tensor, optional): A tensor used in multi-head attention to prevents attention to some unwanted positions, usually the paddings or the subsequent positions. It is a tensor with shape broadcasted to `[batch_size, n_head, sequence_length, sequence_length]`. When the data type is bool, the unwanted positions have `False` values and the others have `True` values. When the data type is int, the unwanted positions have 0 values and the others have 1 values. When the data type is float, the unwanted positions have `-INF` values and the others have 0 values. It can be None when nothing wanted or needed to be prevented attention to. Default None. cache (list, optional): It is a list, and each element in the list is `incremental_cache` produced by `TransformerEncoderLayer.gen_cache`. See `TransformerEncoder.gen_cache` for more details. It is only used for inference and should be None for training. Default None. Returns: Tensor|tuple: It is a tensor that has the same shape and data type \ as `src`, representing the output of Transformer encoder. \ Or a tuple if `cache` is not None, except for encoder output, \ the tuple includes the new cache which is same as input `cache` \ argument but `incremental_cache` in it has an incremental length. \ See `MultiHeadAttention.gen_cache` and `MultiHeadAttention.forward` \ for more details. Nr)rrUr-r) enumerater rr) rDrrrUoutput new_cachesrmod new_cacherrrres "   zTransformerEncoder.forwardcsfdd|jD}|S)a Generates cache for `forward` usage. The generated cache is a list, and each element in it is `incremental_cache` produced by `TransformerEncoderLayer.gen_cache`. See `TransformerEncoderLayer.gen_cache` for more details. Parameters: src (Tensor): The input of Transformer encoder. It is a tensor with shape `[batch_size, source_length, d_model]`. The data type should be float32 or float64. Returns: list: It is a list, and each element in the list is `incremental_cache` produced by `TransformerEncoderLayer.gen_cache`. See `TransformerEncoderLayer.gen_cache` for more details. cg|]}|qSrr[rlayerrrrrrz0TransformerEncoder.gen_cache..r )rDrrUrrrr[szTransformerEncoder.gen_cacherrrrrrFrrxs  4rcrm)TransformerDecoderLayera TransformerDecoderLayer is composed of three sub-layers which are decoder self (multi-head) attention, decoder-encoder cross attention and feedforward network. Before and after each sub-layer, pre-process and post-precess would be applied on the input and output accordingly. If `normalize_before` is True, pre-process is layer normalization and post-precess includes dropout, residual connection. Otherwise, no pre-process and post-precess includes dropout, residual connection, layer normalization. Parameters: d_model (int): The expected feature size in the input and output. nhead (int): The number of heads in multi-head attention(MHA). dim_feedforward (int): The hidden layer size in the feedforward network(FFN). dropout (float, optional): The dropout probability used in pre-process and post-precess of MHA and FFN sub-layer. Default 0.1 activation (str, optional): The activation function in the feedforward network. Default relu. attn_dropout (float, optional): The dropout probability used in MHA to drop some attention target. If None, use the value of `dropout`. Default None act_dropout (float, optional): The dropout probability used after FFN activition. If None, use the value of `dropout`. Default None normalize_before (bool, optional): Indicate whether to put layer normalization into preprocessing of MHA and FFN sub-layers. If True, pre-process is layer normalization and post-precess includes dropout, residual connection. Otherwise, no pre-process and post-precess includes dropout, residual connection, layer normalization. Default False weight_attr(ParamAttr|list|tuple, optional): To specify the weight parameter property. If it is a list/tuple, `weight_attr[0]` would be used as `weight_attr` for self attention, `weight_attr[1]` would be used as `weight_attr` for cross attention, and `weight_attr[2]` would be used as `weight_attr` for linear in FFN. Otherwise, the three sub-layers all uses it as `weight_attr` to create parameters. Default: None, which means the default weight parameter property is used. See usage for details in :ref:`api_paddle_fluid_param_attr_ParamAttr` . bias_attr (ParamAttr|list|tuple|bool, optional): To specify the bias parameter property. If it is a list/tuple, `bias_attr[0]` would be used as `bias_attr` for self attention, `bias_attr[1]` would be used as `bias_attr` for cross attention, and `bias_attr[2]` would be used as `bias_attr` for linear in FFN. Otherwise, the three sub-layers all uses it as `bias_attr` to create parameters. The `False` value means the corresponding layer would not have trainable bias parameter. See usage for details in :code:`ParamAttr` . Default: None,which means the default bias parameter property is used. Examples: .. code-block:: python import paddle from paddle.nn import TransformerDecoderLayer # decoder input: [batch_size, tgt_len, d_model] dec_input = paddle.rand((2, 4, 128)) # encoder output: [batch_size, src_len, d_model] enc_output = paddle.rand((2, 6, 128)) # self attention mask: [batch_size, n_head, tgt_len, tgt_len] self_attn_mask = paddle.rand((2, 2, 4, 4)) # cross attention mask: [batch_size, n_head, tgt_len, src_len] cross_attn_mask = paddle.rand((2, 2, 4, 6)) decoder_layer = TransformerDecoderLayer(128, 2, 512) output = decoder_layer(dec_input, enc_output, self_attn_mask, cross_attn_mask) # [2, 4, 128] rorpNFc s~t|_|jd|jddtt||dks#Jd||dks.Jd||dks9Jd||dur?|n|}|durG|n|}||_t| d} t| d} t |||| d| dd|_ t |||| d | d d|_ t ||| d | d d |_ t|d d |_t ||| d | d d |_t||_t||_t||_t|d d |_t|d d |_t|d d |_tt||_dS)NrDrGrrqrrrsrrtrrr4r]ru)rvrwrxr6rr7r8ryr'r.rz cross_attnrr{rr=r|rr}r~norm3rrdropout3rrarrrFrrr7*sb         z TransformerDecoderLayer.__init__c CsNt||j}t||j}|}|jr||}|dur$|||||d}n ||||||d\}}|||}|js@||}|}|jrJ||}|durX|||||d}n ||||||d\}}|||}|jst||}|}|jr~| |}| | | | |}|||}|js| |}|dur|S|||ffS)a Applies a Transformer decoder layer on the input. Parameters: tgt (Tensor): The input of Transformer decoder layer. It is a tensor with shape `[batch_size, target_length, d_model]`. The data type should be float32 or float64. memory (Tensor): The output of Transformer encoder. It is a tensor with shape `[batch_size, source_length, d_model]`. The data type should be float32 or float64. tgt_mask (Tensor, optional): A tensor used in self attention to prevents attention to some unwanted positions, usually the the subsequent positions. It is a tensor with shape broadcasted to `[batch_size, n_head, target_length, target_length]`. When the data type is bool, the unwanted positions have `False` values and the others have `True` values. When the data type is int, the unwanted positions have 0 values and the others have 1 values. When the data type is float, the unwanted positions have `-INF` values and the others have 0 values. It can be None when nothing wanted or needed to be prevented attention to. Default None. memory_mask (Tensor, optional): A tensor used in decoder-encoder cross attention to prevents attention to some unwanted positions, usually the paddings. It is a tensor with shape broadcasted to `[batch_size, n_head, target_length, source_length]`. When the data type is bool, the unwanted positions have `False` values and the others have `True` values. When the data type is int, the unwanted positions have 0 values and the others have 1 values. When the data type is float, the unwanted positions have `-INF` values and the others have 0 values. It can be None when nothing wanted or needed to be prevented attention to. Default None. cache (tuple, optional): It is a tuple( :code:`(incremental_cache, static_cache)` ), `incremental_cache` is an instance of `MultiHeadAttention.Cache`, `static_cache` is an instance of `MultiHeadAttention.StaticCache. See `TransformerDecoderLayer.gen_cache` for more details. It is only used for inference and should be None for training. Default None. Returns: Tensor|tuple: It is a tensor that has the same shape and data type \ as `tgt`, representing the output of Transformer decoder layer. \ Or a tuple if `cache` is not None, except for decoder layer output, \ the tuple includes the new cache which is same as input `cache` \ argument but `incremental_cache` in it has an incremental length. \ See `MultiHeadAttention.gen_cache` and `MultiHeadAttention.forward` \ for more details. Nrr)r-r)ryr}rzrr~rrrr|r=rr{r) rDtgtmemorytgt_mask memory_maskrUrr static_cacherrrreesD /         zTransformerDecoderLayer.forwardcCs2|jj||jjd}|jj|||jjd}||fS)a Generates cache for `forward` usage. The generated cache is a tuple composed of an instance of `MultiHeadAttention.Cache` and an instance of `MultiHeadAttention.StaticCache`. Parameters: memory (Tensor): The output of Transformer encoder. It is a tensor with shape `[batch_size, source_length, d_model]`. The data type should be float32 or float64. Returns: tuple: It is a tuple( :code:`(incremental_cache, static_cache)` ). \ `incremental_cache` is an instance of `MultiHeadAttention.Cache` \ produced by `self_attn.gen_cache(memory, MultiHeadAttention.Cache)`, \ it reserves two tensors shaped `[batch_size, nhead, 0, d_model // nhead]`. \ `static_cache` is an instance of `MultiHeadAttention.StaticCache` \ produced by `cross_attn.gen_cache(memory, MultiHeadAttention.StaticCache)`, \ it reserves two tensors shaped `[batch_size, nhead, source_length, d_model // nhead]`. See `MultiHeadAttention.gen_cache` and `MultiHeadAttention.forward` \ for more details. r)rzr[r/rr2)rDrrrrrrr[s z!TransformerDecoderLayer.gen_cacherNNNrrrrFrrsG ;Trcs6eZdZdZd fdd Zd ddZd dd ZZS) TransformerDecodera TransformerDecoder is a stack of N decoder layers. Parameters: decoder_layer (Layer): an instance of the `TransformerDecoderLayer`. It would be used as the first layer, and the other layers would be created according to the configurations of it. num_layers (int): The number of decoder layers to be stacked. norm (LayerNorm, optional): the layer normalization component. If provided, apply layer normalization on the output of last encoder layer. Examples: .. code-block:: python import paddle from paddle.nn import TransformerDecoderLayer, TransformerDecoder # decoder input: [batch_size, tgt_len, d_model] dec_input = paddle.rand((2, 4, 128)) # encoder output: [batch_size, src_len, d_model] enc_output = paddle.rand((2, 6, 128)) # self attention mask: [batch_size, n_head, tgt_len, tgt_len] self_attn_mask = paddle.rand((2, 2, 4, 4)) # cross attention mask: [batch_size, n_head, tgt_len, src_len] cross_attn_mask = paddle.rand((2, 2, 4, 6)) decoder_layer = TransformerDecoderLayer(128, 2, 512) decoder = TransformerDecoder(decoder_layer, 2) output = decoder(dec_input, enc_output, self_attn_mask, cross_attn_mask) # [2, 4, 128] Ncr)Ncrrrr decoder_layerrrrrz/TransformerDecoder.__init__..)r6rr7r r r rr)rDrrrrFrrr7rzTransformerDecoder.__init__c Cst||j}t||j}|}g}t|jD]$\}} |dur'| ||||dd}q| ||||||d\}} || q|jdurD||}|durJ|S||fS)a Applies a stack of N Transformer decoder layers on inputs. If `norm` is provided, also applies layer normalization on the output of last decoder layer. Parameters: tgt (Tensor): The input of Transformer decoder. It is a tensor with shape `[batch_size, target_length, d_model]`. The data type should be float32 or float64. memory (Tensor): The output of Transformer encoder. It is a tensor with shape `[batch_size, source_length, d_model]`. The data type should be float32 or float64. tgt_mask (Tensor, optional): A tensor used in self attention to prevents attention to some unwanted positions, usually the the subsequent positions. It is a tensor with shape broadcasted to `[batch_size, n_head, target_length, target_length]`. When the data type is bool, the unwanted positions have `False` values and the others have `True` values. When the data type is int, the unwanted positions have 0 values and the others have 1 values. When the data type is float, the unwanted positions have `-INF` values and the others have 0 values. It can be None when nothing wanted or needed to be prevented attention to. Default None. memory_mask (Tensor, optional): A tensor used in decoder-encoder cross attention to prevents attention to some unwanted positions, usually the paddings. It is a tensor with shape broadcasted to `[batch_size, n_head, target_length, source_length]`. When the data type is bool, the unwanted positions have `False` values and the others have `True` values. When the data type is int, the unwanted positions have 0 values and the others have 1 values. When the data type is float, the unwanted positions have `-INF` values and the others have 0 values. It can be None when nothing wanted or needed to be prevented attention to. Default None. cache (list, optional): It is a list, and each element in the list is a tuple( :code:`(incremental_cache, static_cache)` ). See `TransformerDecoder.gen_cache` for more details. It is only used for inference and should be None for training. Default None. Returns: Tensor|tuple: It is a tensor that has the same shape and data type \ as `tgt`, representing the output of Transformer decoder. \ Or a tuple if `cache` is not None, except for decoder output, \ the tuple includes the new cache which is same as input `cache` \ argument but `incremental_cache` in it has an incremental length. \ See `MultiHeadAttention.gen_cache` and `MultiHeadAttention.forward` \ for more details. N)rrrUr) rDrrrrrUrrrrrrrrres, /    zTransformerDecoder.forwardFcs(fdd|jD}|rtt|}|S)aZ Generates cache for `forward` usage. The generated cache is a list, and each element in it is a tuple( :code:`(incremental_cache, static_cache)` ) produced by `TransformerDecoderLayer.gen_cache`. See `TransformerDecoderLayer.gen_cache` for more details. If `do_zip` is True, apply `zip` on these tuples to get a list with two elements. Parameters: memory (Tensor): The output of Transformer encoder. It is a tensor with shape `[batch_size, source_length, d_model]`. The data type should be float32 or float64. do_zip (bool, optional): Indicate whether to apply `zip` on the tuples. If True, return a list with two elements. Default False Returns: list: It is a list, and each element in the list is a tuple produced \ by `TransformerDecoderLayer.gen_cache(memory)`. See `TransformerDecoderLayer.gen_cache` \ for more details. If `do_zip` is True, apply `zip` on these tuples \ and return a list with two elements. crrrrrrrr`rz0TransformerDecoder.gen_cache..)r rzip)rDrZdo_ziprUrrrr[Js zTransformerDecoder.gen_cacherr)FrrrrFrrs " HrcsPeZdZdZ             dfd d Zdd d ZddZZS) Transformera" A Transformer model composed of an instance of `TransformerEncoder` and an instance of `TransformerDecoder`. While the embedding layer and output layer are not included. Please refer to `Attention is all you need `_ , and see `TransformerEncoder` and `TransformerDecoder` for more details. Users can configurate the model architecture with corresponding parameters. Note the usage of `normalize_before` representing where to apply layer normalization (in pre-process or post-precess of multi-head attention or FFN), and some transformer like models are different on this, such as `BERT `_ and `GPT2 `_ . The default architecture here places layer normalization in post-process and applies another layer normalization on the output of last encoder/decoder layer. Parameters: d_model (int, optional): The expected feature size in the encoder/decoder input and output. Default 512 nhead (int, optional): The number of heads in multi-head attention(MHA). Default 8 num_encoder_layers (int, optional): The number of layers in encoder. Default 6 num_decoder_layers (int, optional): The number of layers in decoder. Default 6 dim_feedforward (int, optional): The hidden layer size in the feedforward network(FFN). Default 2048 dropout (float, optional): The dropout probability used in pre-process and post-precess of MHA and FFN sub-layer. Default 0.1 activation (str, optional): The activation function in the feedforward network. Default relu. attn_dropout (float, optional): The dropout probability used in MHA to drop some attention target. If None, use the value of `dropout`. Default None act_dropout (float, optional): The dropout probability used after FFN activition. If None, use the value of `dropout`. Default None normalize_before (bool, optional): Indicate whether to put layer normalization into preprocessing of MHA and FFN sub-layers. If True, pre-process is layer normalization and post-precess includes dropout, residual connection. Otherwise, no pre-process and post-precess includes dropout, residual connection, layer normalization. Default False weight_attr(ParamAttr|list|tuple, optional): To specify the weight parameter property. If it is a list/tuple, the length of `weight_attr` could be 1, 2 or 3. If it is 3, `weight_attr[0]` would be used as `weight_attr` for self attention, `weight_attr[1]` would be used as `weight_attr` for cross attention of `TransformerDecoder`, and `weight_attr[2]` would be used as `weight_attr` for linear in FFN. If it is 2, `weight_attr[0]` would be used as `weight_attr` both for self attention and cross attntion and `weight_attr[1]` would be used as `weight_attr` for linear in FFN. If it is 1, `weight_attr[0]` would be used as `weight_attr` for self attention, cross attention and linear in FFN. Otherwise, the three sub-layers all uses it as `weight_attr` to create parameters. Default: None, which means the default weight parameter property is used. See usage for details in :code:`ParamAttr` . bias_attr (ParamAttr|list|tuple|bool, optional): To specify the bias parameter property. If it is a list/tuple, the length of `bias_attr` could be 1, 2 or 3. If it is 3, `bias_attr[0]` would be used as `bias_attr` for self attention, `bias_attr[1]` would be used as `bias_attr` for cross attention of `TransformerDecoder`, and `bias_attr[2]` would be used as `bias_attr` for linear in FFN. If it is 2, `bias_attr[0]` would be used as `bias_attr` both for self attention and cross attntion and `bias_attr[1]` would be used as `bias_attr` for linear in FFN. If it is 1, `bias_attr[0]` would be used as `bias_attr` for self attention, cross attention and linear in FFN. Otherwise, the three sub-layers all uses it as `bias_attr` to create parameters. The `False` value means the corresponding layer would not have trainable bias parameter. See usage for details in :code:`ParamAttr` . Default: None,which means the default bias parameter property is used. custom_encoder (Layer, optional): If custom encoder is provided, use it as the encoder. Default None custom_decoder (Layer, optional): If custom decoder is provided, use it as the decoder. Default None Examples: .. code-block:: python import paddle from paddle.nn import Transformer # src: [batch_size, tgt_len, d_model] enc_input = paddle.rand((2, 4, 128)) # tgt: [batch_size, src_len, d_model] dec_input = paddle.rand((2, 6, 128)) # src_mask: [batch_size, n_head, src_len, src_len] enc_self_attn_mask = paddle.rand((2, 2, 4, 4)) # tgt_mask: [batch_size, n_head, tgt_len, tgt_len] dec_self_attn_mask = paddle.rand((2, 2, 6, 6)) # memory_mask: [batch_size, n_head, tgt_len, src_len] cross_attn_mask = paddle.rand((2, 2, 6, 4)) transformer = Transformer(128, 2, 4, 4, 512) output = transformer(enc_input, dec_input, enc_self_attn_mask, dec_self_attn_mask, cross_attn_mask) # [2, 6, 128] rorpNFc stt||dksJd||dksJd||dks(Jd|t| ttfrmt| dkrD| dgd}| dgd}n-t| dkrX| }| d| d| dg}nt| dkri| d| dg}| }nJd | }| }t| ttfrt| dkr| dgd}| dgd}n-t| dkr| }| d| d| dg}nt| dkr| d| dg}| }nJd | }| }| dur| |_nt ||||||| | || }t |}t ||||_|dur||_ nt ||||||| | || }t |}t||||_ ||_||_dS) NrrqrrrsrrrrXFzAlength of bias_attr should be 1 or 2 or 3 when it is a list/tuplezClength of weight_attr should be 1 or 2 or 3 when it is a list/tuple)r6rr7r8rrrrencoderrnrrdecoderrrrr)rDrrZnum_encoder_layersZnum_decoder_layersrr=rrrryrEr5Zcustom_encoderZcustom_decoderZencoder_bias_attrZdecoder_bias_attrZencoder_weight_attrZdecoder_weight_attrrZ encoder_normrZ decoder_normrFrrr7s            zTransformer.__init__cCsHt||j}|j||d}t||j}t||j}|j||||d}|S)a Applies a Transformer model on the inputs. Parameters: src (Tensor): The input of Transformer encoder. It is a tensor with shape `[batch_size, source_length, d_model]`. The data type should be float32 or float64. tgt (Tensor): The input of Transformer decoder. It is a tensor with shape `[batch_size, target_length, d_model]`. The data type should be float32 or float64. memory (Tensor): The output of Transformer encoder. It is a tensor with shape `[batch_size, source_length, d_model]`. The data type should be float32 or float64. src_mask (Tensor, optional): A tensor used in multi-head attention to prevents attention to some unwanted positions, usually the paddings or the subsequent positions. It is a tensor with shape broadcasted to `[batch_size, n_head, sequence_length, sequence_length]`. When the data type is bool, the unwanted positions have `False` values and the others have `True` values. When the data type is int, the unwanted positions have 0 values and the others have 1 values. When the data type is float, the unwanted positions have `-INF` values and the others have 0 values. It can be None when nothing wanted or needed to be prevented attention to. Default None. tgt_mask (Tensor, optional): A tensor used in self attention to prevents attention to some unwanted positions, usually the the subsequent positions. It is a tensor with shape broadcasted to `[batch_size, n_head, target_length, target_length]`. When the data type is bool, the unwanted positions have `False` values and the others have `True` values. When the data type is int, the unwanted positions have 0 values and the others have 1 values. When the data type is float, the unwanted positions have `-INF` values and the others have 0 values. It can be None when nothing wanted or needed to be prevented attention to. Default None. memory_mask (Tensor, optional): A tensor used in decoder-encoder cross attention to prevents attention to some unwanted positions, usually the paddings. It is a tensor with shape broadcasted to `[batch_size, n_head, target_length, source_length]`. When the data type is bool, the unwanted positions have `False` values and the others have `True` values. When the data type is int, the unwanted positions have 0 values and the others have 1 values. When the data type is float, the unwanted positions have `-INF` values and the others have 0 values. It can be None when nothing wanted or needed to be prevented attention to. Default None. Returns: Tensor: It is a tensor that has the same shape and data type \ as `tgt`, representing the output of Transformer decoder. r)rr)r-r)rr)rDrrrrrrrrrrres 1  zTransformer.forwardcCs&tjtj||gtj tddS)a" Generate a square mask for the sequence. The mask ensures that the predictions for position i can depend only on the known outputs at positions less than i. Parameters: length (int|Tensor): The length of sequence. Returns: Tensor: Generated square mask according to the given length. Examples: .. code-block:: python import paddle from paddle.nn.layer.transformer import Transformer length = 5 d_model, n_head, dim_feedforward = 8, 4, 64 transformer_paddle = Transformer( d_model, n_head, dim_feedforward=dim_feedforward) mask = transformer_paddle.generate_square_subsequent_mask(length) print(mask) # [[ 0. -inf -inf -inf -inf] # [ 0. 0. -inf -inf -inf] # [ 0. 0. 0. -inf -inf] # [ 0. 0. 0. 0. -inf] # [ 0. 0. 0. 0. 0.]] )rJZ fill_valuer)r)r*r ZtriufullnpinfZget_default_dtype)rDlengthrrrgenerate_square_subsequent_maskXs z+Transformer.generate_square_subsequent_mask)rrrrrrorpNNFNNNNr)rfrgrhrir7rerrlrrrFrrfs&^ Xs4      .Q=nq