o Qe@sddlZddlZddlZddlmZmZmZddlm Z m Z m Z ddl m Z ddl mZddlmZdd lmZmZdd lmZmZmZdd lmZdd lmZmZmZddlZddlZddlZdd l m!Z!gZ"ddZ#ddZ$ddZ%ddZ&ddZ'ddZ(ddZ)ddZ*ddZ+d d!Z,d"d#Z-d$d%Z.d&d'Z/d(d)Z0Gd*d+d+Z1d,d-Z2d.d/Z3d0d1Z4d2d3Z5d4d5Z6d6d7Z7d8d9Z8d:d;Z9dS)<N)dotmatmul transpose)squeeze unsqueezereshape)multiply)sum)_in_legacy_dygraph)_C_ops _legacy_C_ops)check_variable_and_dtype check_type check_dtype) LayerHelper)_non_static_modein_dygraph_moder ) dygraph_onlycCs|ddD]}|sJd|dq|dddddks%Jdt|j}|d ks0J|dd|t|d }t||ksLJd |d |S) a Parse labels for an input operand. Parameters ---------- labelstr: the input label string operand: the input operand Returns ------- the input operand's full label string in which all anonymous dimensions are labeled in dots. .zInvalid equation: z/ is not a valid label, which should be letters....rz6Invalid equation: `.` is found outside of an ellipsis.rz$Invalid equation: the label string 'z' misses dimensions.)replaceisalphafindlenshape)labelstroperandcZndimsZ full_labelstrr#DD:\Projects\ConvertPro\env\Lib\site-packages\paddle/tensor/einsum.pyparse_op_labels%s     r%cCsH|d}t|t|ksJdt|dt|dttt||S)a Parse label strings for all input operands. Parameters ---------- labelstr: The equation's label string operands: The input operands Returns ------- list of full label strings for all input operands ,,Invalid equation: the number of operands is , but found segments in the label equation.)splitrlistmapr%)r operands nop_labelsr#r#r$ parse_labelsJs  r/cCsv|dkr d|vs Jd|dd}t|}d|vsJ||}|r-Jdt|dt|t|ks9Jdd S) z@ Check whether the equation's right hand side is valid rr4Invalid equation: missing ellipsis in output labels.rrzInvalid equation: output label z not used by any input.4Invalid equation: duplicate output labels are found.N)rset differencesortedr)rhsZ input_labels n_bcast_dimsZrhs_setZnon_input_labelsr#r#r$ validate_rhsbs     r7c Csdgt|}td|}|rB||}}td|}|rBtt||dddt||dddD]\}}|||<q9|rRtt|t|t|} nt tt|} | D] } | || || <q\|S)a@ Build an inverse map of dimension indices. Three conditions must hold for the result to be meaningful. First, no duplicate letter labels in each label string. Second, the number of dots in dimout_labels >= that in in_labels. Third, dots are contiguous in each label string. Parameters ---------- in_labels: The dimension labels to map to out_labels: The dimension labels to map from Returns ------- The inverse map from out_labels to in_labels. The length of the inverse map equals that of out_labels. -1 is filled if there's no matching intput dimension for a specific label. Examples -------- in_labels = 'ij..', out_labels = '..ji' inv_map = [2, 3, 1, 0] in_labels = 'ij..', out_labels = '..kji' inv_map = [2, 3, -1, 1, 0] rz\.+N) rresearchstartendziprange itertoolschainiterr) Z in_labelsZ out_labelsZinv_maprr:r;saxdimitir#r#r$ build_view{s"   rGcs,td|dd}gg}}tdg||D]\}}||kr+|||dq|dd7<q|dkrGt||||dd|}nd|dddt||D}tt|dddD]} || |vru|| || qcd|} || t t fd d |} t|} |} | | | fS) a1 Build the global view, which is a layout of all dimension labels plus an index table that maps from the layout to the dimensions in each operand. In the global view, the dimensions are arranged such that output ones are put on the left and contraction ones are put on the right. Parameters ---------- nop_labels: The input full label strings of all input operands rhs: The equation right hand side n_bcast_dims: The maxium number of broadcast dimensions Returns ------- A tuple of g_labels, g_view, g_nout, g_count g_labels: the layout of all labels in a string g_view: the index table g_nout: the number of output dimensions g_count: the counter array for dimension contractions rrrrNrcss |] \}}|dkr|VqdS)rNr#).0lr"r#r#r$ sz$build_global_view..cs t|SN)rG)rFg_labelsr#r$ z#build_global_view..) r4joinrr<appendr7r=rpopr+r,)r.r5r6concatlabelscountabZ g_labels_outrFZ g_labels_sumg_viewg_noutg_countr#rLr$build_global_views0          r[csg}g}t||D]\}|fdd|Dq ddt|D}ddt|D}|r:Jd||dddd|D}d d|D}||fS) a The global shape is the shape of all dimensions rearranged and broadcasting to the global view. It's a reference data structure for einsum planning. Parameters ---------- g_view: the global view op_shapes: the shapes of the all operands Returns ------- g_shape: the global shape vector g_masks: list of shape masks for each operand. A dimension's shape mask is a boolean indicating whether its size > 1, in other words, it's not squeezable cs g|] }|dkr |ndqS)rrr#rHrDZop_shaper#r$  z&build_global_shape..cSsg|] }t|dhqSr)r2)rHZ sizes_per_axr#r#r$r^scSs g|] \}}t|dkr|qSr`)r)rHrCsizesr#r#r$r^ r_zInvalid operands: label rz- corresponds to non-broadcastable dimensions.cSs$g|]}t|dkr|ndqS)rr)rrR)rHrar#r#r$r^s$cSsg|] }dd|DqS)cSsg|] }|dkp |dkqS)rrr#)rHrBr#r#r$r^z1build_global_shape...r#)rHZ view_shaper#r#r$r^s)r<rQ enumerate)rXrMZ op_shapesZ view_shapesZg_masksviewg_shapeZ non_bcastabler#r]r$build_global_shapesrfcCs |dd}t|tt|kS)z7 Returns True if there is any duplicate label. rr)rrr2)rTr#r#r$has_duplicated_labelss rgcCst|rJd||fS)aE Merges dimensions with duplicate labels. For those dimensions with duplicate labels, merge them into one dimension which represents the diagonal elements. This requires the dimensions with duplicate labels are equal sized. Examples -------- 'ijj...i' would be merged into 'ij...' #Duplicate labels are not supported.)rg)rTr!r#r#r$ diagonalize s rics2d|}fdd}||g||f}||dS)z Add reduce to the plan opcst||dS)Nkeepdim paddle_sum)vardimsrkr#r$rN8szplan_reduce..Nadd_step)planrj reduce_dimsrlvarnamefstepr#rkr$ plan_reduce2s  rxcCs8d|d|g}dd}|||df}||dS)NrjcSs t||SrKrm)var1var2r#r#r$rN?s z"plan_scalar_prod..rrq)rsop1op2varnamesrvrwr#r#r$plan_scalar_prod=sr~c "sxd|d|} } fdd||fD\fdd|D} fdd|D} t| || }t| || }fdd||fD\}}tddt||D}tddt||D}t||}tt| | ||| f\}}}}}t|tt|krt| g| t |f}| |t|tt|krt| g| t |f}| |||d kr2|d kr2d t ||fvr2t ||t ||gt || g}t ||t ||gt || g}t | g| |f}| |t | g| |f}| |t| | g| d d f}| |t ||||}t | g| |f}| |n<||krD|krDd krVnnt| | g| d d f}| |n|rrt t|||||}t| g| |f}| ||rt t|||}t| g| |f}| ||d krt| | g| f}| |n||d kr|d krt| g| dgf}| |t| g| d gf}| |t| | g| f}| |t| g| d dgf}| |n||d krRd t || || fvrRt|| || ksJt | g| t ||d gt || gf}| |t | g| t ||d gt || gf}| |t| | g| d d f}| |t| g| d dgf}| |nt| | g| f}| |t t| d }t|||d d|||D]} || d kp|| d k|| <qt| D]} d || <qttD]} d | <qd }!|||D] } |!|!d | <}!qt |<dS)z plan matmul rjcg|]}|qSr#r#rHrj)rXr#r$r^Mzplan_matmul..cg|] }|dkr|qSrr#rHidx)op1_viewr#r$r^Orbcrrr#r)op2_viewr#r$r^Prbcrr#r#r) g_supportsr#r$r^WrcSg|] \}}|r |ndqSr`r#rHrBmr#r#r$r^XrbcSrr`r#rr#r#r$r^YrbrrFTrrkN)nparrayr<maximumr,ranyZarangerr+rrZ concatenateprodrrr=rr rallrx)"rsrXr{r|rreIJ1J2KryrzZI1ZI2Zop1_dimsZop2_dimsop1_maskop2_maskZ op1_vshapeZ op2_vshapeZvshapei1i2Zj1Zj2krwZ op1_shapeZ op2_shaperfillrtrCrDr#)rrXrrr$ plan_matmulEs        (        " "    "  rc Csf||||}} ||||} } t|} | t|} dg| |}tt|gggf\}}}}tt|| ||d| |dD]W\}}}|dk|dkkr_|dkrY||qB||qB|dkrt| |t| |}|| kr|||kr|||||8<qB||||t|dd8<qB|| d|dd<t|||||||||| dS)z) Plan various kinds of summation rNrr)rr+r=r<rQintmaxr)rsrXr{r|rrerZn_bcastrrrrndimnoutrUrrrrrCZdim1Zdim2foldr#r#r$plan_summations.      rcCs\gg}}t|D]\}}|dkr||q ||q tddt|Dr*g}||fS)Nrcss|] \}}||kVqdSrKr#)rHrFrDr#r#r$rJzrearrange..)rcrQr)ZaxespermrrCrDr#r#r$ rearranges   rc st|}ddt|Dtt||D]*\}}t|\}}|}|r0t|g||f} || |r>t|g||f} || qfdd} | df} || dS)z Plan broadcast across cSg|]}d|qSrjr#rHrFr#r#r$r^z"plan_broadcast..csd}t|tt|S)Nz * )rPevaldictr<)argsexprr}r#r$rv s zplan_broadcast..fN)rr=r<rrrrr) rsr-Znop_axesnoprFZop_axesrrrorwrvr#rr$plan_broadcasts     rc@s<eZdZddZddZddZddZd d Zd d Zd S)PlancCsi|_g|_dSrK)envsteps)selfr#r#r$__init__s z Plan.__init__cCs|j|dSrK)rrQ)rrwr#r#r$rrsz Plan.add_stepcCs||jvr |j|SdSrKr)rrur#r#r$get_varsz Plan.get_varcCs||j|<dSrKr)rruror#r#r$set_var!sz Plan.set_varcCs6d}|jD]^}}}}tt||g||Rq|SrK)rprintreprrresrvZ in_varnamesZ out_varnamerr#r#r$show$sz Plan.showcCsFd}|jD]^}}}}|gt|j||R}|r |||q|SrK)rr,rrrr#r#r$execute*s z Plan.executeN) __name__ __module__ __qualname__rrrrrrrr#r#r#r$rs rc CsJt|}t|d}|t|}t} ddt|D} tt| j| ||s/t| ||| St||D]'\} } ddt| |d| |dD} t| D] \}}|||8<qNq4tt|||D]`\}} }g}t| |d||d|D]\}}}| |r|dkr|ndqytt dd |}|rt | ||d d t|D] \}}||}||o|dk||<|||dkrdnd8<qqdt|D]%}|dkrqt ||dst | |d|qt| ||d|||||qtd d ||d|dDsJ|d} t dd t| d|Drdd| D}t||kr8d|d}t|g||f}| |d}g}t| D]\}}|dkrR||d| |<}q@t| d|D]\}}|dkrj| |q\|rd|d}t|g||f}| |dd| |dD}|rd|d}t|g||f}| || S)zZ Plans the actual execution steps. Results ------- the execution plan rcSrrr#rr#r#r$r^@rzplan_einsum..cSs"g|] \}}t|do | qSr`)r)rHdrBr#r#r$r^KsNrrcSs|dkS)Nrr#)xr#r#r$rNXszplan_einsum..Trkcss|]}| VqdSrKr#)rHmaskedr#r#r$rJszplan_einsum..css|] \}}||kVqdSrKr#)rHrCrDr#r#r$rJrcSsg|]}|dkr|qSrr#r\r#r#r$r^rjcSsg|]}|dkr|qS)rr#r\r#r#r$r^r)rrr=r+r,rrr<rcrQfilterrxrr~rrr4rrrrr)r-rXrerrZrrrrrsZop_namesrdZsupportZ down_countrFrUmaskZ to_reducerDrrtrrCrrurwZunsqueeze_dimsZ squeeze_dimsr#r#r$ plan_einsum3s   & (       rcGs|dd}t|}|dksJd||d^}}t|dks'Jdt||}|r2|dnd}|durr +Invalid equation: multiple `->` were found.Nr&r'r(r)rr0rhr1) rrlowerr*r/ rhs_inferencer+rrg)equationr-rlhsr5rTr#r#r$ preprocesss8     rcs.tddgfdd}tt|||}|S)z this shape is just used for operands planning. may differ with the original shape. for example: ... is replaced by 1 -1 is replaced by 1 Results ------- list of shape shapedrcsvt|jt|ksJdt|jt|fddtt||jD}ttt|}d|vr7||dd|S)NzKlength of shape and length of label must be the same, but received %d != %dcSs g|] \}\}}|dkr|qS)rr#)rHrFrIrBr#r#r$r^r_z8parse_fake_shape..fake_shape..rr) rrrcr<r+r,absinsertindex)labelrjZfakesrr#r$ fake_shapes z$parse_fake_shape..fake_shape) collections namedtupler+r,)rr-rTroutr#rr$parse_fake_shapes  rcsFfdd}t|d|vrdnd}|dt|t}|S)Ncs|dko |dvS)Nr)rr&)get)keycntr#r$is_freeszrhs_inference..is_freerr)rCounterrPrr4elements)rrr5r#rr$rs  rcCsRdd}t|}||}|durt|}|d|}|d|}|d||fS)z8 1. gen rhs if rhs is None 2. '...' -> 'A' cSs0t|}tjD] }||vr|Sq td)NzNYou have used all `a` - `z`, there can't find a unused for einsum optimization)r2rstringascii_lowercase ValueError)counterusedr"r#r#r$get_used_labels  z2gen_equation_for_opteinsum..get_used_labelNrr)rrrr)rr5rrbroadcast_labelr#r#r$gen_equation_for_opteinsums   rcGst|}t|g|R\}}}|dkrt|d|g|RSt|||}t||\}}tj|g|Rddi\} } t|} | D]-} | ^\} }} }}| |ksRJd| | | |g}| |d}| t|g|Rq@t| dks|Jdt| | d S) z einsum v2 implementation. 1. Implement C++ EinsumOp. 2. V2 create the EinsumOp to calculate, so just a little verifty work in python. 3. V2 use opt_einsum.contract_path to optimize the multivariable einsum. r rZ einsum_callTzUAssume the first var_idx is smaller than the second_idx. opt_einsum can guarantee it.rrz4There must be one elements in list, but received %d.r) rr gen_einsum_oprr opt_einsumZ contract_pathr+rRrrQ)rr-Zn_oprr5rTZshapesZ opt_equationr_ZconsZvar_listpathrVrWeq__Zvar_sr#r#r$ einsum_v2s0  rcstdks Jdtrt|dStr'tttd|dSD] }t|dddgdq)t|dtdt dit j dj d }t }||d<fd d ttD}fd d ttD}jdd i|||d|d|S)z& EinsumOp Python Interface: r z&Only support two operands in EinsumOp.rrdtypeZfloat32Zfloat64einsumrcg|] }jdjdqSrr"create_variable_for_type_inferencerrhelperr-r#r$r^4z!gen_einsum_op..crrrrrr#r$r^8rZOperands)ZOutZ InnerCacheZXShape)typeZinputsZoutputsattrsN)r)rrr rr rrrstrrlocalsrrrr=Z append_op)rr-inprrcachesZxshaper#rr$r s>    rcGsddl}t|jddrt|g|RSt|}|dks!Jd|ddd^}}t|d ks8Jd |r>|dnd}t ||}t t t t ||\}}tt d d |}t|||\}} } } t| |d d|D\} } || | | | |f}t|}|}|S)a_ einsum(equation, *operands) The current version of this API should be used in dygraph only mode. Einsum offers a tensor operation API which allows using the Einstein summation convention or Einstain notation. It takes as input one or multiple tensors and produces as output one tensor. Einsum is able to perform a variety of tensor operations. Following lists a few: - for single operand - trace - diagonal - transpose - sum - for double operands - dot - outer - broadcasting and elementwise multiply - matrix multiply - batched matrix multiply - for many operads - broadcasting multiply - chained matrix multiply **The summation notation** - The tensor dimensions are labeled using uncased English letters. E.g., `ijk` relates to a three dimensional tensor whose dimensions are labeled i, j, and k. - The equation is `,` separated into terms, each being a distinct input's dimension label string. - Ellipsis `...` enables broadcasting by automatically converting the unlabeled dimensions into broadcasting dimensions. - Singular labels are called free labels, duplicate are dummy labels. Dummy labeled dimensions will be reduced and removed in the output. - Output labels can be explicitly specified on the right hand side of `->` or omitted. In the latter case, the output labels will be inferred from the input labels. - Inference of output labels - Broadcasting label `...`, if present, is put on the leftmost position. - Free labels are reordered alphabetically and put after `...`. - On explicit output labels - If broadcasting is enabled, then `...` must be present. - The output labels can be an empty, an indication to output as a scalar the sum over the original output. - Non-input labels are invalid. - Duplicate labels are invalid. - For any dummmy label which is present for the output, it's promoted to a free label. - For any free label which is not present for the output, it's lowered to a dummy label. - Examples - '...ij, ...jk', where i and k are free labels, j is dummy. The output label string is '...ik' - 'ij -> i', where i is a free label and j is a dummy label. - '...ij, ...jk -> ...ijk', where i, j and k are all free labels. - '...ij, ...jk -> ij', an invalid equation since `...` is not present for the output. **The summation rule** The summation procedure can be outlined as follows, although the actual steps taken may vary significantly due to implementation specific optimization. - Step 1: preparation for broadcasting, that is, transposing and unsqueezing the input operands to have each resulting dimension identically labeled across all the input operands. - Step 2: broadcasting multiply all the resulting operands from step 1. - Step 3: reducing dummy labeled dimensions. - Step 4: transposing the result tensor to match the output labels. **On trace and diagonal** The trace and diagonal are planned yet unimplemented features. Args: equation (`str`): The summation terms using the Einstein summation notation. operands (`list|Tensor`): The input tensors over which to compute the Einstein summation. The number of operands should equal the number of input terms in the equation. Returns: result (`Tensor`): the result tensor. Examples: .. code-block:: python import paddle paddle.seed(102) x = paddle.rand([4]) y = paddle.rand([5]) # sum print(paddle.einsum('i->', x)) # Tensor(shape=[], dtype=float32, place=CUDAPlace(0), stop_gradient=True, # 1.95791852) # dot print(paddle.einsum('i,i->', x, x)) # Tensor(shape=[1], dtype=float32, place=CUDAPlace(0), stop_gradient=True, # [1.45936954]) # outer print(paddle.einsum("i,j->ij", x, y)) # Tensor(shape=[4, 5], dtype=float32, place=CUDAPlace(0), stop_gradient=True, # [[0.00079869, 0.00120950, 0.00136844, 0.00187187, 0.00192194], # [0.23455200, 0.35519385, 0.40186870, 0.54970956, 0.56441545], # [0.11773264, 0.17828843, 0.20171674, 0.27592498, 0.28330654], # [0.32897076, 0.49817693, 0.56364071, 0.77099484, 0.79162055]]) A = paddle.rand([2, 3, 2]) B = paddle.rand([2, 2, 3]) # transpose print(paddle.einsum('ijk->kji', A)) # Tensor(shape=[2, 3, 2], dtype=float32, place=CUDAPlace(0), stop_gradient=True, # [[[0.95649719, 0.49684682], # [0.80071914, 0.46258664], # [0.49814570, 0.33383518]], # # [[0.07637714, 0.29374704], # [0.51470858, 0.51907635], # [0.99066722, 0.55802226]]]) # batch matrix multiplication print(paddle.einsum('ijk, ikl->ijl', A,B)) # Tensor(shape=[2, 3, 3], dtype=float32, place=CUDAPlace(0), stop_gradient=True, # [[[0.32172769, 0.50617385, 0.41394392], # [0.51736701, 0.49921003, 0.38730967], # [0.69078457, 0.42282537, 0.30161136]], # # [[0.32043904, 0.18164253, 0.27810261], # [0.50226176, 0.24512935, 0.39881429], # [0.51476848, 0.23367381, 0.39229113]]]) # Ellipsis transpose print(paddle.einsum('...jk->...kj', A)) # Tensor(shape=[2, 2, 3], dtype=float32, place=CUDAPlace(0), stop_gradient=True, # [[[0.95649719, 0.80071914, 0.49814570], # [0.07637714, 0.51470858, 0.99066722]], # # [[0.49684682, 0.46258664, 0.33383518], # [0.29374704, 0.51907635, 0.55802226]]]) # Ellipsis batch matrix multiplication print(paddle.einsum('...jk, ...kl->...jl', A,B)) # Tensor(shape=[2, 3, 3], dtype=float32, place=CUDAPlace(0), stop_gradient=True, # [[[0.32172769, 0.50617385, 0.41394392], # [0.51736701, 0.49921003, 0.38730967], # [0.69078457, 0.42282537, 0.30161136]], # # [[0.32043904, 0.18164253, 0.27810261], # [0.50226176, 0.24512935, 0.39881429], # [0.51476848, 0.23367381, 0.39229113]]]) rNZFLAGS_new_einsum1z!At least one operand is expected.rrrr rcSs |dS)Nr)rU)rBr#r#r$rNrOzeinsum..cSsg|]}|jqSr#)rrr#r#r$r^szeinsum..)osrenvironrrrrrr*r/r+r<r,rirr[rfrr)rr-rrrr5r.r6rMrXrYrZrerrrsresultr#r#r$rGs,   r):r>numpyrr8ZlinalgrrrZ manipulationrrrmathr r rnZfluid.frameworkr Zpaddler rZfluid.data_feederrrrZfluid.layer_helperrrrrrrZpaddle.common_ops_importr__all__r%r/r7rGr[rfrgrirxr~rrrrrrrrrrrrrr#r#r#r$sP     %7<* )p"  '