o MeOã@s^ddlZddlmmZddlmZddd„Zddd„Zdd „Zd d „Z  ddd„Z dS)éN)Ú collectiveécCs|j|kr tdƒ‚dS)Nz/The tensor for alltoall is not correctly-sized.)ÚshapeÚ RuntimeError)ÚtensorrÚnranks©rú`D:\Projects\ConvertPro\env\Lib\site-packages\paddle/distributed/communication/stream/alltoall.pyÚ_check_tensor_shapes ÿr cCs4t|ƒ|kr tdƒ‚|D] }|j|krtdƒ‚q dS)Nz4The tensor_list for alltoall is not correctly-sized.)Úlenrr)Z tensor_listrrrrrr Ú_check_tensor_list_shapes ÿ ÿÿÿr cCsV|durt ¡n|}t||j|jƒ|r|j ||¡S|j |||¡}|r)| ¡|S)N) rÚ_get_default_groupr rrÚ process_groupZalltoall_tensor_on_calc_streamZalltoall_tensorÚwait)Z out_tensorZ in_tensorÚgroupÚsync_opÚuse_calc_streamÚtaskrrr Ú_alltoall_tensor_in_dygraph#sÿrcCsŽ|durt ¡n|}t|ƒdkrtdƒ‚t|ƒdkr$|dd„|Dƒ7}n t||dj|jƒ|r7|j ||¡S|j  |||¡}|rE|  ¡|S)Nrz*The input tensor_list should not be empty.cSsg|]}t |¡‘qSr)ÚpaddleZ empty_like)Ú.0rrrr Ú <s ÿz(_alltoall_in_dygraph..) rr r rr rrrZalltoall_on_calc_streamÚalltoallr)Zout_tensor_listZin_tensor_listrrrrrrr Ú_alltoall_in_dygraph4s(   ÿ ÿÿ ÿrTFcCs¤|dur | ¡s tdƒ‚|s|rtdƒ‚|durtdƒ‚|dur$tdƒ‚t ¡rNt |¡}t |¡}|r>|r>t|||||ƒS|sJ|sJt|||||ƒStdƒ‚tdƒ‚)aÞ Scatter a tensor (or a tensor list) across devices and gather outputs to another tensor (or a tensor list, respectively). Args: out_tensor_or_tensor_list (Union[Tensor, List[Tensor]]): The output. If it is a tensor, it should be correctly-sized. If it is a list, it should be empty or contain correctly-sized tensors. Its data type should be the same as the input. in_tensor_or_tensor_list (Union[Tensor, List[Tensor]]): The input to scatter (must be specified on the source rank). If it is a tensor, it should be correctly-sized. If it is a list, it should contain correctly-sized tensors. Support float16, float32, float64, int32, int64, int8, uint8 or bool as the input data type. group (Group, optional): Communicate in which group. If none is given, use the global group as default. sync_op (bool, optional): Indicate whether the communication is sync or not. If none is given, use true as default. use_calc_stream (bool, optional): Indicate whether the communication is done on calculation stream. If none is given, use false as default. This option is designed for high performance demand, be careful to turn it on except you are clearly know its meaning. Returns: Return a task object. Warning: This API only supports the dygraph mode now. Examples: .. code-block:: python # required: distributed import paddle import paddle.distributed as dist dist.init_parallel_env() out_tensor_list = [] if dist.get_rank() == 0: data1 = paddle.to_tensor([[1, 2, 3], [4, 5, 6]]) data2 = paddle.to_tensor([[7, 8, 9], [10, 11, 12]]) else: data1 = paddle.to_tensor([[13, 14, 15], [16, 17, 18]]) data2 = paddle.to_tensor([[19, 20, 21], [22, 23, 24]]) task = dist.stream.alltoall(out_tensor_list, [data1, data2], sync_op=False) task.wait() print(out_tensor_list) # [[[1, 2, 3], [4, 5, 6]], [[13, 14, 15], [16, 17, 18]]] (2 GPUs, out for rank 0) # [[[7, 8, 9], [10, 11, 12]], [[19, 20, 21], [22, 23, 24]]] (2 GPUs, out for rank 1) NzjThe group should not be None and all ranks which invoke this operation should be the member of this group.z5use_calc_stream can only be true in sync op behavior.zThe output should be specified.zThe input should be specified.z:The output and input should be both tensor or tensor list.zIpaddle.distributed.stream.alltoall is only supported in dygraph mode now.)Z is_memberrÚ frameworkZin_dygraph_moderZ is_tensorrr)Zout_tensor_or_tensor_listZin_tensor_or_tensor_listrrrZ out_is_tensorZ in_is_tensorrrr rOs>/ÿÿ  þþÿÿr)r)NTF) rZpaddle.fluid.frameworkZfluidrZpaddle.distributedrr r rrrrrrr Ús    ü