o Ne2@sPddlZddlmZddlmZdaddZddZd d Zd d Z d dZ dS)N) collective) ParametercCstduo ttdddkS)NZPADDLE_TRAINERS_NUM1)__parallel_ctx__clz__intosgetenvr r TD:\Projects\ConvertPro\env\Lib\site-packages\paddle/fluid/dygraph/parallel_helper.py_is_data_parallel_modes  r cCstduS)Nrr r r r _is_parallel_ctx_initializedsrcCstdusJd|adS)Nz-ParallelContext can only be initialized once.r)Zccl_parallel_contextr r r _set_parallel_ctx s rcCstdusJdtdS)Nz&ParallelContext should be initialized.)rinitr r r r _init_parallel_ctx's  rcCs6|D]}|jrqt|tr|jrtj|dddqdS)NrT)Z sync_mode)Zis_distributed isinstancerZ trainablerZ _broadcast) parametersparamr r r _broadcast_parameters.s r) r ZlayersrZ frameworkrrr rrrrr r r r s