o Me7-@sddlZddlZddlmZddlZddlmZddlmZddl m Z ddl m Z ddl mZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZeejZ       dddZdddZdS)N)Enum) Optimizer) get_logger)in_dygraph_mode)ShardingOptimizerStage2)ShardingStage2)ShardingStage3)ShardingScaler)GroupShardedOptimizerStage2)GroupShardedStage2)GroupShardedStage3)GroupShardedScalerFc Cs|t|tjjs Jdt|tsJd|dvsJddd} tt| |} |dur7t| dkr7t d |d vrtt d t d t d t r`t |j|||d }t|||||d}n7t||||d }t|||||d}n#|dkrt rt||||||| d}nt||||||| d}nt dt|tjjrt rt|}nt|}t d t dt d |||fS)a Use group_sharded_parallel can perform group shared configuration on the model, optimizer and GradScaler. Level has three string options, 'os', 'os_g' and 'p_g_os' corresponds to three different usage scenarios: optimizer state segmentation, optimizer state + gradient segmentation, and parameter + gradient + optimizer state segmentation. Usually, optimizer state + gradient segmentation is actually a re optimization of optimizer state segmentation, so optimizer state + gradient segmentation can be used to realize optimizer state segmentation. Args: model (Layer): The layer to be wrapped with group_sharded_parallel. optimizer (Optimizer): The optimizer to be wrapped with group_sharded_parallel. level (str): The different level of the group sharded. Such as `os`, `os_g`, `p_g_os`. scaler (GradScaler, optional): If AMP is used, you need to pass GradScaler. Defaults to None, indicating that GradScaler is not used. group (Group, optional): The group instance. Defaults to None, indicating that the default environment group is used. offload (bool, optional): Whether to use the offload function. Defaults to False, which means that the offload function is not used. sync_buffers (bool, optional): Whether to broadcast model buffers. It is generally used when there are registered model buffers. Defaults to False, indicating that model buffers are not used. buffer_max_size (int, optional): The max size of the buffer used to integrate gradient in `os_g`. The larger the size, the more GPU memory will be used. Defaults to 2**23, which means that the dimension of the buffer is 2**23. segment_size (int, optional): The smallest size of parameter to be sharded in `p_g_os`. Defaults to 2**20, indicating that the dimension of the minimum segmented parameter is 2**20. sync_comm (bool, optional): Whether to use synchronous communication, only in `p_g_os` used. Defaults to False, indicating that asynchronous communication is used. Returns: model: A wrapper for group sharded given model. optimizer: A wrapper for group sharded given optimizer. scaler: A wrapper for group sharded given scaler. Examples: .. code-block:: python # required: distributed import paddle from paddle.fluid.dygraph.nn import Linear from paddle.distributed import fleet from paddle.distributed.sharding import group_sharded_parallel fleet.init(is_collective=True) group = paddle.distributed.new_group([0, 1]) model = Linear(1000, 1000) clip = paddle.nn.ClipGradByGlobalNorm(clip_norm=1.0) optimizer = paddle.optimizer.AdamW(learning_rate=0.001, parameters=model.parameters(), weight_decay=0.00001, grad_clip=clip) # wrap sharding model, optimizer and scaler model, optimizer, scaler = group_sharded_parallel(model, optimizer, "p_g", scaler=scaler) img, label = data label.stop_gradient = True img.stop_gradient = True out = model(img) loss = paddle.nn.functional.cross_entropy(input=out, label=label) loss.backward() optimizer.step() optimizer.clear_grad() z2The model must be the instance of paddle.nn.Layer.zAThe optimizer must be the instance of paddle.optimizer.Optimizer.)osos_gp_g_osz%The level must be os, os_g or p_g_os.cSs |jtjkSN)ZdtypepaddleZfloat16)paramrYD:\Projects\ConvertPro\env\Lib\site-packages\paddle/distributed/sharding/group_sharded.py check_dtypes z+group_sharded_parallel..check_dtypeNrz Please enter the correct scaler.)rrz******************************z6Sharded level os uses sharded level os_g achieved now.)paramsZoptimgroupoffload)r sync_buffersbuffer_max_sizer) optimizerrr segment_sizer sync_commzPlease enter the correct level.zIf there is a communication hang using group sharded, please check whether the communication operations of each process are unified.) isinstancernnZLayerrlistfilter parameterslen ValueErrorlogger_inforr Z_parameter_listr rrr rampZ GradScalerr r ) modelrlevelZscalerrrrrrr rZ params_fp16rrrgroup_sharded_parallel8s@          r-cCstdtj|rJd|tj|ddtj|d}t|t t fr1t |j |n"t|ttfrO|jr=dnd}|j|dt |j |ntd|d urpt|d s`Jd tj|d }t |j|td d S)a Group sharded encapsulated model and optimizer state saving module. Note: If using save_group_sharded_model saves the model. When loading again, you need to set the model or optimizer state before using group_sharded_parallel. Args: model (Layer): A wrapper for group sharded given model. output (str): Save directory. optimizer (Optimizer, optional): Group sharded encapsulated optimizer. Defaults to None, indicating that the optimizer state is not saved. Examples: .. code-block:: python # required: distributed import paddle from paddle.fluid.dygraph.nn import Linear from paddle.distributed import fleet from paddle.distributed.sharding import group_sharded_parallel, save_group_sharded_model fleet.init(is_collective=True) group = paddle.distributed.new_group([0, 1]) model = Linear(1000, 1000) clip = paddle.nn.ClipGradByGlobalNorm(clip_norm=1.0) optimizer = paddle.optimizer.AdamW(learning_rate=0.001, parameters=model.parameters(), weight_decay=0.00001, grad_clip=clip) # wrap sharding model, optimizer and scaler model, optimizer, scaler = group_sharded_parallel(model, optimizer, "p_g", scaler=scaler) img, label = data label.stop_gradient = True img.stop_gradient = True out = model(img) loss = paddle.nn.functional.cross_entropy(input=out, label=label) loss.backward() optimizer.step() optimizer.clear_grad() # save model and optimizer state_dict save_group_sharded_model(model, optimizer, output=output_dir) zC==========Begin to save group sharded model and optimizer==========z7Saving directory ({}) should be a directory, not a fileT)exist_okz model.pdmodelF) convert2cpuzBPlease use the layer which is wrapped with group_sharded_parallel.N_optimzFPlease use the optimizer which is wrapped with group_sharded_parallel.z model.pdoptzA==========End to save group sharded model and optimizer==========)r(r)rpathisfileformatmakedirsjoinr!rr rsaveZ_layerZ state_dictrr Z_offloadZget_all_parametersr'hasattrr0)r+outputrZ output_modelr/Z output_optrrrsave_group_sharded_models<- r9)NNFFrrFr)rloggingenumrrZpaddle.optimizerrZ"paddle.distributed.utils.log_utilsrZpaddle.fluid.frameworkrZTpaddle.distributed.fleet.meta_optimizers.dygraph_optimizer.sharding_optimizer_stage2rZ?paddle.distributed.fleet.meta_parallel.sharding.sharding_stage2rZ?paddle.distributed.fleet.meta_parallel.sharding.sharding_stage3rZ>paddle.distributed.fleet.meta_parallel.sharding.sharding_utilsr ZNpaddle.distributed.fleet.meta_parallel.sharding.group_sharded_optimizer_stage2r ZDpaddle.distributed.fleet.meta_parallel.sharding.group_sharded_stage2r ZDpaddle.distributed.fleet.meta_parallel.sharding.group_sharded_stage3r ZCpaddle.distributed.fleet.meta_parallel.sharding.group_sharded_utilsr WARNINGr(r-r9rrrrs4