o Me@sddlZddlZddlZddlZddlZddlZddlmZddl Z ddl m Z ddl m Z m Z ddlmZddlmZddlmZddlmZddlmZdd lmZmZdd lmZmZdd lmZdd lm Z dd l!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*m+Z+ddl,m-Z-ddl.m/Z/ddl0m1Z1ddl2m3Z3ddl4m5Z5m6Z6ddl7m8Z8m9Z9ddl:m;Z;mZ>ddl?m@Z@mAZAddl mBZBmCZCmDZDmEZEddl mFZFmGZGddlHmIZIdd lJmKZKGd!d"d"ZLdS)#N) defaultdict)fluidstatic)Metric) InputSpec)core)Variable)flatten) global_scope _to_name_str)Operator_non_static_mode)_current_expected_place) ParallelEnv)fleet)config_callbacks) Converter) ProgramHelper)Clusterget_default_cluster)Planner) Parallelizer)DistributedOperator)DistributedSaver)"DistributedDataLoaderFromGeneratorDistributedDataLoader)new_process_groupget_all_process_groups)DistributedContextget_default_distributed_context)Strategy)CollectionNamesget_collection)to_list get_dist_attrget_lr validate_opt)initialize_pg_in_full_modeget_input_split_info)get_cost_from_engine) get_loggerc@sheZdZdZ      dbddZddZddZd d Zd d Zd dZ      dbddZ ddZ ddZ ddZ ddZdcddZddZddZ   !       "ddd#d$Z   !   "ded%d&Z     "dfd'd(Z    ) * * )   dgd+d,Z - * *  *   dhd.d/Z       did0d1Zdjd2d3Z *    ) * * )  dkd4d5Z  * *   *  dld6d7Zdmd8d9Zd:d;Zdd?Zd@dAZdBdCZ dDdEZ!dFdGZ"dHdIZ#dJdKZ$dndLdMZ%dodNdOZ&dpdPdQZ'e(dRdSZ)e(dTdUZ*e(dVdWZ+e(dXdYZ,e(dZd[Z-e(d\d]Z.e(d^d_Z/e(d`daZ0dS)qEnginea An Engine object can provide the full power of auto parallel to users. With the help of it, users can easily obtain the abilities of the distributed training and inference. It also support the dynamic graph and static graph at the same time. Args: model (paddle.nn.Layer, optional): The model is an instance of paddle.nn.Layer. loss (Loss|Callable|None, optional): The loss can be a `paddle.nn.Layer` instance or any callable function taken the predicted values and ground truth values as input. It can be None when there is no loss. Default: None. optimizer (Optimizer|None, optional): The optimizer need to be set in training and should be None in eval and predict mode. Default: None. metrics (Metric|list[Metric]|None, optional): If metrics is set, all metrics will be calculated and output in train/eval mode. Default: None. cluster (Cluster|None, optional): The cluster represents the topology information about the used physical devices. Default: None. (Unused for now) strategy (Strategy|None, optional): The strategy is used to configure the parallelization and optimization behaviors. Default: None. Examples: .. code-block:: python import paddle import paddle.vision.transforms as T from paddle.distributed.fleet import auto from paddle.vision.datasets import MNIST transform = T.Compose([ T.Transpose(), T.Normalize([127.5], [127.5]) ]) train_dataset = MNIST(mode='train', transform=transform) valid_dataset = MNIST(mode='test', transform=transform) model = paddle.vision.models.LeNet() loss = paddle.nn.CrossEntropyLoss() optimizer = paddle.optimizer.Adam( learning_rate=0.001, parameters=model.parameters()) metrics = paddle.metric.Accuracy(topk=(1, 2)) engine = auto.Engine(model, loss, optimizer, metrics) # fit engine.fit(train_dataset, epochs=2, batch_size=64) # evaluate engine.evaluate(valid_dataset, batch_size=64) # predict engine.predict(valid_dataset, batch_size=64) # save engine.save("./my_model") # load engine.load("./my_model") NcCs4|rt|tjjst|std||_|r't|tjjtfs't|s'td||_|ri|_?d d d d |_@d d d d |_Ag|_Bg|_Cg|_Dg|_Eg|_Fd|_Gd |_Hd |_Id|_Jd |_K|jjL|_Md|_NdS) NzI'model must be sub classes of `paddle.nn.Layer` or any callable function.zW'loss' must be sub classes of `paddle.nn.Layer` or any callable function or a Variable.zg'optimizer' must be object of class `paddle.optimizer.Optimizer` or `paddle.fluid.optimizer.Optimizer`.z{} is not sub class of MetriczP'cluster' must be the object or class `paddle.distributed.auto_parallel.Cluster`zN'strategy' must be object of class `paddle.distributed.auto_parallel.Strategy`ZPOD_NAMEz0Distribute training by paddle.distributed.launchT)Z is_collectiveFtrainevalpredict)O isinstancepaddlennLayercallable TypeError_modelr_loss optimizerZ Optimizerrr' _optimizercopydeepcopyZ_orig_optimizerr$rformat __class____name___metricsrr_clusterr! _strategyr,loggingINFO_loggerosgetenvinforinit _executor distributedZget_rank _cur_rankZget_world_size_nranksr_saverrdefault_main_program_orig_main_progdefault_startup_program_orig_startup_progr Z_orig_dist_context_dist_contexts_fwd_main_progs_fwd_dist_contexts_serial_main_progs_serial_startup_progsrdict_dist_main_progs_dist_startup_progs _feed_vars _fetch_vars _planners _has_prepared_has_prepared_reader _inputs_spec _labels_spec_inputs_labels_losses_mode _skip_build_outside_dataloader _planned_mode _dygraph_modeZtuning_tuninghistory)selfmodellossr:metricsclusterstrategymetricrtWD:\Projects\ConvertPro\env\Lib\site-packages\paddle/distributed/auto_parallel/engine.py__init__|s                  zEngine.__init__c sg}g}t|tjjr+|durtt|\}}n@tt|}|d|}||d}n-t|tjjrN|dur=|d\}}n|d}|d|}||d}n tdt |j t |}t |}|j j jddfdd} |durt|D]\} } | dusJddt| } | | | ||qx|durt|D]\} } | dusJdd t| } | | | ||q||}||}||fS) Nrz._adjust_item_speccst|tjr&t||}|dur|||dS|||dSt|ttj tj j frQt ||}||durG||dS|||dSt|t jre|t|gt||dStdt|j)NzYThe sample's dtype returned of dataset should be number, np.ndarray or Tensor, but got {})r2npZndarrayrZ from_numpyappendbatchrrZVarBaseeagerZTensorZ from_tensornumbersNumbertyper7r>r@)itemname batch_sizespecsrzr{ryrtru_infer_item_spec s&      z3Engine._prepare_data_spec.._infer_item_speczReceive None input.inputlabel)r2r3ioZIterableDatasetnextiterZDatasetr7r>rr@r$rCdatasetry enumeratestr_validate_spec) rmdatasplitr inputs_spec labels_specinputslabelssamplerirrrtrru_prepare_data_specsJ        zEngine._prepare_data_specc Cs ts|jr td|rKt|tsJdt|t|ts(Jdt|t|t|ks4Jdt||D]\}}|j |j krJ|j |j q9|rt|ts[Jdt|t|tsiJdt|t|t|ksuJdt||D]\}}|j |j kr|j |j qz||fS)NzOnly support static graph mode.z&inputs should be list, but received {}z:the number of `inputs_spec` should be equal to `inputs`'s.z&labels should be list, but received {}z:the number of `labels_spec` should be equal to `labels`'s.) r rj ValueErrorr2listr>rrwziprxdesc set_shape) rmrrrrZ input_specrZ label_specrrtrtru_prepare_data_tensor0s^     zEngine._prepare_data_tensorc Cs|j|j|j}|j|j}|}gd}|jdjdkr7tt|D]}|jdj|vr6|j dddq%| g}t |jD]\}}|j|vrP| |qBg} t |D]'}|j} | |j|jt|| | d} | | t| } || qW| D] } |jd| qtt|D] }||t|7<qt |D]}|j|}|j ||dq| d|j|j<dS) N)create_py_readerZcreate_double_buffer_readerreadrrF)Zsync)rrT)rZrfrMrT global_blockopsrrangerwZ _remove_opZ_sync_with_cpprr}reversedrZ _prepend_opZ copy_fromr rZadd_dist_op_for_programinsertpopr`) rmdist_main_prog dist_contextdist_main_blockZrelated_reader_opsrZreader_op_indicesidxopZnew_reader_opsZ new_op_descZnew_opdist_oprtrtru_prepare_readerUsD          zEngine._prepare_readercCsi}|durKt|ttfr1t|dkr*t|dtr*|dD]\}}|||<q n!td|t|trD|D]\}}|||<q:ntd||durkt|ts^Jdt|j |D]\}}|||<qb|S)NrrzUnsupported data {}z)user_feeds must be a dict, but receive {}) r2rtuplerwrYitemsrr>rr@)rmrZ user_feedsmodeZfeedsrrtrtru _prepare_feeds0    zEngine._prepare_feedcs|durt|tsJdt|jggfdd}|dkr-|dj|d|dkrJj|d}t|D] \}}|dt||q<|dkrX|dj|dd d tt j D}|peg|phg}|d |fS) Nz+user_fetches must be a list, but receive {}cs^g}|D]}|r t|}|vr|||q|s(g|dSN) _is_local_varr r}index) group_namevar_listZ group_indicesvarvar_name fetch_indices fetch_namesrmrtru_process_fetch_groups   z3Engine._prepare_fetch.._process_fetch_groupr1rorpZmetrics_outputscSsg|]}|dqS)rrt).0rrtrtru z)Engine._prepare_fetch..fetches) r2rr>rr@r]rrr#r"FETCHES)rmZ user_fetchesrrrprrZuser_fetches_collectionrtrru_prepare_fetchs2  zEngine._prepare_fetchcCsi}|dur ||d<|dur|d|d<|dur||d<d} |dkr|| } t| dks.J| D] } || d|d<q0| d7} |j|d} | r|jD]6} || }g}|D] } ||| qU|r}| j|| }tt|D] \}}||| |<qp| d7} qKn|dkr|| }i}|D] } || |d | <q||d <| d7} t t j }i}|D]\}}|j|vr| |j} || ||p|j<q||d <|S) Nepochrsteplrrr1rorpzout%drr) rwr]rAr}update accumulaterr$rr#r"rr)rmoutsrrrrrrlogsZ group_idxZ loss_indicesr metric_varsrsZmetrics_indicesZ metric_outresultsrresZoutputs_indicesZlogs_outZcollect_fetchesZ logs_fetchrrrtrtru_prepare_loggersX         zEngine._prepare_loggercCs6||||||||d|j|<dS)NT)_build_plan _parallel _initializer_rmrrtrtru_prepare_programs    zEngine._prepare_programc Csts|jrPtd|_|jdt|j|j|j |j |j |_ |j ||j j|_|j j}|j j}|j j|_|j j|_|j j}|j j|_|j j}tn|j|d}|dur]dSg}g}g|_|j}|j}|jst !||vt"j#$add|j D|_dd|j D|_t%|j|j}|dkr|jrt&|jtj'j(st)|jsJdt%|j||j|_|dkr|s|jr|j D]}|*t%|j+||jqWdn1swYWdn1swYnt&|jt,sJdt%|j|_t-}|j.st/t0t1|j2d|_3|j|jd }t4||j|d } |d kr5|jdd }|5t6|||j7|j|| |j8|j9|j:|<t6|||j7|j|| |j8|j9|j;|<|j9j<|j:|_<||j=|<dS) NTz'Building model with 'to_static' method.cSg|]}|qSrtZ_create_feed_layerrsrtrtrur.rz!Engine._build..cSrrtrrrtrtrur1rr1zothe type of `loss` of the Engine arguments should be sub classes of `paddle.nn.Layer` or any callable function.z>the type of `loss` of the Engine arguments should be Variable.)rr)rrorpr/)Zfor_test)>r rjr3Zdisable_staticrFrIrr8r9rArarbprogram_helperZ build_programZconcrete_program main_programstartup_programZ input_varsrcZ label_varsrdZ output_varsZ loss_varsrerZ enable_staticrWgetrQclonerSrgr program_guardutilsZ unique_nameguardr$r2r4r5r6r}Zcomputerr Zhas_annotationrrrrNZ data_parallelr _set_recompute_ckptsrr;rBrCrTrVZgradient_scalerU) rmrZserial_main_progZserial_startup_progrrprsZ default_ctx feed_vars fetch_varsrtrtrurs                   z Engine._buildc Cs|jjstd|dksJ|||||j|_|j|_ddl m }||j |j |||j |j||jd|_|j|jjrO|j|j |_dSdS)Nz Please set `tuning.enable=True`.r/r)OptimizationTuner)rZrank)rkenablerrr_dp_world_sizes dp_world_size _dp_ranksdp_rankZtuner.optimization_tunerrto_dictrTrarbrMZ_optimization_tunerZtuneZrun_after_tuningZget_best_configrC)rmrrrrrtrtru_optimization_tuningxs2     zEngine._optimization_tuningc Cs|jdur ||_n||t||j||j|<|j||j|jd}|j|jd}|j|j}g}||D]}|j |j vrO| |j |j q>g|_ g|_ |D]}t|j||j|\}} |j ||j | qXdS)Nrr)ri_init_dist_contextrrTr^Zplanserial_feed_varsserial_main_programrrvarsr}rrr)rM) rmr inputs_var labels_varblock feed_listrZfeed_varrrrtrtrurs,     z Engine._planFcCs:t||j|j|j|}|s||jdS|dSr)rr^Z completerrTZparallelrMZ parallel_all)rmrZ all_ranksZ parallelizerrtrtrurs  zEngine._parallelc Cs|j|}|j}|j}|j|}|j}|j}t|jD]2\}} t| jD](\} } ||j| } | j| jksAJd|| j|| j|| } | | | q$qdS)Nz2'{}' mode op '{}' is different with '{}' op '{}'. ) rTZ_original_serial_main_programriblocksrrrr>Zget_op_dist_attr_for_programZset_op_dist_attr_for_program)rmrrZorigin_main_progZref_modeZref_dist_contextZref_origin_main_progZ ref_blocksZibrZioprZref_opZref_op_dist_attrrtrtrurs&   zEngine._init_dist_contextc CsL|j|j|j|<|j|j|j|<|j|j|j|<|j|j|j|<|j|j |j |<|j|j |j |<|j|j |_|jdkrat}|jjdkrQt|tn|D] }|j|jvr\qS|qSt}t|tjrqttj}|jjrt|jj|j dt!j"|jj|j dt"|jj|j d|j#r|j|}|j||j}|j$%||||j&dur tj'(||_&g}|j||j}|)D]} t*+| j,} | r| -.rq|/| q|r|0|} |j&1| t2|dr t2|dr |3||j4|j5|j6|jj7r$|j89d|j||j}|j&1|dSdS)Nrfullr _state_dict _dist_attrz(NOTE: parameters will be re-initialized.):rTrrWserial_startup_programrXZdist_main_programsrZZdist_startup_programsr[rr\Zserial_fetch_varsr]_serial_optimizerr;rNrrC auto_moder(Zcur_rankrMZranksZ instantiate _get_devicer2rZ CUDAPlacerZdev_idseedr3rr|randomrjrrJrKrExecutorZ list_varsr Zfind_varrZ get_tensorZ_is_initializedr}Z_prunerunhasattr_set_state_dict_strictrrZreinitrFrI) rmrZall_process_groups process_groupZplacerdist_main_programZ uninitializeddist_startup_progrZ scope_varZprune_startup_progrtrtrurs~              zEngine._initializer r+c Csd|_||||\|_|_|j|js||jn||j|j|dd|||| d}|jd|jd\}}t |||||j ||||| |j d }| dt|D]}i}||t|D]@\}}|d||z|jj|j||jj|jjd}Wn tjyYnwt|j}||||||||j}|d||q`| r|d | d kr|| | || || ||}d d |D}| ||dn|!|"||qS|#d||j$S) a Trains the model for a fixed number of epochs. If `valid_data` is set, evaluation will be done at the end of each epoch. Args: train_data (Dataset): An instance of paddle paddle.io.Dataset. Default: None. train_sample_split (int, optional): Each sample of the train dataset is assumed to be a (input, label) pair by default and has two items. If each sample has more than two items, train_sample_split specifies how to split these items into input and label. The items before it are input and the left are label. Default: None. batch_size (int, optional): The batch size of train_data and valid_data if provided. The user's data will be used directly without batching if set to None. Default: 1. epochs (int, optional): The number of epochs to train the model. Default: 1. steps_per_epoch (int, optional): The total number of steps (batches of samples) is executed in one epoch before stating the next one. If None, it is equal to the number samples in your dataset divided by the batch size. Default: None. valid_data (Dataset, optional): An instance of paddle paddle.io.Dataset used for evaluation at the end of epoch. No evaluation will be done if set to None. Default: None. (Unsupported for now) valid_freq (int, optional): Only relevant if valid_data is provided. This specifies how many training epochs before a new evaluation is performed. Default: 1. valid_sample_split (int, optional): Only relevant if valid_data is provided. Each sample of the valid dataset is assumed to be a (input, label) pair by default and has two items. If each sample has more than two items, valid_sample_split specifies how to split these items into input and label. The items before it are input and the left are label. Default: None. valid_steps (int, optional): Only relevant if valid_data is provided. It is the total number of steps (batches of samples) to draw before stopping validation at the end of every epoch. If None, validation will run until the `valid_data` dataset is exhausted. The validation will start from the beginning of the dataset at each epoch. Default: None. collate_fn(callable, optional): function to generate mini-batch data by merging the sample list, None for only stack each fields of sample in axis 0. Default None. callbacks (Callback|None, optional): A list of `Callback` instances to apply during training. Default: None. (Unused for now) Returns: None Examples: .. code-block:: python import paddle import paddle.vision.transforms as T from paddle.distributed.fleet import auto from paddle.vision.datasets import MNIST transform = T.Compose([ T.Transpose(), T.Normalize([127.5], [127.5]) ]) train_dataset = MNIST(mode='train', transform=transform) model = paddle.vision.models.LeNet() loss = paddle.nn.CrossEntropyLoss() optimizer = paddle.optimizer.Adam( learning_rate=0.001, parameters=model.parameters()) metrics = paddle.metric.Accuracy(topk=(1, 2)) engine = auto.Engine(model, loss, optimizer, metrics) engine.fit(train_dataset, epochs=2, batch_size=64) r/FF)rcapacityiterablerepochssteps_per_epoch collate_fnNr) enginerr stepslog_freq save_freqsave_dirverboserpZacc_step fetch_listuse_program_cache return_numpyrrcSsi|] \}}d||qS)Zval_rt)rrvalrtrtru szEngine.fit..)%rfrrarbr_r _switch_mode"_prepare_dataloader_from_generatorrr_steps _metrics_name_k_stepson_beginrZon_epoch_beginron_batch_beginrKrrrC use_cacherr EOFExceptionr&r;r on_batch_endevaluaterr_reset_metricsZ on_epoch_endon_endrl)rmZ train_dataZtrain_sample_splitrr r rrr valid_datavalid_sample_splitZ valid_freqZ valid_stepsr callbacksrZtrain_dataloaderrrcbksrrr_rrZval_logsrtrtrufitsT             z Engine.fitc  Cs<d|_||||\|_|_|j|js||jn||j|j|dd|||d} |jd|jd\} } t |||||| d} | j } | d| | di}t | D];\}}| d||z|jj|j| |jj|jjd }Wn tjy}Ynw||d|d| | |j}| d||qV| d|||S) a Evaluate the loss and metrics of the model on evaluation data. Args: valid_data (Dataset): An instance of paddle paddle.io.Dataset. Default: None. valid_sample_split (int, optional): Each sample of the eval dataset is assumed to be a (input, label) pair by default and has two items. If each sample has more than two items, valid_sample_split specifies how to split these items into input and label. The items before it are input and the left are label. Default: None. batch_size (int, optional): The batch size of valid_data. The user's data will be used directly without batching if set to None. Default: 1. steps (int, optional): It is the total number of steps (batches of samples) to draw before stopping evaluation. If None, evaluation will run until the `valid_data` dataset is exhausted. The evaluation will start from the beginning of the dataset in each run. Default: None. collate_fn(callable, optional): function to generate mini-batch data by merging the sample list, None for only stack each fields of sample in axis 0. Default None. callbacks (Callback|None, optional): A list of `Callback` instances to apply during evaluating. Default: None. (Unused for now) Returns: None Examples: .. code-block:: python import paddle import paddle.vision.transforms as T from paddle.distributed.fleet import auto from paddle.vision.datasets import MNIST transform = T.Compose([ T.Transpose(), T.Normalize([127.5], [127.5]) ]) valid_dataset = MNIST(mode='test', transform=transform) model = paddle.vision.models.LeNet() loss = paddle.nn.CrossEntropyLoss() metrics = paddle.metric.Accuracy(topk=(1, 2)) engine = auto.Engine(model, loss, metrics=metrics) engine.evaluate(valid_dataset, batch_size=64) r0r Frr r rr rNr)rrrrrp)rrpr)rfrrarbr_rrrrrrrr!rr"rKrrrCr#rrr$rr%r(r')rmr)r*rrrrr+rZvalid_dataloaderrrr,Z eval_stepsrrr-rrtrtrur&s`9      zEngine.evaluatec Cs>d|_||||\|_|_|j|js||jn||j|j|dd|||d}|jd|jd\} } g} t |||d} |j } | dd| ii}t |D]F\}}| d||z|jj|j| |jj|jjd }Wn tjywYn w||d|d| | |j}| d||| t|d qP| d|| S) a Compute the output predictions on testing data. Args: test_data (Dataset): An instance of paddle paddle.io.Dataset. Default: None. test_sample_split (int, optional): Each sample of the test dataset is assumed to be a (input, label) pair by default and has two items. If each sample has more than two items, test_sample_split specifies how to split these items into input and label. The items before it are input and the left are label. Default: None. batch_size (int, optional): The batch size of test_data. The user's data will be used directly without batching if set to None. Default: 1. steps (int, optional): It is the total number of steps (batches of samples) to draw before stopping predict. If None, predict will run until the `test_data` dataset is exhausted. The predict will start from the beginning of the dataset in each run. Default: None. collate_fn(callable, optional): function to generate mini-batch data by merging the sample list, None for only stack each fields of sample in axis 0. Default None. callbacks (Callback|None, optional): A list of `Callback` instances to apply during testing. Default: None. (Unused for now) Returns: None Examples: .. code-block:: python import paddle import paddle.vision.transforms as T from paddle.distributed.fleet import auto from paddle.vision.datasets import MNIST transform = T.Compose([ T.Transpose(), T.Normalize([127.5], [127.5]) ]) valid_dataset = MNIST(mode='test', transform=transform) model = paddle.vision.models.LeNet() engine = auto.Engine(model) engine.predict(valid_dataset, batch_size=64) r1r Fr/Nr)rrrrr)rfrrarbr_rrrrrrr!rr"rKrrrCr#rrr$rr%r}rvaluesr()rmZ test_dataZtest_sample_splitrrrr+rZtest_dataloaderrrrr,Z test_stepsrrr-rrtrtrur13sP5     zEngine.predictrTcCsv|dur ||||| |\|_|_|j|js!||jn||j|j|d|||||||| | | | d }|S)NF) return_listrshuffle drop_lastr num_workersuse_buffer_readeruse_shared_memorytimeoutworker_init_fnr r ) to_moderrarbr_rfrr_prepare_dataloader)rmrrr2r3rr4r5r6r7r8r r  sample_splitr dataloaderrtrtrur<s0   zEngine.dataloaderr c  Csr| dur || ||| |\|_|_|j|js!||jn||j|j||||d||||| | d } | S)NF) rr use_double_bufferr r1use_multiprocessr3rr r r) r9rrarbr_rfrrr)rmrr r=r r>r3rr r rr;rr<rtrtrudataloader_from_generators,    z Engine.dataloader_from_generatorcCsD|dur |||jstd|j|jrdS||}||}||}||}||_||_|s6|rXd|_| ||||\}}|jdurMt |_|jdurWt |_n&|s\|rtd|_ |jdurit |_|jdurst |_n |jrz|js~Jd|||_|_|||_|_|j|js||jdS||jdS)Nz7Please set mode to be prepared with `prepare(mode=...)`Tz;Please call the dataloader(...) before calling prepare(...))r9rfrr_r_validate_varsrQrSrgrrrPrRrhrarbrcrdrr)rmrrrrrrrrtrtrupreparesT               zEngine.preparec Cs|dur ||||||j}|||j\}}|jr'|j|js'||jj|j |||j j |j j d}| |ddd|||j} | S)N)feedrrr)r9rrfrrhr`rrKrrrCr#rr) rmrrBrrZ feed_dictrrrrrtrtrurs(  z Engine.runcCs|jjr|dur||jdksJd||j||j}|j|j|j}|j|j|j}|}|j |jd}|j |jd}g}||D]'}|j |j vr[| |j |j qI| ||j}|j|j| |qItj}t||Ft|fid|d|d|d|d |d |d |d |d |d| d| d| d| d| d|jjd|jd|j}Wd|S1swY|S)Nr9Requires batch_size:[{}] to be divisible by k_steps:[{}].rrrplacesr1rr2r3rr4r5r6r7r8r r  split_datadata_parallel_world_sizedata_parallel_rank)rCgradient_merger r>rZrfrMr[rr\rrr}_clone_variable persistablerset_original_id original_idr3r cuda_placesrrrErr)rmrr1rr2r3rr4r5r6r7r8r r rrrrrrrcopy_varrDr<rtrtrur:5s|           zEngine._prepare_dataloaderc Cs|jjr|dur||jdksJd||j||j}|j|j|j} |j|j|j} | }|j |jd}|j |jd}g}||D]'}|j |j vr[| |j |j qI| ||j}|j|j| |qItj}t| | Atdid|d|d|d|d |d |d |d |d |d|d| d| d| d|jjd|jd|j}Wdn1swY||S)NrrCrrrrr r=r r1r>r3rDrr r rrErFrGrt)rCrHr r>rZrfrMr[rr\rrr}rIrJrrKrLr3rrMrrrErrr)rmrr r=r r1r>r3rr r rrrrrrrrrNrDr<rtrtrur{sr          z)Engine._prepare_dataloader_from_generatorcCs0d|_||||\|_|_||j||dS)Nr/)rfrrarbr)rmZ tune_dataZtune_sample_splitrrtrtru_tunes  z Engine._tunecCst|}|jjj|_|durXt|D]E\}}t|tstd|j dur,t d |||jdkrWt |j }|d|jdksKJd |j d|j|d|j<||_ q|p[gS)Nz9'spec' must be object of class `paddle.static.InputSpec`.zrrx)rmrrrzrxrtrtrurs2      zEngine._validate_speccCs<t|}|durt|D] \}}t|tstdq |pgS)Nz'var' must be a `Variable`.)r$rr2rr7)rmrrrrtrtrur@s zEngine._validate_varscCst|}||jjvSr)r rrr)rmrrrtrtrurszEngine._is_local_varcCs|jj}t|jtjjr#t|jdr|jjj dvr|jj j }n|j }n|j }|j r@|dd|_ |jjj |d}|j |dSdS)Ngpt)ZGPTForPretrainingZGPTForPretrainingAuto)z Model ClasszApplied Recompute ckpts)rC recomputer2r8r3r4r5rr?r@rPZ checkpointsrrFrI)rmrQZ exact_ckptsrrtrtrurs  zEngine._set_recompute_ckptscCs|jD]}|qdSr)rAreset)rmrsrtrtrur's  zEngine._reset_metricscCs2|jrdgng}|jD] }|t|q |S)Nro)r9rAextendr$r)rmZ metrics_namemrtrtrur s zEngine._metrics_namecCs4||jvs Jd||||j|j|_dS)Nz5{} model is not ready, please call `prepare()` first.)rZr>r9rTrr;rrtrtrurs  zEngine._switch_modecCs |dvs Jd|||_dS)Nr.z5mode {} should be one of ['train', 'eval', 'predict'])r>rfrrtrtrur9s  zEngine.to_modec CsJ|j||j}|j|}t||}t|||}|j|d}||dS)N)strict)rZrMrTr%rconvertZset_state_dict) rmrrUZ state_dictZ dist_attrprogramrZ cur_dist_attr converterrtrtrurs     zEngine._set_state_dictcCs|r+|j|jvs J|j|j}|j|j|j}|j|j}|jj||||ddSd|jvs2J|jdd}|jdd}|jd|j}|jj ||||j |ddS)a Saves the model, parameters, optimizer state to path. If `training` is set to False, only inference model will be saved. Args: path (str): The file prefix to save model. The format is 'dirname/file_prefix' or 'file_prefix'. if empty str. A exception will be raised. training (bool, optional): Whether to save for training. If not, save for inference only. If `training` is set to True, the optimizer state will be saved. Otherwise, only the model and parameters are saved. This function will silently overwrite existing file at the target location. Default: True. Returns: None Examples: .. code-block:: python import paddle import paddle.vision.transforms as T from paddle.distributed.fleet import auto from paddle.vision.datasets import MNIST transform = T.Compose([ T.Transpose(), T.Normalize([127.5], [127.5]) ]) train_dataset = MNIST(mode='train', transform=transform) model = paddle.vision.models.LeNet() loss = paddle.nn.CrossEntropyLoss() optimizer = paddle.optimizer.Adam( learning_rate=0.001, parameters=model.parameters()) metrics = paddle.metric.Accuracy(topk=(1, 2)) engine = auto.Engine(model, loss, optimizer, metrics) engine.fit(train_dataset, epochs=1, batch_size=64) engine.save("./my_model") )serial_programrrr1rr)rWN) rfrWrZrMrTrOsaver\r]Zsave_inference_modelrK)rmpathZtrainingrYrrrrrtrtrurZ's,-    z Engine.savecCs(||_|j||\|_|_|j|jfS)a Load the stored model, parameters and optimizer states. Args: path (str): The prefix of files storing the model states and optimizer states. strict (bool, optional): Whether to skip the loading of mismatch parameter or raise an error when mismatch happens (not found the parameter in file storing model states of or receives a mismatch shape). Default: True. load_optimizer (bool, optional): If True, the stored optimizer states is restored. Otherwise, the optimizer states is initialized from scratch. Default: True. Returns: None Examples: .. code-block:: python import paddle import paddle.vision.transforms as T from paddle.distributed.fleet import auto from paddle.vision.datasets import MNIST transform = T.Compose([ T.Transpose(), T.Normalize([127.5], [127.5]) ]) train_dataset = MNIST(mode='train', transform=transform) model = paddle.vision.models.LeNet() loss = paddle.nn.CrossEntropyLoss() optimizer = paddle.optimizer.Adam( learning_rate=0.001, parameters=model.parameters()) metrics = paddle.metric.Accuracy(topk=(1, 2)) engine = auto.Engine(model, loss, optimizer, metrics) engine.fit(train_dataset, epochs=1, batch_size=64) engine.save("./my_model") engine.load("./my_model") )rrOloadrr)rmr[rUZload_optimizerrtrtrur\ls . z Engine.loadcCs|jjdkr|jddS|dur|n|j}|dusJd||jvr1td|t|j | ||durV|j|sV| ||_ | ||_ ||||n(ts\|jr`td|jdtj}|jru|js~|j|s~tdt||\}}|j|fS)a Get and Print cost, including memory of every rank, max memory among all ranks, and the global cost of one step based on communication cost(computation cost is 0 by default). In the future, the flops information of every rank and global cost including computation cost will be added. Args: inputs_spec(InputSpec): The specification of inputs. Default: None. labels_spec(InputSpec): The specification of labels. Default: None. mode (str): The engine mode must be in ["train", "predict", "eval"]. Default: None. Returns: Return the global execution time (ms) and max memory (B). rzMThe cost will be calcudated in the search process when the auto mode is full.NzPlease set mode.z'The mode {} is not in accepted modes {}z\Please call `prepare()` or `fit()` or `evaluate()` or `predict()` before calling `cost()`.zThe program whose cost to be estimated must be static default program. Otherwise, please call `prepare()`before calling `cost()`.)rCrrFrIrfr_rr>rkeysr9rrarbrrr rjr3rrPrrr*time)rmrrrrWZ global_costZ max_memoryrtrtrucostsL          z Engine.costcC|j|j|jSr)rZrfrMrmrtrtrurzEngine.main_programcCr`r)r[rfrMrartrtrurrbzEngine.startup_programcC |j|jSr)rTrfrartrtrur zEngine.dist_contextcCrcr)rWrfrartrtrurrdzEngine.serial_main_programcCrcr)rXrfrartrtrurrdzEngine.serial_startup_programcCrcr)r]rfrartrtrurrdzEngine.fetch_varscC|jSr)rcrartrtrurz Engine.inputscCrer)rdrartrtrurrfz Engine.labels)NNNNNN)F)NrrNrNrNNrNNNr+)NrNrNNr+)NrNNNr+) rFFNrTTrNrNrN) r TTFTrrNNrN)NNNNNNN)NNNN) TrFFNrTTrNrN) NTTFFTrrNN)Nr)T)TT)NNN)1r@ __module__ __qualname____doc__rvrrrrrrrrrrrrrr.r&r1r<r?rArr:rrOrr@rrr'rrr9rrZr\r_propertyrrrrrrrrrtrtrtrur-=s&@ mG%-) 9 t!  G . q d / * 7 I A  E 4?       r-)MrGr<rDrrnumpyr| collectionsrr3Z paddle.utilsrrrZ paddle.metricrZ paddle.staticrZ paddle.fluidrrZpaddle.fluid.layers.utilsr Zpaddle.fluid.executorr r Zpaddle.fluid.frameworkr r rrZpaddle.fluid.dygraph.parallelrZpaddle.distributedrr+rrXrhelperrrqrrZ planner_v2rZparallelizer_v2rrrZ dist_saverrZ dist_loaderrrrrrrrr rrr!Z interfacer"r#r$r%r&r'r(r)Zcost.estimate_costr*Zutils.log_utilsr,r-rtrtrtrusL