o eT-@sddlZddlZddlZddlZddlmZddlmZmZm Z ddl m Z m Z m Z mZGdddZde d ed dfd d Zd e d ed dfddZ    dBde dede ede ede ede ed dfddZdCd e dede ededed df ddZd e d dfdd Zde d!ed dfd"d#Zd$e d ee fd%d&Zd'e d(eegee fd ee fd)d*Zd+ed ee fd,d-Zd$e d ee fd.d/Zd+ed ee fd0d1Zd$e d ee fd2d3Zd4ed efd5d6Zd7ed efd8d9Z de d efd:d;Z!de dZ"d e d?ed e fd@dAZ#dS)DN)chain)CallableIterableOptional) TensorProto ModelProtoAttributeProto GraphProtoc@seZdZdeddfddZdS)ExternalDataInfotensorreturnNcCsfd|_d|_d|_d|_d|_|jD] }t||j|jq|jr&t |j|_|jr1t |j|_dSdS)N) locationoffsetlengthchecksumbasepath external_datasetattrkeyvalueint)selfr entryrID:\Projects\ConvertPro\env\Lib\site-packages\onnx/external_data_helper.py__init__s  zExternalDataInfo.__init__)__name__ __module__ __qualname__rrrrrrr sr r base_dirr cCst|}t|j}tj||}t|d*}|jr||j|j r*| |j |_ n | |_ WddSWddS1sBwYdS)a Loads data from an external file for tensor. Ideally TensorProto should not hold any raw data but if it does it will be ignored. Arguments: tensor: a TensorProto object. base_dir: directory that contains the external data. rbN) r _sanitize_pathrospathjoinopenrseekrreadraw_data)r r!infoZ file_locationexternal_data_file_path data_filerrrload_external_data_for_tensors    "r.modelcCs8t|D]}t|rt||tj|_|jdd=qdS)z Loads external tensors into model Arguments: model: ModelProto to load external data to base_dir: directory that contains external data N)_get_all_tensorsuses_external_datar.rDEFAULT data_locationr)r/r!r rrrload_external_data_for_model7s   r4rrrrrc Cs|dstd|jd|jdd=tj|_||dur!t|nd|dur*t|nd||dD]\}}|durG|j }||_ t ||_ q2dS)Nr*zTensor zGdoes not have raw_data field. Cannot set external data for this tensor.)rrrrr) HasField ValueErrornamerrEXTERNALr3ritemsaddrstrr) r rrrrrkvrrrrset_external_dataHs$     r>TFall_tensors_to_one_filesize_thresholdconvert_attributec Cst|}|r t|}|r/tt}|r|}|D]}|dr,t|j|kr,t ||qdS|D]!}|drRt|j|krR|j }t |sMtt}t ||q1dS)a Call to set all tensors with raw data as external data. This call should preceed 'save_model'. 'save_model' saves all the tensors data as external data after calling this function. Arguments: model (ModelProto): Model to be converted. all_tensors_to_one_file (bool): If true, save all tensors to one external file specified by location. If false, save each tensor to a file named with the tensor name. location: specify the external file that all tensors to save to. If not specified, will use the model name. size_threshold: Threshold for size of data. Only when tensor's data is >= the size_threshold it will be converted to external data. To convert every tensor with raw data to external data set size_threshold=0. convert_attribute (bool): If true, convert all tensors to external data If false, convert only non-attribute tensors to external data r*N) _get_initializer_tensorsr0r;uuiduuid1r5sys getsizeofr*r>r7_is_valid_filename) r/r@rrArBtensors file_namer Ztensor_locationrrrconvert_model_to_external_dataas(    rKcCs@t|D]}t|r|dstd|jdd=tj|_qdS)z Call to set all tensors which use external data as embedded data. save_model saves all the tensors data as embedded data after calling this function. Arguments: model (ModelProto): Model to be converted. r*raw_data field doesn't exist.N)r0r1r5r6rrr2r3)r/r rrr convert_model_from_external_datas   rM base_pathcCst|}tj||j}|dstdtj|s"t|d t|dC}| dd|j durL| }|j |krF| d|j || |j | }| |jt||j|| |WddS1smwYdS) a Writes tensor data to an external file according to information in the `external_data` field. Arguments: tensor (TensorProto): Tensor object to be serialized base_path: System path of a folder where tensor data is to be stored r*rLabzr+brN)r r$r%r&rr5r6isfiler'closer(rtellwriter*r>)r rNr+r,r- file_sizerrrrsave_external_datas"        "rWonnx_model_protocCstt|t|S)z=Scan an ONNX model for all tensors and return as an iterator.)rrC_get_attribute_tensorsrXrrrr0sr0 attributefuncccsL|jtjkr||jEdH|jtjkr"|jD] }||EdHqdSdS)zICreate an iterator through processing ONNX model attributes with functor.N)typer ZGRAPHgZGRAPHSZgraphs)r[r\graphrrr_recursive_attribute_processors   r`onnx_model_proto_graphccs:|jEdH|jD]}|jD] }t|tEdHqq dS)z@Create an iterator of initializer tensors from ONNX model graph.N)Z initializernoder[r`#_get_initializer_tensors_from_graphrarbr[rrrrcs   rccct|jEdHdS)z:Create an iterator of initializer tensors from ONNX model.N)rcr_rZrrrrCrCccsL|jD]}|jD]}|dr|jV|jEdHt|tEdHq qdS)zJCreate an iterator of tensors from node attributes of an ONNX model graph.tN)rbr[r5rgrIr`!_get_attribute_tensors_from_graphrdrrrrhs    rhccre)zDCreate an iterator of tensors from node attributes of an ONNX model.N)rhr_rZrrrrYrfrYr%cCs |dS)zRemove path components which would allow traversing up a directory tree from a base path. Note: This method is currently very basic and should be expanded. z/.)lstrip)r%rrrr#s r#filenamecCs td}||}|rdSdS)z8Utility to check whether the provided filename is valid.z^[^<>:;,?"*|/]+$TF)recompilematch)rjexprmrrrrHs  rHcCs|do |jtjkS)z?Returns true if the tensor stores data in an external location.r3)r5r3rr8)r rrrr1sr1 field_keycCs*t|jD] \}}|j|kr|j|=qdS)a  Removes a field from a Tensor's external_data key-value store. Modifies tensor object in place. Arguments: tensor (TensorProto): Tensor object from which value will be removed field_key (string): The key of the field to be removed N) enumeraterr)r roifieldrrrremove_external_data_fields  rsfilepathcCs8t|D]}t|r|drt|||dq|S)a Serializes data for all the tensors which have data location set to TensorProto.External. Note: This function also strips basepath information from all tensors' external_data fields. Arguments: model (ModelProto): Model object which is the source of tensors to serialize. filepath: System path to the directory which should be treated as base path for external data. Returns: ModelProto: The modified model object. r*)r0r1r5rWZ ClearField)r/rtr rrrwrite_external_data_tensorss   ru)NNNN)TNr?F)$rDr$rkrF itertoolsrtypingrrrZonnx_pbrrr r r r;r.r4rr>boolrKrMrWr0r`rcrCrhrYr#rHr1rsrurrrrsR  ($"(