o ed@sdZddlZddlZddlZddlZddlmZddlmZddl m Z m Z ddl m Z ddlmZmZddlmZmZmZmZmZmZmZmZmZmZmZdd lmZdd lm Z d d l!m"Z"m#Z#d d l$m%Z%m&Z&m'Z'd dl(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/d dl(m0Z1erd dl2m3Z3e-4e5Z6edZ7dZ8e GdddZ9e GdddZ:e GdddZ;dedee>ddfddZ?e/dd d!d"ee;d#ed2ee:d#ed6eer)r2r4r4r5r6is z!CommitOperationCopy.__post_init__) r7r8r9r:r;r<r?rr6r4r4r4r5r=Ps  r=c@sveZdZUdZeed<eeeee fed<e dddZ e ed<dd d Z edd edee fd d ZdefddZdS)CommitOperationAdda Data structure holding necessary info to upload a file to a repository on the Hub. Args: path_in_repo (`str`): Relative filepath in the repo, for example: `"checkpoints/1fec34a/weights.bin"` path_or_fileobj (`str`, `Path`, `bytes`, or `BinaryIO`): Either: - a path to a local file (as `str` or `pathlib.Path`) to upload - a buffer of bytes (`bytes`) holding the content of the file to upload - a "file object" (subclass of `io.BufferedIOBase`), typically obtained with `open(path, "rb")`. It must support `seek()` and `tell()` methods. Raises: [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If `path_or_fileobj` is not one of `str`, `Path`, `bytes` or `io.BufferedIOBase`. [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If `path_or_fileobj` is a `str` or `Path` but not a path to an existing file. [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If `path_or_fileobj` is a `io.BufferedIOBase` but it doesn't support both `seek()` and `tell()`. r)path_or_fileobjF)initrepr upload_inforeturnNc Cs t|j|_t|jtrt|j|_t|jtr2tjtj |j}tj |s1t d|dn t|jt j tfs?t dt|jt j riz|j|jdtjWnttfyh}zt d|d}~wwt|jtrxt|j|_dSt|jtrt|j|_dSt|j|_dS)z6Validates `path_or_fileobj` and compute `upload_info`.zProvided path: 'z(' is not a file on the local file systemzpath_or_fileobj must be either an instance of str, bytes or io.BufferedIOBase. If you passed a file-like object, make sure it is in binary mode.rzNpath_or_fileobj is a file-like object but does not implement seek() and tell()N)r-r)r/rBrr;ospathnormpath expanduserisfiler1ioBufferedIOBasebytestellseekSEEK_CUROSErrorAttributeErrorr from_pathrE from_bytesZ from_fileobj)r3rBexcr4r4r5r6s:        z CommitOperationAdd.__post_init__ with_tqdmccst|jts t|jtrF|r*t|j }|VWddS1s#wYdSt|jd }|VWddS1s?wYdSt|jtrUt|jVdSt|jtj ro|j }|jV|j |tj dSdS)u A context manager that yields a file-like object allowing to read the underlying data behind `path_or_fileobj`. Args: with_tqdm (`bool`, *optional*, defaults to `False`): If True, iterating over the file object will display a progress bar. Only works if the file-like object is a path to a file. Pure bytes and buffers are not supported. Example: ```python >>> operation = CommitOperationAdd( ... path_in_repo="remote/dir/weights.h5", ... path_or_fileobj="./local/weights.h5", ... ) CommitOperationAdd(path_in_repo='remote/dir/weights.h5', path_or_fileobj='./local/weights.h5') >>> with operation.as_file() as file: ... content = file.read() >>> with operation.as_file(with_tqdm=True) as file: ... while True: ... data = file.read(1024) ... if not data: ... break config.json: 100%|█████████████████████████| 8.19k/8.19k [00:02<00:00, 3.72kB/s] >>> with operation.as_file(with_tqdm=True) as file: ... requests.put(..., data=file) config.json: 100%|█████████████████████████| 8.19k/8.19k [00:02<00:00, 3.72kB/s] ``` Nrb) r/rBr;rr!openrNrLBytesIOrMrOrPSEEK_SET)r3rWfileZprev_posr4r4r5as_files $ ""  zCommitOperationAdd.as_filecCs:|}t|WdS1swYdS)z[ The base64-encoded content of `path_or_fileobj` Returns: `bytes` N)r]base64 b64encoderead)r3r\r4r4r5 b64contents  $zCommitOperationAdd.b64contentrFN)F)r7r8r9r:r;r<rrrNr rrErr6rr0rr]rar4r4r4r5rAns  #1rAr)rFcCs|dr |dd}|dks|dks|dr td|d|dr+|d d}td d |dDr?td |d |S)Nr,r.z..z../z,Invalid `path_in_repo` in CommitOperation: ''z./css|]}|dkVqdS)z.gitNr4).0partr4r4r5 sz)_validate_path_in_repo..z^Invalid `path_in_repo` in CommitOperation: cannot update files under a '.git/' folder (path: 'z').) startswithr1anysplit)r)r4r4r5r-s    r- operationscCstt}|D]X}|j}t|tr9||dkrtd|d||d7<t|jD] }|t |d7<q,t|t r^|t t|dkr^|j rUtd|dqtd|dqdS)a Warn user when a list of operations is expected to overwrite itself in a single commit. Rules: - If a filepath is updated by multiple `CommitOperationAdd` operations, a warning message is triggered. - If a filepath is updated at least once by a `CommitOperationAdd` and then deleted by a `CommitOperationDelete`, a warning is triggered. - If a `CommitOperationDelete` deletes a filepath that is then updated by a `CommitOperationAdd`, no warning is triggered. This is usually useless (no need to delete before upload) but can happen if a user deletes an entire folder and then add new files to it. rzBAbout to update multiple times the same file in the same commit: 'z9'. This can cause undesired inconsistencies in your repo.rz_About to delete a folder containing files that have just been updated within the same commit: 'zLAbout to delete a file that have just been updated within the same commit: 'N) rintr)r/rAwarningswarnrparentsr;r(r+)rlZnb_additions_per_path operationr)parentr4r4r5warn_on_overwriting_operationss8   rs)endpoint num_threads additions repo_typerepo_idtokenrurvcsxg}t|ddD])}tdd|D|||d\}} | r-ddd| D} td| ||7}qd d |Dg} |D]} | d d urUtd | djdq=| | q=t | dkrhtdd Sdfdd } t rtdt | dt | D]} | | qd St | dkrtd| | dd Stdt | d|dt | | dt | d|t dd S)a Uploads the content of `additions` to the Hub using the large file storage protocol. Relevant external documentation: - LFS Batch API: https://github.com/git-lfs/git-lfs/blob/main/docs/api/batch.md Args: additions (`List` of `CommitOperationAdd`): The files to be uploaded repo_type (`str`): Type of the repo to upload to: `"model"`, `"dataset"` or `"space"`. repo_id (`str`): A namespace (user or an organization) and a repo name separated by a `/`. token (`str`, *optional*): An authentication token ( See https://huggingface.co/settings/tokens ) num_threads (`int`, *optional*): The number of concurrent threads to use when uploading. Defaults to 5. Raises: `RuntimeError` if an upload failed for any reason Raises: `ValueError` if the server returns malformed responses Raises: `requests.HTTPError` if the LFS batch endpoint returned an HTTP error ) chunk_sizecSg|]}|jqSr4)rErfopr4r4r5 Wz$upload_lfs_files..)Z upload_infosrzryrxru c Ss0g|]}d|dd|didqS)z$Encountered error for file with OID oidz: `errormessage)get)rferrr4r4r5ras"z$LFS batch endpoint returned errors: cSsi|] }|jj|qSr4)rEsha256hex)rfZadd_opr4r4r5 iz$upload_lfs_files..actionsNzContent of file rz/ is already present upstream - skipping upload.rzNo LFS files to upload.rFc sNz|d}t||dWdSty&}z td|jd|d}~ww)Nr)rqZlfs_batch_actionrzzError while uploading 'z ' to the Hub.)r Exception RuntimeErrorr))Z batch_actionrqrVZ oid2addoprzr4r5_wrapped_lfs_upload{s z-upload_lfs_files.._wrapped_lfs_uploadz Uploading z* LFS files to the Hub using `hf_transfer`.rzUploading 1 LFS file to the Hubz" LFS files to the Hub using up to z threads concurrentlyzUpload z LFS files)desc max_workersZ tqdm_classrb) rrjoinr1rloggerdebugr)appendlenrhf_tqdmr)rwrxryrzrurvZ batch_actionschunkZbatch_actions_chunkZbatch_errors_chunkrZfiltered_actionsactionrr4rr5upload_lfs_files+s^)            rpreupload_infocCsh|d}t|tstd|D]!}t|tr-t|dtr-t|dtr-|ddvs1tdq|S)Nfilesz&preupload_info is improperly formattedrH uploadModer%z'preupload_info is improperly formatted:)rr/listr1dictr;)rr file_infor4r4r5_validate_preupload_infos   rFrevision create_prc Cs|dur|nt}t|d}i}t|dD]?} ddd| Di} tj|d|d|d || ||r6d d indd } t| t| } |jdid d| dDq|D]} | j j dkre| j }d||<qV|S)av Requests the Hub "preupload" endpoint to determine whether each input file should be uploaded as a regular git blob or as git LFS blob. Args: additions (`Iterable` of :class:`CommitOperationAdd`): Iterable of :class:`CommitOperationAdd` describing the files to upload to the Hub. repo_type (`str`): Type of the repo to upload to: `"model"`, `"dataset"` or `"space"`. repo_id (`str`): A namespace (user or an organization) and a repo name separated by a `/`. token (`str`, *optional*): An authentication token ( See https://huggingface.co/settings/tokens ) revision (`str`): The git revision to upload the files to. Can be any valid git revision. Returns: `Dict[str, UploadMode]` Key is the file path, value is the upload mode ("regular" or "lfs"). Raises: [`~utils.HfHubHTTPError`] If the Hub API returned an error. [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If the Hub API response is improperly formatted. N)rzr{rcSs8g|]}|jt|jjd|jj|jjdqS)ascii)rHsamplesizesha) r)r^r_rErdecoderrrr~r4r4r5rs z&fetch_upload_modes..z/api/zs/z /preupload/r1)jsonheadersparamscSsi|] }|d|dqS)rHrr4)rfr\r4r4r5rrz&fetch_upload_modes..rr'r4) rrrrpostrrrupdaterErr))rwrxryrzrrurr upload_modesrpayloadresprZadditionrHr4r4r5fetch_upload_modess.%     rcopiesr$cCsddlm}|||d}i}t|dddD]X\} } t| } dd| D} td t| tD]&} |j|| | | t| p<||d } | D]}|jsKt d |||j | f<qBq-| D]}|j | f|vrmt d |j d | ph|dqVq|S)a` Requests the Hub files information of the LFS files to be copied, including their sha256. Args: copies (`Iterable` of :class:`CommitOperationCopy`): Iterable of :class:`CommitOperationCopy` describing the files to copy on the Hub. repo_type (`str`): Type of the repo to upload to: `"model"`, `"dataset"` or `"space"`. repo_id (`str`): A namespace (user or an organization) and a repo name separated by a `/`. token (`str`, *optional*): An authentication token ( See https://huggingface.co/settings/tokens ) revision (`str`): The git revision to upload the files to. Can be any valid git revision. Returns: `Dict[Tuple[str, Optional[str]], RepoFile]]` Key is the file path and revision of the file to copy, value is the repo file. Raises: [`~utils.HfHubHTTPError`] If the Hub API returned an error. [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If the Hub API response is improperly formatted. r)HfApi)rurzcSs|jSr@)r?)rr4r4r5sz)fetch_lfs_files_to_copy..)keycSr}r4)r>r~r4r4r5rrz+fetch_lfs_files_to_copy..r)rypathsrrx)Copying a non-LFS file is not implementedz Cannot copy z at revision z: file is missing on repo.) hf_apirrrrangerFETCH_LFS_BATCH_SIZEZlist_files_infor&NotImplementedErrorZ rfilenamer>r)rrxryrzrrurr files_to_copyr?rlroffsetZsrc_repo_filesZ src_repo_filerqr4r4r5fetch_lfs_files_to_copys8 #  rrrcommit_messagecommit_description parent_commitc csH|dur|nd}||d}|dur||d<d|dV|D]}t|tr=||jdkr=d||jd d dVqt|tr]||jd kr]d |jd |jj|jj ddVqt|t rq|j rgdndd|jidVqt|t r||j |jf}|jstdd |jd |jd ddVqtd|d||jdS)aG Builds the payload to POST to the `/commit` API of the Hub. Payload is returned as an iterator so that it can be streamed as a ndjson in the POST request. For more information, see: - https://github.com/huggingface/huggingface_hub/issues/1085#issuecomment-1265208073 - http://ndjson.org/ N)summary descriptionZ parentCommitheader)rvaluer'r\r^)contentrHencodingr&ZlfsFiler)rHalgorrZ deletedFolderZ deletedFilerHr)rHrrz(Unknown operation to commit. Operation: z. Upload mode: )r/rArr)rarrErrrr(r+r=r>r?r&rr1) rlrrrrr header_valuerqZ file_to_copyr4r4r5prepare_commit_payload.sT           r)NFr@)NN)Hr:r^rLrGrn collectionsr contextlibr dataclassesrr itertoolsrpathlibrrtypingr r r r r rrrrrrZtqdm.contrib.concurrentrZhuggingface_hubr constantsrrr&rrrutilsrrrrr r!r"r#rrr$Z get_loggerr7rZ UploadModerr(r=rAr;r-ZCommitOperationrsrmrrrr0rrrr4r4r4r5s   4  $   {. j  J B