o
e. @ s U d Z ddlZddlZddlZddlZddlZddlmZ ddlm Z ddl
m Z m Z m
Z
mZ ddlZddlmZ ddlmZ ddlmZmZ dd lmZ d
d lmZ d
d lmZ e e Z d
Z dZ G dd deZ!dej"fddZ#e g ej"f Z$e#a%e$e&d< e#fde$ddfddZ'dej"fddZ(ede)de)dej"fd d Z*d d
d eefe j+d d!ed"e,d#e)d$e-d%e-d&ee
e. e e
e. d'f f d(ee)e e)d'f f defd)d*Z/dS )+z>Contains utilities to handle HTTP requests in Huggingface Hub. N) lru_cache)
HTTPStatus)CallableTupleTypeUnion)Response) HTTPAdapter)
ProxyErrorTimeout)PreparedRequest )logging)
HTTP_METHOD_TX-Amzn-Trace-Idz x-request-idc s6 e Zd ZdZ fddZdedef fdd Z ZS )UniqueRequestIdAdapterr c s t j|fi | t|jvr |jtptt |jt< t|jdd d}t
d|jt d|j d|j
d| d d S ) N
authorization z
Bearer hf_zRequest z: z (authenticated: ))super add_headersX_AMZN_TRACE_IDheadersget X_REQUEST_IDstruuiduuid4
startswithloggerdebugmethodurl)selfrequestkwargsZ has_token __class__ KD:\Projects\ConvertPro\env\Lib\site-packages\huggingface_hub/utils/_http.pyr / s
&z"UniqueRequestIdAdapter.add_headersr% returnc
sh zt j|g|R i |W S tjy3 } z|jt}|dur.g |jd| dR |_ d}~ww )zSCatch any RequestException to append request id to the error message for debugging.Nz
(Request ID: r )r sendrequestsRequestExceptionr r r args)r$ r% r/ r& eZ
request_idr' r) r* r, < s zUniqueRequestIdAdapter.send) __name__
__module__ __qualname__r r r r r,
__classcell__r) r) r' r* r , s
r r+ c C s( t } | dt | dt | S )Nzhttp://zhttps://)r- Sessionmountr )sessionr) r) r* _default_backend_factoryH s r8 _GLOBAL_BACKEND_FACTORYbackend_factoryc C s | a t dS )a
Configure the HTTP backend by providing a `backend_factory`. Any HTTP calls made by `huggingface_hub` will use a
Session object instantiated by this factory. This can be useful if you are running your scripts in a specific
environment requiring custom configuration (e.g. custom proxy or certifications).
Use [`get_session`] to get a configured Session. Since `requests.Session` is not guaranteed to be thread-safe,
`huggingface_hub` creates 1 Session instance per thread. They are all instantiated using the same `backend_factory`
set in [`configure_http_backend`]. A LRU cache is used to cache the created sessions (and connections) between
calls. Max size is 128 to avoid memory leaks if thousands of threads are spawned.
See [this issue](https://github.com/psf/requests/issues/2766) to know more about thread-safety in `requests`.
Example:
```py
import requests
from huggingface_hub import configure_http_backend, get_session
# Create a factory function that returns a Session with configured proxies
def backend_factory() -> requests.Session:
session = requests.Session()
session.proxies = {"http": "http://10.10.1.10:3128", "https": "https://10.10.1.11:1080"}
return session
# Set it as the default session factory
configure_http_backend(backend_factory=backend_factory)
# In practice, this is mostly done internally in `huggingface_hub`
session = get_session()
```
N)r9 _get_session_from_cache cache_clear)r: r) r) r* configure_http_backendS s r= c C s t t t dS )a
Get a `requests.Session` object, using the session factory from the user.
Use [`get_session`] to get a configured Session. Since `requests.Session` is not guaranteed to be thread-safe,
`huggingface_hub` creates 1 Session instance per thread. They are all instantiated using the same `backend_factory`
set in [`configure_http_backend`]. A LRU cache is used to cache the created sessions (and connections) between
calls. Max size is 128 to avoid memory leaks if thousands of threads are spawned.
See [this issue](https://github.com/psf/requests/issues/2766) to know more about thread-safety in `requests`.
Example:
```py
import requests
from huggingface_hub import configure_http_backend, get_session
# Create a factory function that returns a Session with configured proxies
def backend_factory() -> requests.Session:
session = requests.Session()
session.proxies = {"http": "http://10.10.1.10:3128", "https": "https://10.10.1.11:1080"}
return session
# Set it as the default session factory
configure_http_backend(backend_factory=backend_factory)
# In practice, this is mostly done internally in `huggingface_hub`
session = get_session()
```
process_id thread_id)r; osgetpid threading get_identr) r) r) r* get_sessionw s rE r? r@ c C s t S )z
Create a new session per thread using global factory. Using LRU cache (maxsize 128) to avoid memory leaks when
using thousands of threads. Cache is cleared when `configure_http_backend` is called.
)r9 r> r) r) r* r; s r; ) max_retriesbase_wait_time
max_wait_timeretry_on_exceptionsretry_on_status_codesr" r# rH rI rJ rK .rL c K sZ t |tr|f}t |tr|f}d}|} d}
d|v r(t |d tjr(|d }
t } |d7 }z:|
dur<|d |
| jd| |d|} | j |vrO| W S t
d| j d| d | ||kri| | W S W n& |y }
zt
d
|
d | d | ||kr|
W Y d}
~
nd}
~
ww t
d | d
| d| d t
| t|| d } q,)a> Wrapper around requests to retry calls on an endpoint, with exponential backoff.
Endpoint call is retried on exceptions (ex: connection timeout, proxy error,...)
and/or on specific status codes (ex: service unavailable). If the call failed more
than `max_retries`, the exception is thrown or `raise_for_status` is called on the
response object.
Re-implement mechanisms from the `backoff` library to avoid adding an external
dependencies to `hugging_face_hub`. See https://github.com/litl/backoff.
Args:
method (`Literal["GET", "OPTIONS", "HEAD", "POST", "PUT", "PATCH", "DELETE"]`):
HTTP method to perform.
url (`str`):
The URL of the resource to fetch.
max_retries (`int`, *optional*, defaults to `5`):
Maximum number of retries, defaults to 5 (no retries).
base_wait_time (`float`, *optional*, defaults to `1`):
Duration (in seconds) to wait before retrying the first time.
Wait time between retries then grows exponentially, capped by
`max_wait_time`.
max_wait_time (`float`, *optional*, defaults to `8`):
Maximum duration (in seconds) to wait before retrying.
retry_on_exceptions (`Type[Exception]` or `Tuple[Type[Exception]]`, *optional*, defaults to `(Timeout, ProxyError,)`):
Define which exceptions must be caught to retry the request. Can be a single
type or a tuple of types.
By default, retry on `Timeout` and `ProxyError`.
retry_on_status_codes (`int` or `Tuple[int]`, *optional*, defaults to `503`):
Define on which status codes the request must be retried. By default, only
HTTP 503 Service Unavailable is retried.
**kwargs (`dict`, *optional*):
kwargs to pass to `requests.request`.
Example:
```
>>> from huggingface_hub.utils import http_backoff
# Same usage as "requests.request".
>>> response = http_backoff("GET", "https://www.google.com")
>>> response.raise_for_status()
# If you expect a Gateway Timeout from time to time
>>> http_backoff("PUT", upload_url, data=data, retry_on_status_codes=504)
>>> response.raise_for_status()
```
When using `requests` it is possible to stream data by passing an iterator to the
`data` argument. On http backoff this is a problem as the iterator is not reset
after a failed call. This issue is mitigated for file objects or any IO streams
by saving the initial position of the cursor (with `data.tell()`) and resetting the
cursor between each call (with `data.seek()`). For arbitrary iterators, http backoff
will fail. If this is a hard constraint for you, please let us know by opening an
issue on [Github](https://github.com/huggingface/huggingface_hub).
r NdataTr
)r" r# z HTTP Error z thrown while requesting r 'z' thrown while requesting z Retrying in z s [Retry /z]. r) )
isinstancetypeintioIOBasetellrE seekr% status_coder warningraise_for_statustimesleepmin)r" r# rH rI rJ rK rL r& Znb_triesZ
sleep_timeZio_obj_initial_posr7 responseerrr) r) r* http_backoff sD
H
r` )0__doc__rT rA rC r[ r functoolsr httpr typingr r r r r- r Zrequests.adaptersr Zrequests.exceptionsr
r Zrequests.modelsr r r Z_typingr Z
get_loggerr1 r r r r r5 r8 ZBACKEND_FACTORY_Tr9 __annotations__r= rE rS r; SERVICE_UNAVAILABLEr float Exceptionr` r) r) r) r* sf
$