o es @sddlZddlmZddlmZddlmZmZmZm Z m Z m Z ddl m Z ddlmZeeZeGdd d ZeGd d d ZGd d d eZGdddeZGdddeZdeeeefde eeeffddZddZdedeedeeeeffddZdS)N) defaultdict) dataclass)AnyDictListOptionalTupleUnion) yaml_dump) get_loggerc@seZdZUdZeed<eed<eed<eed<eed<dZeeed<dZ eeed <dZ eeed <dZ eeed <dZ ee eefed <dZeeed <dZeeed<dZee eefed<dZeeed<dZeeed<edefddZdddefddZdS) EvalResultu Flattened representation of individual evaluation results found in model-index of Model Cards. For more information on the model-index spec, see https://github.com/huggingface/hub-docs/blob/main/modelcard.md?plain=1. Args: task_type (`str`): The task identifier. Example: "image-classification". dataset_type (`str`): The dataset identifier. Example: "common_voice". Use dataset id from https://hf.co/datasets. dataset_name (`str`): A pretty name for the dataset. Example: "Common Voice (French)". metric_type (`str`): The metric identifier. Example: "wer". Use metric id from https://hf.co/metrics. metric_value (`Any`): The metric value. Example: 0.9 or "20.0 ± 1.2". task_name (`str`, *optional*): A pretty name for the task. Example: "Speech Recognition". dataset_config (`str`, *optional*): The name of the dataset configuration used in `load_dataset()`. Example: fr in `load_dataset("common_voice", "fr")`. See the `datasets` docs for more info: https://hf.co/docs/datasets/package_reference/loading_methods#datasets.load_dataset.name dataset_split (`str`, *optional*): The split used in `load_dataset()`. Example: "test". dataset_revision (`str`, *optional*): The revision (AKA Git Sha) of the dataset used in `load_dataset()`. Example: 5503434ddd753f426f4b38109466949a1217c2bb dataset_args (`Dict[str, Any]`, *optional*): The arguments passed during `Metric.compute()`. Example for `bleu`: `{"max_order": 4}` metric_name (`str`, *optional*): A pretty name for the metric. Example: "Test WER". metric_config (`str`, *optional*): The name of the metric configuration used in `load_metric()`. Example: bleurt-large-512 in `load_metric("bleurt", "bleurt-large-512")`. See the `datasets` docs for more info: https://huggingface.co/docs/datasets/v2.1.0/en/loading#load-configurations metric_args (`Dict[str, Any]`, *optional*): The arguments passed during `Metric.compute()`. Example for `bleu`: max_order: 4 verified (`bool`, *optional*): Indicates whether the metrics originate from Hugging Face's [evaluation service](https://huggingface.co/spaces/autoevaluate/model-evaluator) or not. Automatically computed by Hugging Face, do not set. verify_token (`str`, *optional*): A JSON Web Token that is used to verify whether the metrics originate from Hugging Face's [evaluation service](https://huggingface.co/spaces/autoevaluate/model-evaluator) or not. task_type dataset_type dataset_name metric_type metric_valueN task_namedataset_config dataset_splitdataset_revision dataset_args metric_name metric_config metric_argsverified verify_tokenreturncCs|j|j|j|j|jfS)z9Returns a tuple that uniquely identifies this evaluation.)rrrrrselfr MD:\Projects\ConvertPro\env\Lib\site-packages\huggingface_hub/repocard_data.pyunique_identifier|s zEvalResult.unique_identifierothercCsD|jD]\}}|dkrq|dkrt||t||krdSqdS)zx Return True if `self` and `other` describe exactly the same metric but with a different value. rrFT)__dict__itemsgetattr)rr#key_r r r!is_equal_except_valuesz EvalResult.is_equal_except_value)__name__ __module__ __qualname____doc__str__annotations__rrrrrrrrrrrrboolrpropertytupler"r)r r r r!r s( / r c@seZdZdZddefddZdeeeffddZ d d Z ddefd d Z ddZ ddededefddZ ddededefddZdedefddZdededd fddZdedefddZd S)CardDataaStructure containing metadata from a RepoCard. [`CardData`] is the parent class of [`ModelCardData`] and [`DatasetCardData`]. Metadata can be exported as a dictionary or YAML. Export can be customized to alter the representation of the data (example: flatten evaluation results). `CardData` behaves as a dictionary (can get, pop, set values) but do not inherit from `dict` to allow this export step. Fignore_metadata_errorscKs|j|dSN)r$update)rr4kwargsr r r!__init__szCardData.__init__rcCst|j}||t|S)zConverts CardData to a dict. Returns: `dict`: CardData represented as a dictionary ready to be dumped to a YAML block for inclusion in a README.md file. )copydeepcopyr$_to_dict _remove_nonerZ data_dictr r r!to_dicts  zCardData.to_dictcCsdS)zUse this method in child classes to alter the dict representation of the data. Alter the dict in-place. Args: data_dict (`dict`): The raw dict representation of the card data. Nr r=r r r!r;szCardData._to_dictNcCst|d|dS)a Dumps CardData to a YAML block for inclusion in a README.md file. Args: line_break (str, *optional*): The line break to use when dumping to yaml. Returns: `str`: CardData represented as a YAML block. F) sort_keys line_break)r r>strip)rr@r r r!to_yamls zCardData.to_yamlcCs|Sr5)rBrr r r!__repr__szCardData.__repr__r'defaultcC|j||Sz#Get value for a given metadata key.)r$getrr'rDr r r!rGz CardData.getcCrE)z#Pop value for a given metadata key.)r$poprHr r r!rJrIz CardData.popcCs |j|SrFr$rr'r r r! __getitem__ zCardData.__getitem__valuecCs||j|<dS)z#Set value for a given metadata key.NrK)rr'rOr r r! __setitem__rIzCardData.__setitem__cCs ||jvS)z%Check if a given metadata key is set.rKrLr r r! __contains__rNzCardData.__contains__)Fr5)r*r+r,r-r0r8rr.rr>r;rBrCrGrJrMrPrQr r r r!r3s   r3cseZdZdZdddddddddd deeeeefdeedeedeeed eeed eeed eeed eed e ffddZ ddZ Z S) ModelCardDataa Model Card Metadata that is used by Hugging Face Hub when included at the top of your README.md Args: language (`Union[str, List[str]]`, *optional*): Language of model's training data or metadata. It must be an ISO 639-1, 639-2 or 639-3 code (two/three letters), or a special value like "code", "multilingual". Defaults to `None`. license (`str`, *optional*): License of this model. Example: apache-2.0 or any license from https://huggingface.co/docs/hub/repositories-licenses. Defaults to None. library_name (`str`, *optional*): Name of library used by this model. Example: keras or any library from https://github.com/huggingface/hub-docs/blob/main/js/src/lib/interfaces/Libraries.ts. Defaults to None. tags (`List[str]`, *optional*): List of tags to add to your model that can be used when filtering on the Hugging Face Hub. Defaults to None. datasets (`List[str]`, *optional*): List of datasets that were used to train this model. Should be a dataset ID found on https://hf.co/datasets. Defaults to None. metrics (`List[str]`, *optional*): List of metrics used to evaluate this model. Should be a metric name that can be found at https://hf.co/metrics. Example: 'accuracy'. Defaults to None. eval_results (`Union[List[EvalResult], EvalResult]`, *optional*): List of `huggingface_hub.EvalResult` that define evaluation results of the model. If provided, `model_name` is used to as a name on PapersWithCode's leaderboards. Defaults to `None`. model_name (`str`, *optional*): A name for this model. It is used along with `eval_results` to construct the `model-index` within the card's metadata. The name you supply here is what will be used on PapersWithCode's leaderboards. If None is provided then the repo name is used as a default. Defaults to None. ignore_metadata_errors (`str`): If True, errors while parsing the metadata section will be ignored. Some information might be lost during the process. Use it at your own risk. kwargs (`dict`, *optional*): Additional metadata that will be added to the model card. Defaults to None. Example: ```python >>> from huggingface_hub import ModelCardData >>> card_data = ModelCardData( ... language="en", ... license="mit", ... library_name="timm", ... tags=['image-classification', 'resnet'], ... ) >>> card_data.to_dict() {'language': 'en', 'license': 'mit', 'library_name': 'timm', 'tags': ['image-classification', 'resnet']} ``` NF) languagelicense library_nametagsdatasetsmetrics eval_results model_namer4rSrTrUrVrWrXrYrZr4c s||_||_||_||_||_||_||_||_| dd} | rQzt | \}}||_||_Wn"t yP} z| r>t dnt d| dWYd} ~ nd} ~ wwtjdi| |jrrt|jtkri|jg|_|jdurtt ddSdS)N model-indexz         zModelCardData.__init__cCs0|jdurt|j|j|d<|d=|d=dSdS)z[Format the internal data dict. In this case, we convert eval results to a valid model indexNr[rYrZ)rYeval_results_to_model_indexrZr=r r r!r;?s zModelCardData._to_dict) r*r+r,r-rr r.rr r0r8r; __classcell__r r rer!rRs>6      /rRcs&eZdZdZddddddddddddddddeeeeefdeeeeefdeeeeefdeeeeefd eeeeefd eeeeefd eeed eeeeefd eeeeefdeedeedeedeeeeefde ffddZ ddZ Z S)DatasetCardDataa Dataset Card Metadata that is used by Hugging Face Hub when included at the top of your README.md Args: language (`List[str]`, *optional*): Language of dataset's data or metadata. It must be an ISO 639-1, 639-2 or 639-3 code (two/three letters), or a special value like "code", "multilingual". license (`Union[str, List[str]]`, *optional*): License(s) of this dataset. Example: apache-2.0 or any license from https://huggingface.co/docs/hub/repositories-licenses. annotations_creators (`Union[str, List[str]]`, *optional*): How the annotations for the dataset were created. Options are: 'found', 'crowdsourced', 'expert-generated', 'machine-generated', 'no-annotation', 'other'. language_creators (`Union[str, List[str]]`, *optional*): How the text-based data in the dataset was created. Options are: 'found', 'crowdsourced', 'expert-generated', 'machine-generated', 'other' multilinguality (`Union[str, List[str]]`, *optional*): Whether the dataset is multilingual. Options are: 'monolingual', 'multilingual', 'translation', 'other'. size_categories (`Union[str, List[str]]`, *optional*): The number of examples in the dataset. Options are: 'n<1K', '1K1T', and 'other'. source_datasets (`List[str]]`, *optional*): Indicates whether the dataset is an original dataset or extended from another existing dataset. Options are: 'original' and 'extended'. task_categories (`Union[str, List[str]]`, *optional*): What categories of task does the dataset support? task_ids (`Union[str, List[str]]`, *optional*): What specific tasks does the dataset support? paperswithcode_id (`str`, *optional*): ID of the dataset on PapersWithCode. pretty_name (`str`, *optional*): A more human-readable name for the dataset. (ex. "Cats vs. Dogs") train_eval_index (`Dict`, *optional*): A dictionary that describes the necessary spec for doing evaluation on the Hub. If not provided, it will be gathered from the 'train-eval-index' key of the kwargs. config_names (`Union[str, List[str]]`, *optional*): A list of the available dataset configs for the dataset. NF)rSrTannotations_creatorslanguage_creatorsmultilingualitysize_categoriessource_datasetstask_categoriestask_idspaperswithcode_id pretty_nametrain_eval_index config_namesr4rSrTrjrkrlrmrnrorprqrrrsrtr4c sp||_||_||_||_||_||_||_||_| |_| |_ | |_ | |_ | p+| dd|_ tjdi|dS)Ntrain-eval-indexr )rjrkrSrTrlrmrnrorprqrrrtrJrsrar8)rrSrTrjrkrlrmrnrorprqrrrsrtr4r7rer r!r8nszDatasetCardData.__init__cCs|d|d<dS)Nrsru)rJr=r r r!r;szDatasetCardData._to_dict) r*r+r,r-rr r.rrr0r8r;rhr r rer!riFs\*     $ricseZdZdZddddddddddddd deedeedeedeed eed eed eed eed eeedeeedeeedeffddZ Z S) SpaceCardDataa Space Card Metadata that is used by Hugging Face Hub when included at the top of your README.md To get an exhaustive reference of Spaces configuration, please visit https://huggingface.co/docs/hub/spaces-config-reference#spaces-configuration-reference. Args: title (`str`, *optional*) Title of the Space. sdk (`str`, *optional*) SDK of the Space (one of `gradio`, `streamlit`, `docker`, or `static`). sdk_version (`str`, *optional*) Version of the used SDK (if Gradio/Streamlit sdk). python_version (`str`, *optional*) Python version used in the Space (if Gradio/Streamlit sdk). app_file (`str`, *optional*) Path to your main application file (which contains either gradio or streamlit Python code, or static html code). Path is relative to the root of the repository. app_port (`str`, *optional*) Port on which your application is running. Used only if sdk is `docker`. license (`str`, *optional*) License of this model. Example: apache-2.0 or any license from https://huggingface.co/docs/hub/repositories-licenses. duplicated_from (`str`, *optional*) ID of the original Space if this is a duplicated Space. models (List[`str`], *optional*) List of models related to this Space. Should be a dataset ID found on https://hf.co/models. datasets (`List[str]`, *optional*) List of datasets related to this Space. Should be a dataset ID found on https://hf.co/datasets. tags (`List[str]`, *optional*) List of tags to add to your Space that can be used when filtering on the Hub. ignore_metadata_errors (`str`): If True, errors while parsing the metadata section will be ignored. Some information might be lost during the process. Use it at your own risk. kwargs (`dict`, *optional*): Additional metadata that will be added to the space card. Example: ```python >>> from huggingface_hub import SpaceCardData >>> card_data = SpaceCardData( ... title="Dreambooth Training", ... license="mit", ... sdk="gradio", ... duplicated_from="multimodalart/dreambooth-training" ... ) >>> card_data.to_dict() {'title': 'Dreambooth Training', 'sdk': 'gradio', 'license': 'mit', 'duplicated_from': 'multimodalart/dreambooth-training'} ``` NF) titlesdk sdk_versionpython_versionapp_fileapp_portrTduplicated_frommodelsrWrVr4rwrxryrzr{r|rTr}r~rWrVr4c  sX||_||_||_||_||_||_||_||_| |_| |_ | |_ t j di| dS)Nr ) rwrxryrzr{r|rTr}r~rWrVrar8)rrwrxryrzr{r|rTr}r~rWrVr4r7rer r!r8szSpaceCardData.__init__) r*r+r,r-rr.intrr0r8rhr r rer!rvsN4     rvrcrcCsg}|D]}|d}|d}|D]w}|dd}|dd}|dd}|dd} |dd} |dd} |dd} |dd } |d D];}|d}|d }|d}|d }|d}|d }|d }t||| |||| | | | |||||d}||qKqq||fS)aTakes in a model index and returns the model name and a list of `huggingface_hub.EvalResult` objects. A detailed spec of the model index can be found here: https://github.com/huggingface/hub-docs/blob/main/modelcard.md?plain=1 Args: model_index (`List[Dict[str, Any]]`): A model index data structure, likely coming from a README.md file on the Hugging Face Hub. Returns: model_name (`str`): The name of the model as found in the model index. This is used as the identifier for the model on leaderboards like PapersWithCode. eval_results (`List[EvalResult]`): A list of `huggingface_hub.EvalResult` objects containing the metrics reported in the provided model_index. Example: ```python >>> from huggingface_hub.repocard_data import model_index_to_eval_results >>> # Define a minimal model index >>> model_index = [ ... { ... "name": "my-cool-model", ... "results": [ ... { ... "task": { ... "type": "image-classification" ... }, ... "dataset": { ... "type": "beans", ... "name": "Beans" ... }, ... "metrics": [ ... { ... "type": "accuracy", ... "value": 0.9 ... } ... ] ... } ... ] ... } ... ] >>> model_name, eval_results = model_index_to_eval_results(model_index) >>> model_name 'my-cool-model' >>> eval_results[0].task_type 'image-classification' >>> eval_results[0].metric_type 'accuracy' ``` nameresultstaskrbdatasetconfigsplitrevisionargsrXrOr verifyToken)rrrrrrrrrrrrrrr)rGr append)rcrYelemrrresultrrrrrrrrZmetricrrrrrrr eval_resultr r r!r\sT8          %r\cCsNt|tttfrt|dd|DSt|tr%t|dd|DS|S)zk Recursively remove `None` values from a dict. Borrowed from: https://stackoverflow.com/a/20558778 css |] }|durt|VqdSr5r<).0xr r r! Psz_remove_none..css4|]\}}|dur|durt|t|fVqdSr5r)rkvr r r!rRs2) isinstancelistr2setrbdictr%)objr r r!r<Ks  r<rZrYc Cstt}|D] }||j|qg}|D](}|d}|j|jd|j|j|j |j |j |j ddd|Dd}||q||dg}t |S)aTakes in given model name and list of `huggingface_hub.EvalResult` and returns a valid model-index that will be compatible with the format expected by the Hugging Face Hub. Args: model_name (`str`): Name of the model (ex. "my-cool-model"). This is used as the identifier for the model on leaderboards like PapersWithCode. eval_results (`List[EvalResult]`): List of `huggingface_hub.EvalResult` objects containing the metrics to be reported in the model-index. Returns: model_index (`List[Dict[str, Any]]`): The eval_results converted to a model-index. Example: ```python >>> from huggingface_hub.repocard_data import eval_results_to_model_index, EvalResult >>> # Define minimal eval_results >>> eval_results = [ ... EvalResult( ... task_type="image-classification", # Required ... dataset_type="beans", # Required ... dataset_name="Beans", # Required ... metric_type="accuracy", # Required ... metric_value=0.9, # Required ... ) ... ] >>> eval_results_to_model_index("my-cool-model", eval_results) [{'name': 'my-cool-model', 'results': [{'task': {'type': 'image-classification'}, 'dataset': {'name': 'Beans', 'type': 'beans'}, 'metrics': [{'type': 'accuracy', 'value': 0.9}]}]}] ``` r)rbr)rrbrrrrc Ss.g|]}|j|j|j|j|j|j|jdqS))rbrOrrrrr)rrrrrrr)rrr r r! s z/eval_results_to_model_index..)rrrX)rr)rrr"rvaluesrrrrrrrrr<) rZrYZtask_and_ds_types_maprZmodel_index_datarZ sample_resultdatarcr r r!rgWs4%   rg)r9 collectionsr dataclassesrtypingrrrrrr Zhuggingface_hub.utilsr Z utils.loggingr r*r^r r3rRrirvr.r\r<rgr r r r!s$     EjP*Qd*