o Me6@s(dZddlmZddlZddlZddlZddlZddlmZddl Z ddl m Z ddl mmZgZdZdZdZdZd Zd Zd Zd d Zd%ddZddZddZedddddd&ddZedddddd&ddZedddddd&dd Zedddddd%d!d"Zedddddd#d$Z dS)'aW ACL2016 Multimodal Machine Translation. Please see this website for more details: http://www.statmt.org/wmt16/multimodal-task.html#task1 If you use the dataset created for your task, please cite the following paper: Multi30K: Multilingual English-German Image Descriptions. @article{elliott-EtAl:2016:VL16, author = {{Elliott}, D. and {Frank}, S. and {Sima"an}, K. and {Specia}, L.}, title = {Multi30K: Multilingual English-German Image Descriptions}, booktitle = {Proceedings of the 6th Workshop on Vision and Language}, year = {2016}, pages = {70--74}, year = 2016 } )print_functionN) defaultdictz2http://paddlemodels.bj.bcebos.com/wmt/wmt16.tar.gzZ 0c38be43600334966403524a40dcd81ei+iKzzzc CsVtt}tj|dd>}|dD]0}t|}|d}t |dkr&q|dkr.|dn|d}|D] } || d7<q6qWdn1sLwYt|d K} | t d t t tfttt|d d d dD]\} } | d|kr~n| t | d| t dqrWddSWddS1swYdS)Nrmode wmt16/train enrwbz %s %s %s cSs|dS)Nr )xr r DD:\Projects\ConvertPro\env\Lib\site-packages\paddle/dataset/wmt16.pyHsz__build_dict..T)keyreverse )rinttarfileopen extractfilecptto_textstripsplitlenwriteto_bytes START_MARKEND_MARKUNK_MARK enumeratesortedsix iteritems) tar_file dict_sizeZ save_pathlang word_dictfline line_splitZsenwZfoutidxwordr r r __build_dict8s:     "r1Fc Cstjtjjjd||f}tj|rtt |d |kr&t ||||i}t |d'}t |D]\}}|rBt |||<q2||t |<q2Wd|S1sWwY|S)Nwmt16/%s_%d.dictrb)ospathjoinpaddledatasetcommon DATA_HOMEexistsrr readlinesr1r#rrr) r'r(r)r dict_pathr*Zfdictr/r,r r r __load_dictOs$   r>cCs4t||dkrtnt}t||dkrtnt}||fS)Nr )minTOTAL_EN_WORDSTOTAL_DE_WORDS src_dict_size trg_dict_sizesrc_langr r r__get_dict_size`srFcsfdd}|S)Nc 3s,ttdkrdndt}t}tdkr%dnd}d|}tjddZ}|D]K}t|}| d}t |dkrMq8|| }|gfd d |D|g}|| } fd d | D} | |g} |g| } || | fVq8WddS1swYdS) Nr derr rrrr cg|]}|qSr get.0r.)src_dictunk_idr r sz2reader_creator..reader..crHr rIrK)trg_dictrNr rrOs) r>r r!r"rrrrrrrr) Zstart_idZend_idZsrc_colZtrg_colr+r,r-Z src_wordsZsrc_idsZ trg_wordsZtrg_idsZ trg_ids_next file_namerCrEr'rD)rMrPrNrreaderjs>        "zreader_creator..readerr )r'rRrCrDrErSr rQrreader_creatorhs"rTz2.0.0zpaddle.text.datasets.WMT16r z>Please use new dataset API which supports paddle.io.DataLoader)ZsinceZ update_tolevelreasonr cCB|dvrtdt|||\}}ttjjtdtdd|||dS)a} WMT16 train set reader. This function returns the reader for train data. Each sample the reader returns is made up of three fields: the source language word index sequence, target language word index sequence and next word index sequence. NOTE: The original like for training data is: http://www.quest.dcs.shef.ac.uk/wmt16_files_mmt/training.tar.gz paddle.dataset.wmt16 provides a tokenized version of the original dataset by using moses's tokenization script: https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/tokenizer.perl Args: src_dict_size(int): Size of the source language dictionary. Three special tokens will be added into the dictionary: for start mark, for end mark, and for unknown word. trg_dict_size(int): Size of the target language dictionary. Three special tokens will be added into the dictionary: for start mark, for end mark, and for unknown word. src_lang(string): A string indicating which language is the source language. Available options are: "en" for English and "de" for Germany. Returns: callable: The train reader. r rGzIAn error language type. Only support: en (for English); de(for Germany).wmt16 wmt16.tar.gzrr'rRrCrDrE ValueErrorrFrTr7r8r9downloadDATA_URLDATA_MD5rBr r rtrains' racCrW)a} WMT16 test set reader. This function returns the reader for test data. Each sample the reader returns is made up of three fields: the source language word index sequence, target language word index sequence and next word index sequence. NOTE: The original like for test data is: http://www.quest.dcs.shef.ac.uk/wmt16_files_mmt/mmt16_task1_test.tar.gz paddle.dataset.wmt16 provides a tokenized version of the original dataset by using moses's tokenization script: https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/tokenizer.perl Args: src_dict_size(int): Size of the source language dictionary. Three special tokens will be added into the dictionary: for start mark, for end mark, and for unknown word. trg_dict_size(int): Size of the target language dictionary. Three special tokens will be added into the dictionary: for start mark, for end mark, and for unknown word. src_lang(string): A string indicating which language is the source language. Available options are: "en" for English and "de" for Germany. Returns: callable: The test reader. rXHAn error language type. Only support: en (for English); de(for Germany).rYrZz wmt16/testr[r\rBr r rtests& rccCrW)a WMT16 validation set reader. This function returns the reader for validation data. Each sample the reader returns is made up of three fields: the source language word index sequence, target language word index sequence and next word index sequence. NOTE: The original like for validation data is: http://www.quest.dcs.shef.ac.uk/wmt16_files_mmt/validation.tar.gz paddle.dataset.wmt16 provides a tokenized version of the original dataset by using moses's tokenization script: https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/tokenizer.perl Args: src_dict_size(int): Size of the source language dictionary. Three special tokens will be added into the dictionary: for start mark, for end mark, and for unknown word. trg_dict_size(int): Size of the target language dictionary. Three special tokens will be added into the dictionary: for start mark, for end mark, and for unknown word. src_lang(string): A string indicating which language is the source language. Available options are: "en" for English and "de" for Germany. Returns: callable: The validation reader. rXrbrYrZz wmt16/valr[r\rBr r r validations% rdcCst|dkr t|t}nt|t}tjtjjj d||f}tj |s'Jd tjtjjj d}t ||||S)a return the word dictionary for the specified language. Args: lang(string): A string indicating which language is the source language. Available options are: "en" for English and "de" for Germany. dict_size(int): Size of the specified language dictionary. reverse(bool): If reverse is set to False, the returned python dictionary will use word as key and use index as value. If reverse is set to True, the returned python dictionary will use index as key and word as value. Returns: dict: The word dictionary for the specific language. r r2z Word dictionary does not exist. rZ) r?r@rAr4r5r6r7r8r9r:r;r>)r)r(rr=r'r r rget_dict,s  recCstjjjtdtddS)z!download the entire dataset. rYrZN)r7Zv4r8r9r^r_r`r r r rfetchOsrf)F)r )!__doc__ __future__rr4r%rgzip collectionsrr7Z paddle.compatcompatrZpaddle.utils.deprecatedutils deprecated__all__r_r`r@rAr r!r"r1r>rFrTrarcrdrerfr r r rsr    ' 0 0 .