o Me@sdZddlmZddlZddlmmZddlZddl Z ddl Z gZ dZ dZ GdddeZddd Zdd d Zd dZedddddejfddZedddddejfddZedddddddZdS)z imikolov's simple dataset. This module will download dataset from http://www.fit.vutbr.cz/~imikolov/rnnlm/ and parse training set and test set into paddle reader creators. )print_functionNz) collections defaultdictintstripsplit)f word_freqlwr r r word_count*s r2c sd}d}ttjjtjjjdtjjjR}| |}| |}t |t |}d|vr/|d=fddt |D}t |ddd }tt|\}} ttt|t jt|} t|| d<Wd | S1skwY| S) z Build a word dictionary from the corpus, Keys of the dictionary are words, and values are zero-based IDs of these words. $./simple-examples/data/ptb.train.txt$./simple-examples/data/ptb.valid.txtimikolovcsg|] }|dkr|qS)rr ).0x min_word_freqr r Iszbuild_dict..cSs|d |dfS)Nrrr )rr r r Mszbuild_dict..)keyN)tarfileopenpaddledatasetcommondownloadrURLMD5 extractfilersix iteritemssortedlistzipdictmovesrangelen) r!Ztrain_filenameZ test_filenametfZtrainfZtestfrZword_freq_sortedwords_word_idxr r r build_dict7s2    r;csfdd}|S)Nc3s\ttjjtjjjdtjjj}| }d|D]|}t j krddks.Jddg| dg}t|krcfdd|D}tjt|d D] }t|||VqUqt jkr| }fd d|D}dg|}|dg}d krt|krq||fVqJd WddS1swYdS)NrrzInvalid gram lengthr rcg|]}|qSr getrrUNKr:r r r"dz2reader_creator..reader..rcr=r r>r@rAr r r"irCrFzUnknow data type)r%r&r'r(r)r*rr+r,r-rr rrr6r.r4r5tupler )r7rriZsrc_seqZtrg_seq data_typefilenamenr:)rBr readerWs:       "zreader_creator..readerr )rHr:rIrGrJr rFr reader_creatorUsrKz2.0.0zpaddle.text.datasets.Imikolovrz>Please use new dataset API which supports paddle.io.DataLoader)ZsinceZ update_tolevelreasoncCtd|||S)a imikolov training set creator. It returns a reader creator, each sample in the reader is a word ID tuple. :param word_idx: word dictionary :type word_idx: dict :param n: sliding window size if type is ngram, otherwise max length of sequence :type n: int :param data_type: data type (ngram or sequence) :type data_type: member variable of DataType (NGRAM or SEQ) :return: Training reader creator :rtype: callable rrKr:rIrGr r r traintrQcCrN)a imikolov test set creator. It returns a reader creator, each sample in the reader is a word ID tuple. :param word_idx: word dictionary :type word_idx: dict :param n: sliding window size if type is ngram, otherwise max length of sequence :type n: int :param data_type: data type (ngram or sequence) :type data_type: member variable of DataType (NGRAM or SEQ) :return: Test reader creator :rtype: callable rrOrPr r r testrRrScCstjjtdtdS)Nr)r'r(r)r*r+r,r r r r fetchsrT)N)r)__doc__ __future__rZpaddle.dataset.commonr'Zpaddle.utils.deprecatedutils deprecatedrr%r.__all__r+r,objectrrr;rKr rQrSrTr r r r sF