o Me@sdZddlmZddlZddlmmZddlZddl Z ddl Z ddl Z ddl Z gZ dZdZddZdd Zed d d d dddZed d d d dddZed d d d dddZed d d d dddZed d d d dddZdS)a IMDB dataset. This module downloads IMDB dataset from http://ai.stanford.edu/%7Eamaas/data/sentiment/. This dataset contains a set of 25,000 highly polar movie reviews for training, and 25,000 for testing. Besides, this module also provides API for building dictionary. )print_functionNz6https://dataset.bj.bcebos.com/imdb%2FaclImdb_v1.tar.gzZ 7c2ac02c03563afcf9b574c7e56c153accsttjjtdtC}|}|dkrHt | |j r8| | tddttjV|}|dksWddSWddS1sSwYdS)zQ Read files that match the given pattern. Tokenize and yield each file. imdbNz )tarfileopenpaddledatasetcommondownloadURLMD5nextboolmatchname extractfilereadrstripsixb translatestring punctuationlowersplit)patternZtarftfrCD:\Projects\ConvertPro\env\Lib\site-packages\paddle/dataset/imdb.pytokenize(s$ "rc stt}t|D]}|D] }||d7<q q fddt|D}t|ddd}tt|\}}t tt|tj t |}t ||d<|S)z Build a word dictionary from the corpus. Keys of the dictionary are words, and values are zero-based IDs of these words. csg|] }|dkr|qS)rr).0xcutoffrr Gszbuild_dict..cSs|d |dfS)Nrrr)r!rrrIszbuild_dict..)key) collections defaultdictintrr iteritemssortedlistzipdictmovesrangelen) rr#Z word_freqdocword dictionarywords_word_idxrr"r build_dict<s   r9z2.0.0zpaddle.text.datasets.Imdbrz>Please use new dataset API which supports paddle.io.DataLoader)ZsinceZ update_tolevelreasoncsBdgfdd}||d||dfdd}|S)Nr'cs0t|D]}|fdd|D|fqdS)Ncsg|]}|qSr)get)r wUNKr8rrr$[sz0reader_creator..load..)rappend)routlabelr3r>rrloadYs  zreader_creator..loadrrc3sD] \}}||fVqdS)Nr)r3rB)INSrrreader`s  zreader_creator..readerr)Z pos_patternZ neg_patternr8rCrEr)rDr?r8rreader_creatorPs   rFcCttdtd|S)a IMDB training set creator. It returns a reader creator, each sample in the reader is an zero-based ID sequence and label in [0, 1]. :param word_idx: word dictionary :type word_idx: dict :return: Training reader creator :rtype: callable zaclImdb/train/pos/.*\.txt$zaclImdb/train/neg/.*\.txt$rFrecompiler8rrrtraing  rLcCrG)a IMDB test set creator. It returns a reader creator, each sample in the reader is an zero-based ID sequence and label in [0, 1]. :param word_idx: word dictionary :type word_idx: dict :return: Test reader creator :rtype: callable zaclImdb/test/pos/.*\.txt$zaclImdb/test/neg/.*\.txt$rHrKrrrtest|rMrNcCsttddS)za Build a word dictionary from the corpus. :return: Word dictionary :rtype: dict z/aclImdb/((train)|(test))/((pos)|(neg))/.*\.txt$)r9rIrJrrrr word_dicts rPcCstjjtdtdS)Nr)rrrr r r rrrrfetchsrQ)__doc__ __future__rZpaddle.dataset.commonrZpaddle.utils.deprecatedutils deprecatedr(rrIrr__all__r r rr9rFrLrNrPrQrrrrsb