o Me'@sdZddlmZddlZddlZddlZddlZddlm Z ddl m m Z ddlmZmZgZdZdZdZdZd Zd Zd Zd Zd ZdZdZddZddZddZ   d$ddZ e dddddddZ!e dddddddZ"e dddddd d!Z#e dddddd"d#Z$dS)%a^ Conll05 dataset. Paddle semantic role labeling Book and demo use this dataset as an example. Because Conll05 is not free in public, the default downloaded URL is test set of Conll05 (which is public). Users can change URL and MD5 to their Conll dataset. And a pre-trained word vector model based on Wikipedia corpus is used to initialize SRL model. )print_functionN)ziprangezBhttp://paddlemodels.bj.bcebos.com/conll05st/conll05st-tests.tar.gzZ 387719152ae52d60422c016e92a742fcz:http://paddlemodels.bj.bcebos.com/conll05st%2FwordDict.txtZ ea7fb7d4c75cc6254716f0177a506baaz:http://paddlemodels.bj.bcebos.com/conll05st%2FverbDict.txtZ 0d2977293bbb6cbefab5b0f97db1e77czesz1corpus_reader..reader..r -r F*Trz*)()rzUnexpected label: %s)tarfiler extractfilegzipGzipFilercptZto_textrsplitlenrappendrfind RuntimeErrorclose)tfZwfpfZ words_fileZ props_fileZ sentenceslabelsZone_segwordlabelZ a_kind_lableZ verb_listrZlblZcur_tagZ is_in_bracketZlbl_seqZ verb_wordl data_path props_name words_namer rreaderVsz         6 zcorpus_reader..readerr)r:r<r;r=rr9r corpus_readerMs >r>csfdd}|S)Nc 3sD]\}}}t|}|d}dgt|}|dkr*d||d<||d}nd}|dkr=d||d<||d}nd}d||<||}|t|dkr\d||d<||d} nd} |t|dkrsd||d<||d} nd} fdd|D} |tg|} |tg|} |tg|}| tg|}| tg|}|g|}fd d|D}| | | ||||||f VqdS) NzB-Vrr ZbosrZeoscsg|]}|tqSr)getUNK_IDXrw) word_dictrrr!sz2reader_creator..reader..csg|]}|qSr)r?rA) label_dictrrr!s)r.rr?r@)Zsentence predicater5Zsen_lenZ verb_indexmarkZctx_n1Zctx_n2Zctx_0Zctx_p1Zctx_p2Zword_idxZ ctx_n2_idxZ ctx_n1_idxZ ctx_0_idxZ ctx_p1_idxZ ctx_p2_idxZpred_idxZ label_idxr>rDpredicate_dictrCrrr=sF      zreader_creator..readerr)r>rCrHrDr=rrGrreader_creators0rIz2.0.0zpaddle.text.datasets.Conll05str z>Please use new dataset API which supports paddle.io.DataLoader)ZsinceZ update_tolevelreasoncCsLttjjtdt}ttjjtdt}t tjjt dt }|||fS)zF Get the word, verb and label dictionary of Wikipedia corpus. conll05st) rpaddledatasetcommondownload WORDDICT_URL WORDDICT_MD5 VERBDICT_URL VERBDICT_MD5r TRGDICT_URL TRGDICT_MD5)rC verb_dictrDrrrget_dicts  rXcCstjjtdtS)z@ Get the trained word vector based on Wikipedia corpus. rL)rMrNrOrPEMB_URLEMB_MD5rrrr get_embeddings r[cCs6t\}}}ttjjtdtddd}t||||S)ap Conll05 test set creator. Because the training dataset is not free, the test dataset is used for training. It returns a reader creator, each sample in the reader is nine features, including sentence sequence, predicate, predicate context, predicate context flag and tagged sequence. :return: Training reader creator :rtype: callable rLz2conll05st-release/test.wsj/words/test.wsj.words.gzz2conll05st-release/test.wsj/props/test.wsj.props.gz)r<r;) rXr>rMrNrOrPDATA_URLDATA_MD5rI)rCrWrDr=rrrtests r^cCs^tjjtdttjjtdttjjtdt tjjt dt tjjt dt dS)NrL)rMrNrOrPrQrRrSrTrUrVrYrZr\r]rrrrfetchs r_)NNN)%__doc__ __future__rr(r* itertoolsZpaddle.dataset.commonrMZ paddle.compatcompatr,Zpaddle.utils.deprecatedutils deprecatedZ six.movesrr__all__r\r]rQrRrSrTrUrVrYrZr@rrr>rIrXr[r^r_rrrrsp  K 8