o e3@sddlZddlmZddlmZddlmZddlmZddl m Z ddl m Z ddl mZdd l mZdd d ZGd ddeZdS)N)partial) components)bfile) record_pb2) RecordReader)default_data_manager) decode_tag encode_tagFcCsLd|vrdS|r$t|d}|}t}||d|jdjkr$dSdS)a*Determine whether it is a VDL log file according to the file name. File name of a VDL log file must contain `vdlrecords`. Args: path: File name to determine. check: Check file is valid or not. Returns: True if the file is a VDL log file, otherwise false. Z vdlrecordsFfilepathZ meta_data_tagrT)rZget_nextrRecordParseFromStringvaluestag)pathcheck_reader meta_datarecordrFD:\Projects\ConvertPro\env\Lib\site-packages\visualdl/reader/reader.pyis_VDLRecord_files   rc@s eZdZdZd?ddZeddZejddZedd Zd d Z d d Z ddZ ddZ ddZ ddZddZddZd@ddZddZdd ZdAd"d#ZdBd%d&Zd@d'd(Zd)d*ZdCd+d,ZdAd-d.Zed/d0ZdAd1d2Zd3d4Zd@d5d6Zd@d7d8ZdAd9d:Zd;d<Z d=d>Z!d$S)D LogReaderzELog reader to read vdl log, support for frontend api in lib.py. cCst|tr |g|_n||_d|_i|_d|_i|_i|_i|_i|_ t dd|_ |rWt dd|_ |j|d|}|j|dt}|D] }td||fqJdS|rjt|_|jdd i|_d |_dSdS) z}Instance of LogReader Args: logdir: The dir include vdl log files, multiple subfolders allowed. NcSstddS)NcS ttSN collections defaultdictlistrrrrNz6LogReader.__init__....)rrrrrrr!Nsz$LogReader.__init__..cSrrrrrrrr!Rs) file_path)remainz(self.get_%s=partial(self.get_data, '%s')Tupdater) isinstancestrdirreaderreaderswalks_tags name2tags tags2nameZ file_readersrr _log_datas _log_dataget_file_reader get_remain read_log_datarkeysexecr data_manager load_new_dataZ_a_tags_model)selflogdirr#r$Zcomponents_namenamerrr__init__8s8      zLogReader.__init__cC|jSr)r9r:rrrmodelazLogReader.modelcCs||_t|d4}||std|nt|r||_ntd|WddSWddSWddS1s@wYdS)Nrbz:Model path %s should be file path, please check this path.z1Model path %s is invalid, please check this path.)r9rZBFileisfileprintexists)r:Z model_pathZbfprrrr@es$  "cCr>r)r)r?rrrr;trAzLogReader.logdircCspg}d|kr|D]}||j|j|j|jgq|Sd|kr6|D]}||j|j|jj|j|jjgq |S|S)Nscalarscalars)appendidr timestampvalue tag_value)r: componentZ proto_datasdataitemrrrparsing_from_protoxszLogReader.parsing_from_protocCsH|j}i}|D]}t|j|}ttdd|}|||<q |S)NcSst|Srr )xrrrr!sz)LogReader._get_log_tags..)r1r5r map)r:Zcomponent_keysZlog_tagskeyr-tagsrrr _get_log_tagss  zLogReader._get_log_tagscCs||jvr$||j|vr$||j||vr$|j|||St||j|}|j|dd}||_||_|j|d}|j |dd||}| ||}||j|||<|S)NF)r#r&)r*)r$r&) r0r5rjoinr,_get_file_readerr)r*r3r4rP)r:rMrunrr#r*r$rNrrr get_log_datas&  zLogReader.get_log_datacCs|Sr)rUr?rrrget_tagsszLogReader.get_tagscCs|j|t|Sr)r1r)r:rMrrrrget_dataszLogReader.get_datacCst}|||jd}t|j}t|jj |}||j vr| d}d|kr.d}nQd|kr5d}nJd|krsz"LogReader.logs..)reverser)r,rgitemssortlenr&)r:r&Z walks_temprXrTZ tags_temprrrlogss   zLogReader.logscCsn|jdur ||j|d|krtd|d|t||}||jvr.||||j||_ |j S)zGet log reader for every vdl log file. Get instance of class RecordReader base on BFile. Note that each `log` may contain multi `tag`. Args: dir: Dir name of log. log: Vdl log file name. NzCan't find file %s./) r,rngetFileNotFoundErrorrrVr+r5register_readerr*)r:r)logr rrrget_log_readers    zLogReader.get_log_readercCs ||dS)zGet file reader for specified vdl log file. Get instance of class RecordReader base on BFile. Args: file_path: Vdl log file path. T)rW)r:r#rrrr2 s zLogReader.get_file_readerTcCs.|r|||j||_|jSt|d}|S)Nr )rrr+r*r)r:r#r&r*rrrrWs    zLogReader._get_file_readerNcCs@|r|t|jvrt||d}||_||j|<dSdS dS)N)r r))r r+r5rr))r:rr)r&r*rrrrrs zLogReader.register_readercCs:|||jD]\}}t||}|||q dS)zxRegister all readers for all vdl log files. Args: update: Need update if `update` is True. N)rnr,rkrrVrr)r:r&r)rr rrrregister_readers%s  zLogReader.register_readerscCsV|jD]#}||_|j}|D]}||\}}}}|j||jj||qqdS)zAdd remain data to data_manager. Add remain data to data manager according its component type and tag one by one. N)r+rr*r3rar7Zadd_itemr))r:r*r$rOrMr)rrrrr add_remain0s zLogReader.add_remaincCs4|jdur |dur td|dur|jS|S)z,Get all remain data by self.reader. NzPlease specify log path!)r* RuntimeErrorr3)r:r*rrrr3?s  zLogReader.get_remainc CsJtdd}|D]}||\}}}}||||q |r#||_|S)znParse data from log file without sampling. Args: remain: Raw data from log file. cSrrrrrrrr!Mr"z)LogReader.read_log_data..)rrrarHr1) r:r$r&r1rOrMr)rrrrrr4GszLogReader.read_log_datacCr>r)r1r?rrrlog_dataVrAzLogReader.log_datacCs|j|dt|jS)Nr%)rnr r,r5r:r&rrrrunsZs zLogReader.runscCs|jdur ||jSr)r-rvr?rrrrT^s zLogReader.tagscCs6|jdurtS|dur|j|dt|j}|S)z)Get components type used by vdl. NTr%)r;setr8r-r)r:r&Zcomponents_setrrrrcs  zLogReader.componentscCsPt}|js|S|dur|j|dt|jD] }|dkr q||q|S)z1Get component tabs used by vdl frontend. Tr%r)r{r;r8r-radd)r:r&component_tabsrMrrrr}ns  zLogReader.component_tabscCs&|jdur|j|d|dSdS)z~Load remain data. Make sure all readers for every vdl log file are registered, load all remain data. Nr%)r;rurvryrrrr8|s   zLogReader.load_new_datacCs|Srrr?rrr __enter__zLogReader.__enter__cCsdSrr)r:exc_typeexc_valexc_tbrrr__exit__rzLogReader.__exit__)rrF)T)NTr)"__name__ __module__ __qualname____doc__r=propertyr@setterr;rPrUrYrZr[rar_rgrnrtr2rWrrrurvr3r4rxrzrTrr}r8r~rrrrrr3sF )    +         rr)r functoolsrZvisualdl.componentrZ visualdl.iorZvisualdl.protorZvisualdl.reader.record_readerrZvisualdl.server.data_managerrZvisualdl.utils.string_utilrr robjectrrrrrs