o eR@s<dZddlmZddlZddlZddlZddlmZddlm Z m Z ddl m Z m Z mZmZddlmZmZddlmZmZdd lmZmZmZdd lmZmZdd lmZdd lm Z dd l!m"Z"Gddde Z#d*ddZ$ddZ%Gdddej&j'Z(d+d d!Z)Gd"d#d#eZ*ed$d%Z+ed&d'Z,ed(d)Z-dS),aSemaphore-free alternate executor. There are two popular environments that do not fully support the standard Python multiprocessing module: AWS Lambda, and Termux (a terminal emulator for Android). This alternate executor divvies up work among worker processes before processing, rather than having each worker consume work from a shared queue when they finish their task. This means workers have no need to coordinate with each other. Each worker communicates only with the main process. This is not without drawbacks. If the tasks are not "even" in size, which cannot be guaranteed, some workers may end up with too much work while others are idle. It is less efficient than the standard implementation, so not th edefault. ) annotationsN)suppress)Enumauto)islicerepeat takewhile zip_longest)PipeProcess) Connectionwait)CallableIterableIterator)ExecutorhookimplNullProgressBarInputFileError)remove_all_log_handlersc@s"eZdZdZeZeZeZdS) MessageTypezImplement basic IPC messaging.N)__name__ __module__ __qualname____doc__r exceptionresultcompleter r ND:\Projects\ConvertPro\env\Lib\site-packages\ocrmypdf/extra_plugins/semfree.pyr$s  rnintiterablerreturnrcs&t|ttfddtdDS)zSplit iterable into groups of n. >>> list(split_every(4, range(10))) [[0, 1, 2], [3, 4, 5], [6, 7, 8], [9]] https://stackoverflow.com/a/22919323 c3s|] }ttVqdSN)listr).0_iteratorr"r r! 5szsplit_every..N)iterrboolr)r"r$r r*r! split_every,sr/cGstd)Nz-A worker process lost access to an input filer)argsr r r!process_sigbus8sr1cs*eZdZdZd fdd Zdd ZZS) ConnectionLogHandlerzBHandler used by child processes to forward log messages to parent.connr r%Nonecstd||_dSr&)super__init__r3)selfr3 __class__r r!r6?s  zConnectionLogHandler.__init__cCs|jd|fdS)Nlog)r3send)r7recordr r r!enqueueEszConnectionLogHandler.enqueue)r3r r%r4)rrrrr6r= __classcell__r r r8r!r2<sr2r3r user_initCallable[[], None]c CsttttjtWdn1swYt|}t}t|| || |||D],}z||}Wnt yY} z| t j| fWYd} ~ nd} ~ ww| t j|fq6| t jdf|dS)z Initialize a process pool workerN)rAttributeErrorsignalZSIGBUSr1r2logging getLoggerrsetLevel addHandler Exceptionr;rrrrclose) r3r?ZlogleveltaskZ task_argshrootr0rer r r! process_loopIs*    rMc@seZdZdZdddZdS)LambdaExecutorzEExecutor for AWS Lambda or similar environments that lack semaphores. use_threadsr. max_workersr# tqdm_kwargsdictworker_initializerrrItask_argumentsr task_finishedc Cs|r/|dkr/|jd i|}|D] } || } || |qWddS1s(wYdSt|}tttt||} | sBdSg} g} | D]*}t\}}dd|D}tt||tdj ||fd}d|_ | || |qH| D]}| qu|jd i|m}| rt | D]\} t| tstdz| \}}Wnty| | Yqw|tjkr|r|||q|dkr|}t|j}||q|tjkr| | q|tjkr| D]}|q|q| sWdn1swY| D]}|qdS) NcSsg|]}|dur|qSr&r )r(r0r r r! sz+LambdaExecutor._execute..)targetr0TzWe only support Connection()r:r ) pbar_classr'r r/r r rMrCrDleveldaemonappendstartr isinstancer NotImplementedErrorrecvEOFErrorremoverrnamehandlerr terminatejoin)r7rOrPrQrSrIrTrUZpbarr0rZ grouped_argsZ processes connectionschunkZ parent_connZ child_connZ worker_argsprocessmsg_typemsgr<loggerr r r!_executems                     zLambdaExecutor._executeN)rOr.rPr#rQrRrSrrIrrTrrUr)rrrrrnr r r r!rNjsrNcCs t|dS)N)rZ)rN)Zprogressbar_classr r r! get_executors rocCstSr&)rC StreamHandlerr r r r!get_logging_consolesrqcCstSr&rr r r r!get_progressbar_classsrr)r"r#r$rr%r)r3r r?r@).r __future__rrClogging.handlersrB contextlibrenumrr itertoolsrrrr multiprocessingr r Zmultiprocessing.connectionr r typingrrrZocrmypdfrrZocrmypdf._concurrentrZocrmypdf.exceptionsrZocrmypdf.helpersrrr/r1handlers QueueHandlerr2rMrNrorqrrr r r r!s6        !P