o e ^@s6dZddlmZddlZddlZddlZddlZddlmZddl m Z ddl m Z ddl mZmZmZmZmZmZddlmZddlZdd lmZmZmZmZmZmZmZmZm Z dd l!m"Z"dd l#m$Z$m%Z%dd l&m'Z'm(Z(dd l)m*Z*ddl+m,Z,ddl-m.Z.m/Z/e0e1Z2dZ3dZ4ede5Z6GdddeZ7ddddZ8deddZ9dedd Z:dfd&d'Z;dgd)d*Zdid3d4Z?djd6d7Z@dkdd?ZBdmdCdDZCdndGdHZDdgdIdJZEdodMdNZFdpdOdPZGdqdSdTZHdrdXdYZIe%ZJeJfdsd\d]ZKdtd_d`ZLe1dakreLejMd^ejMdbejMdcdSdS)uz/Post-processing image optimization of OCR PDFs.) annotationsN) defaultdict)fspath)Path)CallableIterator MutableSet NamedTupleNewTypeSequence)compress) DictionaryNameObjectObjectStreamModePdfPdfErrorPdfImageStreamUnsupportedImageTypeError)Image)ExecutorSerialExecutor)jbig2encpngquant) PdfContext)OutputFileAccessError)IMG2PDF_KWARGS safe_symlinkKFXrefc@s"eZdZUdZded<ded<dS)XrefExtz$A PDF xref and image extension pair.r!xrefstrextN)__name__ __module__ __qualname____doc____annotations__r+r+AD:\Projects\ConvertPro\env\Lib\site-packages\ocrmypdf/optimize.pyr"0s  r"rootrr#r%r$returncCs||d|S)N08dr+)r-r#r%r+r+r,img_name7sr0cC t||dS)N.pngr0r-r#r+r+r,png_name; r5cCr1)N.jpgr3r4r+r+r,jpg_name?r6r8pikerimager+tuple[PdfImage, tuple[Name, Object]] | NonecCs~~|jtjkr dS|jdkrtd|ddS|jdks$|jdkr/td|ddSt|}t |j dkr|j d}|j d}t |j dkrw|dtj krw|d tj ddkrw|dtjkrw|dswtd|d |j d}ntd|d dS|j d}|jdkrtd|d dS|dtjkrtd|d dS|dtjkr|d d ddkrtd|ddStj|vrtd|ddS||fS)Ndxref z': skipping image with small stream sizez0: skipping image with unusually small dimensionsrzQ: found image compressed as /FlateDecode /DCTDecode, marked for JPEG optimizationz2: skipping image with multiple compression filtersz: skipping wide gamut imagez: skipping JPEG2000 imagez/Kz: skipping CCITT Group 3 imagez": skipping image with Decode table)ZSubtyperrLengthlogdebugZWidthZHeightrlenZfilter_decodeparms FlateDecodegetZ Predictor DCTDecodebits_per_componentZ JPXDecodeZCCITTFaxDecodeZDecode)r9r-r:r#pimZ first_filtdpZ second_filtdpfiltdpr+r+r,extract_image_filterCsP        " rKXrefExt | Nonec Cs:~t||||}|durdS|\}}|jdkr|dtjkrtr|jtjd}|dus1|j rzWz0tj |j_||d} | d} |j | d} Wdn1sTwY| | | WntyzYW|duru||j_dS|j`dSwW|dur||j_n|j`n |dur||j_w|j`wt|| SdS)Nr?rr/wbstream)rKrHr JBIG2Decoder availableobjrFZ ColorSpaceZ image_maskZ DeviceGrayopen extract_torename with_suffixrr") r9r-r:r#optionsresultrIrJ colorspaceimgnamefr%r+r+r,extract_image_jbig2|sB          r\c Cst||||}|dur dS|\}}|jdkrdS|dtjkr_|jdkr_z+||d}|d} |j| d} Wdn1sAwY||| Wn t yYYdSwt || S|j r||j |j vr||jdkr||t||t |dS|j s|j |j vrz |t||Wntytd YdSwt |dS|j s|j tjkr|jdkr|js|t||t |dSdS) Nr?rr@r/rMrNr2z8PDF contains an atypical image that cannot be optimized.)rKrHrrGoptimizerSrTrUrVrr"ZindexedrYZSIMPLE_COLORSPACESZ as_pil_imagesaver5NotImplementedErrorrBwarningZICCBased jbig2_lossy) r9r-r:r#rWrXrIrJrZr[r%r+r+r,extract_image_genericsP              rc extract_fnCallable[..., XrefExt | None]Iterator[tuple[int, XrefExt]]c csht}t}i}d}t|jD]_\}} z| jj} Wn ty#Yqwt| D]D\} } | jddkr6q*t | jd} t j | vrXt | j jd}| |t d|d| | t d| d| |vrn||| <q*q||}|D];} || df} z |||| | |d}Wntyt d| d|d7}Yqvw|r|\}}|| t| |fVqvdS) aExtract image using extract_fn Enumerate images on each page, lookup their xref/ID number in the PDF. Exclude images that are soft masks (i.e. alpha transparency related). Record the page number on which an image is first used, since images may be used on multiple pages (or multiple times on the same page). Current we do not check Form XObjects or other objects that may contain images, and we don't evaluate alternate images or thumbnails. extract_fn must decide if wants to extract the image in this context. If it does a tuple should be returned: (xref, ext) where .ext is the file extension. extract_fn must also extract the file it finds interesting. rr?r=z': skipping image because it is an SMaskz': treating as an optimization candidate)r9r-r:r#rWz0: While extracting this image, an error occurredN)set enumeratepagesZ ResourcesZXObjectAttributeErrordictitemsZobjgenr!rZSMaskaddrBrC get_object Exception exceptionr")r9r-rWrdZ include_xrefsZ exclude_xrefsZpageno_for_xreferrorspagenopageZxobjsZ_imnamer:r#Z smask_xrefZ working_xrefsrX_r%r+r+r,extract_imagessV         rutuple[list[Xref], list[Xref]]cCsg}g}t|||tD]!\}}td||jdkr!||jq |jdkr,||jq tdt|dt|||fS)z0Extract any >=2bpp image we think we can improvez%sr2r7zOptimizable images: JPEGs: z PNGs: )rurcrBrCr%appendr#rD)r9r-rWjpegspngsrtxref_extr+r+r,extract_images_generic!s    r{dict[int, list[XrefExt]]cCsPtt}t|||tD]\}}||j}|||q tdt||S)z?Extract any bitonal image that we think we can improve as JBIG2z"Optimizable images: JBIG2 groups: ) rlistrur\jbig2_page_group_sizerwrBrCrD)r9r-rW jbig2_groupsrrrzgroupr+r+r,extract_images_jbig22s  rrexecutorrNonec Csfddd}ddd}|jd kr|}tj}n|}tj}|d |jtt|d d |j d ||||ddS)z&Produce JBIG2 images from their groupsr-rgroupsr|c3sB|D]\}}d|d}tfdd|D|fVqdS)Nrr/c3s |] \}}t||VqdSNr3).0r#r%r-r+r, HszB_produce_jbig2_images..jbig2_group_args..)rlr)r-rr xref_extsprefixr+rr,jbig2_group_argsCs z/_produce_jbig2_images..jbig2_group_argsc ssf|D]+\}}d|d}t|D]\}}|\}}t|t|||||d|dfVqqdS)Nrr/.04d)rlrhrr0) r-rrrrnrzr#r%r+r+r,jbig2_single_argsLs  z0_produce_jbig2_images..jbig2_single_argsr?TZJBIG2item)totaldescunitdisable use_threads max_workers tqdm_kwargstasktask_argumentsN)r-rrr|)rr|)r~rZconvert_group_mpZconvert_single_mpjobsrkrD progress_bar)rr-rWrrrZ jbig2_argsZ jbig2_convertr+r+r,_produce_jbig2_images>s&   rcCst|||||D]Y\}}d|d}||d}|r.|} t|| } t| d} n |jdkr6d} nt|t|D]%\} } | \}}||d| d}|}| |d }|j |t j | d q>q dS) aConvert images to JBIG2 and insert into PDF. When the JBIG2 page group size is > 1 we do several JBIG2 images at once and build a symbol dictionary that will span several pages. Each JBIG2 image must reference to its symbol dictionary. If too many pages shared the same dictionary JBIG2 encoding becomes more expensive and less efficient. The default value of 10 was determined through testing. Currently this must be lossy encoding since jbig2enc does not support refinement coding. When the JBIG2 symbolic coder is not used, each JBIG2 stands on its own and needs no dictionary. Currently this must be lossless JBIG2. rr/z.sym)Z JBIG2Globalsr?NrrrfilterZ decode_parms) rrlexists read_bytesrr r~FileNotFoundErrorrhrnwriterrP)r9rr-rWrrrrZ jbig2_symfileZjbig2_globals_dataZ jbig2_globalsZjbig2_globals_dictrrzr#rtZ jbig2_im_fileZ jbig2_im_dataim_objr+r+r,convert_to_jbig2ms*      rargstuple[Xref, Path, Path, int]tuple[Xref, Path | None]cCs|\}}}}t|}|j|d|dWdn1swY|j|jkr>td|d||dfS||fS)NT)r^Zqualityr=z, jpeg, made larger - skip)rrSr_statst_sizerBrCunlink)rr#in_jpgopt_jpg jpeg_qualityZimr+r+r,_optimize_jpegs  rrxSequence[Xref]c sPdfdd }dfdd }|d jtd td j d t||d dS)Nr.&Iterator[tuple[Xref, Path, Path, int]]c3s4D]}t|}|d}|||jfVqdS)Nz.opt.jpg)r8rVr)r#rr)rxrWr-r+r, jpeg_argss   z"transcode_jpegs..jpeg_argsrXrcs<|\}}|r|}|d}|j|tjd|dSNr)r)rrnrrrGupdate)rXpbarr#rcompdatar)r9r+r, finish_jpegs   z$transcode_jpegs..finish_jpegTzRecompressing JPEGsr:rrrrrrrrrZ task_finished)r.r)rXr)rrkrDrr)r9rxr-rWrrrr+)rxrWr9r-r,transcode_jpegss rcCsPt||||}|dur dS|\}}|dtjkr&|ds&|jdkr&t|dSdS)Nrr?z.memory)rKrrGr^r")r9r-r:r#rWrXZ_pimrJr+r+r,_find_deflatable_jpegs  r%tuple[Pdf, threading.Lock, Xref, int]tuple[Xref, bytes]c Cs|\}}}}|)||d}z|}Wnty)|dfYWdSwWdn1s4wYt||}t|t|krJ|dfS||fS)Nr)rnread_raw_bytesrr rD)rr9lockr# complevelxobjdatarr+r+r, _deflate_jpegs      rc sgt||tD]\}}|j}td|d|q |jdkr&dndtdfdd }fd d }|d |j t d t d|j dt ||ddS)Nr=z!: marking this JPEG as deflatabler] r.rc3sD] }|fVqdSrr+r#rrxrr9r+r, deflate_argssz#deflate_jpegs..deflate_argscsf|\}}t|dkr-|d}|j|tjtjgdWdn1s(wY|dSr)rDrnrrrErGr)rXrr#rr)rr9r+r,finishs   zdeflate_jpegs..finishTzDeflating JPEGsr:rr)r.r)rurr#rBrCrwr^ threadingLockrrkrDrr) r9r-rWrZ_pagenorzr#rrr+rr, deflate_jpegss,  rfilenameboolc Cs,|d}|d}tjt|fd|itWdn1s"wYt|`}tt|j dj }| |}| |d}|j||j|jdt|t|} hd} | | 8} |D]} | tjkr{t| | vr{|| || <qh| D]} || =q~WddS1swYdS)Nz.png.pdfrMZ outputstreamrr> z/IDz/OCz /Interpolatez/SMaskz/OPIz/Maskz/Intentz /StructParentz /MetadataT)rVrSimg2pdfconvertrrrnextiterriimagesvaluesZ copy_foreignrnrrFilterZ DecodeParmsrgkeysrrAr$) r9rr#outputr[Z pdf_imageZ foreign_imageZ local_imagerZdel_keysZ keep_fieldskeyr+r+r,_transcode_png s6        !!rr image_name_fnCallable[[Path, Xref], Path]c st|jdkr8td|jdtd|jdffdd}|d|jtdtd|j d t j |d D] }t |}t |||q:dS) Nr@ r<c3sLD] }t||t|ddfV|qdS)Nrr?)rBrCr5rmrrrmodified png_qualityr-r+r, pngquant_argsBs z%transcode_pngs..pngquant_argsTZPNGsr:rr) rgr^maxrminrrkrDrrZ quantize_mpr5r) r9rrr-rWrrr#rr+rr,transcode_pngs3s,   r input_file output_filecCs|j}|jdkrt|||S|jdkr|jdkrtnd|_|jdkr-|jdkr*tnd|_|jdkr:|jr7dnd|_t |S}|j d}|j dd t |||\}} t|||||t||||t|| t|||t|||} t|| ||||d } ||j| fi|Wdn1swY|j} | j} | dkrtd td d| | }|dkrtd t |}||j|fi|Wd|S1swY|St| ||S)Nrr](rr?rT)exist_okz.opt.pdfzeOutput file not created after optimizing. We probably ran out of disk space in the temporary folder: rzLImage optimization did not improve the file - optimizations will not be used)rWr^rrDEFAULT_JPEG_QUALITYrDEFAULT_PNG_QUALITYr~rbrrSparentmkdirr{rrrr5rrrVZremove_unreferenced_resourcesr_rrrtempfile gettempdirrBinfo)rrcontextZ save_settingsrrWr9r-rxryrZ target_fileZ input_sizeZ output_sizeZsavingsr+r+r,r^bsZ                r^r?c Csddlm}ddlm}Gddd}t|}|||t|dddd}|-}t|||dd} t|d} t|| | td d t j d |t | t |WddS1sVwYdS) Nr)copy)TemporaryDirectoryc@seZdZdZddZdS)zmain..OptimizeOptionszEmulate ocrmypdf's optionscSs:||_||_||_||_||_d|_||_d|_d|_dS)NrTF) rrr^rrr~rbquietr)selfrr optimize_rrjb2lossyr+r+r,__init__s z&main..OptimizeOptions.__init__N)r&r'r(r)rr+r+r+r,OptimizeOptionss rF)rrrrrrzout.pdfT)Zcompress_streamsZ preserve_pdfaZobject_stream_mode) shutilrrrrintrr^rkrgenerater) infileoutfilelevelrrrrrWZtmpdirrZtmpoutr+r+r,mains6     "r__main__r@r])r-rr#r!r%r$r.r)r-rr#r!r.r) r9rr-rr:rr#r!r.r;) r9rr-rr:rr#r!r.rL)r9rr-rrdrer.rf)r9rr-rr.rv)r9rr-rr.r|)rr|r-rrrr.r) r9rrr|r-rrrr.r)rrr.r) r9rrxrr-rrrr.r)rrr.r)r9rr-rrrr.r)r9rrrr#r!r.r) r9rrrrrr-rr.r)rrrrrrr.r)r?)Nr) __future__rloggingsysrr collectionsrosrpathlibrtypingrrrr r r zlibr rZpikepdfr rrrrrrrrZPILrZocrmypdf._concurrentrrZocrmypdf._execrrZocrmypdf._jobcontextrZocrmypdf.exceptionsrZocrmypdf.helpersrr getLoggerr&rBrrrr!r"r0r5r8rKr\rcrur{rrrrrrrrrrZDEFAULT_EXECUTORr^rargvr+r+r+r,sb      ,          9 ) > >  / -   ' ), > ."