hhhRdZddlmZddlZddlZddlZddlZddlZddlmZddl m Z ddl m Z ddl mZmZmZmZmZmZmZmZmZmZmZddlmZddlZddlZdd lmZdd lm Z dd l!m"Z"dd l#m$Z$m%Z%erddl&Z&ddl'Z'ddl(Z(ddl)Z)dd l*m+Z+gdZ,gdZ-d6dZ.ej/e0Z1dZ2dZ3dZ4dZ5hdZ6d7dZ7d8d Z8d8d!Z9d"d#gZ:d9d'Z;Gd(d)e Z<Gd*d+e Z=Gd,d-e Z>Gd.d/e Z?Gd0d1e Z@Gd2d3e ZAGd4d5e ZBdS):z(Module contains common parsers for PDFs.) annotationsN)datetime)Path)TemporaryDirectory) TYPE_CHECKINGAnyBinaryIOIterableIteratorLiteralMappingOptionalSequenceUnioncast)urlparse)Document)BaseBlobParser)Blob)BaseImageBlobParserRapidOCRBlobParser)TextLinearizationConfig) DCTDecodeDCT JPXDecode) LZWDecodeLZW FlateDecodeFl ASCII85DecodeA85ASCIIHexDecodeAHxRunLengthDecodeRLCCITTFaxDecodeCCF JBIG2Decodeimages,Sequence[Union[Iterable[np.ndarray], bytes]]returnstrc ddlm}n#t$rtdwxYw|}d}|D]6}||\}}|r$d|D}|d|z }7|S)zExtract text from images with RapidOCR. Args: images: Images to extract text from. Returns: Text extracted from images. Raises: ImportError: If `rapidocr-onnxruntime` package is not installed. r)RapidOCRzc`rapidocr-onnxruntime` package not found, please install it with `pip install rapidocr-onnxruntime`cg|] }|d S)).0texts m/var/www/FlaskApp/flask-venv/lib/python3.11/site-packages/langchain_community/document_loaders/parsers/pdf.py z5extract_from_images_with_rapidocr..Zs111$d1g111 )rapidocr_onnxruntimer. ImportErrorjoin)r)r.ocrr4imgresult_s r5!extract_from_images_with_rapidocrr@@s 1111111     1    (**C D&&CHH   &11&111F DIIf%% %D Ks #z {image_text} r8z >sourcecreatorproducer total_pages creationdateblobrcontentformatc|rR|jpd}|dkr |dd}d|d|d}n#|dkrd tj|d d |d }|S)aFormat the content of the image with the source of the blob. blob: The blob containing the image. format:: The format for the parsed output. - "text" = return the content as is - "markdown-img" = wrap the content into an image markdown link, w/ link pointing to (`![body)(#)`] - "html-img" = wrap the content as the `alt` text of an tag and link to (`{body}`) #z markdown-img]z\\]z![z]()zhtml-imgz T)quotez src=)rAreplacehtmlescape)rFrGrHrAs r5_format_inner_imagerQisX# ^ # #ooc622G/7//f///GG z ! !W4;wd#C#C#CWW6WWWG Nr7metadatadict[str, Any]ct|stdt |ddt std|S)zValidate that the metadata has all the standard keys and the page is an integer. The standard keys are: - source - total_page - creationdate - creator - producer Validate that page is an integer if it is present. z3The PDF parser must valorize the standard metadata.pagerz(The PDF metadata page must be a integer.)_STD_METADATA_KEYSissubsetkeys ValueError isinstancegetint)rRs r5_validate_metadatar]sg  & &x}} 7 7PNOOO hll61--s 3 3ECDDD Or7cxi}ddd}|D]\}}t|ttfvrt|}|dr |dd}|}|dvrV t j|dd d  d ||<#t$r|||<YwxYw||vr||||<|||<t|tr| ||<t|tr|||<|S) zPurge metadata from unwanted keys and normalize key names. Args: metadata: The original metadata dictionary. Returns: The cleaned and normalized the key format of metadata dictionary. rDrA) page_count file_path/r1N)rEmoddate'r/zD:%Y%m%d%H%M%S%zT) itemstyper,r\ startswithlowerrstrptimerN isoformatrYrZstrip)rR new_metadatamap_keykvs r5_purge_metadatarpsf$&L#G    1 773* $ $AA <<   !""A GGII + + + $"*"3IIc2&&(:##)C..Q $ $ $"# Q $ '\\'(L $LOO 3   ggiiLOO 3   LO s?CCCz  extras list[str]text_from_pagecdfd ||d }|s=d }d td |}|rtd|z}||z}|S)a5Insert extras such as image/table in a text between two paragraphs if possible, else at the end of the text. Args: extras: List of extra content (images/tables) to insert. text_from_page: The text content from the page. Returns: The merged text with extras inserted. rrrsrtr,recursboolr+ Optional[str]c@|rtD]}||}|dkrpd}|r ||d|d}|r|||dz}nEd}dtd|}|r||z}|d||z||dz}nd}n|}|S)NFr/rqc|SNr2xs r5zO_merge_text_and_extras.._recurs_merge_text_and_extras..s!r7)_PARAGRAPH_DELIMITERrfindr;filter) rrrtrvdelimpos previous_textall_text all_extras str_extras_recurs_merge_text_and_extrass r5rz=_merge_text_and_extras.._recurs_merge_text_and_extrass  &-  $**511"99$(M(E(E"N4C4$8%)) % #0>#$$3G#G%' %+[[ V1L1L%M%M %<).);J*4C40:=stt@TT!E#& %Hr7Tr/rqc|Sr|r2r}s r5rz(_merge_text_and_extras..s!r7rz)rrrsrtr,rvrwr+rx)r;rr)rrrtrrrrs @r5_merge_text_and_extrasrs<-,V^TJJH / [[ V!J!J. Or7cJeZdZdZ d dedddddd!fdZd"dZd#dZxZS)$ PyPDFParseraParse a blob from a PDF using `pypdf` library. This class provides methods to parse a blob from a PDF document, supporting various configurations such as handling password-protected PDFs, extracting images. It integrates the 'pypdf' library for PDF processing and offers synchronous blob parsing. Examples: Setup: .. code-block:: bash pip install -U langchain-community pypdf Load a blob from a PDF file: .. code-block:: python from langchain_core.documents.base import Blob blob = Blob.from_path("./example_data/layout-parser-paper.pdf") Instantiate the parser: .. code-block:: python from langchain_community.document_loaders.parsers import PyPDFParser parser = PyPDFParser( # password = None, mode = "single", pages_delimiter = " ", # images_parser = TesseractBlobParser(), ) Lazily parse the blob: .. code-block:: python docs = [] docs_lazy = parser.lazy_parse(blob) for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) NFrUr4plain)modepages_delimiter images_parserimages_inner_formatextraction_modeextraction_kwargspasswordOptional[Union[str, bytes]]extract_imagesrwrLiteral['single', 'page']rr,rOptional[BaseImageBlobParser]r+Literal['text', 'markdown-img', 'html-img']rLiteral['plain', 'layout']rOptional[dict[str, Any]]ct|dvrtd||_|r|st }||_||_||_||_||_ ||_ |pi|_ dS)uInitialize a parser based on PyPDF. Args: password: Optional password for opening encrypted PDFs. extract_images: Whether to extract images from the PDF. mode: The extraction mode, either "single" for the entire document or "page" for page-wise extraction. pages_delimiter: A string delimiter to separate pages in single-mode extraction. images_parser: Optional image blob parser. images_inner_format: The format for the parsed output. - "text" = return the content as is - "markdown-img" = wrap the content into an image markdown link, w/ link pointing to (`![body)(#)`] - "html-img" = wrap the content as the `alt` text of an tag and link to (`{body}`) extraction_mode: “plain” for legacy functionality, “layout” extract text in a fixed width format that closely adheres to the rendered layout in the source pdf. extraction_kwargs: Optional additional parameters for the extraction process. Raises: ValueError: If the `mode` is not "single" or "page". singlerUmode must be single or pageN) super__init__rYrrrrrrrrr) selfrrrrrrrr __class__s r5rzPyPDFParser.__init__$sJ  ) ) ):;; ;,  1- 1.00M*#6    ..!2!8br7rFrr+Iterator[Document]c #n K ddl n#t$rtdwxYwd fd }|5} j|j }t d d d d t t|jpiz|j t|j d z}g}t|j D]\}}||} |} t| g| } jdkr2t#| t%|||j|dzV|| jdkr8t#j|t%|VddddS#1swxYwYdS)m Lazily parse the blob. Insert image, if possible, between two paragraphs. In this way, a paragraph can be continued on the next page. Args: blob: The blob to parse. Raises: ImportError: If the `pypdf` package is not found. Yield: An iterator over the parsed documents. rNzE`pypdf` package not found, please install it with `pip install pypdf`rUpypdf.PageObjectr+r,cjdr|S|jddjijS)z Extract text from image given the version of pypdf. Args: page: The page object to extract text from. Returns: str: The extracted text. 3rNr2) __version__rg extract_textrr)rUpypdfrs r5_extract_text_from_pagez7PyPDFParser.lazy_parse.._extract_text_from_pagemsb ++C00 ((***(t($($8,r7rPyPDFr/rCrBrE)rArD)rU)rU page_label page_contentrRr)rUrr+r,)rr: as_bytes_io PdfReaderrrprdictrRrAlenpages enumerateextract_images_from_pagerrkrrr] page_labelsappendrr;) rrFr pdf_file_obj pdf_reader doc_metadata single_texts page_numberrUrtimages_from_pagerrs ` @r5 lazy_parsezPyPDFParser.lazy_parseWs  LLLL   W          $    # <( NNNJ*$"MMtZ06B778#k#&z'7#8#8LL%.z/?%@%@ 2 2! T!8!8d!C!C!C#'#@#@#F#F 1%&%''9&&"%-!3((3.8.D[.Q""       !''1111yH$$!%!5!:!:S>S>U>U3V3V(W(WXXHHNN#8999'"$*,,KOOH--22;u2MMM>+*>*>*@*@KXXXD!%d&8&C&CD&I&I!J!J!WJMM+D*d>VWW!''#((f)=)=>>(   r7NF)rrrrwrrrr,rrrrrrrrrFrr+r)rUrr+r,) __name__ __module__ __qualname____doc___DEFAULT_PAGES_DELIMITERrrr __classcell__rs@r5rrs..d15$19 +177;KQ6=6:1919191919191919fKKKKZ0 0 0 0 0 0 0 0 r7rceZdZdZdZ d(ddeddddd)fdZed*dZed+dZ d,d-d#Z d.d'Z xZ S)/PDFMinerParseraParse a blob from a PDF using `pdfminer.six` library. This class provides methods to parse a blob from a PDF document, supporting various configurations such as handling password-protected PDFs, extracting images, and defining extraction mode. It integrates the 'pdfminer.six' library for PDF processing and offers synchronous blob parsing. Examples: Setup: .. code-block:: bash pip install -U langchain-community pdfminer.six pillow Load a blob from a PDF file: .. code-block:: python from langchain_core.documents.base import Blob blob = Blob.from_path("./example_data/layout-parser-paper.pdf") Instantiate the parser: .. code-block:: python from langchain_community.document_loaders.parsers import PDFMinerParser parser = PDFMinerParser( # password = None, mode = "single", pages_delimiter = " ", # extract_images = True, # images_to_text = convert_images_to_text_with_tesseract(), ) Lazily parse the blob: .. code-block:: python docs = [] docs_lazy = parser.lazy_parse(blob) for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) FNrr4)rrrrrconcatenate_pagesrrwrrxrrrr,rrrrrOptional[bool]cht|dvrtd|r|st}||_||_||_||_||_||_ |?tj s&dt_ t d|rdnd|_dSdS)aHInitialize a parser based on PDFMiner. Args: password: Optional password for opening encrypted PDFs. mode: Extraction mode to use. Either "single" or "page" for page-wise extraction. pages_delimiter: A string delimiter to separate pages in single-mode extraction. extract_images: Whether to extract images from PDF. images_inner_format: The format for the parsed output. - "text" = return the content as is - "markdown-img" = wrap the content into an image markdown link, w/ link pointing to (`![body)(#)`] - "html-img" = wrap the content as the `alt` text of an tag and link to (`{body}`) concatenate_pages: Deprecated. If True, concatenate all PDF pages into one a single document. Otherwise, return one document per page. Returns: This method does not directly return data. Use the `parse` or `lazy_parse` methods to retrieve parsed documents with content and metadata. Raises: ValueError: If the `mode` is not "single" or "page". Warnings: `concatenate_pages` parameter is deprecated. Use `mode='single' or 'page' instead. rrNTzS`concatenate_pages` parameter is deprecated. Use `mode='single' or 'page'` instead.rrU)rrrYrrrrrrrr_warn_concatenate_pagesrr) rrrrrrrrrs r5rzPDFMinerParser.__init__ sP  ) ) ):;; ;  1- 1.00M,*#6    .  (!9 9=6=%6A6DIII ) (r7sUnion[bytes, str]r+c0ddlmt|tr.|drt |ddddS d|D}d fd |DS#t$rt |cYSwxYw) z Decodes a PDFDocEncoding string to Unicode. Adds py3 compatibility to pdfminer's version. Args: s: The string to decode. Returns: str: The decoded Unicode string. r)PDFDocEncodingsNzutf-16beignorec3bK|]*}t|trt|n|V+dSr|)rZr,ord)r3cs r5 z-PDFMinerParser.decode_text..Ys;CCAjC007CFFFaCCCCCCr7r/c3(K|] }|V dSr|r2)r3ors r5r z-PDFMinerParser.decode_text..Zs(;;>!,;;;;;;r7)pdfminer.utilsrrZbytesrgr,r; IndexError)rordsrs @r5 decode_textzPDFMinerParser.decode_textHs 211111 a   4ALL$=$= 4quj(33 3 CCCCCD77;;;;d;;;;; ;   q66MMM s ,A99BBrrc8ddlm}t|dr|}t |t r't t tj|St ||rt |j St |ttfrt |St |tr9|D]"\}}t|||<#|S|S)z Recursively resolve the metadata values. Args: obj: The object to resolve and decode. It can be of any type. Returns: The resolved and decoded object. r) PSLiteralresolve)pdfminer.psparserrhasattrrrZlistmaprresolve_and_decodernamer,rrre)rrrnros r5rz!PDFMinerParser.resolve_and_decode^s 0///// 3 " " ++--C c4 N=sCCDD D Y ' ' !--ch77 7 c5\ * * !--c22 2 T " "   > >1'::1==AJ r7r/Tfpr cachingrSc ddlm}m}m}||}||||}i} |jD]} | | | D]c\} } t| | | <$#t$r3} t d| t| Yd} ~ \d} ~ wwxYwtt||| d<| S)ag Extract metadata from a PDF file. Args: fp: The file pointer to the PDF file. password: The password for the PDF file, if encrypted. Defaults to an empty string. caching: Whether to cache the PDF structure. Defaults to True. Returns: Metadata of the PDF file. r) PDFDocumentPDFPage PDFParser)rrzD[WARNING] Metadata key "%s" could not be parsed due to exception: %sNrD)pdfminer.pdfpagerrr infoupdatererr Exceptionrrr,rr create_pages)rrrrrrr parserdocrRr"rnroes r5 _get_metadatazPDFMinerParser._get_metadatazs>$ EDDDDDDDDD2k&8WEEEH " "D OOD ! ! ! !NN$$  DAq ,??BB    $FF  #&d7+?+?+D+D&E&E"F"FsA== B:)B55B:rFrrc #K ddl}ddlm}ddlm}mmm}m}m m ddl m }m }ddlm} t!|jdkrt%dn#t$$rt%d wxYw|5} t)5| | jpd } |} t/d d d d | jpd z} |j| d<Gfdd|}t5j|| || |}g}t9| D]\}}dd|| }|!}j"dkrPddtG|tI| d|izV|%dr |dd}|&|j"dkr:j'(|}tG|tI| Vdddn #1swxYwYddddS#1swxYwYdS)a Lazily parse the blob. Insert image, if possible, between two paragraphs. In this way, a paragraph can be continued on the next page. Args: blob: The blob to parse. Raises: ImportError: If the `pdfminer.six` or `pillow` package is not found. Yield: An iterator over the parsed documents. rN)PDFLayoutAnalyzer)LAParams LTContainerLTImageLTItemLTPageLTText LTTextBox)PDFPageInterpreterPDFResourceManager)ri:>4zThis parser is tested with pdfminer.six version 20201018 or later. Remove pdfminer, and install pdfminer.six with `pip uninstall pdfminer && pip install pdfminer.six`.zMpdfminer package not found, please install it with `pip install pdfminer.six`r/rPDFMinerrrAc@eZdZ ddfd Zdfd ZxZS)*PDFMinerParser.lazy_parse..Visitorr1Nrsrcmgrr4pagenor\laparamsOptional[LAParams]r+NonecPt|||dS)N)r9r:)rr)rr8r9r:rs r5rz3PDFMinerParser.lazy_parse..Visitor.__init__s* GG$$WVh$OOOOOr7ltpager0c :d fd |dS)Nitemr/r+r<ct|r|D] } |n7t| r'|t| rddSt|r jrddlm}| }||}tjt |z }d|j d<t j |j }t|| jdSdSdS)Nr8r) ImageWriterrJrA)rZwriteget_textrpdfminer.imagerB export_imager from_pathrrRrrrrQr)r@childrB image_writerfilenamerFrr-r.r1r2renderrtempdirtext_ios r5rKzIPDFMinerParser.lazy_parse..Visitor.receive_layout..rendersx%dK88;)-.. &u .'f55;#MM$--//:::%dI66!#MM$/////'g66!#1" F F F F F F/:{7/C/C +7+D+DT+J+J'+~d7mmh6N'O'O:= h 7-1$($6$A$A$$G$G."."".!+!( $7(,j$:R%&%&!"!"!"!"!""""!Dr7)r@r/r+r<r2) mer>rKr-r.r1r2rrLrMs @r5receive_layoutz9PDFMinerParser.lazy_parse..Visitor.receive_layoutsX!!!!!!!!!!!!!8F6NNNNNr7)r1N)r8r4r9r\r:r;r+r<)r>r0r+r<)rrrrrOr)rr-r.r1r2rrLrMs@r5Visitorr7s#$37 PPPPPPP################r7rP)r:rUr rzr))pdfminerpdfminer.converterr+pdfminer.layoutr,r-r.r/r0r1r2pdfminer.pdfinterpr3r4r!rr\rr:rr get_pagesrrpr)rArStringIOrtruncateseek process_pagerrkrrr]endswithrrr;)rrFrRr+r,r/r0r3r4rrrr8rrPvisitor_for_all all_contentirUrdocument_contentr-r.r1r2rLrMs` @@@@@@r5rzPDFMinerParser.lazy_parses  OOO < < < < < <                   R Q Q Q Q Q Q Q 0 0 0 0 0 08'((833!L4    2       O <1C1E1EO %%lT]=Pb%QQE((**G*'JPRSS$$\DM>>OK$U++ 1 14  ### Q,,T222"++--#>>++9&&$$Q'''LLOOO"%-!3LFA;4N!O!O  ((..1#+CRC=&&x0000yH$$#'#7#<#<[#I#I !1/ ==YO O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O sCAAA7KG?J( K(J, ,K/J, 0KKKF)rrwrrxrrrr,rrrrrr)rrr+r,)rrr+r)r/T)rr rr,rrwr+rSr) rrrrrrr staticmethodrrr)rrrs@r5rrs00d$ %:B#'*277;KQ,0:B:B:B:B:B:B:B:Bx\*\< ,,,,,\yyyyyyyyr7rc eZdZdZejZ d)ddedddddd*fdZd+dZ d,d-dZ d.d$Z d/d&Z d0d'Z d1d(ZxZS)2 PyMuPDFParseraParse a blob from a PDF using `PyMuPDF` library. This class provides methods to parse a blob from a PDF document, supporting various configurations such as handling password-protected PDFs, extracting images, and defining extraction mode. It integrates the 'PyMuPDF' library for PDF processing and offers synchronous blob parsing. Examples: Setup: .. code-block:: bash pip install -U langchain-community pymupdf Load a blob from a PDF file: .. code-block:: python from langchain_core.documents.base import Blob blob = Blob.from_path("./example_data/layout-parser-paper.pdf") Instantiate the parser: .. code-block:: python from langchain_community.document_loaders.parsers import PyMuPDFParser parser = PyMuPDFParser( # password = None, mode = "single", pages_delimiter = " ", # images_parser = TesseractBlobParser(), # extract_tables="markdown", # extract_tables_settings=None, # text_kwargs=None, ) Lazily parse the blob: .. code-block:: python docs = [] docs_lazy = parser.lazy_parse(blob) for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) NFrUr4)rrrrrextract_tablesextract_tables_settings text_kwargsrrrwrrxrrrr,rrrrrd/Union[Literal['csv', 'markdown', 'html'], None]rer+r<c>t|dvrtd|r|dvrtd||_||_||_|pi|_|r|st}||_||_ ||_ ||_ | |_ dS)aInitialize a parser based on PyMuPDF. Args: password: Optional password for opening encrypted PDFs. mode: The extraction mode, either "single" for the entire document or "page" for page-wise extraction. pages_delimiter: A string delimiter to separate pages in single-mode extraction. extract_images: Whether to extract images from the PDF. images_parser: Optional image blob parser. images_inner_format: The format for the parsed output. - "text" = return the content as is - "markdown-img" = wrap the content into an image markdown link, w/ link pointing to (`![body)(#)`] - "html-img" = wrap the content as the `alt` text of an tag and link to (`{body}`) extract_tables: Whether to extract tables in a specific format, such as "csv", "markdown", or "html". extract_tables_settings: Optional dictionary of settings for customizing table extraction. Returns: This method does not directly return data. Use the `parse` or `lazy_parse` methods to retrieve parsed documents with content and metadata. Raises: ValueError: If the mode is not "single" or "page". ValueError: If the extract_tables format is not "markdown", "html", or "csv". rr)markdownrOcsvzmode must be markdownN) rrrYrrrrfrrrrrdre) rrfrrrrrrrdrers r5rzPyMuPDFParser.__init__]sV  ) ) ):;; ;  6n4OOO455 5 .  &,"  1- 1.00M,#6 *,'>$$$r7rFrrc,||Sr|) _lazy_parse)rrFs r5rzPyMuPDFParser.lazy_parses    r7c #K ddl}|p|j}|jsNddlm}m}m}m}iddddddddd dd |d dd dd |ddddddd|d|dddddddddddd|_n#t$rtdwxYwtj 5| 5}|j |j |} n|j |d} | jr| |jdddd|| |z} g} | D]v} || | |} |jdkr+t+| t-| d| jizVa| | w|jd kr8t+|j| t-| Vdddn #1swxYwYddddS#1swxYwYdS)!a Lazily parse the blob. Insert image, if possible, between two paragraphs. In this way, a paragraph can be continued on the next page. Args: blob: The blob to parse. text_kwargs: Optional keyword arguments to pass to the `get_text` method. If provided at run time, it will override the default text_kwargs. Raises: ImportError: If the `pypdf` package is not found. Yield: An iterator over the parsed documents. rN)DEFAULT_JOIN_TOLERANCEDEFAULT_MIN_WORDS_HORIZONTALDEFAULT_MIN_WORDS_VERTICALDEFAULT_SNAP_TOLERANCEclipvertical_strategylineshorizontal_strategyvertical_lineshorizontal_linessnap_tolerancesnap_x_tolerancesnap_y_tolerancejoin_tolerancejoin_x_tolerancejoin_y_toleranceedge_min_lengthmin_words_verticalmin_words_horizontalintersection_toleranceintersection_x_toleranceintersection_y_tolerance)text_tolerancetext_x_tolerancetext_y_tolerancestrategy add_lineszGpymupdf package not found, please install it with `pip install pymupdf`pdf)streamfiletypePyMuPDFr/rrUrr)pymupdfrfre pymupdf.tablernrorprqr:rc_lockrdatar is_encrypted authenticater_extract_metadata_get_page_contentrkrrr]numberrrr;)rrFrfrrnrorprqr`r'r full_contentrUrs r5rlzPyMuPDFParser._lazy_parses,)  NNN%9)9K/ 0D0(0*7 0 %d 0 ' 0%&<0'0'0%&<0'0'0&q0)*D0+,H0 -a!0"/#0$/%0&'(()() $!%/000,2   -      !!## y9$&',y11CC&',i%HHHC#4$$T]333 )($&  **355  6 "  6 6D#55c4MMSSUUHyF**&)1%7 , /D D&&%++H55559(("%)%9%>%>|%L%L!3L!A!A5                                 sCA"A''BG0%D'G G0G G0G G00G47G4r'pymupdf.Document pymupdf.PagerSc|jdii|j|}|||}||}g}|r|||r||t ||}|S)a:Get the text of the page using PyMuPDF and RapidOCR and issue a warning if it is empty. Args: doc: The PyMuPDF document object. page: The PyMuPDF page object. blob: The blob being parsed. Returns: str: The text content of the page. r2)rDrf_extract_images_from_page_extract_tables_from_pagerr) rr'rUrfrtrtables_from_pagerrrs r5rzPyMuPDFParser._get_page_contents"'MM)LD,<)L )LMM99#tDD99$??  , MM* + + +  , MM* + + +)&.AAr7rc tiddd|j|jtdfdjD}dD]}|jvrj|||<|S)zExtract metadata from the document and page. Args: doc: The PyMuPDF document object. blob: The blob being parsed. Returns: dict: The extracted metadata. rr/)rCrBrErAr`rDczi|]7}tj|ttf)|j|8Sr2)rZrRr,r\r3rnr's r5 z3PyMuPDFParser._extract_metadata..1sK!#,q/C:>>s|Ar7)modDate creationDate)rprArrR)rr'rFrRrns ` r5rzPyMuPDFParser._extract_metadatas#  )($&"k!%#&s88   \   "- . .ACL  !l1o r7c|jsdSddl}|}g}|D] }|jr|d}|j||}t j|jtj|j |j d} tj } tj| | tj| d} t%|j| j} |t-| | |jt0t4t9d|S) a Extract images from a PDF page and get the text using images_to_text. Args: doc: The PyMuPDF document object. page: The PyMuPDF page object. Returns: str: The extracted text from the images on the page. r/rNrrzapplication/x-npyrr)rr get_imagesPixmaprrsamplesrrrrrrnumpyrrrrrrrrrQrrrHrr;r) rr'rUrimg_listr)r=xrefpiximagerrFrs r5rz'PyMuPDFParser._extract_images_from_page=sg! 2??$$  C! 1v$gnS$// ckBBBJJJ 2!jll  ;...~((**6I"$"4"?"?"E"EFFS  'j$:RSS!''#((f)=)=>>(   r7c|jdSddl}t|jj|fi|j}|r|jdkr$t d|DS|jdkr$t d|DS|jdkr$t d |DStd |jd dS) zExtract tables from a PDF page. Args: page: The PyMuPDF page object. Returns: str: The extracted tables in the specified format. Nr/rric6g|]}|Sr2) to_markdownr3tables r5r6z;PyMuPDFParser._extract_tables_from_page..vs$)W)W)W%%*;*;*=*=)W)W)Wr7rOcbg|],}|ddd-S)F)headerindex bold_rows) to_pandasto_htmlrs r5r6z;PyMuPDFParser._extract_tables_from_page..ysR " ))11#("'&+2r7rjc`g|]+}|dd,S)F)rr)rto_csvrs r5r6z;PyMuPDFParser._extract_tables_from_page..sO " ))00#("'1r7zextract_tables z not implemented) rdrrr find_tablesre _JOIN_TABLESr;rY)rrUr tables_lists r5rz'PyMuPDFParser._extract_tables_from_pageds8   &2 %GM %d K Kd.J K K    "j00#(()W)W;)W)W)WXXX$..#(( &1    $--#(( &1 !Kd&9KKKrr7r)rfrrrwrrxrrrr,rrrrrdrgrerr+r<rr|)rFrrfrr+r)r'rrUrrfrSr+r,)r'rrFrr+r)r'rrUrr+r,)rUrr+r,)rrrr threadingLockrrrrrlrrrrrrs@r5rcrc$s22l IN  E15$;? #'*077;KQJN<@;?;?;?;?;?;?;?;?z    15 _____B:@% % % % N,,,,,,,,r7rccdeZdZdZejZ dddeddddfdZddZ ddZ xZ S) PyPDFium2ParseraoParse a blob from a PDF using `PyPDFium2` library. This class provides methods to parse a blob from a PDF document, supporting various configurations such as handling password-protected PDFs, extracting images, and defining extraction mode. It integrates the 'PyPDFium2' library for PDF processing and offers synchronous blob parsing. Examples: Setup: .. code-block:: bash pip install -U langchain-community pypdfium2 Load a blob from a PDF file: .. code-block:: python from langchain_core.documents.base import Blob blob = Blob.from_path("./example_data/layout-parser-paper.pdf") Instantiate the parser: .. code-block:: python from langchain_community.document_loaders.parsers import PyPDFium2Parser parser = PyPDFium2Parser( # password=None, mode="page", pages_delimiter=" ", # extract_images = True, # images_to_text = convert_images_to_text_with_tesseract(), ) Lazily parse the blob: .. code-block:: python docs = [] docs_lazy = parser.lazy_parse(blob) for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) FNrUr4)rrrrrrrwrrxrrrr,rrrrr+r<ct|dvrtd||_|r|st }||_||_||_||_||_ dS)ukInitialize a parser based on PyPDFium2. Args: password: Optional password for opening encrypted PDFs. mode: The extraction mode, either "single" for the entire document or "page" for page-wise extraction. pages_delimiter: A string delimiter to separate pages in single-mode extraction. extract_images: Whether to extract images from the PDF. images_parser: Optional image blob parser. images_inner_format: The format for the parsed output. - "text" = return the content as is - "markdown-img" = wrap the content into an image markdown link, w/ link pointing to (`![body)(#)`] - "html-img" = wrap the content as the `alt` text of an tag and link to (`{body}`) extraction_mode: “plain” for legacy functionality, “layout” for experimental layout mode functionality extraction_kwargs: Optional additional parameters for the extraction process. Returns: This method does not directly return data. Use the `parse` or `lazy_parse` methods to retrieve parsed documents with content and metadata. Raises: ValueError: If the mode is not "single" or "page". rrN) rrrYrrrrrrr)rrrrrrrrs r5rzPyPDFium2Parser.__init__sL  ) ) ):;; ;,  1- 1.00M*#6    .r7rFrrc #K ddl}n#t$rtdwxYwtj5|5}d} |j||jd}g}ddddt|z}|j |d <t||d <t|D]\}}| } d | } | ||} t%| g| } ||jd krA| d s| d z } t-| t/i|d |i V|| |jdkr8t-|j |t/| V|r|n#|r|wwxYwdddn #1swxYwYddddS#1swxYwYdS)rrNzKpypdfium2 package not found, please install it with `pip install pypdfium2`T)r autoclose PyPDFium2r/rrArDr8rUrr) pypdfium2r:rrr PdfDocumentrrpget_metadata_dictrArr get_textpager;get_text_range splitlinescloserrrkrr[rr]rr) rrFrr`rrrrrU text_pagertimage_from_pagers r5rzPyPDFium2Parser.lazy_parses         +   "4 +4 +!!##3 +y! 1+!6!6!DMT"""J$&L%0#.(*$$( (D(D(F(FGG $HL .2[L*25j//L/-6z-B-B::) T$($5$5$7$7 )-%4466AACC**")))*.*H*H*N*N#9,-~$$%''! 9..#+#4#4T#:#:1 (D 0"*-5);%&*6%&(. %&%&*"*"###)//9999yH,,&)-)=)B)B<)P)P%7 %E%E "+"((***"+"((****+e3 +3 +3 +3 +3 +3 +3 +3 +3 +3 +3 +3 +3 +3 +3 +4 +4 +4 +4 +4 +4 +4 +4 +4 +4 +4 +4 +4 +4 +4 +4 +4 +4 +sV #I(I GH(I(II I(I I(I I((I,/I,pypdfium2._helpers.page.PdfPagec|jsdSddlm}t||jf}|sdSg}|D]}t j}| }|j dkrHtj || tj|d}t!|j|j} |t)|| |j|t.t2|S) rr/rN)rrrrr)r pypdfium2.rawrawr get_objectsFPDF_PAGEOBJ_IMAGErr get_bitmapto_numpysizerrrrrrrrrrQrrrrHrr;) rrUpdfium_cr) str_imagesrrrrFtext_from_images r5rz)PyPDFium2Parser._extract_images_from_pageKso! 2((((((d&&x/J.L&MMNN 2   E*,,K''))2244H}q   J{E$4$4$6$6$?$?$A$A B B B>+"6"6"8"8DWXXXD"4#5#@#@#F#FGGTO   #D/4;STT    KKMMMM ''<3D3DZ3P3P'QQQr7r`)rrwrrxrrrr,rrrrr+r<r)rUrr+r,) rrrrrrrrrrrrrs@r5rrs00h IN  E %0/#'*077;KQ0/0/0/0/0/0/0/0/dM+M+M+M+^RRRRRRRRr7rc:eZdZdZ ddd ZddZddZddZdS)PDFPlumberParserzParse `PDF` with `PDFPlumber`.NFrfOptional[Mapping[str, Any]]deduperwrr+r<cz ddl}n#t$rtdwxYw|pi|_||_||_dS)zInitialize the parser. Args: text_kwargs: Keyword arguments to pass to ``pdfplumber.Page.extract_text()`` dedupe: Avoiding the error of duplicate characters if `dedupe=True`. rNzEpillow package not found, please install it with `pip install pillow`)rr:rfrr)rrfrrrs r5rzPDFPlumberParser.__init__osf  JJJJ   W  '," ,s!rFrrc#Kddl}5}|j|fdjDEd{VddddS#1swxYwYdS)Lazily parse the blob.rNc g|]}t|dz|ztjj|jdz t jdfifdjDS)r8r1)rAr`rUrDc|i|]8}tj|ttfv*|j|9Sr2)rfrRr,r\rs r5rz:PDFPlumberParser.lazy_parse...sH !#CLO44c BBs|ABBBr7r) r_process_page_contentrrrArrrrR)r3rUrFr'rs r5r6z/PDFPlumberParser.lazy_parse..s&%!%!;!;D!A!A"44T::";"&*k)-$($4q$8+.sy>>   %(\   r7) pdfplumberrrr)rrFrr`r's`` @r5rzPDFPlumberParser.lazy_parses      9!*/),,C& I'                         s,AAArUpdfplumber.page.Pager,c||jr$|jdi|jS|jdi|jS)z)Process the page content based on dedupe.r2)r dedupe_charsrrf)rrUs r5rz&PDFPlumberParser._process_page_contentsQ ; H34$$&&3GGd6FGG G t 444#3444r7c fddlm}|jsdSg}|jD]}|ddjt vr|dddkr|tj| d|dd |dd f|d  d |tj |d tj |dd |dd d ,|ddjtvr/||d ut!jdt%|S)z8Extract images from page and get the text with RapidOCR.rrr/rFilterBitsPerComponentr11WidthHeightLrrzr)rrrr)rrrrr frombytesrconvertrrrrwarningswarnr@)rrUrr)r=s r5rz*PDFPlumberParser._extract_images_from_pages" 2; 5 5C8}X&+/GGGx=!3499MM!OO #!$Xw!7Xx9P Q #H 6 6 8 8&gcll MM c(m&<&<&>&>bhOOOWWM(3S]75KR Xx(-1FFF c(m44667777 344440888r7)NFF)rfrrrwrrwr+r<r)rUrr+r,)rrrrrrrrr2r7r5rrlsz((48$ -----,:5555 999999r7rc.eZdZdZ ddddd ZddZdS)AmazonTextractPDFParseraSend `PDF` files to `Amazon Textract` and parse them. For parsing multi-page PDFs, they have to reside on S3. The AmazonTextractPDFLoader calls the [Amazon Textract Service](https://aws.amazon.com/textract/) to convert PDFs into a Document structure. Single and multi-page documents are supported with up to 3000 pages and 512 MB of size. For the call to be successful an AWS account is required, similar to the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) requirements. Besides the AWS configuration, it is very similar to the other PDF loaders, while also supporting JPEG, PNG and TIFF and non-native PDF formats. ```python from langchain_community.document_loaders import AmazonTextractPDFLoader loader=AmazonTextractPDFLoader("example_data/alejandro_rosalez_sample-small.jpeg") documents = loader.load() ``` One feature is the linearization of the output. When using the features LAYOUT, FORMS or TABLES together with Textract ```python from langchain_community.document_loaders import AmazonTextractPDFLoader # you can mix and match each of the features loader=AmazonTextractPDFLoader( "example_data/alejandro_rosalez_sample-small.jpeg", textract_features=["TABLES", "LAYOUT"]) documents = loader.load() ``` it will generate output that formats the text in reading order and try to output the information in a tabular structure or output the key/value pairs with a colon (key: value). This helps most LLMs to achieve better accuracy when processing these texts. ``Document`` objects are returned with metadata that includes the ``source`` and a 1-based index of the page number in ``page``. Note that ``page`` represents the index of the result returned from Textract, not necessarily the as-written page number in the document. N)linearization_configtextract_featuresOptional[Sequence[int]]client Optional[Any]r!Optional[TextLinearizationConfig]r+r<c ddlddlmcm}|_||_|fd|D|_ng|_|||_n#|jdddd|_n#t$rtd wxYw|s> ddl }| d |_ dS#t$rtd wxYw||_ dS) a5Initializes the parser. Args: textract_features: Features to be used for extraction, each feature should be passed as an int that conforms to the enum `Textract_Features`, see `amazon-textract-caller` pkg client: boto3 textract client linearization_config: Config to be used for linearization of the output should be an instance of TextLinearizationConfig from the `textractor` pkg rNc:g|]}|Sr2)Textract_Features)r3ftcs r5r6z4AmazonTextractPDFParser.__init__..s4***01B((++***r7Tz# z## *)hide_figure_layout title_prefixsection_header_prefixlist_element_prefixzCould not import amazon-textract-caller or amazon-textract-textractor python package. Please install it with `pip install amazon-textract-caller` & `pip install amazon-textract-textractor`.textractzRCould not import boto3 python package. Please install it with `pip install boto3`.) textractcallertextractor.entities.documententitiesdocumentr textractorrrrr:boto3rboto3_textract_client)rrrrr r rs @r5rz AmazonTextractPDFParser.__init__sb&  ' ' ' ' = = = = = = = = =DG(DO ,****5F***&&*,&#/,@)),0O,S,S'+!%*/(+ -T--)    <   0  -2\\*-E-E***   !B  *0D & & &sA%A))B B))CrFrrc#lK|jr!tt|jnd}|rL|jdkrA|jr:|jt|j|j|j}nI|j| |j|jj j |j}|j j |}t|jD]<\}}t||j|j|dzdV=dS) zIterates over the Blob pages and returns an Iterator with a Document for each page, like the other parsers If multi-page document, blob.path has to be set to the S3 URI and for single page docs the blob.data is taken Ns3)input_documentfeaturesr )rr call_moder )configr1rArUr)pathrr,schemenetlocr call_textractrr as_bytesTextract_Call_Mode FORCE_SYNCr rrrrrDrrA)rrFurl_parse_resulttextract_response_jsonr idxrUs r5rz"AmazonTextractPDFParser.lazy_parse<sT822  IC!]]$2K]LL$(KqAA      r7)NN)rrrrrrr+r<r)rrrrrrr2r7r5rrsf00h6: $=0 CG =0=0=0=0=0=0~!!!!!!r7rc*eZdZdZddZdd Zdd ZdS)DocumentIntelligenceParserzjLoads a PDF with Azure Document Intelligence (formerly Form Recognizer) and chunks at character level.rrmodelr,cJtjd||_||_dS)Na<langchain_community.document_loaders.parsers.pdf.DocumentIntelligenceParserand langchain_community.document_loaders.pdf.DocumentIntelligenceLoader are deprecated. Please upgrade to langchain_community.document_loaders.DocumentIntelligenceLoader for any file parsing purpose using Azure Document Intelligence service.)rrrr!)rrr!s r5rz#DocumentIntelligenceParser.__init__ds/       r7rFrr>r+rc#K|jD]H}dd|jD}t||j|jd}|VIdS)N cg|] }|j Sr2)rG)r3lines r5r6z=DocumentIntelligenceParser._generate_docs..rsAAA AAAr7rr)rr;rtrrAr)rrFr>prGds r5_generate_docsz)DocumentIntelligenceParser._generate_docsps{  AhhAAAAABBG$"kMAGGGG  r7c#K|5}|j|j|}|}|||}|Ed{VddddS#1swxYwYdS)rN)rrbegin_analyze_documentr!r>r))rrFfile_objpollerr>docss r5rz%DocumentIntelligenceParser.lazy_parse}s     8[77 HMMF]]__F&&tV44DOOOOOOO                   sAA77A;>A;N)rrr!r,)rFrr>rr+rr)rrrrrr)rr2r7r5r r `s\AA              r7r )r)r*r+r,)rFrrGr,rHr,r+r,)rRrSr+rS)rrrsrtr,r+r,)Cr __future__rrOrloggingrrrpathlibrtempfilertypingrrr r r r r rrrr urllib.parserrrlangchain_core.documentsr)langchain_community.document_loaders.baser1langchain_community.document_loaders.blob_loadersr3langchain_community.document_loaders.parsers.imagesrrrrrr)textractor.data.text_linearization_configrrrr@ getLoggerrrrrrrrVrQr]rprrrrrcrrrr r2r7r5r;s .."""""" ''''''                          "!!!!! ------DDDDDDBBBBBB RNNNLLLQQQQQQ999">  8 $ $*  !UUU,&####N   2222ja a a a a .a a a HJJJJJ^JJJZ lllllNlll^ VRVRVRVRVRnVRVRVRr[9[9[9[9[9~[9[9[9|SSSSSnSSSl&&&&&&&&&&r7