hhRDddlmZddlZddlZddlZddlZddlmZmZm Z m Z m Z m Z m Z mZddlZddlZddlmZddlmZddlmZejeZddZGddeZeeegefZeeee ejej fgefZ!e ee!fZ"ddZ#dS)) annotationsN)CallableIteratorListOptionalSequenceSetUnioncast)Document)extract_sub_links) BaseLoaderraw_htmlstrurlresponse0Union[requests.Response, aiohttp.ClientResponse]returndictct|ddd}||d} ddlm}n,#t$rt d|cYSwxYw||d}|d x}r||d <|d d d i x}r|dd|d <|dx} r| dd|d<|S)z3Extract metadata from raw html using BeautifulSoup.headersz Content-Type)source content_typer) BeautifulSoupztThe bs4 package is required for default metadata extraction. Please install it with `pip install -U beautifulsoup4`.z html.parsertitlemetaname description)attrscontentNhtmllanglanguage) getattrgetbs4r ImportErrorloggerwarningfindget_text) rrrrmetadatarsouprrr"s v/var/www/FlaskApp/flask-venv/lib/python3.11/site-packages/langchain_community/document_loaders/recursive_url_loader.py_metadata_extractorr0s=8Y//33NBGGL|< < {'source': 'https://docs.python.org/3.9/', 'content_type': 'text/html', 'title': '3.9.19 Documentation', 'language': None} Async load: .. code-block:: python docs = await loader.aload() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python < {'source': 'https://docs.python.org/3.9/', 'content_type': 'text/html', 'title': '3.9.19 Documentation', 'language': None} Content parsing / extraction: By default the loader sets the raw HTML from each link as the Document page content. To parse this HTML into a more human/LLM-friendly format you can pass in a custom ``extractor`` method: .. code-block:: python # This example uses `beautifulsoup4` and `lxml` import re from bs4 import BeautifulSoup def bs4_extractor(html: str) -> str: soup = BeautifulSoup(html, "lxml") return re.sub(r"\n\n+", "\n\n", soup.text).strip() loader = RecursiveUrlLoader( "https://docs.python.org/3.9/", extractor=bs4_extractor, ) print(loader.load()[0].page_content[:200]) .. code-block:: python 3.9.19 Documentation Download Download these documents Docs by version Python 3.13 (in development) Python 3.12 (stable) Python 3.11 (security-fixes) Python 3.10 (security-fixes) Python 3.9 (securit Metadata extraction: Similarly to content extraction, you can specify a metadata extraction function to customize how Document metadata is extracted from the HTTP response. .. code-block:: python import aiohttp import requests from typing import Union def simple_metadata_extractor( raw_html: str, url: str, response: Union[requests.Response, aiohttp.ClientResponse] ) -> dict: content_type = getattr(response, "headers").get("Content-Type", "") return {"source": url, "content_type": content_type} loader = RecursiveUrlLoader( "https://docs.python.org/3.9/", metadata_extractor=simple_metadata_extractor, ) loader.load()[0].metadata .. code-block:: python {'source': 'https://docs.python.org/3.9/', 'content_type': 'text/html'} Filtering URLs: You may not always want to pull every URL from a website. There are four parameters that allow us to control what URLs we pull recursively. First, we can set the ``prevent_outside`` parameter to prevent URLs outside of the ``base_url`` from being pulled. Note that the ``base_url`` does not need to be the same as the URL we pass in, as shown below. We can also use ``link_regex`` and ``exclude_dirs`` to be more specific with the URLs that we select. In this example, we only pull websites from the python docs, which contain the string "index" somewhere and are not located in the FAQ section of the website. .. code-block:: python loader = RecursiveUrlLoader( "https://docs.python.org/3.9/", prevent_outside=True, base_url="https://docs.python.org", link_regex=r']*?\s+)?href="([^"]*(?=index)[^"]*)"', exclude_dirs=['https://docs.python.org/3.9/faq'] ) docs = loader.load() .. code-block:: python ['https://docs.python.org/3.9/', 'https://docs.python.org/3.9/py-modindex.html', 'https://docs.python.org/3.9/genindex.html', 'https://docs.python.org/3.9/tutorial/index.html', 'https://docs.python.org/3.9/using/index.html', 'https://docs.python.org/3.9/extending/index.html', 'https://docs.python.org/3.9/installing/index.html', 'https://docs.python.org/3.9/library/index.html', 'https://docs.python.org/3.9/c-api/index.html', 'https://docs.python.org/3.9/howto/index.html', 'https://docs.python.org/3.9/distributing/index.html', 'https://docs.python.org/3.9/reference/index.html', 'https://docs.python.org/3.9/whatsnew/index.html'] N TF)base_urlautoset_encodingencodingproxiesrr max_depth Optional[int] use_asyncOptional[bool] extractorOptional[Callable[[str], str]]metadata_extractor Optional[_MetadataExtractorType] exclude_dirsOptional[Sequence[str]]timeoutprevent_outsidebool link_regexUnion[str, re.Pattern, None]rOptional[dict]check_response_statuscontinue_on_failurer6 Optional[str]r7r8r9rNonec |_||nd|_||nd|_||nd|_||nt}||_||_t||_||nd|_ tfd|j Drtdd|j ||_ ||nd |_ | |_| |_| |_| |_| | n|_||_dS) a Initialize with URL to crawl and any subdirectories to exclude. Args: url: The URL to crawl. max_depth: The max depth of the recursive loading. use_async: Whether to use asynchronous loading. If True, lazy_load function will not be lazy, but it will still work in the expected way, just not lazy. extractor: A function to extract document contents from raw HTML. When extract function returns an empty string, the document is ignored. Default returns the raw HTML. metadata_extractor: A function to extract metadata from args: raw HTML, the source url, and the requests.Response/aiohttp.ClientResponse object (args in that order). Default extractor will attempt to use BeautifulSoup4 to extract the title, description and language of the page. ..code-block:: python import requests import aiohttp def simple_metadata_extractor( raw_html: str, url: str, response: Union[requests.Response, aiohttp.ClientResponse] ) -> dict: content_type = getattr(response, "headers").get("Content-Type", "") return {"source": url, "content_type": content_type} exclude_dirs: A list of subdirectories to exclude. timeout: The timeout for the requests, in the unit of seconds. If None then connection will not timeout. prevent_outside: If True, prevent loading from urls which are not children of the root url. link_regex: Regex for extracting sub-links from the raw html of a web page. headers: Default request headers to use for all requests. check_response_status: If True, check HTTP response status and skip URLs with error responses (400-599). continue_on_failure: If True, continue if getting or parsing a link raises an exception. Otherwise, raise the exception. base_url: The base url to check for outside links against. autoset_encoding: Whether to automatically set the encoding of the response. If True, the encoding of the response will be set to the apparent encoding, unless the `encoding` argument has already been explicitly set. encoding: The encoding of the response. If manually set, the encoding will be set to given value, regardless of the `autoset_encoding` argument. proxies: A dictionary mapping protocol names to the proxy URLs to be used for requests. This allows the crawler to route its requests through specified proxy servers. If None, no proxies will be used and requests will go directly to the target URL. Example usage: ..code-block:: python proxies = { "http": "http://10.10.1.10:3128", "https": "https://10.10.1.10:1080", } Nr3Fc|SNr4)xs r/z-RecursiveUrlLoader.__init__..Ms1r4c3BK|]}|VdSrP) startswith).0 exclude_dirrs r/ z.RecursiveUrlLoader.__init__..Xs/PP{s~~k**PPPPPPrSz9Base url is included in exclude_dirs. Received base_url: z and exclude_dirs: T)rr:r<r>r0r7r8_wrap_metadata_extractorr@rBany ValueErrorrDrErGrrJrKr6r9)selfrr:r<r>r@rBrDrErGrrJrKr6r7r8r9s ` r/__init__zRecursiveUrlLoader.__init__s9X&/&;&/&;&/&;"-  $  !1  ":;M"N"N,8,DLL" PPPPd>OPPP P P 5C55!%!255   2A2MSW$ %:"#6 $,$8c  rSrdepthvisitedSet[str]r_intIterator[Document]c #>K||jkrdS|| tj||j|j|j}|j |j|_n|jr |j |_|j r,d|j cxkrdkrnntd|j nN#t$rA}|jr3td|d|d|jjYd}~dS|d}~wwxYw||j}|r-t+|||j|| Vt/|j||j|j|j|j|j }|D]'}||vr!||||d z Ed{V(dS) zRecursively get all child links starting with the path of the input URL. Args: url: The URL to crawl. visited: A set of visited URLs. depth: Current depth of recursion. Stop when depth >= max_depth. N)rDrr9WReceived HTTP status zUnable to load from . Received error of type  page_contentr-r6patternrEexclude_prefixesrKr^)r:addrequestsr&rDrr9r8r7apparent_encodingrJ status_coder[ ExceptionrKr)r* __class____name__r>textr r@r r6rGrErB_get_child_links_recursive) r\rr`r_rer! sub_linkslinks r/rxz-RecursiveUrlLoader._get_child_links_recursivegsR DN " " F  C |T\4<H}($(M!!& ?$,$>!) QcX5I.P.P.P.PS.P.P.P.P.P !O9M!O!OPPP   ' .3....{+.. ..//  $00XNN    & M ]O 0!. $ 8     D7""::';  sBB(( C324C.,C..C3sessionr_r}Optional[aiohttp.ClientSession]List[Document]c K|jstd||jkrgS|du}||nFtjtjdtj|j|j}| | | |4d{V}| d{V}|j r,d|j cxkrdkrnntd |j  dddd{Vn#1d{VswxYwYn|#tjjt f$r^}|r|d{V|jr4t&d |d |d |jjgcYd}~S|d}~wwxYwg||} | r9t3| |||| ||jdz krt7|||j|j|j|j|j} g} tA| !|} | D]1} | |"| |||dz2tGj$| d{V}|D]+}tK|t s|fd|Dz ,|r|d{VS)zRecursively get all child links starting with the path of the input URL. Args: url: The URL to crawl. visited: A set of visited URLs. depth: To reach the current url, how many pages have been visited. z?Async functions forbidden when not initialized with `use_async`NF)ssl)total) connectorrDrrerfrgzUnable to load rhrirjrorlr|cg|]}|v| Sr4r4)rVrresultss r/ zGRecursiveUrlLoader._async_get_child_links_recursive..s#FFF!Qg5E5EA5E5E5ErS)&r<r[r:aiohttp ClientSession TCPConnector ClientTimeoutrDrrpr&rwrJstatusclient_exceptions InvalidURLrtcloserKr)r*rurvr>appendr r@r r6rGrErBset difference _async_get_child_links_recursiveasynciogather isinstance)r\rr`r}r_ close_sessionrrwryr!rz sub_tasksto_visitr{ next_results sub_resultrs @r/rz3RecursiveUrlLoader._async_get_child_links_recursivesm~ Q  DN " "I 4 " G&!.5999-DLAAA    C {{3'' P P P P P P P8%]]__,,,,,,-P#2O2O2O2OC2O2O2O2O2O$%NX_%N%NOOOO P P P P P P P P P P P P P P P P P P P P P P P P P P P)4i@    &mmoo%%%%%%%' .c..A..{+..  ..&&   NN!(!44T3II    4>A% % %) $ 4!%!2$($<II9~~0099H     99gweai: ")!;;;;;;;L* G G j)44 8JFFFFzFFFF  "--// ! ! ! ! ! ! !sJ D'AD6D DDDDF5AFF FFct}|jr>tj||j|}t |pgS||j|S)zLazy load web pages. When use_async is True, this function will not be lazy, but it will still work in the expected way, just not lazy.)rr<rrunrriterrx)r\r`rs r/ lazy_loadzRecursiveUrlLoader.lazy_loadsj EE > Fk55dhHHG 2&& &2248WEE ErS) r3NNNr4r5TNNFT)"rrr:r;r<r=r>r?r@rArBrCrDr;rErFrGrHrrIrJrFrKrFr6rLr7rFr8rLr9rIrrM)rrr`rar_rbrrc) rrr`rar}r~r_rbrr)rrc)rv __module__ __qualname____doc__r]rxrrr4rSr/r2r25sFFV$%$(48?C02!# $37"&&+$(g#'!%"&"&%ggggggT<=;;;;;;D48 [[[[[[z F F F F F FrSr2r@_MetadataExtractorType_MetadataExtractorType2cttjjdkrt t Sd fd }|S) NrrrrrrrcBtt||SrP)r _MetadataExtractorType1)rrrr@s r/_metadata_extractor_wrapperz=_wrap_metadata_extractor.._metadata_extractor_wrappers$ E4/1CDDXsSS SrSrrrrrrrr)leninspect signature parametersr r)r@rs` r/rYrYse 7 / 0 0 ;<<AA+-?@@@ T T T T T T+*rSr)r@rrr)$ __future__rrrloggingretypingrrrrrr r r rrqlangchain_core.documentsr langchain_core.utils.htmlr )langchain_community.document_loaders.baser getLoggerrvr)r0r2rrrResponseClientResponserrrYr4rSr/rs""""""                     ------777777@@@@@@  8 $ $2WFWFWFWFWFWFWFWFt#C:t#34"#uX&(>>?@$F68OOP++++++rS