hh ^ddlZddlmZmZmZmZmZddlmZddl m Z Gdde Z dS)N)IteratorMappingOptionalSequenceUnion)Document) BaseLoaderc<eZdZdZ ddededeedeed eeeeeeeeeeefffd eed ee d e d eee efdee fdZ de e fdZdeeefdefdZdS)HuggingFaceDatasetLoaderz&Load from `Hugging Face Hub` datasets.textNFpathpage_content_columnnamedata_dir data_files cache_dirkeep_in_memory save_infosuse_auth_tokennum_procc ||_||_||_||_||_||_||_||_| |_| |_ dS)aInitialize the HuggingFaceDatasetLoader. Args: path: Path or name of the dataset. page_content_column: Page content column name. Default is "text". name: Name of the dataset configuration. data_dir: Data directory of the dataset configuration. data_files: Path(s) to source data file(s). cache_dir: Directory to read/write data. keep_in_memory: Whether to copy the dataset in-memory. save_infos: Save the dataset information (checksums/size/splits/...). Default is False. use_auth_token: Bearer token for remote files on the Dataset Hub. num_proc: Number of processes. N) r rrrrrrrrr) selfr rrrrrrrrrs v/var/www/FlaskApp/flask-venv/lib/python3.11/site-packages/langchain_community/document_loaders/hugging_face_dataset.py__init__z!HuggingFaceDatasetLoader.__init__ sQ> #6    $",$,  returnc #(K ddlm}n#t$rtdwxYw|jjjjjjj j j  fd DEd{VdS)zLoad documents lazily.r) load_datasetzXCould not import datasets python package. Please install it with `pip install datasets`.) r rrrrrrrrc 3K|]K}|D]@}t|j|VALdS)) page_contentmetadataN)r parse_objpopr).0keyrowdatasetrs r z5HuggingFaceDatasetLoader.lazy_load..Ns  s|    !^^CGGD4L,M,MNN          rN) datasetsr ImportErrorr rrrrrrrrkeys)rrr's` @r lazy_loadz"HuggingFaceDatasetLoader.lazy_load6s  - - - - - - -   A   ,]n..]         ||~~             s 'r cXt|trtj|S|S)N) isinstanceobjectjsondumps)rr s rr"z"HuggingFaceDatasetLoader.parse_objWs* lF + + ,:l++ +r) r NNNNNFNN)__name__ __module__ __qualname____doc__strrrrrboolintrrrr,r/r"rrr r sT00 $*""& #')- 59"&(!(!(!!(!sm (! 3- (!  #x}gc5hsm9K3L.L&MM N (!C=(!!(!(!!tSy!12(!3-(!(!(!(!T (     BeCK&8Srr ) r0typingrrrrrlangchain_core.documentsr)langchain_community.document_loaders.baser r r9rrr=s ??????????????------@@@@@@QQQQQzQQQQQr