_j0e$ dZddlZddlZddlmZddlZddlmZddlm Z m Z m Z m Z m Z ddlmZmZmZddlmZd d lmZmZmZmZd d lmZmZd d lmZmZeje Z!ed Z"e"#dgd ddee$ej%dfdee&ej'ddfdeedzej'ddfdee&ej'ddfdeddf d Z(d!e$de&fd"Z)e"#d#gd$ dd!ee$dzej%d%fd&ee&ej'd'd(d)fd*ee&ej'd+d,fd-ee&ej'd.d/d0fd1ededdfd2Z*d3e$dzd1e$dzd&e&d*e&d-e&de$dzddfd4Z+d!e$d&e&d*e&d-e&de$dzddf d5Z,e"#d6d7d8g ddee$ej%d9fdeddfd:Z-e"#d;gd< ddee$ej%d9fd=ee&ej'd>d?d@fdAee&ej'dBdCfdeddf dDZ.e"#dEgdF dd!ee$ej%dGfd-ee&ej'd.d/dHfd=ee&ej'd>d?d@fdIee&ej'dJdKfdLee/e$dzej'dMfdNee/e$dzej'dOfdeddfdPZ0e"#dQgdR ddSee$ej%dTfdUee$ej%dVfdeddfdWZ1e"#dXgdY ddZee$dzej%d[fd\ee$dzej%d]fd;ee&ej'd^fd_ee&ej'd`dafdbee&ej'dcddfdeee$dzej'dffdgee$dzej'dhfdIee&ej'dJdifdLee/e$dzej'djfdNee/e$dzej'dkfdlee$dzej'dmfdnee&ej'dodpfdqee&ej'drdsfdtee&ej'dudvdwfdeddf dxZ2e"#dygdz dd{ee$ej%d|fd}ee$dzej%d~fdeddfdZ3dS)z7Contains commands to interact with buckets via the CLI.N) Annotated)logging) BUCKET_PREFIX BucketFile FilterMatcher_is_bucket_path_parse_bucket_uri)SoftTemporaryDirectorydisable_progress_bars is_hf_uri) REPO_REGIONS) SearchOptTokenOpt get_hf_api typer_factory) format_sizeprint_file_listing)OutputFormatWithAutooutz"Commands to interact with buckets.)helpcreate)zhf buckets create my-bucketz hf buckets create user/my-bucketz-hf buckets create hf://buckets/user/my-bucketz*hf buckets create user/my-bucket --privatez+hf buckets create user/my-bucket --exist-okz,hf buckets create user/my-bucket --region us)nameexamplesF bucket_idzTBucket ID: bucket_name, namespace/bucket_name, or hf://buckets/namespace/bucket_nameprivatez --privatezCreate a private bucket.regionz--regionzdCloud region in which to create the bucket. Can be one of 'us' or 'eu'. Requires Team plan or above.exist_okz --exist-okz3Do not raise an error if the bucket already exists.tokenreturnctt|}|tr=t|}|jr t jd|dtd|j}|||r|nd||}tj d|j |j dS) zCreate a new bucket.r z-Cannot specify a prefix for bucket creation: . Use namespace/bucket_name or namespace/bucket_name.N)rrrzBucket created)uriurl)r startswithrr path_in_repotyper BadParameterid create_bucketrresultr&to_urir')rrrrr apiparsed bucket_urls X/var/www/FlaskApp/flask-venv/lib/python3.11/site-packages/huggingface_hub/cli/buckets.pyrr1sT 5 ! ! !CM**"9--   $W WW1>WWW I """, #J JZ^%:%:%<%<*.QQQQQQargumentc||tr|ttd}n|}d|vS)zFCheck if argument is a bucket ID (namespace/name) vs just a namespace.N/)r(rlen)r5paths r3 _is_bucket_idr:osA=))M**,,- $;r4z list | ls) zhf buckets listzhf buckets list huggingfacez$hf buckets list --search "my-prefix"zhf buckets list user/my-bucketz!hf buckets list user/my-bucket -Rz!hf buckets list user/my-bucket -hz%hf buckets list user/my-bucket --treez(hf buckets list user/my-bucket --tree -hz+hf buckets list hf://buckets/user/my-bucketz%hf buckets list user/my-bucket/sub -RzyNamespace (user or org) to list buckets, or bucket ID (namespace/bucket_name(/prefix) or hf://buckets/...) to list files.human_readablez--human-readablez-hz$Show sizes in human readable format.as_treez--treez3List files in tree format (only for listing files). recursivez --recursivez-Rz0List files recursively (only for listing files).searchc|duot|}|r,|tjdt|||||dSt ||||||dS)zList buckets or files in a bucket. When called with no argument or a namespace, lists buckets. When called with a bucket ID (namespace/bucket_name), lists files in the bucket. Nz'Cannot use --search when listing files.)r5r;r<r=r ) namespacer>r;r<r=r )r:r*r+ _list_files _list_buckets)r5r;r<r=r>r is_file_modes r3list_cmdrDxsr4'CM(,C,CL  $%NOO O)        )       r4r@c|rtjd|rtjd|K|tr1|t td}|d}t |}fd|||D}tj |ddd  dS) zList buckets in a namespace.z'Cannot use --tree when listing buckets.z,Cannot use --recursive when listing buckets.Nr7r#cg|]:}|j|jrt|jn|j|j|jd;S))r,rsize total_files created_at)r,rrrGrHrI).0bucketr;s r3 z!_list_buckets..se     )~@N_K ^<<right)rGrH) alignments) r*r+r(rr8rstripr list_bucketsrtable)r@r>r;r<r=r r0itemss ` r3rBrBsL !JKKKQ !OPPP!5!5m!D!Dc-00223 $$S)) 5 ! ! !C    &&6&JJ   EIe I IJJJJJJr4c,|r.tjtjkrt jdt |}t|}t| |j |j pd|}t||||dS)zList files in a bucket.z%Cannot use --tree with --format json.r#Nprefixr=)r;r<r=) rmoderjsonr*r+rr listlist_bucket_treer,r)r)r5r;r<r=r r0r1rRs r3rArAsJ383888 !HIII 5 ! ! !C x ( (F   I&.$     Eu^WXabbbbbbr4infozhf buckets info user/my-bucketz+hf buckets info hf://buckets/user/my-bucketzFBucket ID: namespace/bucket_name or hf://buckets/namespace/bucket_namect|}t|}||j}t j|ddS)zGet info about a bucket.r#r,)id_keyN)rr bucket_infor,rdict)rr r0r1rKs r3rZrZsO" 5 ! ! !C y ) )F __VY ' 'FHVD!!!!!!r4delete)z hf buckets delete user/my-bucketz-hf buckets delete hf://buckets/user/my-bucketz&hf buckets delete user/my-bucket --yesz-hf buckets delete user/my-bucket --missing-okyesz--yesz-yzSkip confirmation prompt. missing_okz --missing-okz3Do not raise an error if the bucket does not exist.c|tr>t|}|jr t jd|dtd|j}n$d|vr t jd|dtdtjd|d| t| }| || tj d | dS)zDelete a bucket. This deletes the entire bucket and all its contents. Use `hf buckets rm` to remove individual files. z-Cannot specify a prefix for bucket deletion: r$r%r7zInvalid bucket ID: z-. Must be in format namespace/bucket_name or z(Are you sure you want to delete bucket ''?r`r#)razBucket deleted)rN) r(rr r)r*r+r,rconfirmr delete_bucketr.)rr`rar r1r0s r3r_r_s-JM**  "9--   $W WW1>WWW I I    a) a a;H a a a   KH9HHHcRRRR 5 ! ! !CiJ777J9555555r4z remove | rm)z)hf buckets remove user/my-bucket/file.txtz2hf buckets rm hf://buckets/user/my-bucket/file.txtz.hf buckets rm user/my-bucket/logs/ --recursivez:hf buckets rm user/my-bucket --recursive --include "*.tmp"z8hf buckets rm user/my-bucket/data/ --recursive --dry-runzBucket path: namespace/bucket_name/path or hf://buckets/namespace/bucket_name/path. With --recursive, namespace/bucket_name is also accepted to target all files.z0Remove files recursively under the given prefix.dry_runz --dry-runz8Preview what would be deleted without actually deleting.includezQInclude only files matching pattern (can specify multiple). Requires --recursive.excludezLExclude files matching pattern (can specify multiple). Requires --recursive.cLt|}|j|j}|dkr|stjddd|s|r|stjdt |} |rt jd} g} | |pdd D]R} t| tr;| | | d t| d S| d t| d |s|r t|| fd| D} n| } d| D}t!d| D}t#|d }|st jddSt|d|}|sP|sNt jdd|Dt jd|ddd|rMt jdfd|Dt jd|ddS| |t jd|dd t||!dS|}|stjd"|r7t jd#t.d$|t jd%dSt jd&|d'd|| |gt jd(|)dS)*zbRemove files from a bucket. To delete an entire bucket, use `hf buckets delete` instead. z?No file path specified. To remove files, provide a path (e.g. 'zd/FILE') or use --recursive to remove all files. To delete the entire bucket, use `hf buckets delete z`.z,--include and --exclude require --recursive.r#zListing files from remoteNTrTzListing files from remote (z files))include_patternsexclude_patternscHg|]}|j|S)matchesr9)rJfmatchers r3rLzremove..s,MMM1W__QV5L5LMQMMMr4cg|] }|j Sro)r9rJrqs r3rLzremove..s444af444r4c3$K|] }|jV dSN)rGrts r3 zremove..s$77A777777r4)r;zNo files to remove.z file(s) totaling  c3 K|] }d|V dS)z Nro)rJr9s r3rwzremove..s(BBt{D{{BBBBBBr4zRemove z from 'rcFrdc36K|]}dtd|VdS)delete: r7N)r)rJr9rs r3rwzremove..s:ccPTK-KKKTKKccccccr4z (dry run) z would be removed.)r_zRemoved ')r files_deletedrGzFile path cannot be empty.r{r7z"(dry run) 1 file would be removed.zRemove 'z' from 'z File removed)r9r)r r,r)r*r+rrstatusrY isinstancerappendupdater8donersumrtextjoinrebatch_bucket_filesr.r)r5r=r`rgrhrir r1rUr0r~ all_filesitem matched_files file_paths total_sizesize_str count_label file_pathrrrs @@r3removerTszx ( (F I  F ||I|  R  R RDM R R R   Q7QIQ !OPPP 5 ! ! !C<H788&( (( >T)   U UD $ ++ U  &&& SC NNSSSTTT I#i..IIIJJJ  &g &#WwWWWGMMMM MMMMM%M44m444 7777777 z$???  H* + + + FZFFHFF  P7 P HTYYBBzBBBBB C C C KC+CCiCCC O O O O   HTYYccccXbccccc d d d HA+AAA B B B F y<<< 7{ 7 79 7 7 7j//         C$%ABB B   HF FyFF9FF G G G H9 : : : F ?y??)???SIIII y)=== > YGGGGGGr4move)z/hf buckets move user/old-bucket user/new-bucketz/hf buckets move user/my-bucket my-org/my-bucketzIhf buckets move hf://buckets/user/old-bucket hf://buckets/user/new-bucketfrom_idzMSource bucket ID: namespace/bucket_name or hf://buckets/namespace/bucket_nameto_idzRDestination bucket ID: namespace/bucket_name or hf://buckets/namespace/bucket_namect|}|jr tjd|dtdt|}|jr tjd|dtdt |}||j|jtj d|j|jdS)z2Move (rename) a bucket to a new name or namespace.z)Cannot specify a prefix for bucket move: r$r%r#)rrz Bucket movedN) r r)r*r+rr move_bucketr,rr.)rrr parsed_from parsed_tor0s r3rrs2$G,,K   S S S-: S S S   "%((I   S S S-: S S S   5 ! ! !COOKN),O???J~{~Y\JJJJJJr4sync)z2hf buckets sync ./data hf://buckets/user/my-bucketz2hf buckets sync hf://buckets/user/my-bucket ./dataz;hf buckets sync ./data hf://buckets/user/my-bucket --deletez^hf buckets sync hf://buckets/user/my-bucket ./data --include "*.safetensors" --exclude "*.tmp"zIhf buckets sync ./data hf://buckets/user/my-bucket --plan sync-plan.jsonlz'hf buckets sync --apply sync-plan.jsonlzhf buckets cp my-config.json hf://buckets/user/my-bucket/logs/zKhf buckets cp my-config.json hf://buckets/user/my-bucket/remote-config.jsonz7hf buckets cp - hf://buckets/user/my-bucket/config.jsonzchf buckets cp hf://buckets/user/my-bucket/logs hf://buckets/user/archive-bucket/ # nests logs/ dirzihf buckets cp hf://buckets/user/my-bucket/logs/ hf://buckets/user/archive-bucket/ # copies contents onlyzehf buckets cp hf://datasets/user/my-dataset/processed/ hf://buckets/user/my-bucket/dataset/processed/srczJSource: local file, any hf:// URI (model, dataset, bucket), or - for stdindstz=Destination: local path, bucket hf://... URI, or - for stdoutc t|}t|}|duot|}t|}|duot|}|dk}|dk} |rf|rd |||n3#t$r&} t jt| d} ~ wwxYwtj d||dS|s.|s,|s*|t jdt jd|r|st jd|rE|rCt|} | j d ks| d rt jd | r|st jd |s5|s3tj|rt jd |rRt|} | j| j } } | d ks| d rt jd| d dd}| rt%5t'5}tj||}|| | |fgt-|d5}|dx}r;t0jj||dx};dddn #1swxYwYdddn #1swxYwYdddn #1swxYwYdSdS||}nvtj|s4| tjs| d r!tj||}n|}tj|}|rtj|d|| | |fgtj d||dS|rot|} t0jj}| | j|| j fgtj dd|dStj!|st jd|t|} | j| j } } | d kr tj"|}n=| d r&| d ztj"|z}n| }| | ||fgtj d|tF| d |dS)zCopy files to or from buckets.r#N-Copied)rrz2Missing destination. Provide a bucket path as DST.z;One of SRC or DST must be a bucket path (hf://buckets/...).z+Stdin upload requires a bucket destination.rkr7zAStdin upload requires a full destination path including filename.z"Cannot pipe to stdout for uploads.zNSource must be a file, not a directory. Use `hf buckets sync` for directories.zJSource path must include a file name, not just a bucket or directory path.rrbiHT)r Downloaded)addUploadedstdinzSource file not found: )$rr r copy_files ValueErrorr*r+strrr.r r)endswithosr9isdirr,rsplitr r rdownload_bucket_filesopenreadsysstdoutbufferwritesepdirnamemakedirsrrisfilebasenamer)rrr r0 src_is_hf dst_is_hf src_is_bucket dst_is_bucket src_is_stdin dst_is_stdouter1rrUfilenametmp_dirtmp_pathrqchunk local_path parent_dirdata remote_paths r3rrs_4 5 ! ! !C#I42IcNNI#C((MtO<(<('(( ; ;+--;!w||GX>>H--i68:L9MNNNh--;'(vvj'9'99e;J-33E:::()vvj'9'99e;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ;{% s## !s||BF';'; !s||C?P?P !W\\#x88  44J 7 J6666  % %i6:2F1G H H H J|* = = = = = = Y"3''y$$&& vyf6I/J.KLLL :7444444w~~c"" F$%Ds%D%DEE E"3''"Iv':6 R<<'**3//KK \\#   ! 3,)9)9#)>)>>KK K y[/A.BCCC :3},Wi,W,W+,W,WXXXXXXst&A== B-!B((B-#L 2A K2rs >=  ######[ZZZZZZZZZ!!!!!! ;:::::::........  H % %m!EFFF     0   90R0R g    0R  +    0R t w     0R*  F    +0R8 90R: ;0R0R0R  0RfCD     0    E> >  d W    >    7    > $  F    %> 2   C    3> B C> D E> F G> > > > B KTz K $J K K K  K : K  K K K KFcccc c : c  cccc0  (5 " " Y     "  "  " " " "   .  -.6.6 Y    .6    ,     .6  F    .6, -.6. /.6.6.6.6b    6    [HH a    H   C    H$    ,     %H4  K    5HB S D d    CHN S D _    OHZ [H\ ]HHH  HD  *"K"K  `    "K  e     "K "K "K"K"K"KT       &         yrPrP  d ^     rP  d c     rP   B     rP&  N    'rP4  O    5rPB  d  P     CrPN  d  6     OrPZ  H    [rPh S D I    irPt S D I    urP@ d  ;    ArPL  T    MrPZ  V    [rPh   8    irPx yrPz {rPrPrP  rPt     , nYnY  ^U^!m n n nn nY  d NEN(ghhhh  nY nY nYnYnY! nYnYnYr4