_jhdZddlZddlZddlmZddlmZmZddlm Z ddl m Z ddl m Z mZddlZddlmZd d lmZmZmZmZmZmZmZmZd d lmZmZd d lmZm Z m!Z!m"Z"m#Z#m$Z$d dl%m&Z&e$dZ'e dGddZ(ej)dZ*hdZ+hdZ,hdZ-ej)dZ.dddddZ/iZ0e1e-D]Z2e2e0e2<e2de0e2d<e2de0e2d <e d!e0e3e4"Z5e dGd#d$Z6e7eedzfZ8e9ee:e3fZ;d%eee:efd&e6fd'Zd+e3d&e3fd,Z?d)ed-e@d&e7eAe8e;ffd.ZBd/e3d0e;d&eeedzeCge@ffd1ZDd2eCdzd3e3d4eCd&e@fd5ZEd6e3d&e7ee8ge7ed7ffe@ffd8ZFd)ed9eAe3d&e(fd:ZGe'Hd;gd<= dgd?e e3dzejId@fdAe e@ejIdBfdCe eAe3dzejIdDdEdFfdGe e5dzejIdHfdIe eJdzejIdJfd&df dKZKe'HgdL= dhd9e eAe3ejLdMfd?e e3dzejId@fdNe e@ejIdOdPdQfdRe e@ejIdSfd&df dTZMe'HdUdVg= dhd?e e3dzejId@fdNe e@ejIdOdPdQfdRe e@ejIdSfd&dfdWZNe'HgdX=e jOdddd>d>dfdYedZe d[e!d?e e3dzejId\fd]e e3dzejId^fd_e e@ejId`dafdbe e@ejIdcddfdee"d&dfdfZPdS)izHContains the 'hf cache' command group with cache management subcommands.N) defaultdict)CallableMapping) dataclass)Enum) AnnotatedAny)CLIError)ANSICachedRepoInfoCachedRevisionInfo CacheNotFound HFCacheInfo _format_size parse_hf_uriscan_cache_dir)parse_duration parse_size) RepoIdArg RepoTypeOpt RevisionOptTokenOpt get_hf_api typer_factory)outzManage local cache directory.)helpT)frozencfeZdZUeeed<eeeefed<e edfed<dS)_DeletionResolution revisionsselected.missingN) __name__ __module__ __qualname__ frozensetstr__annotations__dictr rtupleV/var/www/FlaskApp/flask-venv/lib/python3.11/site-packages/huggingface_hub/cli/cache.pyr!r!1sR~>9-?#@@AAAA 38_r.r!zA^(?P[a-zA-Z_]+)\s*(?P==|!=|>=|<=|>|<|=)\s*(?P.+)$><=>!=<=>=>refssizetypeaccessedmodified>namer7r9r:z.^(?P[a-zA-Z_]+)(?::(?Pasc|desc))?$descasc)r9r:r7r;z:asc_ascz:desc_desc SortOptions)r8modulec2eZdZUdZeed<eed<eed<dS)CacheDeletionCountsz>Simple counters summarizing cache deletions for CLI messaging. repo_countpartial_revision_counttotal_revision_countN)r%r&r'__doc__intr*r-r.r/rCrCPs9HHOOOr.rCselected_by_reporeturnc d}d}d}|D]S\}}|t|z }t|t|jkr|dz }|t|z }T||z }t|||S)z(Summarize deletions across repositories.rr)itemslenr"rC)rIrDtotal_revisionsrevisions_in_full_reposrepor"rEs r/summarize_deletionsrQ]sJO+113366i3y>>) y>>S00 0 0 !OJ #s9~~ 5 #,/FF z+A? S SSr.c t|dD]}|jd|j}t||d}t |t |jkrt jd|dit jd|d|D]R}dt|j pd }t jd |j d |d |j Sd S)zBPretty-print selected cache revisions during confirmation prompts.cB|j|jfSN repo_typerepo_idlowerrPs r/z0print_cache_selected_revisions..qs$.RVR^RdRdRfRfAgr.key/c|jSrT commit_hashrevs r/rZz0print_cache_selected_revisions..s3?r. - z (entire repo): z (detached)z z [z] N) sortedkeysrVrWrMr"rtextjoinr6r`size_on_disk_str)rIrPrepo_keyr"revisionr6s r/print_cache_selected_revisionsrnos/',,..4g4ghhh [ [n55t|55+D17R7RSSS y>>S00 0 0 H4H444 5 5 5  ####$$$! [ [H88F8=1122BlD HYh2YYdYYh>WYY Z Z Z Z [ [ [r. hf_cache_infoci}i}|jD]H}|j}|||<|jD] }||f||j<!I||fS)zPCreate lookup tables so CLI commands can resolve repo ids and revisions quickly.)reposcache_idrXr"r`)ro repo_lookuprevision_lookuprPrlrms r/build_cache_indexru~s.0KLNO#MM=&&(( $ H M MH=A8.sQUQ]QcQcQeQe@fr.r[c&h|]}|jD]}|Sr-)r6).0rmrefs r/ z(collect_cache_entries..s*(e(e(eW_Wd(e(ePS(e(e(e(er.c|jSrTr_ras r/rZz'collect_cache_entries..rcr.NcJ|dj|d |djndfS)Nrr)rrr`entrys r/rZz'collect_cache_entries..s+a!(-a(<a$$"r.c|djS)Nr)rrrs r/rZz'collect_cache_entries..s uQx'8r.)rgrqr(r"appendsort)ror~entries repo_refs_map sorted_reposrPrms r/collect_cache_entriesrs!#G!#M--3f3fgggL))'(e(eT^(e(e(eff d  )"4>7R7RSSS 1 1h/0000 1 NND$< ( ( ( (:        88 999 M !!r.exprrc 0 t|}|std|d|d|d|d t vr+tdd|dtt d tvr+td d |dttd d krt   fd SdvrJt  dtdtdzdtdtf fd }|SdkrT dkrtdddtdtdzdtdtffd }|Sdkrtdd dtdtdzdtdtf fd }|S)zqConvert a `hf cache ls` filter expression into the yes/no test we apply to each cache entry before displaying it.zInvalid filter expression: 'z'.r\opvaluezUnsupported operator 'z ' in filter ''. Must be one of .zUnsupported filter key '' in 'r7cBt||jn|jSrT)_compare_numeric size_on_disk)rPrm_rsize_thresholds r/rZz&compile_cache_filter..s+)9%-%9H ! !t?P  * * r.>r9r:rPrmNnowrJcndkr|jn||jn|j}|dSt||z S)Nr9F) last_accessed last_modifiedr)rPrmr timestampr\rsecondss r/ _time_filterz*compile_cache_filter.._time_filters\*$$""'++'  u#C)ORAA Ar.r8r1z/Only '=' is supported for 'type' filters. Got 'rc>|jkSrT)rVrX)rPrmrexpecteds r/ _type_filterz*compile_cache_filter.._type_filters>''))X5 5r.z.Only '=' is supported for 'refs' filters. Got c||jn!|t}d|DvS)Nc6g|]}|Sr-)rX)rrs r/ z>compile_cache_filter.._refs_filter..s (E(E(E(E(E(Er.)r6getr(rX)rPrmrr6r value_raws r/ _refs_filterz*compile_cache_filter.._refs_filtersO$,$88==m>O>OPTV_VaVa>b>bD??$$(E(E(E(E(EE Er.)_FILTER_PATTERNmatchstrip ValueErrorgrouprX_ALLOWED_OPERATORSlist _FILTER_KEYSrrr rfloatbool) rrrrrrrr\rrrrs ` @@@@@@r/compile_cache_filterrs  ! !$**,, / /E B@@@@AAA ++e   " " $ $C T  B G$$**,,I ###v"vv4vv[_`r[s[svvvwww ,lClltllW[\hWiWilllmmm f}}#I..       &&& !2!233 B~ B9Kd9R BY^ Bcg B B B B B B B B f}}??$$ 99UrUUUVV V 6~ 69Kd9R 6W\ 6ae 6 6 6 6 6 6 99SbSSSTT T F~ F9Kd9R FW\ Fae F F F F F F Fr.leftrrightc|dS||k||k||k||k||k||kd}||vrtd|||S)z)Evaluate numeric comparisons for filters.NF)r1r3r2r0r5r4z)Unsupported numeric comparison operator: )r)rrr comparisonss r/rrsr |uU]em E\ E\emem K IRIIJJJ r?r. sort_expr.c t|}|st d|d|d|d}t vr+t dd|dtt d|r|n t}|d k}d td ttd fffd }||fS)zConvert a `hf cache ls` sort expression into a key function for sorting entries. Returns: A tuple of (key_function, reverse_flag) where reverse_flag indicates whether to sort in descending order (True) or ascending order (False). zInvalid sort expression: 'z5'. Expected format: 'key' or 'key:asc' or 'key:desc'.r\orderzUnsupported sort key 'rrrr<rrJ.c2|\}}dkr|j}|fSdkr||jn|j}|fSdkr|j|jnd}|fSdkr&||j|jnd}n|j|jnd}|fSt d)Nr;r7r9gr:zUnsupported sort key: )rrrXrrrr)rrPrmrr\s r/ _sort_keyz%compile_cache_sort.._sort_key sh &==,,..E8O &==-5-AH))tGXE8O *  +/*<*HD&&cE8O *  #2:2H2T..Z].2.@.L**RU8O7#77888r.) _SORT_PATTERNrrrXrr _SORT_KEYSr_SORT_DEFAULT_ORDER CacheEntryr,r )rrexplicit_orderrreverserr\s @r/compile_cache_sortr s     1 1 7 7 9 9 : :E xvivvvwww ++e   " " $ $C[[))N *m#mmYmmZ^_iZjZjmmmnnn- JNN2Ec2JEvoG99c3h999999< g r.targetscvt|\}}tt}t}g}|D]6}|}|s|} t jd| rh|| } | ||p| \} } ||  | | | j |t|} | ||| j D]7} ||  | | | j 88d| D}tt||t!|S)z8Resolve the deletion targets into a deletion resolution.z[0-9a-fA-F]{40}Nc4i|]\}}|t|Sr-)r()rrPrevss r/ z-_resolve_deletion_targets..bs$PPPttYt__PPPr.r"r#r$)rursetrrXre fullmatchrraddr`r}r"rLr!r(r,)rorrsrtr#r"r$ raw_targetrvloweredrrPrm matched_repofrozen_selecteds r/_resolve_deletion_targetsrAs#4]#C#C K>I#>N>NH%%IG00 !!##  ,,.. <*G 4 4 #''00E}z***"ND( TN  x ( ( ( MM(. / / / "'A&'I'I'O'O'Q'QRR   NN: & & & $. 0 0H \ " & &x 0 0 0 MM(. / / / / 0QPx~~?O?OPPPO I&& g   r.z list | ls)z hf cache lszhf cache ls --revisionsz*hf cache ls --filter "size>1GB" --limit 20zhf cache ls --format json)examplesF cache_dirz9Cache directory to scan (defaults to Hugging Face cache).r"zCInclude revisions in the output instead of aggregated repositories.filterz-fz--filterzZFilter entries (e.g. 'size>1GB', 'type=model', 'accessed>7d'). Can be used multiple times.raSort entries by key. Supported keys: 'accessed', 'modified', 'name', 'size'. Append ':asc' or ':desc' to explicitly set the order (e.g., 'modified:asc'). Defaults: 'accessed', 'modified', 'size' default to 'desc' (newest/biggest first); 'name' defaults to 'asc' (alphabetical).limitzSLimit the number of results returned. Returns only the top N entries after sorting.c  t|}n*#t$r}td|j|d}~wwxYw|pg}t ||\} fd|D} n4#t $r'}t jt||d}~wwxYwtj | D]fd|D}|rd t|j \} } | | | n4#t $r'}t jt||d}~wwxYw|(|dkrt jd|d |d|}|r)d |D} tj| gd d d din*fd|D} tj| gddd di|rd|D} t| }|r3t!d|D}t!d|D}n2t!d| D}t!d| D}tjt%jd|d|dt)|ddSdS)z&List cached repositories or revisions.Cache directory not found: N)r~c0g|]}t|Sr-)r)rrrs r/rzls..s$TTTD*4??TTTr.cFg|]}|d|d|S)rrr-)rrfnrs r/rzls..s4MMMUE!HeAh1L1LM5MMMr.)r\rrz&Limit must be a positive integer, got rc g|]T\}}||j|j|j|jt |j|j|jt|j dUS)N)r{rWrVrm snapshot_pathr7rr6) rrrWrVr`r)rrklast_modified_strrgr6)rrPrms r/rzls..ss    h#m<!^$0!$X%;!.s{    am<!^-!%!7!=2!%!7}00y{{CCDD      r.)r{r7rrr6r{ch|]\}}|Sr-r-)rrPrs r/rzls..s444q444r.c3$K|] \}}|dV dS)Nrr-rrrbs r/ zls..s& L Lvq#COOOOO L Lr.c3.K|]\}}||jVdSrTrrs r/rzls..s)WW&!SsS-WWr.c3>K|]}t|jVdSrT)rMr"rrPs r/rzls..s, N NT^!4!4 N N N N N Nr.c3$K|] }|jV dSrTrrs r/rzls..s%HH4T.HHHHHHr.z Found z repo(s) for a total of z revision(s) and z on disk.)rrr rrrtyper BadParameterr)timerrrrtablerMsumrir boldr)rr"rrrroexcfiltersr filter_fns sort_key_fnrrL unique_reposrDrevision_count total_sizerrrs @@@r/lsrms^O&y11 OOODS]DDEE3NOlG2=T]^^^G]4TTTTGTTT 444 S**34 )++CNNMMMMMgMMM 8 8#5dj#A#A K LL['L : : : : 8 8 8$SXX..C 7 8  199$%Ve%V%V%VWW W&5&/'    #*     GGG(           #     LLL(      44G444 &&  I L L L L LLLNWWGWWWWWJJ N N N N NNNNHH<HHHHHJ  I<:<<~<<$Z00<<<         s@ <7<A'' B1"BB.C44 D%>"D  D%)zhf cache rm model/gpt2z-hf cache rm hf://models/openai-community/gpt2zhf cache rm z hf cache rm model/gpt2 --dry-runzhf cache rm model/gpt2 --yesziOne or more repo IDs (e.g. model/bert-base-uncased), repo-level hf:// URIs, or revision hashes to delete.yesz-yz--yeszSkip confirmation prompt.dry_runz,Preview deletions without removing anything.c  t|}n*#t$r}td|j|d}~wwxYwt ||}|jr;dd|jD}tjd|t|j dkr)tj dtj d|jt|j }t!|j} g} | jr| | jd | jr| | jd | s| | jd d | } tj d | d |jdt/|j|r*tjdd| j| j|jdStjd||t!|j} tjd| jd| jd|jd| j| j|jdS)z(Remove cached repositories or revisions.rN c3 K|] }d|V dSrdNr-)rrs r/rzrm..#s(KKuN5NNKKKKKKr.zCould not find in cache: rzNothing to delete.codez repo(s)z revision(s)z and About to delete z totalling rDry run: no files were deleted.T)rrqr"r7zProceed with deletion?rDeleted z repo(s) and z revision(s); freed ) repos_deletedrevisions_deletedfreed)rrr rrr$rjrwarningrMr"rirExitdelete_revisionsrgrQr#rDrrErFexpected_freed_size_strrnresultconfirmexecute) rrrrror resolutiondetailsstrategycounts summary_parts summary_texts r/rmrsLO&y11 OOODS]DDEE3NO+='BBJ<))KK 8JKKKKK :::;;; :   A%% %&&&ja    -}-vj6J/K/KLH !4 5 5F!M = 1;;;<<< $M =KKKLLL K ;IIIJJJ<< ..LH \ \ \9Y \ \ \]]]":#6777 -#11      K(c2222  !4 5 5FJ 66$ 6 663N 6 62 6 6 6' 5.  949zhf cache prunezhf cache prune --dry-runc t|}n*#t$r}td|j|d}~wwxYwi}t }|jD]G}t d|jD}|s#|||<|d|DHt|dkrtj ddStt ||d} |j t| j} t|} tj d | jd | jd t%||r$tjd d | j| jdStjd|| tjd| jd| jd| j| jdS)z)Remove detached revisions from the cache.rNc3JK|]}t|jdk|VdS)rN)rMr6rrms r/rzprune..ps7``(HMHZHZ^_H_H_XH_H_H_H_``r.c3$K|] }|jV dSrTr_rs r/rzprune..ts%GG(-GGGGGGr.rz2No unreferenced revisions found. Nothing to prune.r-rrz unreferenced revision(s) (z total).rT)rr"r7zProceed?r r z! unreferenced revision(s); freed r)r r )rrr rrrqr(r"updaterMrrir!rrgrQrFrrnrrr) rrrrorr#r"rPdetachedrrrs r/pruner#Ps60O&y11 OOODS]DDEE3NOEGH%%I#HH``dn`````  !GGhGGGGGGG 9~~ EFFF$I&&J .}-vj6J/K/KLH  * *FH}66}}S[Ss}}}#8,,, -11     K $$$$ Jt6.ttQYQqttt 5.r)zhf cache verify gpt2z)hf cache verify gpt2 --revision refs/pr/1z.hf cache verify my-dataset --repo-type datasetrWrVrmzXCache directory to use when verifying files from cache (defaults to Hugging Face cache). local_dirz?If set, verify files under this directory instead of the cache.fail_on_missing_filesz--fail-on-missing-filesz?Fail if some files exist on the remote but are missing locally.fail_on_extra_filesz--fail-on-extra-fileszLFail if some files exist locally but are not present on the remote revision.tokenc j|+|)tjdtjdt |}||t |dr|jnt|||||} d} | j r=d d | j D} tj d | d } | j ri|r>d d | j D} tj d| d } n)tj t| j d| jri|r>d d| jD} tj d| d } n)tj t| jd| j} | dkr?tjd|d|jd| d| jtj| tjd| jd|jd|d||j| jt| dS)aVerify checksums for a single repo revision from cache or a local directory. Examples: - Verify main revision in cache: `hf cache verify gpt2` - Verify specific revision: `hf cache verify gpt2 --revision refs/pr/1` - Verify dataset: `hf cache verify karpathy/fineweb-edu-100b-shuffle --repo-type dataset` - Verify local dir: `hf cache verify deepseek-ai/DeepSeek-OCR --local-dir /path/to/repo` NzCCannot pass both --local-dir and --cache-dir. Use one or the other.r r)r'r)rWrVrmr$rr'rrc 3bK|]*}d|dd|dd|dd|dV+dS) rdpathz : expected rz ( algorithmz), got actualNr-)rms r/rzverify..sj   _1V9 ^ ^: ^ ^!K. ^ ^QRS[Q\ ^ ^      r.u<❌ Checksum verification failed for the following file(s): rc3 K|] }d|V dSrr-rps r/rzverify..s(IIq q IIIIIIr.u6❌ Missing files (present remotely, absent locally): zM remote file(s) are missing locally. Use --fail-on-missing-files for details.c3 K|] }d|V dSrr-r/s r/rzverify..s(GGq q GGGGGGr.u4❌ Extra files (present locally, absent remotely): zV local file(s) do not exist on the remote repo. Use --fail-on-extra-files for details.zVerification failed for 'z' (z) in z. Revision: z Verified z file(s) for z 'z'. All checksums match.)rWrVcheckedr*)rerrorrrrverify_repo_checksumshasattrrr) mismatchesrjri missing_pathsrrM extra_paths verified_pathrmr checked_count) rWrVrmrr$r%r&r'apir exit_coderverified_locations r/verifyr>s`!6 WXXXja    5 ! ! !C  & &%,Y%@%@T)//c)nn 'FI ))  &      ZQXZZ[[[     iiIIF4HIIIIIG HXwXX Y Y YII Kv+,,;;;       iiGGF4FGGGGGG HVWVV W W WII Kv)**999    ,A~~ } } }IO } }J[ } }lrl{ } }   ji((((JjF(jjyjj'jjj/$ " # # r.)NFNNN)NFF)QrGrr collectionsrcollections.abcrr dataclassesrenumrtypingrr rhuggingface_hub.errorsr utilsr r rrrrrrutils._parsingrr _cli_utilsrrrrrr_outputr cache_clir!compilerrrrrr_sort_options_dictrgr\r)r%r@rCr,rr+r( RepoRefsMaprQrnrur}rrrrrrrrcommandOptionrHrArgumentrr#modelr>r-r.r/rQsQ ON ######--------!!!!!!!!!!!!!! ++++++                    87777777```````````````` M> ? ? ?  $ "*abb666??? 5 5 5  LMM     6*  66C!s*-|||#|||$+. #}}}%%d="43xPPP  $>#5#<< = >9S>12 Tni8J.KKLTTTTT$ [W^YWiMj=j5k [pt [ [ [ [(( n eN$667 78:((((" "s "s " " " """6:" 4 [ ()""""4> >)> ~1D8%@$FG>>>>B54<S4(3#3%*uSRUX9V0WY]0]*^3333l&[&49&Qd&&&&X    G| |  d  L    |   V    |   S D  m     | * d 7     +| <  d  f     =| H I| | | | ~ *   5OO  S |    O d  L    O    ,     O*  ?    +O6 7OOOOd -/IJKK   )FF d  L    F    ,     F  ?    F* +FFFLKFR ).   =ff fff d  k     f d  R    f %  %R    !f.#  #_    /f< =f> ?ffffffr.