_jUddlmZddlZddlmZmZmZmZgdZde d<e eZ de d<Gdd eZ Gd d Z dd ZGddZGddZgdZdS)) annotationsN)IterableListProtocolcast)aaboutaboveafteragainagainstainallamanandanyarearenzaren'tasatbebecausebeenbeforebeingbelowbetweenbothbutbycancouldnzcouldn'tddiddidnzdidn'tdodoesdoesnzdoesn'tdoingdonzdon'tdownduringeachfewforfromfurtherhadhadnzhadn'thashasnzhasn'thavehavenzhaven'thavingheherherehersherselfhimhimselfhishowiifinintoisisnzisn'titzit'sitsitselfjustllmmamemightnzmightn'tmoremostmustnzmustn'tmymyselfneednzneedn'tnonornotnowoofoffononceonlyorotherourours ourselvesoutoverownressameshanzshan'tshezshe'sshouldz should'veshouldnz shouldn'tsosomesuchtthanthatzthat'llthetheirtheirsthem themselvesthentherethesetheythisthosethroughtotoounderuntilupveverywaswasnzwasn'twewerewerenzweren'twhatwhenwherewhichwhilewhowhomwhywillwithwonzwon'twouldnzwouldn'tyyouzyou'dzyou'llzyou'rezyou'veyouryoursyourself yourselves List[str]DEFAULT_ENGLISH_STOPWORDSDEFAULT_CHROMA_BM25_STOPWORDSceZdZddZdS)SnowballStemmertokenstrreturncdSNselfrs v/var/www/FlaskApp/flask-venv/lib/python3.11/site-packages/chromadb/utils/embedding_functions/schemas/bm25_tokenizer.pystemzSnowballStemmer.stems Nrrrr)__name__ __module__ __qualname__rrrrrrs(      rrc"eZdZdZd dZd dZdS) _SnowballStemmerAdapterzCAdapter that provides the uniform `stem` API used across languages.rNonec ddl}n#t$rtdwxYw|d|_dS)NrziThe snowballstemmer python package is not installed. Please install it with `pip install snowballstemmer`english)snowballstemmer ImportError ValueErrorstemmer_stemmer)rrs r__init__z _SnowballStemmerAdapter.__init__sa  " " " " "   {   (// :: !rrc\tt|j|Sr)rrrstemWordrs rrz_SnowballStemmerAdapter.stems"C//66777rN)rrr)rrr__doc__rrrrrrrsBMM;;;;888888rrrctS)z&Return a Snowball stemmer for English.)rrrrget_english_stemmerrs " $ $$rcBeZdZdZdd Zdd ZeddZddZdS) Bm25TokenizerzGTokenizer with stopword filtering and stemming used by BM25 embeddings.rr stopwords Iterable[str]token_max_lengthintrrc||_d|D|_||_tjdtj|_dS)Nc6h|]}|Sr)lower).0words r z)Bm25Tokenizer.__init__..s >>>D4::<<>>>rz[^\w\s]+)flags)r _stopwords_token_max_lengthrjcompileUNICODE_non_alphanumeric_pattern)rrrrs rrzBm25Tokenizer.__init__sG   >>I>>>!1)+Krz)R)R)R&&&rtextrc8|jd|S)N )rsub)rrs r_remove_non_alphanumericz&Bm25Tokenizer._remove_non_alphanumerics-11#t<<z2Bm25Tokenizer._simple_tokenize..sAAA%5AAAAr)rsplit)rs r_simple_tokenizezBm25Tokenizer._simple_tokenizes*AA4::<<#5#5#7#7AAAArc4||}||}g}|D]h}||jvr t||jkr%|j|}|r||i|Sr) rrrlenrrrstripappend)rrcleaned raw_tokenstokensrstemmeds rtokenizezBm25Tokenizer.tokenizes//55**733  ' 'E''5zzD222m((//5577G ' g&&& rN)rrrrrrrr)rrrr)rrrr) rrrrrr staticmethodrrrrrrrs~QQ S S S S====BBB\Brrc eZdZd d dZd d Zd S)Murmur3AbsHasherrseedrrrcr ddl}n#t$rtdwxYw|j|_||_dS)NrzaThe murmurhash3 python package is not installed. Please install it with `pip install murmurhash3`)mmh3rrhashhasherr)rrrs rrzMurmur3AbsHasher.__init__sX  KKKK   s  i  rrrc zttt|||jS)N)r)rrabsrrrs rrzMurmur3AbsHasher.hashs,CT[[TY[??@@AAArN)r)rrrr)rrrr)rrrrrrrrrrsGBBBBBBrr)rrrrrr)rr) __future__rrjtypingrrrrr__annotations__listrrrrrr__all__rrrrsh""""""" 111111111111t(t(t(ttttn,040I+J+JJJJJ     h   88888888"%%%% %%%%%%%%P B B B B B B B B   r