U ʗRe3@sdZddlZddlZddlZddlmZddlmZmZm Z ddl m Z ddl m Z ddlmZdd lmZdd lmZGd d d ZdS) a Module containing the UniversalDetector detector class, which is the primary class a user of ``chardet`` should use. :author: Mark Pilgrim (initial port to Python) :author: Shy Shalom (original C code) :author: Dan Blanchard (major refactoring for 3.0) :author: Ian Cordasco N)CharSetGroupProber) InputStateLanguageFilter ProbingState)EscCharSetProber) Latin1Prober)MBCSGroupProber)SBCSGroupProber) UTF1632Proberc @seZdZdZdZedZedZedZ dddd d d d d dZ e j fddZ eddZeddZeddZddZddZddZdS)UniversalDetectoraq The ``UniversalDetector`` class underlies the ``chardet.detect`` function and coordinates all of the different charset probers. To get a ``dict`` containing an encoding and its confidence, you can simply run: .. code:: u = UniversalDetector() u.feed(some_bytes) u.close() detected = u.result g?s[-]s(|~{)s[-]z Windows-1252z Windows-1250z Windows-1251z Windows-1256z Windows-1253z Windows-1255z Windows-1254z Windows-1257)z iso-8859-1z iso-8859-2z iso-8859-5z iso-8859-6z iso-8859-7z iso-8859-8z iso-8859-9z iso-8859-13cCsTd|_d|_g|_d|_d|_d|_d|_d|_||_t t |_ d|_ |dSN)_esc_charset_prober_utf1632_prober_charset_probersresultdone _got_data _input_state _last_char lang_filterlogging getLogger__name__logger_has_win_bytesreset)selfrr/builddir/build/BUILDROOT/alt-python38-pip-22.2.1-2.el8.x86_64/opt/alt/python38/lib/python3.8/site-packages/pip/_vendor/chardet/universaldetector.py__init__Ts zUniversalDetector.__init__cCs|jSr )rrrrr input_statebszUniversalDetector.input_statecCs|jSr )rr!rrr has_win_bytesfszUniversalDetector.has_win_bytescCs|jSr )rr!rrrcharset_probersjsz!UniversalDetector.charset_proberscCsfdddd|_d|_d|_d|_tj|_d|_|jr>|j |j rN|j |j D] }| qTdS)z Reset the UniversalDetector and all of its probers back to their initial states. This is called by ``__init__``, so you only need to call this directly in between analyses of different documents. Nencoding confidencelanguageF) rrrrr PURE_ASCIIrrrrrr)rproberrrrrns   zUniversalDetector.resetcCs|jr dS|sdSt|ts$t|}|js|tjrFdddd|_nv|tjtj frhdddd|_nT|drdddd|_n:|d rd ddd|_n |tj tj frd ddd|_d |_|jd dk rd |_dS|j t jkr(|j|rt j|_ n*|j t jkr(|j|j|r(t j|_ |dd|_|jsFt|_|jjtjkr|j|tjkr|jj|jdd|_d |_dS|j t jkr|jst|j |_|j|tjkr|jj|j|jj!d|_d |_n|j t jkr|j"s4t#|j g|_"|j t$j%@r&|j"&t'|j"&t(|j"D]:}||tjkr:|j||j!d|_d |_qvq:|j)|rd |_*dS)a Takes a chunk of a document and feeds it through all of the relevant charset probers. After calling ``feed``, you can check the value of the ``done`` attribute to see if you need to continue feeding the ``UniversalDetector`` more data, or if it has made a prediction (in the ``result`` attribute). .. note:: You should always call ``close`` when you're done feeding in your document if ``done`` is not already ``True``. Nz UTF-8-SIG?r&zUTF-32szX-ISO-10646-UCS-4-3412szX-ISO-10646-UCS-4-2143zUTF-16Tr')+r isinstance bytearrayr startswithcodecsBOM_UTF8r BOM_UTF32_LE BOM_UTF32_BEBOM_LEBOM_BErrr+HIGH_BYTE_DETECTORsearch HIGH_BYTE ESC_DETECTORr ESC_ASCIIrr stater DETECTINGfeedFOUND_IT charset_nameget_confidencerrrr)rr rNON_CJKappendr rWIN_BYTE_DETECTORr)rbyte_strr,rrrr@s             zUniversalDetector.feedc Cst|jr |jSd|_|js&|jdn|jtjkrBdddd|_n|jtjkrd}d}d}|j D]"}|sjq`| }||kr`|}|}q`|r||j kr|j }|j }| }|d r|jr|j||}|||jd|_|jtjkrn|jd dkrn|jd |j D]`}|sq t|trP|jD] }|jd |j |j| q,n|jd |j |j| q |jS) z Stop analyzing the current document and come up with a final prediction. :returns: The ``result`` attribute, a ``dict`` with the keys `encoding`, `confidence`, and `language`. Tzno data received!asciir-r.r&Nr%ziso-8859r'z no probers hit minimum thresholdz%s %s confidence = %s)rrrrdebugrrr+r;rrCMINIMUM_THRESHOLDrBlowerr2r ISO_WIN_MAPgetr)getEffectiveLevelrDEBUGr0rprobers) rprober_confidencemax_prober_confidence max_proberr,rBlower_charset_namer( group_proberrrrclosesj           zUniversalDetector.closeN)r __module__ __qualname____doc__rJrecompiler9r<rFrLrALLr propertyr"r#r$rr@rVrrrrr 4s2       r )rYr3rrZcharsetgroupproberrenumsrrr escproberr latin1proberrmbcsgroupproberr sbcsgroupproberr utf1632proberr r rrrrs