U ʗRea @s,ddlmZddlmZGdddeZdS)) CharSetProber) ProbingStatecseZdZdZdZdZfddZfddZedd Z ed d Z d d Z ddZ ddZ ddZddZddZddZddZddZeddZd d!ZZS)" UTF1632Proberad This class simply looks for occurrences of zero bytes, and infers whether the file is UTF16 or UTF32 (low-endian or big-endian) For instance, files looking like ( [nonzero] )+ have a good probability to be UTF32BE. Files looking like ( [nonzero] )+ may be guessed to be UTF16BE, and inversely for little-endian varieties. gGz?csntd|_dgd|_dgd|_tj|_ddddg|_d|_ d|_ d|_ d|_ d|_ d|_|dSNF)super__init__position zeros_at_modnonzeros_at_modr DETECTING_statequadinvalid_utf16beinvalid_utf16leinvalid_utf32beinvalid_utf32le'first_half_surrogate_pair_detected_16be'first_half_surrogate_pair_detected_16leresetself __class__/builddir/build/BUILDROOT/alt-python38-pip-22.2.1-2.el8.x86_64/opt/alt/python38/lib/python3.8/site-packages/pip/_vendor/chardet/utf1632prober.pyr 's   zUTF1632Prober.__init__csftd|_dgd|_dgd|_tj|_d|_d|_ d|_ d|_ d|_ d|_ ddddg|_dSr)r rr r r rrrrrrrrrrrrrrr6s   zUTF1632Prober.resetcCs4|r dS|rdS|r$dS|r0dSdS)Nzutf-32bezutf-32lezutf-16bezutf-16lezutf-16)is_likely_utf32beis_likely_utf32leis_likely_utf16beis_likely_utf16lerrrr charset_nameDszUTF1632Prober.charset_namecCsdS)NrrrrrlanguageQszUTF1632Prober.languagecCstd|jdS)N?g@maxr rrrrapprox_32bit_charsUsz UTF1632Prober.approx_32bit_charscCstd|jdS)Nr%g@r&rrrrapprox_16bit_charsXsz UTF1632Prober.approx_16bit_charscCsj|}||jkoh|jd||jkoh|jd||jkoh|jd||jkoh|jd||jkoh|j SNrr)r(MIN_CHARS_FOR_DETECTIONr EXPECTED_RATIOr rr approx_charsrrrr[s zUTF1632Prober.is_likely_utf32becCsj|}||jkoh|jd||jkoh|jd||jkoh|jd||jkoh|jd||jkoh|j Sr*)r(r-r r.r rr/rrrres zUTF1632Prober.is_likely_utf32lecCsV|}||jkoT|jd|jd||jkoT|jd|jd||jkoT|j S)Nrr,rr+)r)r-r r.r rr/rrrr os zUTF1632Prober.is_likely_utf16becCsV|}||jkoT|jd|jd||jkoT|jd|jd||jkoT|j S)Nrr+rr,)r)r-r r.r rr/rrrr!ys zUTF1632Prober.is_likely_utf16lecCs|ddksL|ddksL|ddkrR|ddkrRd|dkrHdkrRnnd|_|ddks|ddks|ddkr|ddkrd|dkrdkrnnd|_d S) z Validate if the quad of bytes is valid UTF-32. UTF-32 is valid in the range 0x00000000 - 0x0010FFFF excluding 0x0000D800 - 0x0000DFFF https://en.wikipedia.org/wiki/UTF-32 rrr+Tr,N)rr)rrrrrvalidate_utf32_characterss8          z'UTF1632Prober.validate_utf32_characterscCs|jsNd|dkrdkr*nnd|_qxd|dkrBdkrxnqxd|_n*d|dkrfdkrrnnd|_nd|_|jsd|dkrdkrnnd|_qd|dkrdkrnqd|_n*d|dkrdkrnnd|_nd|_d S) a9 Validate if the pair of bytes is valid UTF-16. UTF-16 is valid in the range 0x0000 - 0xFFFF excluding 0xD800 - 0xFFFF with an exception for surrogate pairs, which must be in the range 0xD800-0xDBFF followed by 0xDC00-0xDFFF https://en.wikipedia.org/wiki/UTF-16 r2rTr3FrN)rrrr)rpairrrrvalidate_utf16_characterss z'UTF1632Prober.validate_utf16_characterscCs|D]}|jd}||j|<|dkrX||j||jdd||jdd|dkrt|j|d7<n|j|d7<|jd7_q|jS)Nrr,rr+r)r rr4r8r r state)rbyte_strcmod4rrrfeeds   zUTF1632Prober.feedcCsF|jtjtjhkr|jS|dkr.tj|_n|jdkr@tj|_|jS)Ng?i)rrNOT_MEFOUND_ITget_confidencer rrrrr9s   zUTF1632Prober.statecCs(|s |s |s |r$dSdS)Ng333333?g)r!r rrrrrrr@szUTF1632Prober.get_confidence)__name__ __module__ __qualname____doc__r-r.r rpropertyr"r$r(r)rrr r!r4r8r=r9r@ __classcell__rrrrrs*           rN) charsetproberrenumsrrrrrrs