a ��Re��@s0ddlZddlZddlmZGdd�de�ZdS)�N�)�ProbingStatec@sneZdZdZddd�Zdd�Zedd��Zd d �Zedd��Z d d�Z edd��Zedd��Z edd��ZdS)� CharSetProbergffffff�?NcCsd|_||_t�t�|_dS�N)�_state�lang_filter�logging� getLogger�__name__�logger)�selfr�r ��/builddir/build/BUILDROOT/alt-python39-pip-21.3.1-2.el8.x86_64/opt/alt/python39/lib/python3.9/site-packages/pip/_vendor/chardet/charsetprober.py�__init__'szCharSetProber.__init__cCstj|_dSr)r� DETECTINGr�rr r r�reset,szCharSetProber.resetcCsdSrr rr r r�charset_name/szCharSetProber.charset_namecCsdSrr )r�bufr r r�feed3szCharSetProber.feedcCs|jSr)rrr r r�state6szCharSetProber.statecCsdS)Ngr rr r r�get_confidence:szCharSetProber.get_confidencecCst�dd|�}|S)Ns([-])+� )�re�sub)rr r r�filter_high_byte_only=sz#CharSetProber.filter_high_byte_onlycCs\t�}t�d|�}|D]@}|�|dd��|dd�}|��sL|dkrLd}|�|�q|S)u9 We define three types of bytes: alphabet: english alphabets [a-zA-Z] international: international characters [-ÿ] marker: everything else [^a-zA-Z-ÿ] The input buffer can be thought to contain a series of words delimited by markers. This function works to filter all words that contain at least one international character. All contiguous sequences of markers are replaced by a single space ascii character. This filter applies to all scripts which do not use English characters. s%[a-zA-Z]*[�-�]+[a-zA-Z]*[^a-zA-Z�-�]?N��r)� bytearrayr�findall�extend�isalpha)r�filtered�words�word� last_charr r r�filter_international_wordsBs�z(CharSetProber.filter_international_wordscCs�t�}d}d}tt|��D]n}|||d�}|dkr characters. Also retains English alphabet and high byte characters immediately before occurrences of >. This filter can be applied to all scripts which contain both English characters and extended ASCII characters, but is currently only used by ``Latin1Prober``. Frr�>�s