B <_9Y��@s0ddlZddlZddlmZGdd�de�ZdS)�N�)�ProbingStatec@sneZdZdZddd�Zdd�Zedd��Zd d �Zedd��Z d d�Z edd��Zedd��Z edd��ZdS)� CharSetProbergffffff�?NcCsd|_||_t�t�|_dS)N)�_state�lang_filter�loggingZ getLogger�__name__Zlogger)�selfr�r �F/opt/alt/python37/lib/python3.7/site-packages/chardet/charsetprober.py�__init__'szCharSetProber.__init__cCstj|_dS)N)rZ DETECTINGr)r r r r�reset,szCharSetProber.resetcCsdS)Nr )r r r r�charset_name/szCharSetProber.charset_namecCsdS)Nr )r �bufr r r�feed3szCharSetProber.feedcCs|jS)N)r)r r r r�state6szCharSetProber.statecCsdS)Ngr )r r r r�get_confidence:szCharSetProber.get_confidencecCst�dd|�}|S)Ns([-])+� )�re�sub)rr r r�filter_high_byte_only=sz#CharSetProber.filter_high_byte_onlycCs`t�}t�d|�}xH|D]@}|�|dd��|dd�}|��sN|dkrNd}|�|�qW|S)u9 We define three types of bytes: alphabet: english alphabets [a-zA-Z] international: international characters [-ÿ] marker: everything else [^a-zA-Z-ÿ] The input buffer can be thought to contain a series of words delimited by markers. This function works to filter all words that contain at least one international character. All contiguous sequences of markers are replaced by a single space ascii character. This filter applies to all scripts which do not use English characters. s%[a-zA-Z]*[�-�]+[a-zA-Z]*[^a-zA-Z�-�]?N��r)� bytearrayr�findall�extend�isalpha)r�filtered�wordsZwordZ last_charr r r�filter_international_wordsBs z(CharSetProber.filter_international_wordscCs�t�}d}d}x~tt|��D]n}|||d�}|dkr>d}n|dkrJd}|dkr|��s||kr�|s�|�|||��|�d�|d}qW|s�|�||d ��|S) a� Returns a copy of ``buf`` that retains only the sequences of English alphabet and high byte characters that are not between <> characters. Also retains English alphabet and high byte characters immediately before occurrences of >. This filter can be applied to all scripts which contain both English characters and extended ASCII characters, but is currently only used by ``Latin1Prober``. Frr�>�s