B �A�[��@s�ddlmZmZmZddlmZddlmZddlm Z mZe e_ ee_dgZ Gdd�de�ZGdd�de�ZGd d �d e�ZdS)�)�absolute_import�division�unicode_literals)�str)�urllib)�parse�request�RobotFileParserc@sZeZdZdZddd�Zdd�Zdd�Zd d �Zdd�Zd d�Z dd�Z dd�Zdd�ZdS)r zs This class provides a set of methods to read, parse and answer questions about a single robots.txt file. �cCs,g|_d|_d|_d|_|�|�d|_dS)NFr)�entries� default_entry�disallow_all� allow_all�set_url�last_checked)�self�url�r�T/opt/alt/python37/lib/python3.7/site-packages/future/backports/urllib/robotparser.py�__init__s zRobotFileParser.__init__cCs|jS)z�Returns the time the robots.txt file was last fetched. This is useful for long-running web spiders that need to check for new robots.txt files periodically. )r)rrrr�mtime&szRobotFileParser.mtimecCsddl}|��|_dS)zYSets the time the robots.txt file was last fetched to the current time. rN)�timer)rrrrr�modified/szRobotFileParser.modifiedcCs&||_tj�|�dd�\|_|_dS)z,Sets the URL referring to a robots.txt file.��N)rrr�urlparse�host�path)rrrrrr7szRobotFileParser.set_urlc Cs|ytj�|j�}WnHtjjk rZ}z&|jdkr:d|_n|jdkrJd|_Wdd}~XYnX|� �}|� |�d��dS)z4Reads the robots.txt URL and feeds it to the parser.)i�i�Ti�Nzutf-8) rrZurlopenr�errorZ HTTPError�coder r�readr�decode� splitlines)r�f�err�rawrrrr <s zRobotFileParser.readcCs,d|jkr|jdkr(||_n|j�|�dS)N�*)� useragentsrr�append)r�entryrrr� _add_entryIs zRobotFileParser._add_entrycCstd}t�}�xN|D�]D}|sL|dkr0t�}d}n|dkrL|�|�t�}d}|�d�}|dkrj|d|�}|��}|sxq|�dd�}t|�dkr|d��|d<tj� |d��|d<|ddkr�|dkr�|�|�t�}|j �|d�d}q|ddk�r*|dk�rX|j�t |dd ��d}q|dd kr|dkr|j�t |dd��d}qW|dk�rp|�|�dS)z�Parse the input lines from a robots.txt file. We allow that a user-agent: line is not preceded by one or more blank lines. rr��#N�:z user-agentZdisallowFZallowT)�Entryr*�find�strip�split�len�lowerrr�unquoter'r(� rulelines�RuleLine)r�lines�stater)�line�irrrrRsJ zRobotFileParser.parsecCs�|jr dS|jrdStj�tj�|��}tj�dd|j|j|j |j f�}tj�|�}|s\d}x"|jD]}|� |�rd|�|�SqdW|jr�|j�|�SdS)z=using the parsed robots.txt decide if useragent can fetch urlFTr �/)r rrrrr4� urlunparser�params�query�fragment�quoter� applies_to� allowancer)r� useragentr� parsed_urlr)rrr� can_fetch�s zRobotFileParser.can_fetchcCsd�dd�|jD��S)Nr cSsg|]}t|�d�qS)� )r)�.0r)rrr� �sz+RobotFileParser.__str__..)�joinr)rrrr�__str__�szRobotFileParser.__str__N)r ) �__name__� __module__�__qualname__�__doc__rrrrr r*rrErJrrrrr s 3c@s(eZdZdZdd�Zdd�Zdd�ZdS) r6zoA rule line is a single "Allow:" (allowance==True) or "Disallow:" (allowance==False) followed by a path.cCs(|dkr|sd}tj�|�|_||_dS)Nr T)rrr@rrB)rrrBrrrr�szRuleLine.__init__cCs|jdkp|�|j�S)Nr&)r� startswith)r�filenamerrrrA�szRuleLine.applies_tocCs|jr dpdd|jS)NZAllowZDisallowz: )rBr)rrrrrJ�szRuleLine.__str__N)rKrLrMrNrrArJrrrrr6�sr6c@s0eZdZdZdd�Zdd�Zdd�Zdd �Zd S)r.z?An entry has one or more user-agents and zero or more rulelinescCsg|_g|_dS)N)r'r5)rrrrr�szEntry.__init__cCsPg}x|jD]}|�d|dg�qWx |jD]}|�t|�dg�q,Wd�|�S)NzUser-agent: rFr )r'�extendr5rrI)r�ret�agentr9rrrrJ�sz Entry.__str__cCsF|�d�d��}x.|jD]$}|dkr*dS|��}||krdSqWdS)z2check if this entry applies to the specified agentr;rr&TF)r1r3r')rrCrSrrrrA�szEntry.applies_tocCs$x|jD]}|�|�r|jSqWdS)zZPreconditions: - our agent applies to this entry - filename is URL decodedT)r5rArB)rrPr9rrrrB�s zEntry.allowanceN)rKrLrMrNrrJrArBrrrrr.�s r.N)� __future__rrrZfuture.builtinsrZfuture.backportsrZfuture.backports.urllibr�_parserZ_request�__all__�objectr r6r.rrrr�s