� e f4�@sddZddlZddlZdgZGdd�d�ZGdd�d�ZGdd�d�ZdS) a% robotparser.py Copyright (C) 2000 Bastian Kleineidam You can choose between two licenses when using this package: 1) GNU GPLv2 2) PSF license for Python 2.2 The robots.txt Exclusion Protocol is implemented as specified in http://www.robotstxt.org/norobots-rfc.txt �N�RobotFileParserc@s�eZdZdZddd�Zdd�Zdd�Zd d �Zdd�Zd d�Z dd�Z dd�Zdd�ZdS)rzs This class provides a set of methods to read, parse and answer questions about a single robots.txt file. �cCs>g|_d|_d|_d|_|j|�d|_dS)NFr)�entries� default_entry�disallow_all� allow_all�set_url�last_checked)�self�url�r�7/opt/alt/python34/lib64/python3.4/urllib/robotparser.py�__init__s zRobotFileParser.__init__cCs|jS)z�Returns the time the robots.txt file was last fetched. This is useful for long-running web spiders that need to check for new robots.txt files periodically. )r )r rrr �mtimeszRobotFileParser.mtimecCsddl}|j�|_dS)zYSets the time the robots.txt file was last fetched to the current time. rN)�timer )r rrrr �modified(szRobotFileParser.modifiedcCs5||_tjj|�dd�\|_|_dS)z,Sets the URL referring to a robots.txt file.��N)r�urllib�parse�urlparse�host�path)r rrrr r0s zRobotFileParser.set_urlcCs�ytjj|j�}Wnptjjk r�}zJ|jdkrOd|_n*|jdkry|jdkryd|_nWYdd}~Xn)X|j �}|j |jd�j��dS) z4Reads the robots.txt URL and feeds it to the parser.��Ti�i�Nzutf-8)rr) rZrequestZurlopenr�errorZ HTTPError�coderr�readr�decode� splitlines)r �f�err�rawrrr r5szRobotFileParser.readcCsAd|jkr-|jdkr=||_q=n|jj|�dS)N�*)� useragentsrr�append)r �entryrrr � _add_entryBszRobotFileParser._add_entrycCsd}t�}|j�x�|D]�}|sx|dkrJt�}d}qx|dkrx|j|�t�}d}qxn|jd�}|dkr�|d|�}n|j�}|s�q n|jdd�}t|�dkr |dj�j�|d�s z+RobotFileParser.__str__..)�joinr)r rrr �__str__�szRobotFileParser.__str__N) �__name__� __module__�__qualname__�__doc__rrrrrr'rr>rDrrrr rs 4c@s:eZdZdZdd�Zdd�Zdd�ZdS) r3zoA rule line is a single "Allow:" (allowance==True) or "Disallow:" (allowance==False) followed by a path.cCs\|dkr|rd}ntjjtjj|��}tjj|�|_||_dS)NrT)rrr9rr:rr<)r rr<rrr r�s zRuleLine.__init__cCs|jdkp|j|j�S)Nr#)r� startswith)r �filenamerrr r;�szRuleLine.applies_tocCs|jrdpdd|jS)NZAllowZDisallowz: )r<r)r rrr rD�szRuleLine.__str__N)rErFrGrHrr;rDrrrr r3�sr3c@sFeZdZdZdd�Zdd�Zdd�Zdd �Zd S)r+z?An entry has one or more user-agents and zero or more rulelinescCsg|_g|_dS)N)r$r2)r rrr r�s zEntry.__init__cCsjg}x'|jD]}|jd|dg�qWx*|jD]}|jt|�dg�q:Wdj|�S)NzUser-agent: r?r)r$�extendr2r@rC)r Zret�agentr6rrr rD�sz Entry.__str__cCs]|jd�dj�}x=|jD]2}|dkr9dS|j�}||kr#dSq#WdS)z2check if this entry applies to the specified agentr8rr#TF)r.r0r$)r r=rLrrr r;�szEntry.applies_tocCs.x'|jD]}|j|�r |jSq WdS)zZPreconditions: - our agent applies to this entry - filename is URL decodedT)r2r;r<)r rJr6rrr r<�szEntry.allowanceN)rErFrGrHrrDr;r<rrrr r+�s r+)rHZurllib.parserZurllib.request�__all__rr3r+rrrr �s �