src_prepare/euscan-ng
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

euscan: don't fail if robots.txt can't be read

Browse Source 
Signed-off-by: Corentin Chary <corentin.chary@gmail.com>
This commit is contained in:
Corentin Chary 2012-07-20 08:24:24 +02:00
parent bcda1a7ce3
commit 634e06b779
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
pym/euscan/helpers.py
View File

@ -416,7 +416,7 @@ def urlallowed(url):
setdefaulttimeout(timeout) setdefaulttimeout(timeout)
return rp.can_fetch(CONFIG['user-agent'], url) if rp else False return rp.can_fetch(CONFIG['user-agent'], url) if rp else True
def urlopen(url, timeout=None, verb="GET"): def urlopen(url, timeout=None, verb="GET"):