euscan: don't fail if robots.txt can't be read

Signed-off-by: Corentin Chary <corentin.chary@gmail.com>
This commit is contained in:
Corentin Chary committed 2012-07-20 08:41:13 +02:00
1 parent bcda1a7ce3
commit 634e06b779
1 file changed
+1 -1
+1 -1
View File
@@ -416,7 +416,7 @@ def urlallowed(url):
setdefaulttimeout(timeout)
return rp.can_fetch(CONFIG['user-agent'], url) if rp else False
return rp.can_fetch(CONFIG['user-agent'], url) if rp else True
def urlopen(url, timeout=None, verb="GET"):