src_prepare/euscan-ng
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

euscan: show a trace when blocked by robots.txt

Browse Source 
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
This commit is contained in:
Corentin Chary 2011-10-08 08:33:48 +02:00
parent 739358cf7c
commit 8be69f2350
1 changed files with 1 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
pym/euscan/helpers.py
View File

@ -253,6 +253,7 @@ def urlallowed(url):
def urlopen(url, timeout=None, verb="GET"): def urlopen(url, timeout=None, verb="GET"):
if not urlallowed(url): if not urlallowed(url):
output.einfo("Url '%s' blocked by robots.txt" % url)
return None return None
if not timeout: if not timeout: