Corentin Chary 14971584af euscan: robots.txt, timeout, user-agent, ... ... - Add a blacklist for robots.txt, we *want* to scan sourceforge - Set a user-agent that doesn't looks like a browser - Handle timeouts more carefully - If brute force detect too much versions, avoid infinite loops - Handle redirections more carefully Signed-off-by: Corentin Chary <corentincj@iksaif.net>		2011-09-21 10:09:50 +02:00
..
euscan	euscan: robots.txt, timeout, user-agent, ...	2011-09-21 10:09:50 +02:00