Corentin Chary 14971584af euscan: robots.txt, timeout, user-agent, ... ...

- Add a blacklist for robots.txt, we *want* to scan sourceforge
- Set a user-agent that doesn't looks like a browser
- Handle timeouts more carefully
- If brute force detect too much versions, avoid infinite loops
- Handle redirections more carefully

Signed-off-by: Corentin Chary <corentincj@iksaif.net>

2011-09-21 10:09:50 +02:00

571 B

Raw Blame History

View Raw