4 Commits

Author	SHA1	Message	Date
Corentin Chary	14971584af	euscan: robots.txt, timeout, user-agent, ... ... - Add a blacklist for robots.txt, we *want* to scan sourceforge - Set a user-agent that doesn't looks like a browser - Handle timeouts more carefully - If brute force detect too much versions, avoid infinite loops - Handle redirections more carefully Signed-off-by: Corentin Chary <corentincj@iksaif.net>	2011-09-21 10:09:50 +02:00
Corentin Chary	8c40a1795c	euscan: blacklist art.gnome.org ... Signed-off-by: Corentin Chary <corentincj@iksaif.net>	2011-09-10 08:25:39 +02:00
Corentin Chary	a137ef60e3	euscan: respect robots.txt ... Signed-off-by: Corentin Chary <corentincj@iksaif.net>	2011-09-06 16:32:29 +02:00
Corentin Chary	752fb04425	euscan: shake the code ... - add custom site handlers - use a custom user agent - fix some bugs in management commands Signed-off-by: Corentin Chary <corentincj@iksaif.net>	2011-08-31 15:38:32 +02:00