euscan-ng

Author	SHA1	Message	Date
Corentin Chary	14971584af	euscan: robots.txt, timeout, user-agent, ... - Add a blacklist for robots.txt, we want to scan sourceforge - Set a user-agent that doesn't looks like a browser - Handle timeouts more carefully - If brute force detect too much versions, avoid infinite loops - Handle redirections more carefully Signed-off-by: Corentin Chary <corentincj@iksaif.net>	2011-09-21 10:09:50 +02:00
Corentin Chary	a137ef60e3	euscan: respect robots.txt Signed-off-by: Corentin Chary <corentincj@iksaif.net>	2011-09-06 16:32:29 +02:00
Corentin Chary	454d369ced	euscan/handlers: fix resursive brute force in generic handler component was modified by the function since it's a reference, do an explicit copy to fix that. Signed-off-by: Corentin Chary <corentincj@iksaif.net>	2011-09-06 15:47:00 +02:00
Corentin Chary	752fb04425	euscan: shake the code - add custom site handlers - use a custom user agent - fix some bugs in management commands Signed-off-by: Corentin Chary <corentincj@iksaif.net>	2011-08-31 15:38:32 +02:00