Management commands are refactored to have the logic moved from the
commands themselves to a separated function in order to make them easily
convertible into Celery tasks
Commands are renamed to be modules importable from other python scripts
Signed-off-by: volpino <fox91@anche.no>
- Add a blacklist for robots.txt, we *want* to scan sourceforge
- Set a user-agent that doesn't looks like a browser
- Handle timeouts more carefully
- If brute force detect too much versions, avoid infinite loops
- Handle redirections more carefully
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
