Corentin Chary
3da5fb5240
euscan: mangle versions the gentoo-way
...
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
2012-04-19 15:36:44 +02:00
Corentin Chary
6a57b44d7c
euscan: force nodejs.org scan
...
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
2012-04-04 14:34:09 +02:00
Corentin Chary
76a1121c9f
euscan: don't keep version from partial paths
...
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
2012-03-28 09:09:08 +02:00
Corentin Chary
5062f29e3c
euscanwww: don't delete packages when deleting related versions
...
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
2012-03-14 14:07:28 +01:00
Corentin Chary
329b20fecf
euscan/rubygem: better gem detection
...
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
2012-03-02 13:54:23 +01:00
Corentin Chary
df1d778365
euscan: add a quick /_rc/-rc/ hack
...
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
2012-02-20 08:21:07 +01:00
Corentin Chary
5bd358968a
euscan: re-indent blacklists
...
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
2012-02-20 08:20:54 +01:00
Corentin Chary
8190779070
euscan: remove a line
...
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
2012-02-14 14:54:34 +01:00
Corentin Chary
72e43ac4fe
euscan: tweak error paths
...
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
2012-01-02 21:31:05 +01:00
Corentin Chary
b78e73038d
euscan: add kde handler and fix some regex issues
...
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
2011-12-30 11:34:39 +01:00
Corentin Chary
fef31c514c
euscan: respect --quiet in pypi handler
...
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
2011-12-12 15:15:00 +01:00
Corentin Chary
3569f1bb97
euscan: better --quiet mode
...
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
2011-12-11 14:32:38 +01:00
Corentin Chary
a11bfdc54f
euscan: fix a bug with htop versions
...
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
2011-11-27 16:25:44 +01:00
Corentin Chary
8be69f2350
euscan: show a trace when blocked by robots.txt
...
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
2011-10-08 08:33:48 +02:00
Corentin Chary
739358cf7c
euscan: really skip all dev versions
...
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
2011-10-08 08:33:25 +02:00
Corentin Chary
277fb4ebe6
euscan: add new robots.txt exceptions
...
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
2011-10-08 08:33:03 +02:00
Corentin Chary
d7f655cdde
euscan: add an optional persistent cache
...
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
2011-10-02 10:04:44 +02:00
Corentin Chary
7fb4e6edff
euscan: better output
...
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
2011-09-21 10:58:41 +02:00
Corentin Chary
14971584af
euscan: robots.txt, timeout, user-agent, ...
...
- Add a blacklist for robots.txt, we *want* to scan sourceforge
- Set a user-agent that doesn't looks like a browser
- Handle timeouts more carefully
- If brute force detect too much versions, avoid infinite loops
- Handle redirections more carefully
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
2011-09-21 10:09:50 +02:00
Corentin Chary
8c40a1795c
euscan: blacklist art.gnome.org
...
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
2011-09-10 08:25:39 +02:00
Corentin Chary
9da62b211b
euscan: fix some robots.txt issues
...
- disable checks for ftp
- fail silently
- use einfo and not eerror
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
2011-09-10 08:23:46 +02:00
Corentin Chary
c5af0e1937
euscan: don't mix spaces and tabs
...
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
2011-09-06 17:35:17 +02:00
Corentin Chary
2210b2610d
euscan: don't get robots.txt on ftp
...
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
2011-09-06 17:34:50 +02:00
Corentin Chary
a137ef60e3
euscan: respect robots.txt
...
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
2011-09-06 16:32:29 +02:00
Corentin Chary
bd75e1af4e
euscan/helpers: use HEAD in tryurl
...
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
2011-09-06 15:47:54 +02:00
Corentin Chary
454d369ced
euscan/handlers: fix resursive brute force in generic handler
...
component was modified by the function since it's a reference,
do an explicit copy to fix that.
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
2011-09-06 15:47:00 +02:00
Corentin Chary
8dc19b9856
euscan: fix some errors
...
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
2011-09-06 09:17:08 +02:00
Corentin Chary
752fb04425
euscan: shake the code
...
- add custom site handlers
- use a custom user agent
- fix some bugs in management commands
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
2011-08-31 15:38:32 +02:00