Corentin Chary
|
2210b2610d
|
euscan: don't get robots.txt on ftp
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
|
2011-09-06 17:34:50 +02:00 |
|
Corentin Chary
|
a137ef60e3
|
euscan: respect robots.txt
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
|
2011-09-06 16:32:29 +02:00 |
|
Corentin Chary
|
bd75e1af4e
|
euscan/helpers: use HEAD in tryurl
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
|
2011-09-06 15:47:54 +02:00 |
|
Corentin Chary
|
454d369ced
|
euscan/handlers: fix resursive brute force in generic handler
component was modified by the function since it's a reference,
do an explicit copy to fix that.
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
|
2011-09-06 15:47:00 +02:00 |
|
Corentin Chary
|
8dc19b9856
|
euscan: fix some errors
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
|
2011-09-06 09:17:08 +02:00 |
|
Corentin Chary
|
752fb04425
|
euscan: shake the code
- add custom site handlers
- use a custom user agent
- fix some bugs in management commands
Signed-off-by: Corentin Chary <corentincj@iksaif.net>
|
2011-08-31 15:38:32 +02:00 |
|