Update README.rst
This commit is contained in:
Federico Scrinzi
parent
e17e0de19a
commit
1c53c60eed
221
README.rst
221
README.rst
@ -2,7 +2,7 @@ What is euscan ?
|
|||||||
================
|
================
|
||||||
|
|
||||||
dev-portage/euscan
|
dev-portage/euscan
|
||||||
------------------
|
==================
|
||||||
|
|
||||||
euscan is available in portage as a dev package (app-portage/euscan-9999).
|
euscan is available in portage as a dev package (app-portage/euscan-9999).
|
||||||
This tool allows to check if a given package/ebuild has new upstream versions
|
This tool allows to check if a given package/ebuild has new upstream versions
|
||||||
@ -17,56 +17,62 @@ to help euscan do its job more efficiently.
|
|||||||
|
|
||||||
euscan heuristics are described in the "How does-it works ?" section.
|
euscan heuristics are described in the "How does-it works ?" section.
|
||||||
|
|
||||||
### Examples
|
Examples
|
||||||
|
--------
|
||||||
|
::
|
||||||
|
|
||||||
$ euscan amatch
|
$ euscan amatch
|
||||||
|
|
||||||
* dev-ruby/amatch-0.2.7 [gentoo]
|
* dev-ruby/amatch-0.2.7 [gentoo]
|
||||||
|
|
||||||
Ebuild: /home/euscan/local/usr/portage/dev-ruby/amatch/amatch-0.2.7.ebuild
|
Ebuild: /home/euscan/local/usr/portage/dev-ruby/amatch/amatch-0.2.7.ebuild
|
||||||
Repository: gentoo
|
Repository: gentoo
|
||||||
Homepage: http://flori.github.com/amatch/
|
Homepage: http://flori.github.com/amatch/
|
||||||
Description: Approximate Matching Extension for Ruby
|
Description: Approximate Matching Extension for Ruby
|
||||||
|
|
||||||
* SRC_URI is 'mirror://rubygems/amatch-0.2.7.gem'
|
* SRC_URI is 'mirror://rubygems/amatch-0.2.7.gem'
|
||||||
* Using: http://rubygems.org/api/v1/versions/amatch.json
|
* Using: http://rubygems.org/api/v1/versions/amatch.json
|
||||||
|
|
||||||
Upstream Version: 0.2.8 http://rubygems.org/gems/amatch-0.2.8.gem
|
Upstream Version: 0.2.8 http://rubygems.org/gems/amatch-0.2.8.gem
|
||||||
|
|
||||||
$ euscan rsyslog
|
::
|
||||||
|
|
||||||
* app-admin/rsyslog-5.8.5 [gentoo]
|
$ euscan rsyslog
|
||||||
|
|
||||||
Ebuild: /home/euscan/local/usr/portage/app-admin/rsyslog/rsyslog-5.8.5.ebuild
|
* app-admin/rsyslog-5.8.5 [gentoo]
|
||||||
Repository: gentoo
|
|
||||||
Homepage: http://www.rsyslog.com/
|
|
||||||
Description: An enhanced multi-threaded syslogd with database support and more.
|
|
||||||
|
|
||||||
* SRC_URI is 'http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.8.5.tar.gz'
|
Ebuild: /home/euscan/local/usr/portage/app-admin/rsyslog/rsyslog-5.8.5.ebuild
|
||||||
* Scanning: http://www.rsyslog.com/files/download/rsyslog/rsyslog-${PV}.tar.gz
|
Repository: gentoo
|
||||||
* Scanning: http://www.rsyslog.com/files/download/rsyslog
|
Homepage: http://www.rsyslog.com/
|
||||||
* Generating version from 5.8.5
|
Description: An enhanced multi-threaded syslogd with database support and more.
|
||||||
* Brute forcing: http://www.rsyslog.com/files/download/rsyslog/rsyslog-${PV}.tar.gz
|
|
||||||
* Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.8.6.tar.gz ... [ !! ]
|
|
||||||
* Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.8.7.tar.gz ... [ !! ]
|
|
||||||
* Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.8.8.tar.gz ... [ !! ]
|
|
||||||
* Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.0.tar.gz ... [ ok ]
|
|
||||||
* Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.10.0.tar.gz ... [ !! ]
|
|
||||||
* Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.11.0.tar.gz ... [ !! ]
|
|
||||||
* Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.1.tar.gz ... [ ok ]
|
|
||||||
* Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.2.tar.gz ... [ ok ]
|
|
||||||
* Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.3.tar.gz ... [ ok ]
|
|
||||||
* Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.12.0.tar.gz ... [ !! ]
|
|
||||||
* Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.4.tar.gz ... [ !! ]
|
|
||||||
* Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.5.tar.gz ... [ !! ]
|
|
||||||
* Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.6.tar.gz ... [ !! ]
|
|
||||||
|
|
||||||
Upstream Version: 5.9.1 http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.1.tar.gz
|
* SRC_URI is 'http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.8.5.tar.gz'
|
||||||
Upstream Version: 5.9.0 http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.0.tar.gz
|
* Scanning: http://www.rsyslog.com/files/download/rsyslog/rsyslog-${PV}.tar.gz
|
||||||
Upstream Version: 5.9.3 http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.3.tar.gz
|
* Scanning: http://www.rsyslog.com/files/download/rsyslog
|
||||||
Upstream Version: 5.9.2 http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.2.tar.gz
|
* Generating version from 5.8.5
|
||||||
|
* Brute forcing: http://www.rsyslog.com/files/download/rsyslog/rsyslog-${PV}.tar.gz
|
||||||
|
* Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.8.6.tar.gz ... [ !! ]
|
||||||
|
* Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.8.7.tar.gz ... [ !! ]
|
||||||
|
* Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.8.8.tar.gz ... [ !! ]
|
||||||
|
* Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.0.tar.gz ... [ ok ]
|
||||||
|
* Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.10.0.tar.gz ... [ !! ]
|
||||||
|
* Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.11.0.tar.gz ... [ !! ]
|
||||||
|
* Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.1.tar.gz ... [ ok ]
|
||||||
|
* Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.2.tar.gz ... [ ok ]
|
||||||
|
* Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.3.tar.gz ... [ ok ]
|
||||||
|
* Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.12.0.tar.gz ... [ !! ]
|
||||||
|
* Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.4.tar.gz ... [ !! ]
|
||||||
|
* Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.5.tar.gz ... [ !! ]
|
||||||
|
* Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.6.tar.gz ... [ !! ]
|
||||||
|
|
||||||
### Hidden settings
|
Upstream Version: 5.9.1 http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.1.tar.gz
|
||||||
|
Upstream Version: 5.9.0 http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.0.tar.gz
|
||||||
|
Upstream Version: 5.9.3 http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.3.tar.gz
|
||||||
|
Upstream Version: 5.9.2 http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.2.tar.gz
|
||||||
|
|
||||||
|
|
||||||
|
Hidden settings
|
||||||
|
---------------
|
||||||
|
|
||||||
You can configure some settings using the command line, but the __init__.py
|
You can configure some settings using the command line, but the __init__.py
|
||||||
file of the euscan package contains more settings, including blacklists and
|
file of the euscan package contains more settings, including blacklists and
|
||||||
@ -76,7 +82,7 @@ Maybe we should add the ability to use /etc/euscan.conf and
|
|||||||
~/.config/euscan/euscan.conf to override these settings.
|
~/.config/euscan/euscan.conf to override these settings.
|
||||||
|
|
||||||
euscan-www: euscan as a service
|
euscan-www: euscan as a service
|
||||||
-------------------------------
|
===============================
|
||||||
|
|
||||||
euscan-www is a web application that aggregates euscan results. For example
|
euscan-www is a web application that aggregates euscan results. For example
|
||||||
there is an instance of euscan-www that monitors gentoo-x86 + some official
|
there is an instance of euscan-www that monitors gentoo-x86 + some official
|
||||||
@ -86,36 +92,40 @@ euscan-www uses django and provides some custom commands to feed the database.
|
|||||||
You can use euscan-www on you system tree, or preferably you can use a local
|
You can use euscan-www on you system tree, or preferably you can use a local
|
||||||
tree to avoid messing with your system.
|
tree to avoid messing with your system.
|
||||||
|
|
||||||
### Installation
|
Installation
|
||||||
|
------------
|
||||||
|
|
||||||
Install requirements from PyPI using
|
Install requirements from PyPI using::
|
||||||
|
|
||||||
$ python setup.py develop
|
$ python setup.py develop
|
||||||
|
|
||||||
Extra dependencies:
|
Extra dependencies:
|
||||||
|
* portage python api
|
||||||
* portage python api
|
* rrdtool[python]
|
||||||
* rrdtool[python]
|
|
||||||
|
|
||||||
Like any django web app, just start by editing settings.py and then run
|
Like any django web app, just start by editing settings.py and then run
|
||||||
these two commands.
|
these two commands.
|
||||||
|
::
|
||||||
|
|
||||||
$ python manage.py syncdb
|
$ python manage.py syncdb
|
||||||
$ python manage.py migrate
|
$ python manage.py migrate
|
||||||
|
|
||||||
Now your instance is ready, you can just run this command to browse it.
|
Now your instance is ready, you can just run this command to browse it.
|
||||||
If you want to host it publicly you should use a real webserver.
|
If you want to host it publicly you should use a real webserver.
|
||||||
|
::
|
||||||
|
|
||||||
$ python manage.py runserver
|
$ python manage.py runserver
|
||||||
|
|
||||||
### Creating a local tree (optional)
|
Creating a local tree (optional)
|
||||||
|
--------------------------------
|
||||||
|
|
||||||
Create a local tree with all that portage (and layman would need).
|
Create a local tree with all that portage (and layman would need).
|
||||||
There is an example in euscanwww/scripts/local-tree/. See escan-update.sh
|
There is an example in euscanwww/scripts/local-tree/. See escan-update.sh
|
||||||
to know what env variables you need to run any portage related command in
|
to know what env variables you need to run any portage related command in
|
||||||
this local tree.
|
this local tree.
|
||||||
|
|
||||||
### Scanning process
|
Scanning process
|
||||||
|
----------------
|
||||||
|
|
||||||
The scanning process is done by euscan-update.sh. You should read carefully
|
The scanning process is done by euscan-update.sh. You should read carefully
|
||||||
this script, and adapt it to your needs. For example it uses gparallel to
|
this script, and adapt it to your needs. For example it uses gparallel to
|
||||||
@ -123,40 +133,36 @@ launch multiple process at a time, and you should adapt that to your number
|
|||||||
of cpu and network bandwith.
|
of cpu and network bandwith.
|
||||||
|
|
||||||
Once your euscan-update.sh is ok, just run it.
|
Once your euscan-update.sh is ok, just run it.
|
||||||
|
::
|
||||||
|
|
||||||
$ sh euscan-update.sh
|
$ sh euscan-update.sh
|
||||||
|
|
||||||
### Custom Django management commands
|
Custom Django management commands
|
||||||
|
---------------------------------
|
||||||
|
|
||||||
euscan-www povides some new management commands, here is a short description
|
euscan-www povides some new management commands, here is a short description
|
||||||
of these commands. Use "help" or read euscan-update.sh to get more informations.
|
of these commands. Use "help" or read euscan-update.sh to get more informations.
|
||||||
|
|
||||||
#### list-packages
|
list-packages
|
||||||
|
List packages stored in database.
|
||||||
|
|
||||||
List packages stored in database.
|
scan-portage
|
||||||
|
Scan the portage tree and store new packages and versions in the database.
|
||||||
|
|
||||||
#### scan-portage
|
scan-metadata.py
|
||||||
|
Scan metadata and looks for homepage, maintainers and herds.
|
||||||
|
|
||||||
Scan the portage tree and store new packages and versions in the database.
|
scan-upstream
|
||||||
|
Scan upstream package. The prefered way to use this script it to first launch
|
||||||
|
euscan on some packages, store the result of the file, and feed this command with
|
||||||
|
the result.
|
||||||
|
|
||||||
#### scan-metadata.py
|
update-counters
|
||||||
|
Update statistics and rrd files.
|
||||||
|
|
||||||
Scan metadata and looks for homepage, maintainers and herds.
|
regen-rrds
|
||||||
|
If you deleted your rrd files, this script will use the database to
|
||||||
#### scan-upstream
|
regen them.
|
||||||
|
|
||||||
Scan upstream package. The prefered way to use this script it to first launch
|
|
||||||
euscan on some packages, store the result of the file, and feed this command with
|
|
||||||
the result.
|
|
||||||
|
|
||||||
#### update-counters
|
|
||||||
|
|
||||||
Update statistics and rrd files.
|
|
||||||
|
|
||||||
#### regen-rrds
|
|
||||||
|
|
||||||
If you deleted your rrd files, this script will use the database to
|
|
||||||
regen them.
|
|
||||||
|
|
||||||
How does it work ?
|
How does it work ?
|
||||||
==================
|
==================
|
||||||
@ -173,9 +179,12 @@ uses a file that describe what url and regexp to use to match packages.
|
|||||||
euscan uses SRC_URI and tries to find the current version (or part of this version)
|
euscan uses SRC_URI and tries to find the current version (or part of this version)
|
||||||
in the resolved SRC_URI and generate a regexp from that.
|
in the resolved SRC_URI and generate a regexp from that.
|
||||||
|
|
||||||
For example for app-accessibility/dash-4.10.1, SRC_URI is:
|
For example for app-accessibility/dash-4.10.1, SRC_URI is::
|
||||||
|
|
||||||
mirror://gnome/sources/dasher/4.10/dasher-4.10.1.tar.bz2
|
mirror://gnome/sources/dasher/4.10/dasher-4.10.1.tar.bz2
|
||||||
euscan will scan pages based on this template:
|
|
||||||
|
euscan will scan pages based on this template::
|
||||||
|
|
||||||
http://ftp.gnome.org/pub/gnome/sources/dasher/${0}.${1}/dasher-${PV}.tar.bz2
|
http://ftp.gnome.org/pub/gnome/sources/dasher/${0}.${1}/dasher-${PV}.tar.bz2
|
||||||
|
|
||||||
Then, from that, it will scan the top-most directory that doesn't depend on
|
Then, from that, it will scan the top-most directory that doesn't depend on
|
||||||
@ -188,57 +197,49 @@ Like when scanning directories, a template of SRC_URI is built. Then euscan
|
|||||||
generate next possible version numbers, and tries to download the url generated
|
generate next possible version numbers, and tries to download the url generated
|
||||||
from the template and the new version number.
|
from the template and the new version number.
|
||||||
|
|
||||||
For example, running euscan on portage/app-accessibility/festival-freebsoft-utils-0.6:
|
For example, running euscan on portage/app-accessibility/festival-freebsoft-utils-0.6::
|
||||||
SRC_URI is 'http://www.freebsoft.org/pub/projects/festival-freebsoft-utils/festival-freebsoft-utils-0.6.tar.gz'
|
|
||||||
Template is http://www.freebsoft.org/pub/projects/festival-freebsoft-utils/festival-freebsoft-utils-${PV}.tar.gz
|
SRC_URI is 'http://www.freebsoft.org/pub/projects/festival-freebsoft-utils/festival-freebsoft-utils-0.6.tar.gz'
|
||||||
Generate version from 0.6: 0.7, 0.8, 0.10, ...
|
Template is http://www.freebsoft.org/pub/projects/festival-freebsoft-utils/festival-freebsoft-utils-${PV}.tar.gz
|
||||||
Try new urls: http://www.freebsoft.org/pub/projects/festival-freebsoft-utils/festival-freebsoft-utils-0.7.tar.gz, etc..
|
Generate version from 0.6: 0.7, 0.8, 0.10, ...
|
||||||
|
Try new urls: http://www.freebsoft.org/pub/projects/festival-freebsoft-utils/festival-freebsoft-utils-0.7.tar.gz, etc..
|
||||||
|
|
||||||
Blacklists
|
Blacklists
|
||||||
----------
|
----------
|
||||||
|
|
||||||
euscan uses blacklist for multiple purposes.
|
euscan uses blacklist for multiple purposes.
|
||||||
|
|
||||||
### BLACKLIST_VERSIONS
|
BLACKLIST_VERSIONS
|
||||||
|
For versions that should not be checked at all. sys-libs/libstdc++-v3-3.4
|
||||||
|
is good example because it's a package which version will always be 3.4
|
||||||
|
(Compatibility package for running binaries linked against a pre gcc 3.4 libstdc++).
|
||||||
|
|
||||||
For versions that should not be checked at all. sys-libs/libstdc++-v3-3.4
|
BLACKLIST_PACKAGES
|
||||||
is good example because it's a package which version will always be 3.4
|
Some packages are dead, but SRC_URI refers to sources that are still being
|
||||||
(Compatibility package for running binaries linked against a pre gcc 3.4 libstdc++).
|
updated, for example: sys-kernel/xbox-sources that uses the same sources as
|
||||||
|
vanilla-sources but is not updated the same way.
|
||||||
|
|
||||||
|
SCANDIR_BLACKLIST_URLS
|
||||||
|
For urls that are not browsable. mirror://gentoo/ is a good example: it's
|
||||||
|
both stupid to scan it and very long/expensive.
|
||||||
|
|
||||||
### BLACKLIST_PACKAGES
|
BRUTEFORCE_BLACKLIST_PACKAGES and BRUTEFORCE_BLACKLIST_URLS
|
||||||
|
Disable brute force on those packages and urls. Most of the time it's because
|
||||||
|
upstream is broken and will answer HTTP 200 even if the file doesn't exist.
|
||||||
|
|
||||||
Some packages are dead, but SRC_URI refers to sources that are still being
|
ROBOTS_TXT_BLACKLIST_DOMAINS
|
||||||
updated, for example: sys-kernel/xbox-sources that uses the same sources as
|
Don't respect robots.txt for these domains (sourcefourge, berlios, github.com).
|
||||||
vanilla-sources but is not updated the same way.
|
|
||||||
|
|
||||||
### SCANDIR_BLACKLIST_URLS
|
|
||||||
|
|
||||||
For urls that are not browsable. mirror://gentoo/ is a good example: it's
|
|
||||||
both stupid to scan it and very long/expensive.
|
|
||||||
|
|
||||||
### BRUTEFORCE_BLACKLIST_PACKAGES and BRUTEFORCE_BLACKLIST_URLS
|
|
||||||
|
|
||||||
Disable brute force on those packages and urls. Most of the time it's because
|
|
||||||
upstream is broken and will answer HTTP 200 even if the file doesn't exist.
|
|
||||||
|
|
||||||
### ROBOTS_TXT_BLACKLIST_DOMAINS
|
|
||||||
|
|
||||||
Don't respect robots.txt for these domains (sourcefourge, berlios, github.com).
|
|
||||||
|
|
||||||
Site handlers
|
Site handlers
|
||||||
-------------
|
-------------
|
||||||
|
|
||||||
### Pecl/PEAR
|
Pecl/PEAR
|
||||||
|
A site handler that uses the Pecl/PEAR rest API
|
||||||
|
(http://pear.php.net/manual/en/core.rest.php).
|
||||||
|
|
||||||
A site handler that uses the Pecl/PEAR rest API
|
Rubygems
|
||||||
(http://pear.php.net/manual/en/core.rest.php).
|
This one uses rubygems's json API
|
||||||
|
(http://guides.rubygems.org/rubygems-org-api/)
|
||||||
|
|
||||||
### Rubygems
|
PyPI
|
||||||
|
Uses PyPI's XML rpc API.
|
||||||
This one uses rubygems's json API
|
|
||||||
(http://guides.rubygems.org/rubygems-org-api/)
|
|
||||||
|
|
||||||
### Pypy
|
|
||||||
|
|
||||||
Uses pypy's XML rpc API.
|
|
||||||
Loading…
Reference in New Issue
Block a user