src_prepare/euscan-ng
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Update README.rst

Browse Source
This commit is contained in:
Federico Scrinzi 2012-05-01 15:29:40 +03:00
parent e17e0de19a
commit 1c53c60eed
1 changed files with 111 additions and 110 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

177
README.rst
View File

@ -2,7 +2,7 @@ What is euscan ?
================ ================
dev-portage/euscan dev-portage/euscan
------------------ ==================
euscan is available in portage as a dev package (app-portage/euscan-9999). euscan is available in portage as a dev package (app-portage/euscan-9999).
This tool allows to check if a given package/ebuild has new upstream versions This tool allows to check if a given package/ebuild has new upstream versions
@ -17,30 +17,34 @@ to help euscan do its job more efficiently.
euscan heuristics are described in the "How does-it works ?" section. euscan heuristics are described in the "How does-it works ?" section.
### Examples Examples
--------
::
$ euscan amatch $ euscan amatch
* dev-ruby/amatch-0.2.7 [gentoo] * dev-ruby/amatch-0.2.7 [gentoo]
Ebuild: /home/euscan/local/usr/portage/dev-ruby/amatch/amatch-0.2.7.ebuild Ebuild: /home/euscan/local/usr/portage/dev-ruby/amatch/amatch-0.2.7.ebuild
Repository: gentoo Repository: gentoo
Homepage: http://flori.github.com/amatch/ Homepage: http://flori.github.com/amatch/
Description: Approximate Matching Extension for Ruby Description: Approximate Matching Extension for Ruby
* SRC_URI is 'mirror://rubygems/amatch-0.2.7.gem' * SRC_URI is 'mirror://rubygems/amatch-0.2.7.gem'
* Using: http://rubygems.org/api/v1/versions/amatch.json * Using: http://rubygems.org/api/v1/versions/amatch.json
Upstream Version: 0.2.8 http://rubygems.org/gems/amatch-0.2.8.gem Upstream Version: 0.2.8 http://rubygems.org/gems/amatch-0.2.8.gem
$ euscan rsyslog ::
$ euscan rsyslog
* app-admin/rsyslog-5.8.5 [gentoo] * app-admin/rsyslog-5.8.5 [gentoo]
Ebuild: /home/euscan/local/usr/portage/app-admin/rsyslog/rsyslog-5.8.5.ebuild Ebuild: /home/euscan/local/usr/portage/app-admin/rsyslog/rsyslog-5.8.5.ebuild
Repository: gentoo Repository: gentoo
Homepage: http://www.rsyslog.com/ Homepage: http://www.rsyslog.com/
Description: An enhanced multi-threaded syslogd with database support and more. Description: An enhanced multi-threaded syslogd with database support and more.
* SRC_URI is 'http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.8.5.tar.gz' * SRC_URI is 'http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.8.5.tar.gz'
* Scanning: http://www.rsyslog.com/files/download/rsyslog/rsyslog-${PV}.tar.gz * Scanning: http://www.rsyslog.com/files/download/rsyslog/rsyslog-${PV}.tar.gz
@ -61,12 +65,14 @@ Description: An enhanced multi-threaded syslogd with database support and more.
* Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.5.tar.gz ... [ !! ] * Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.5.tar.gz ... [ !! ]
* Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.6.tar.gz ... [ !! ] * Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.6.tar.gz ... [ !! ]
Upstream Version: 5.9.1 http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.1.tar.gz Upstream Version: 5.9.1 http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.1.tar.gz
Upstream Version: 5.9.0 http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.0.tar.gz Upstream Version: 5.9.0 http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.0.tar.gz
Upstream Version: 5.9.3 http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.3.tar.gz Upstream Version: 5.9.3 http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.3.tar.gz
Upstream Version: 5.9.2 http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.2.tar.gz Upstream Version: 5.9.2 http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.2.tar.gz
### Hidden settings
Hidden settings
---------------
You can configure some settings using the command line, but the __init__.py You can configure some settings using the command line, but the __init__.py
file of the euscan package contains more settings, including blacklists and file of the euscan package contains more settings, including blacklists and
@ -76,7 +82,7 @@ Maybe we should add the ability to use /etc/euscan.conf and
~/.config/euscan/euscan.conf to override these settings. ~/.config/euscan/euscan.conf to override these settings.
euscan-www: euscan as a service euscan-www: euscan as a service
------------------------------- ===============================
euscan-www is a web application that aggregates euscan results. For example euscan-www is a web application that aggregates euscan results. For example
there is an instance of euscan-www that monitors gentoo-x86 + some official there is an instance of euscan-www that monitors gentoo-x86 + some official
@ -86,36 +92,40 @@ euscan-www uses django and provides some custom commands to feed the database.
You can use euscan-www on you system tree, or preferably you can use a local You can use euscan-www on you system tree, or preferably you can use a local
tree to avoid messing with your system. tree to avoid messing with your system.
### Installation Installation
------------
Install requirements from PyPI using Install requirements from PyPI using::
$ python setup.py develop $ python setup.py develop
Extra dependencies: Extra dependencies:
* portage python api
* portage python api * rrdtool[python]
* rrdtool[python]
Like any django web app, just start by editing settings.py and then run Like any django web app, just start by editing settings.py and then run
these two commands. these two commands.
::
$ python manage.py syncdb $ python manage.py syncdb
$ python manage.py migrate $ python manage.py migrate
Now your instance is ready, you can just run this command to browse it. Now your instance is ready, you can just run this command to browse it.
If you want to host it publicly you should use a real webserver. If you want to host it publicly you should use a real webserver.
::
$ python manage.py runserver $ python manage.py runserver
### Creating a local tree (optional) Creating a local tree (optional)
--------------------------------
Create a local tree with all that portage (and layman would need). Create a local tree with all that portage (and layman would need).
There is an example in euscanwww/scripts/local-tree/. See escan-update.sh There is an example in euscanwww/scripts/local-tree/. See escan-update.sh
to know what env variables you need to run any portage related command in to know what env variables you need to run any portage related command in
this local tree. this local tree.
### Scanning process Scanning process
----------------
The scanning process is done by euscan-update.sh. You should read carefully The scanning process is done by euscan-update.sh. You should read carefully
this script, and adapt it to your needs. For example it uses gparallel to this script, and adapt it to your needs. For example it uses gparallel to
@ -123,40 +133,36 @@ launch multiple process at a time, and you should adapt that to your number
of cpu and network bandwith. of cpu and network bandwith.
Once your euscan-update.sh is ok, just run it. Once your euscan-update.sh is ok, just run it.
::
$ sh euscan-update.sh $ sh euscan-update.sh
### Custom Django management commands Custom Django management commands
---------------------------------
euscan-www povides some new management commands, here is a short description euscan-www povides some new management commands, here is a short description
of these commands. Use "help" or read euscan-update.sh to get more informations. of these commands. Use "help" or read euscan-update.sh to get more informations.
#### list-packages list-packages
List packages stored in database.
List packages stored in database. scan-portage
Scan the portage tree and store new packages and versions in the database.
#### scan-portage scan-metadata.py
Scan metadata and looks for homepage, maintainers and herds.
Scan the portage tree and store new packages and versions in the database. scan-upstream
Scan upstream package. The prefered way to use this script it to first launch
euscan on some packages, store the result of the file, and feed this command with
the result.
#### scan-metadata.py update-counters
Update statistics and rrd files.
Scan metadata and looks for homepage, maintainers and herds. regen-rrds
If you deleted your rrd files, this script will use the database to
#### scan-upstream regen them.
Scan upstream package. The prefered way to use this script it to first launch
euscan on some packages, store the result of the file, and feed this command with
the result.
#### update-counters
Update statistics and rrd files.
#### regen-rrds
If you deleted your rrd files, this script will use the database to
regen them.
How does it work ? How does it work ?
================== ==================
@ -173,9 +179,12 @@ uses a file that describe what url and regexp to use to match packages.
euscan uses SRC_URI and tries to find the current version (or part of this version) euscan uses SRC_URI and tries to find the current version (or part of this version)
in the resolved SRC_URI and generate a regexp from that. in the resolved SRC_URI and generate a regexp from that.
For example for app-accessibility/dash-4.10.1, SRC_URI is: For example for app-accessibility/dash-4.10.1, SRC_URI is::
mirror://gnome/sources/dasher/4.10/dasher-4.10.1.tar.bz2 mirror://gnome/sources/dasher/4.10/dasher-4.10.1.tar.bz2
euscan will scan pages based on this template:
euscan will scan pages based on this template::
http://ftp.gnome.org/pub/gnome/sources/dasher/${0}.${1}/dasher-${PV}.tar.bz2 http://ftp.gnome.org/pub/gnome/sources/dasher/${0}.${1}/dasher-${PV}.tar.bz2
Then, from that, it will scan the top-most directory that doesn't depend on Then, from that, it will scan the top-most directory that doesn't depend on
@ -188,57 +197,49 @@ Like when scanning directories, a template of SRC_URI is built. Then euscan
generate next possible version numbers, and tries to download the url generated generate next possible version numbers, and tries to download the url generated
from the template and the new version number. from the template and the new version number.
For example, running euscan on portage/app-accessibility/festival-freebsoft-utils-0.6: For example, running euscan on portage/app-accessibility/festival-freebsoft-utils-0.6::
SRC_URI is 'http://www.freebsoft.org/pub/projects/festival-freebsoft-utils/festival-freebsoft-utils-0.6.tar.gz'
Template is http://www.freebsoft.org/pub/projects/festival-freebsoft-utils/festival-freebsoft-utils-${PV}.tar.gz SRC_URI is 'http://www.freebsoft.org/pub/projects/festival-freebsoft-utils/festival-freebsoft-utils-0.6.tar.gz'
Generate version from 0.6: 0.7, 0.8, 0.10, ... Template is http://www.freebsoft.org/pub/projects/festival-freebsoft-utils/festival-freebsoft-utils-${PV}.tar.gz
Try new urls: http://www.freebsoft.org/pub/projects/festival-freebsoft-utils/festival-freebsoft-utils-0.7.tar.gz, etc.. Generate version from 0.6: 0.7, 0.8, 0.10, ...
Try new urls: http://www.freebsoft.org/pub/projects/festival-freebsoft-utils/festival-freebsoft-utils-0.7.tar.gz, etc..
Blacklists Blacklists
---------- ----------
euscan uses blacklist for multiple purposes. euscan uses blacklist for multiple purposes.
### BLACKLIST_VERSIONS BLACKLIST_VERSIONS
For versions that should not be checked at all. sys-libs/libstdc++-v3-3.4
is good example because it's a package which version will always be 3.4
(Compatibility package for running binaries linked against a pre gcc 3.4 libstdc++).
For versions that should not be checked at all. sys-libs/libstdc++-v3-3.4 BLACKLIST_PACKAGES
is good example because it's a package which version will always be 3.4 Some packages are dead, but SRC_URI refers to sources that are still being
(Compatibility package for running binaries linked against a pre gcc 3.4 libstdc++). updated, for example: sys-kernel/xbox-sources that uses the same sources as
vanilla-sources but is not updated the same way.
SCANDIR_BLACKLIST_URLS
For urls that are not browsable. mirror://gentoo/ is a good example: it's
both stupid to scan it and very long/expensive.
### BLACKLIST_PACKAGES BRUTEFORCE_BLACKLIST_PACKAGES and BRUTEFORCE_BLACKLIST_URLS
Disable brute force on those packages and urls. Most of the time it's because
upstream is broken and will answer HTTP 200 even if the file doesn't exist.
Some packages are dead, but SRC_URI refers to sources that are still being ROBOTS_TXT_BLACKLIST_DOMAINS
updated, for example: sys-kernel/xbox-sources that uses the same sources as Don't respect robots.txt for these domains (sourcefourge, berlios, github.com).
vanilla-sources but is not updated the same way.
### SCANDIR_BLACKLIST_URLS
For urls that are not browsable. mirror://gentoo/ is a good example: it's
both stupid to scan it and very long/expensive.
### BRUTEFORCE_BLACKLIST_PACKAGES and BRUTEFORCE_BLACKLIST_URLS
Disable brute force on those packages and urls. Most of the time it's because
upstream is broken and will answer HTTP 200 even if the file doesn't exist.
### ROBOTS_TXT_BLACKLIST_DOMAINS
Don't respect robots.txt for these domains (sourcefourge, berlios, github.com).
Site handlers Site handlers
------------- -------------
### Pecl/PEAR Pecl/PEAR
A site handler that uses the Pecl/PEAR rest API
(http://pear.php.net/manual/en/core.rest.php).
A site handler that uses the Pecl/PEAR rest API Rubygems
(http://pear.php.net/manual/en/core.rest.php). This one uses rubygems's json API
(http://guides.rubygems.org/rubygems-org-api/)
### Rubygems PyPI
Uses PyPI's XML rpc API.
This one uses rubygems's json API
(http://guides.rubygems.org/rubygems-org-api/)
### Pypy
Uses pypy's XML rpc API.