diff --git a/README.rst b/README.rst index b6a5ead..f0a1cc3 100644 --- a/README.rst +++ b/README.rst @@ -2,7 +2,7 @@ What is euscan ? ================ dev-portage/euscan ------------------- +================== euscan is available in portage as a dev package (app-portage/euscan-9999). This tool allows to check if a given package/ebuild has new upstream versions @@ -17,56 +17,62 @@ to help euscan do its job more efficiently. euscan heuristics are described in the "How does-it works ?" section. -### Examples +Examples +-------- +:: -$ euscan amatch + $ euscan amatch - * dev-ruby/amatch-0.2.7 [gentoo] + * dev-ruby/amatch-0.2.7 [gentoo] -Ebuild: /home/euscan/local/usr/portage/dev-ruby/amatch/amatch-0.2.7.ebuild -Repository: gentoo -Homepage: http://flori.github.com/amatch/ -Description: Approximate Matching Extension for Ruby + Ebuild: /home/euscan/local/usr/portage/dev-ruby/amatch/amatch-0.2.7.ebuild + Repository: gentoo + Homepage: http://flori.github.com/amatch/ + Description: Approximate Matching Extension for Ruby - * SRC_URI is 'mirror://rubygems/amatch-0.2.7.gem' - * Using: http://rubygems.org/api/v1/versions/amatch.json + * SRC_URI is 'mirror://rubygems/amatch-0.2.7.gem' + * Using: http://rubygems.org/api/v1/versions/amatch.json -Upstream Version: 0.2.8 http://rubygems.org/gems/amatch-0.2.8.gem + Upstream Version: 0.2.8 http://rubygems.org/gems/amatch-0.2.8.gem -$ euscan rsyslog +:: - * app-admin/rsyslog-5.8.5 [gentoo] + $ euscan rsyslog -Ebuild: /home/euscan/local/usr/portage/app-admin/rsyslog/rsyslog-5.8.5.ebuild -Repository: gentoo -Homepage: http://www.rsyslog.com/ -Description: An enhanced multi-threaded syslogd with database support and more. + * app-admin/rsyslog-5.8.5 [gentoo] - * SRC_URI is 'http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.8.5.tar.gz' - * Scanning: http://www.rsyslog.com/files/download/rsyslog/rsyslog-${PV}.tar.gz - * Scanning: http://www.rsyslog.com/files/download/rsyslog - * Generating version from 5.8.5 - * Brute forcing: http://www.rsyslog.com/files/download/rsyslog/rsyslog-${PV}.tar.gz - * Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.8.6.tar.gz ... [ !! ] - * Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.8.7.tar.gz ... [ !! ] - * Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.8.8.tar.gz ... [ !! ] - * Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.0.tar.gz ... [ ok ] - * Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.10.0.tar.gz ... [ !! ] - * Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.11.0.tar.gz ... [ !! ] - * Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.1.tar.gz ... [ ok ] - * Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.2.tar.gz ... [ ok ] - * Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.3.tar.gz ... [ ok ] - * Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.12.0.tar.gz ... [ !! ] - * Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.4.tar.gz ... [ !! ] - * Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.5.tar.gz ... [ !! ] - * Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.6.tar.gz ... [ !! ] + Ebuild: /home/euscan/local/usr/portage/app-admin/rsyslog/rsyslog-5.8.5.ebuild + Repository: gentoo + Homepage: http://www.rsyslog.com/ + Description: An enhanced multi-threaded syslogd with database support and more. -Upstream Version: 5.9.1 http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.1.tar.gz -Upstream Version: 5.9.0 http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.0.tar.gz -Upstream Version: 5.9.3 http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.3.tar.gz -Upstream Version: 5.9.2 http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.2.tar.gz + * SRC_URI is 'http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.8.5.tar.gz' + * Scanning: http://www.rsyslog.com/files/download/rsyslog/rsyslog-${PV}.tar.gz + * Scanning: http://www.rsyslog.com/files/download/rsyslog + * Generating version from 5.8.5 + * Brute forcing: http://www.rsyslog.com/files/download/rsyslog/rsyslog-${PV}.tar.gz + * Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.8.6.tar.gz ... [ !! ] + * Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.8.7.tar.gz ... [ !! ] + * Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.8.8.tar.gz ... [ !! ] + * Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.0.tar.gz ... [ ok ] + * Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.10.0.tar.gz ... [ !! ] + * Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.11.0.tar.gz ... [ !! ] + * Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.1.tar.gz ... [ ok ] + * Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.2.tar.gz ... [ ok ] + * Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.3.tar.gz ... [ ok ] + * Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.12.0.tar.gz ... [ !! ] + * Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.4.tar.gz ... [ !! ] + * Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.5.tar.gz ... [ !! ] + * Trying: http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.6.tar.gz ... [ !! ] -### Hidden settings + Upstream Version: 5.9.1 http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.1.tar.gz + Upstream Version: 5.9.0 http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.0.tar.gz + Upstream Version: 5.9.3 http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.3.tar.gz + Upstream Version: 5.9.2 http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.9.2.tar.gz + + +Hidden settings +--------------- You can configure some settings using the command line, but the __init__.py file of the euscan package contains more settings, including blacklists and @@ -76,7 +82,7 @@ Maybe we should add the ability to use /etc/euscan.conf and ~/.config/euscan/euscan.conf to override these settings. euscan-www: euscan as a service -------------------------------- +=============================== euscan-www is a web application that aggregates euscan results. For example there is an instance of euscan-www that monitors gentoo-x86 + some official @@ -86,36 +92,40 @@ euscan-www uses django and provides some custom commands to feed the database. You can use euscan-www on you system tree, or preferably you can use a local tree to avoid messing with your system. -### Installation +Installation +------------ -Install requirements from PyPI using +Install requirements from PyPI using:: $ python setup.py develop Extra dependencies: - - * portage python api - * rrdtool[python] +* portage python api +* rrdtool[python] Like any django web app, just start by editing settings.py and then run these two commands. +:: $ python manage.py syncdb $ python manage.py migrate Now your instance is ready, you can just run this command to browse it. If you want to host it publicly you should use a real webserver. +:: $ python manage.py runserver -### Creating a local tree (optional) +Creating a local tree (optional) +-------------------------------- Create a local tree with all that portage (and layman would need). There is an example in euscanwww/scripts/local-tree/. See escan-update.sh to know what env variables you need to run any portage related command in this local tree. -### Scanning process +Scanning process +---------------- The scanning process is done by euscan-update.sh. You should read carefully this script, and adapt it to your needs. For example it uses gparallel to @@ -123,40 +133,36 @@ launch multiple process at a time, and you should adapt that to your number of cpu and network bandwith. Once your euscan-update.sh is ok, just run it. +:: $ sh euscan-update.sh -### Custom Django management commands +Custom Django management commands +--------------------------------- euscan-www povides some new management commands, here is a short description of these commands. Use "help" or read euscan-update.sh to get more informations. -#### list-packages +list-packages + List packages stored in database. -List packages stored in database. +scan-portage + Scan the portage tree and store new packages and versions in the database. -#### scan-portage +scan-metadata.py + Scan metadata and looks for homepage, maintainers and herds. -Scan the portage tree and store new packages and versions in the database. +scan-upstream + Scan upstream package. The prefered way to use this script it to first launch + euscan on some packages, store the result of the file, and feed this command with + the result. -#### scan-metadata.py +update-counters + Update statistics and rrd files. -Scan metadata and looks for homepage, maintainers and herds. - -#### scan-upstream - -Scan upstream package. The prefered way to use this script it to first launch -euscan on some packages, store the result of the file, and feed this command with -the result. - -#### update-counters - -Update statistics and rrd files. - -#### regen-rrds - -If you deleted your rrd files, this script will use the database to -regen them. +regen-rrds + If you deleted your rrd files, this script will use the database to + regen them. How does it work ? ================== @@ -173,9 +179,12 @@ uses a file that describe what url and regexp to use to match packages. euscan uses SRC_URI and tries to find the current version (or part of this version) in the resolved SRC_URI and generate a regexp from that. -For example for app-accessibility/dash-4.10.1, SRC_URI is: +For example for app-accessibility/dash-4.10.1, SRC_URI is:: + mirror://gnome/sources/dasher/4.10/dasher-4.10.1.tar.bz2 -euscan will scan pages based on this template: + +euscan will scan pages based on this template:: + http://ftp.gnome.org/pub/gnome/sources/dasher/${0}.${1}/dasher-${PV}.tar.bz2 Then, from that, it will scan the top-most directory that doesn't depend on @@ -188,57 +197,49 @@ Like when scanning directories, a template of SRC_URI is built. Then euscan generate next possible version numbers, and tries to download the url generated from the template and the new version number. -For example, running euscan on portage/app-accessibility/festival-freebsoft-utils-0.6: -SRC_URI is 'http://www.freebsoft.org/pub/projects/festival-freebsoft-utils/festival-freebsoft-utils-0.6.tar.gz' -Template is http://www.freebsoft.org/pub/projects/festival-freebsoft-utils/festival-freebsoft-utils-${PV}.tar.gz -Generate version from 0.6: 0.7, 0.8, 0.10, ... -Try new urls: http://www.freebsoft.org/pub/projects/festival-freebsoft-utils/festival-freebsoft-utils-0.7.tar.gz, etc.. +For example, running euscan on portage/app-accessibility/festival-freebsoft-utils-0.6:: + + SRC_URI is 'http://www.freebsoft.org/pub/projects/festival-freebsoft-utils/festival-freebsoft-utils-0.6.tar.gz' + Template is http://www.freebsoft.org/pub/projects/festival-freebsoft-utils/festival-freebsoft-utils-${PV}.tar.gz + Generate version from 0.6: 0.7, 0.8, 0.10, ... + Try new urls: http://www.freebsoft.org/pub/projects/festival-freebsoft-utils/festival-freebsoft-utils-0.7.tar.gz, etc.. Blacklists ---------- euscan uses blacklist for multiple purposes. -### BLACKLIST_VERSIONS +BLACKLIST_VERSIONS + For versions that should not be checked at all. sys-libs/libstdc++-v3-3.4 + is good example because it's a package which version will always be 3.4 + (Compatibility package for running binaries linked against a pre gcc 3.4 libstdc++). -For versions that should not be checked at all. sys-libs/libstdc++-v3-3.4 -is good example because it's a package which version will always be 3.4 -(Compatibility package for running binaries linked against a pre gcc 3.4 libstdc++). +BLACKLIST_PACKAGES + Some packages are dead, but SRC_URI refers to sources that are still being + updated, for example: sys-kernel/xbox-sources that uses the same sources as + vanilla-sources but is not updated the same way. +SCANDIR_BLACKLIST_URLS + For urls that are not browsable. mirror://gentoo/ is a good example: it's + both stupid to scan it and very long/expensive. -### BLACKLIST_PACKAGES +BRUTEFORCE_BLACKLIST_PACKAGES and BRUTEFORCE_BLACKLIST_URLS + Disable brute force on those packages and urls. Most of the time it's because + upstream is broken and will answer HTTP 200 even if the file doesn't exist. -Some packages are dead, but SRC_URI refers to sources that are still being -updated, for example: sys-kernel/xbox-sources that uses the same sources as -vanilla-sources but is not updated the same way. - -### SCANDIR_BLACKLIST_URLS - -For urls that are not browsable. mirror://gentoo/ is a good example: it's -both stupid to scan it and very long/expensive. - -### BRUTEFORCE_BLACKLIST_PACKAGES and BRUTEFORCE_BLACKLIST_URLS - -Disable brute force on those packages and urls. Most of the time it's because -upstream is broken and will answer HTTP 200 even if the file doesn't exist. - -### ROBOTS_TXT_BLACKLIST_DOMAINS - -Don't respect robots.txt for these domains (sourcefourge, berlios, github.com). +ROBOTS_TXT_BLACKLIST_DOMAINS + Don't respect robots.txt for these domains (sourcefourge, berlios, github.com). Site handlers ------------- -### Pecl/PEAR +Pecl/PEAR + A site handler that uses the Pecl/PEAR rest API + (http://pear.php.net/manual/en/core.rest.php). -A site handler that uses the Pecl/PEAR rest API -(http://pear.php.net/manual/en/core.rest.php). +Rubygems + This one uses rubygems's json API + (http://guides.rubygems.org/rubygems-org-api/) -### Rubygems - -This one uses rubygems's json API -(http://guides.rubygems.org/rubygems-org-api/) - -### Pypy - -Uses pypy's XML rpc API. +PyPI + Uses PyPI's XML rpc API. \ No newline at end of file