euscanwww: update style, prefetch related, ...

Signed-off-by: Corentin Chary <corentincj@iksaif.net>
This commit is contained in:
Corentin Chary 2012-03-05 14:46:34 +01:00
parent 1d65ee1984
commit 043197a3be
18 changed files with 126 additions and 60 deletions

90
README
View File

@ -4,9 +4,16 @@ What is euscan ?
dev-portage/euscan
------------------
euscan is available in portage as a dev package (app-portage/euscan-9999). This tool allow to check if a given package/ebuild has new upstream versions or not. It will use different heuristic to scan upstream and grab new versions and related urls.
euscan is available in portage as a dev package (app-portage/euscan-9999).
This tool allow to check if a given package/ebuild has new upstream versions
or not. It will use different heuristic to scan upstream and grab new versions
and related urls.
This tool was designed to mimic debian's uscan, but there is a major difference between the two: uscan uses a specific "watch" file that describe how it should scan packages, while euscan uses only what can already be found in ebuilds. Of course, we could later add some informations in metadata.xml to help euscan do its job more efficiently.
This tool was designed to mimic debian's uscan, but there is a major
difference between the two: uscan uses a specific "watch" file that describe
how it should scan packages, while euscan uses only what can already be found
in ebuilds. Of course, we could later add some informations in metadata.xml
to help euscan do its job more efficiently.
euscan heuristics are described in the "How does-it works ?" section.
@ -61,35 +68,50 @@ Upstream Version: 5.9.2 http://www.rsyslog.com/files/download/rsyslog/rsyslog-5.
### Hidden settings
You can configure some settings using the command line, but the __init__.py file of the euscan package contains more settings, including blacklists and default settings.
You can configure some settings using the command line, but the __init__.py
file of the euscan package contains more settings, including blacklists and
default settings.
Maybe we should add the ability to use /etc/euscan.conf and ~/.config/euscan/euscan.conf to override these settings.
Maybe we should add the ability to use /etc/euscan.conf and
~/.config/euscan/euscan.conf to override these settings.
euscan-www: euscan as a service
-------------------------------
euscan-www is a web application that aggregates euscan results. For example there is an instance of euscan-www that monitors gentoo-x86 + some official overlays currently hosted at http://euscan.iksaif.net/ .
euscan-www is a web application that aggregates euscan results. For example
there is an instance of euscan-www that monitors gentoo-x86 + some official
overlays currently hosted at http://euscan.iksaif.net/ .
euscan-www uses django and provides some custom commands to feed the database. You can use euscan-www on you system tree, or preferably you can use a local tree to avoid messing with your system.
euscan-www uses django and provides some custom commands to feed the database.
You can use euscan-www on you system tree, or preferably you can use a local
tree to avoid messing with your system.
### Installation
Like any django web app, just start by editing settings.py and then run these two commands.
Like any django web app, just start by editing settings.py and then run
these two commands.
$ python manage.py syncdb
$ python manage.py migrate
Now your instance is ready, you can just run this command to browse it. If you want to host it publicly you should use a real webserver.
Now your instance is ready, you can just run this command to browse it.
If you want to host it publicly you should use a real webserver.
$ python manage.py runserver
### Creating a local tree
### Creating a local tree (optional)
Create a local tree with all that portage (and layman would need). There is an example in euscanwww/scripts/local-tree/. See escan-update.sh to know what env variables you need to run any portage related command in this local tree.
Create a local tree with all that portage (and layman would need).
There is an example in euscanwww/scripts/local-tree/. See escan-update.sh
to know what env variables you need to run any portage related command in
this local tree.
### Scanning process
The scanning process is done by euscan-update.sh. You should read carefully this script, and adapt it to your needs. For example it uses gparallel to launch multiple process at a time, and you should adapt that to your number of cpu and network bandwith.
The scanning process is done by euscan-update.sh. You should read carefully
this script, and adapt it to your needs. For example it uses gparallel to
launch multiple process at a time, and you should adapt that to your number
of cpu and network bandwith.
Once your euscan-update.sh is ok, just run it.
@ -97,7 +119,8 @@ Once your euscan-update.sh is ok, just run it.
### Custom Django management commands
euscan-www povides some new management commands, here is a short description of these commands. Use "help" or read euscan-update.sh to get more informations.
euscan-www povides some new management commands, here is a short description
of these commands. Use "help" or read euscan-update.sh to get more informations.
#### list-packages
@ -113,7 +136,9 @@ Scan metadata and looks for homepage, maintainers and herds.
#### scan-upstream
Scan upstream package. The prefered way to use this script it to first launch euscan on some packages, store the result of the file, and feed this command with the result.
Scan upstream package. The prefered way to use this script it to first launch
euscan on some packages, store the result of the file, and feed this command with
the result.
#### update-counters
@ -121,31 +146,38 @@ Update statistics and rrd files.
#### regen-rrds
If you deleted your rrd files, this script will use the database to regen them.
If you deleted your rrd files, this script will use the database to
regen them.
How does it work ?
==================
euscan has different heuristics to scan upstream and provides multiple "handlers". First, here is a description of the generic handler.
euscan has different heuristics to scan upstream and provides multiple
"handlers". First, here is a description of the generic handler.
Scanning directories
--------------------
The first thing to do is to scan directories. It's also what uscan do, but it uses a file that describe what url and regexp to use to match packages.
The first thing to do is to scan directories. It's also what uscan do, but it
uses a file that describe what url and regexp to use to match packages.
euscan uses SRC_URI and tries to find the current version (or part of this version) in the resolved SRC_URI and generate a regexp from that.
euscan uses SRC_URI and tries to find the current version (or part of this version)
in the resolved SRC_URI and generate a regexp from that.
For example for app-accessibility/dash-4.10.1, SRC_URI is:
mirror://gnome/sources/dasher/4.10/dasher-4.10.1.tar.bz2
euscan will scan pages based on this template:
http://ftp.gnome.org/pub/gnome/sources/dasher/${0}.${1}/dasher-${PV}.tar.bz2
Then, from that, it will scan the top-most directory that doesn't depend on the version, and try to go deeper from here.
Then, from that, it will scan the top-most directory that doesn't depend on
the version, and try to go deeper from here.
Brute force
-----------
Like when scanning directories, a template of SRC_URI is built. Then euscan generate next possible version numbers, and tries to download the url generated from the template and the new version number.
Like when scanning directories, a template of SRC_URI is built. Then euscan
generate next possible version numbers, and tries to download the url generated
from the template and the new version number.
For example, running euscan on portage/app-accessibility/festival-freebsoft-utils-0.6:
SRC_URI is 'http://www.freebsoft.org/pub/projects/festival-freebsoft-utils/festival-freebsoft-utils-0.6.tar.gz'
@ -160,20 +192,26 @@ euscan uses blacklist for multiple purposes.
### BLACKLIST_VERSIONS
For versions that should not be checked at all. sys-libs/libstdc++-v3-3.4 is good example because it's a package which version will always be 3.4 (Compatibility package for running binaries linked against a pre gcc 3.4 libstdc++).
For versions that should not be checked at all. sys-libs/libstdc++-v3-3.4
is good example because it's a package which version will always be 3.4
(Compatibility package for running binaries linked against a pre gcc 3.4 libstdc++).
### BLACKLIST_PACKAGES
Some packages are dead, but SRC_URI refers to sources that are still being updated, for example: sys-kernel/xbox-sources that uses the same sources as vanilla-sources but is not updated the same way.
Some packages are dead, but SRC_URI refers to sources that are still being
updated, for example: sys-kernel/xbox-sources that uses the same sources as
vanilla-sources but is not updated the same way.
### SCANDIR_BLACKLIST_URLS
For urls that are not browsable. mirror://gentoo/ is a good example: it's both stupid to scan it and very long/expensive.
For urls that are not browsable. mirror://gentoo/ is a good example: it's
both stupid to scan it and very long/expensive.
### BRUTEFORCE_BLACKLIST_PACKAGES and BRUTEFORCE_BLACKLIST_URLS
Disable brute force on those packages and urls. Most of the time it's because upstream is broken and will answer HTTP 200 even if the file doesn't exist.
Disable brute force on those packages and urls. Most of the time it's because
upstream is broken and will answer HTTP 200 even if the file doesn't exist.
### ROBOTS_TXT_BLACKLIST_DOMAINS
@ -184,11 +222,13 @@ Site handlers
### Pecl/PEAR
A site handler that uses the Pecl/PEAR rest API (http://pear.php.net/manual/en/core.rest.php).
A site handler that uses the Pecl/PEAR rest API
(http://pear.php.net/manual/en/core.rest.php).
### Rubygems
This one uses rubygems's json API (http://guides.rubygems.org/rubygems-org-api/)
This one uses rubygems's json API
(http://guides.rubygems.org/rubygems-org-api/)
### Pypy

4
TODO
View File

@ -30,11 +30,13 @@ Site Handlers
euscanwww
---------
- quick install doc
- sort verions on package page
- export a sqlite database
- add last scan in the footer [0.2.0]
- add an /about/config page that describe the current config (overlays, stuff in make.conf, euscan default settings, etc..) [0.2.0]
- add a table for overlays, change euscan_versions [0.2.0]
- rss scan world + post ?
- sort verions on package page
### Commands [0.2.0]

View File

@ -115,7 +115,7 @@ class Command(BaseCommand):
package.n_packaged = n_packaged.get(package.id, 0)
package.n_overlay = n_overlay.get(package.id, 0)
default = {'id' : -1}
default = {'id' : None}
package.last_version_gentoo_id = last_versions_gentoo.get(package.id, default)['id']
package.last_version_overlay_id = last_versions_overlay.get(package.id, default)['id']
package.last_version_upstream_id = last_versions_upstream.get(package.id, default)['id']

View File

@ -2,14 +2,17 @@ from django import template
register = template.Library()
@register.inclusion_tag('euscan/_packages.html')
def packages(packages):
return { 'packages' : packages }
@register.inclusion_tag('euscan/_packages.html', takes_context=True)
def packages(context, packages):
context['packages'] = packages
return context
@register.inclusion_tag('euscan/_package_cols.html')
def package_cols(infos):
return { 'infos' : infos }
@register.inclusion_tag('euscan/_package_cols.html', takes_context=True)
def package_cols(context, infos):
context['infos'] = infos
return context
@register.inclusion_tag('euscan/_package_bar.html')
def package_bar(infos):
return { 'infos' : infos }
@register.inclusion_tag('euscan/_package_bar.html', takes_context=True)
def package_bar(context, infos):
context['infos'] = infos
return context

View File

@ -20,7 +20,12 @@ def index(request):
ctx['n_packages'] = Package.objects.count()
ctx['n_herds'] = Herd.objects.count()
ctx['n_maintainers'] = Maintainer.objects.count()
try:
ctx['last_scan'] = EuscanResult.objects.get(id=EuscanResult.objects.aggregate(Max('id'))['id__max']).datetime
except EuscanResult.DoesNotExist:
ctx['last_scan'] = None
return ctx
@render_to('euscan/logs.html')
@ -37,7 +42,8 @@ def categories(request):
@render_to('euscan/category.html')
def category(request, category):
packages = Package.objects.filter(category=category).select_related('last_version_gentoo', 'last_version_overlay', 'last_version_upstream')
packages = Package.objects.filter(category=category)
packages = packages.select_related('last_version_gentoo', 'last_version_overlay', 'last_version_upstream')
print dir(packages[0])
if not packages:
raise Http404
@ -56,6 +62,7 @@ def herds(request):
def herd(request, herd):
herd = get_object_or_404(Herd, herd=herd)
packages = Package.objects.filter(herds__id=herd.id)
packages = packages.select_related('last_version_gentoo', 'last_version_overlay', 'last_version_upstream')
return { 'herd' : herd, 'packages' : packages }
@render_to('euscan/maintainers.html')
@ -72,6 +79,7 @@ def maintainers(request):
def maintainer(request, maintainer_id):
maintainer = get_object_or_404(Maintainer, id=maintainer_id)
packages = Package.objects.filter(maintainers__id=maintainer.id)
packages = packages.select_related('last_version_gentoo', 'last_version_overlay', 'last_version_upstream')
return { 'maintainer' : maintainer, 'packages' : packages }
@render_to('euscan/overlays.html')

View File

@ -78,7 +78,7 @@ a:hover, a:active
.menu {
width:100%;
margin-top: 75px;
margin-top: 30px;
background: #8076A1;
color: #FFF;
}
@ -195,6 +195,7 @@ td
th
{
font-size: 0.8em;
text-align: center;
border-bottom: 3px solid;
}

BIN
euscanwww/media/favicon.ico Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 3.4 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 3.3 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 916 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 558 B

Binary file not shown.

After

Width:  |  Height:  |  Size: 751 B

View File

@ -42,7 +42,7 @@
<li>---</li>
{% block menu_feed %}
<li>
<img src="{{ MEDIA_URL }}/img/feed.png" alt="feed" />
<img src="{{ MEDIA_URL }}img/feed.png" alt="feed" />
<a title="Global Feed" href="{% url global_feed %}">Global Feed</a>
</li>
{% endblock %}

View File

@ -21,3 +21,15 @@
});
</script>
{% endblock %}
{% block menus %}
{{ block.super }}
<div class="menu">
<ul>
<li><img src="{{ MEDIA_URL }}img/gentoo-icon.png" /> Gentoo</li>
<li><img src="{{ MEDIA_URL }}img/overlay-icon.png" /> Overlays</li>
<li><img src="{{ MEDIA_URL }}img/upstream-icon.png" /> Upstream</li>
<li><img src="{{ MEDIA_URL }}img/freshness-icon.png" /> Freshness</li>
</ul>
</div>
{% endblock %}

View File

@ -6,13 +6,13 @@
<table id="table" class="display">
<thead>
<th>Package</th>
<th>Gentoo</th>
<th>Overlay</th>
<th>Upstream</th>
<th>Gentoo</th>
<th>Overlays</th>
<th>Unpackaged</th>
<th>Freshness</th>
<th><img src="{{ MEDIA_URL }}img/gentoo-icon.png" alt="gentoo" title="Last version in Gentoo" /></th>
<th><img src="{{ MEDIA_URL }}img/overlay-icon.png" alt="overlays" title="Last version in Overlays" /></th>
<th><img src="{{ MEDIA_URL }}img/upstream-icon.png" alt="upstream" title="Last version Upstream" /></th>
<th><img src="{{ MEDIA_URL }}img/gentoo-icon.png" alt="gentoo" title="Versions in Gentoo" /></th>
<th><img src="{{ MEDIA_URL }}img/overlay-icon.png" alt="overlays" title="Versions in Overlays" /></th>
<th><img src="{{ MEDIA_URL }}img/upstream-icon.png" alt="upstream" title="Versions only upstream" /></th>
<th><img src="{{ MEDIA_URL }}img/freshness-icon.png" title="Freshness" /></th>
</thead>
<tbody>
{% for package in packages %}

View File

@ -12,7 +12,7 @@
<li><a href="{% url api.views.categories 'json' %}">/api/1.0/categories.json</a></li>
<li>/api/1.0/packages/by-maintainer/&lt;maintainer-id&gt;.json</li>
<li>/api/1.0/packages/by-category/&lt;category&gt;.json</li>
<li>/api/1.0/packages/by-herds/&lt;herd&gt;.json</li>
<li>/api/1.0/packages/by-herd/&lt;herd&gt;.json</li>
<li>/api/1.0/package/&lt;category&gt;/&lt;package&gt;.json</li>
</ul>

View File

@ -12,10 +12,10 @@
<table id="table" class="display">
<thead>
<th>Category</th>
<th>Gentoo</th>
<th>Overlays</th>
<th>Unpackaged</th>
<th>Freshness</th>
<th><img src="{{ MEDIA_URL }}img/gentoo-icon.png" alt="gentoo" title="Versions in Gentoo" /></th>
<th><img src="{{ MEDIA_URL }}img/overlay-icon.png" alt="overlays" title="Versions in Overlays" /></th>
<th><img src="{{ MEDIA_URL }}img/upstream-icon.png" alt="upstream" title="Versions only upstream" /></th>
<th><img src="{{ MEDIA_URL }}img/freshness-icon.png" title="Freshness" /></th>
{% if request.GET.extras %}
<th>Graphs</th>
{% endif %}

View File

@ -11,10 +11,10 @@
<table id="table" class="display">
<thead>
<th>Herd</th>
<th>Gentoo</th>
<th>Overlays</th>
<th>Unpackaged</th>
<th>Freshness</th>
<th><img src="{{ MEDIA_URL }}img/gentoo-icon.png" alt="gentoo" title="Versions in Gentoo" /></th>
<th><img src="{{ MEDIA_URL }}img/overlay-icon.png" alt="overlays" title="Versions in Overlays" /></th>
<th><img src="{{ MEDIA_URL }}img/upstream-icon.png" alt="upstream" title="Versions only upstream" /></th>
<th><img src="{{ MEDIA_URL }}img/freshness-icon.png" title="Freshness" /></th>
{% if request.GET.extras %}
<th>Graphs</th>
{% endif %}

View File

@ -11,10 +11,10 @@
<table id="table" class="display">
<thead>
<th>Maintainer</th>
<th>Gentoo</th>
<th>Overlays</th>
<th>Unpackaged</th>
<th>Freshness</th>
<th><img src="{{ MEDIA_URL }}img/gentoo-icon.png" alt="gentoo" title="Versions in Gentoo" /></th>
<th><img src="{{ MEDIA_URL }}img/overlay-icon.png" alt="overlays" title="Versions in Overlays" /></th>
<th><img src="{{ MEDIA_URL }}img/upstream-icon.png" alt="upstream" title="Versions only upstream" /></th>
<th><img src="{{ MEDIA_URL }}img/freshness-icon.png" title="Freshness" /></th>
{% if request.GET.extras %}
<th>Graphs</th>
{% endif %}