James Antill

XML grep

2011-09-26T19:05:14Z

For a long time I've wanted to be able to do an "XML grep" on XML data, where I see just the nodes I care about in an XML file. After recently hitting this problem again, I found out about xmlstarlet and after a _lot_ of work I managed to get what I wanted. So I figured I'd write it down, for both of us:

xmlstarlet sel -I -t \
  -m '/updates/update' \
  -i 'pkglist/collection/package[@name="raydium"]' \
  -c . \
  /var/cache/yum/x86_64/15/updates/gen/updateinfo.xml

...I'll explain each line of the above:

{xmlstarlet sel -I -t} -- Runs the command, in "select" mode and does automatic indentation of the output.

{-m '/updates/update'} -- Where we are matching "from", comparing this to grep we are kind of saying that each node at <updates><update> is a line. This is an XPATH expression.

{-i 'pkglist/collection/package[@name="raydium"]'} -- The condition we want to match on, here we are saying that we want to match on anything of the form <updates><update><pkglist><collection><package name="raydium">. This is an XPATH expression (it's worth repeating the @ changes the match from the node to the attribute).

{-c .} -- This says display all the nodes from the start of our match #2, that pass our condition.

{ /var/cache/yum/x86_64/15/updates/gen/updateinfo.xml } -- The XML file to operate on (if it's missing it default to stdin.

A few things you might not know about RHEL-6.1+ yum

2011-05-19T18:02:04Z

Time to look at a few features of yum in RHEL-6.1 now that it's released

Search is more userfriendly

As we maintain yum we are always looking for the "minor" changes that can make a big difference to the user, and this is probably one of the biggest minor changes. As of late RHEL-5 and RHEL-6.0 "yum search" was great for finding obscure things that you knew something about but with 6.1 we've hopefully made it useful for finding the "everyday" packages you can't remember the exact name of. We did this by excluding a lot of the "extra" hits, when you get a large search result. For instance "yum search kvm manager" is pretty useless in RHEL-6.0, but in RHEL-6.1 you should find what you want very quickly.
Example commands:
```
yum search kvm manager
yum search python url
```
The updateinfo command

The "yum-security" or "yum-plugin-security" package has been around since early RHEL-5, but the RHEL-6.1 update has introduced the "updateinfo" command to make things a little easier to use, and you can now easily view installed security errata (to more easily make sure you are secure). We've also added a few new pieces of data to the RHEL updateinfo data. Probably the most significant is that as well as errata being marked "security" or not they are now tagged with their "severity". So you can automatically apply only "critical" security updates, for example.
Example commands:
```
yum updateinfo list security all
yum update-minimal --sec-severity=critical
```
The versionlock command

As with the previous point we've had "yum-plugin-version" for a long time, but now we've made it easier to use and put all it's functions under a single "versionlock" sub-command. You can now also "exclude" specific versions you don't want, instead of locking to known good specific ones you had tested.
Example commands:
```
# Lock to the version of yum currently installed.
yum versionlock add yum
# Opposite, disallow versions of yum currently available:
yum versionlock exclude yum

yum versionlock list
yum versionlock delete yum\*
yum versionlock clear

# This will show how many "excluded" packages are in each repo.
yum repolist -x .
```
Manage your own .repo variables

This is actually available in RHEL-6.0, but given that almost nobody knows about it I thought I'd share it here. You can put files in "/etc/yum/vars" and then use the names of those files are variables in any yum configuration, just like $basearch or $releasever. There is also a special $uuid variable, so you can track individual machines if you want to.
yum has it's own DB

Again, this something that was there in RHEL-6.0 but has improved (and is likely to improve more over time). The most noticeable addition is that we now store the "installed_by" and "changed_by" attributes, this could be worked out from "yum history" before, but now it's easily available directly from the installed package.
Example commands:
```
yumdb
yumdb info yum
yumdb set installonly keep kernel-2.6.32-71.7.1.el6
yumdb sync
```
Additional data in "yum history"

Again, this something that was there in RHEL-6.0 but has improved (and is likely to improve more over time). The most noticeable additions are that we now store the command line and we store a "transaction file" that you can use on other machines.
Example commands:
```
yum history
yum history pkgs yum
yum history summary

yum history undo last

yum history addon-info 1    config-main
yum history addon-info last saved_tx
```
"yum install" is now fully kickstart compatible

As of RHEL-6.0 there was one thing you could do in a kickstart package list that you couldn't do in "yum install" and that was to "remove" packages with "-package". As of the RHEL-6.1 yum you can do that, and we also added that functionality to upgrade/downgrade/remove. Apart from anything else, this should make it very easy to turn the kickstart package list into "yum shell" files (which can even be run in kickstart's %post).
Example commands:
```
 yum install 'config(postfix) >= 2.7.0'
 yum install MTA
 yum install '/usr/kerberos/sbin/*'
 yum -- install @books -javanotes
```
Easier to change yum configuration

We tended to get a lot of feature requests for a plugin to add a command line option so the user could change a single yum.conf variable, and we had to evaluate those requests for general distribution based on how much we thought all users would want/need them. With the RHEL-6.1 yum we created the --setopt so that any option can be changed easily, without having to create a specific bit of code. There were also some updates to the yum-config-manager command.
Example commands:
```
yum --setopt=alwaysprompt=false upgrade yum

yum-config-manager
yum-config-manager --enable myrepo
yum-config-manager --add-repo https://example.com/myrepo.repo
```
Working towards managing 10 machines easily

yum is the best way to manage a single machine, but it isn't quite as good at managing 10 identical machines. While the RHEL-6.1 yum still isn't great at this, we've made a few improvements that should help significantly. The biggest is probably the "load-ts" command, and the infrastructure around it, which allows you to easily create a transaction on one machine, test it, and then "deploy" it to a number of other machines. This is done with checking on the yum side that the machines started from the same place (via. rpmdb versions), so that you know you are doing the same operation.

Also worth noting is that we have added a plugin hook to the "package verify" operation, allowing things like "puppet" to hook into the verification process. A prototype of what that should allow those kinds of tools to do was written by Seth Vidal here.
Example commands:
```
# Find the current rpmdb version for this machine (available in RHEL-6.0)
yum version nogroups

# Completely re-image a machine, or dump it's "package image"
yum-debug-dump
yum-debug-restore 
    --install-latest
    --ignore-arch
    --filter-types=install,remove,update,downgrade

# This is the easiest way to get a transaction file without modifying the rpmdb
echo | yum update blah
ls ${TMPDIR:-/tmp}/yum_save_tx-* | sort | tail -1

# You can now load a transaction and/or see the previous transaction from the history
yum load-ts /tmp/yum_save_tx-2011-01-17-01-00ToIFXK.yumtx
yum -q history addon-info last saved_tx > my-yum-saved-tx.yumtx
```

Yum "update" from F-11 to F-13, with some help from distro-sync

2010-06-11T20:34:28Z

Here is my experience of updating a machine from F-11 to F-13, on 2010-05-30, with plain yum over ssh. I got all the info. from "yum history", and had a little help from "distro-sync" which is only in rawhide's yum atm.

The "simple" start.

First off I updated yum to the latest from rawhide "yum-3.2.27-13.fc14.noarch". Then I updated to the new release via. "yum --releasever=13 update fedora-release". Then I ran my first "yum distro-sync" to do the update, and hit depsolving problems.

First problem.

The first problem was that libgnomedb-1:3.99.7-3.fc11.x86_64 had to be removed, as it didn't exist anymore in F-13 and was depending on things that needed to be upgraded. Then I had a similar problem with a bunch fc8 packages, that were also depending on things that were going away:

    Erase        chkfontpath-1.10.1-2.fc8.x86_64
    Erase        fonts-arabic-2.1-2.fc8.noarch
    Erase        fonts-bengali-2.1.5-3.fc8.noarch
    Erase        fonts-gujarati-2.1.5-3.fc8.noarch
    Erase        fonts-hebrew-0.101-2.fc8.noarch
    Erase        fonts-hindi-2.1.5-3.fc8.noarch
    Erase        fonts-kannada-2.1.5-3.fc8.noarch
    Erase        fonts-malayalam-2.1.5-3.fc8.noarch
    Erase        fonts-oriya-2.1.5-3.fc8.noarch
    Erase        fonts-punjabi-2.1.5-3.fc8.noarch
    Erase        fonts-sinhala-0.2.2-3.fc8.noarch
    Erase        fonts-tamil-2.1.5-3.fc8.noarch
    Erase        fonts-telugu-2.1.5-3.fc8.noarch
    Erase        system-config-soundcard-2.0.6-11.fc8.noarch

Finally I had to erase cryptsetup-luks-1.0.6-7.fc11.i586, because that had stopped being multilib. and didn't have obsoletes. All of these were relatively painless, esp. with the new code in yum which tries to explain depsolving problems.

distro-sync

Then I ran distro-sync (excluding yum), and let it do it's thing. This is a new command in yum, and means you don't have to worry about bugs in NEVR packaging (because it will downgrade packages to the latest available). These are the packages it "fixed" for me:

    Downgrade    compat-db45-4.5.20-2.fc13.x86_64
    Downgraded               4.5.20-5.fc10.x86_64
    Downgrade    nss-softokn-freebl-3.12.4-19.fc13.x86_64
    Downgraded                      3.12.6-1.2.fc11.x86_64
    Downgrade    preupgrade-1.1.4-1.fc13.noarch
    Downgraded              1.1.5-1.fc11.noarch

The compat-db45 is esp. amusing, as that had probably been broken all through fc11.

Explaining: "Warning: RPMDB altered outside of yum."

2010-01-22T22:04:57Z

What does it mean?

The yum message "Warning: RPMDB altered outside of yum." or, as the yum message said for a few months, "Warning: RPMDB has been altered since the last yum transaction." means some application has altered the rpm database (installed or removed a package) without going through the Yum APIs. This is almost always due to someone using rpm directly (Ie. rpm -ivh blah.rpm), but another possibility is an application built on top of the rpm APIs (Ie. smart, apt, zypp). While it's possible that someone has hacked your machine and altered the rpmdb maliciously, it would have to be done poorly to trigger this warning.

Why has yum started to emit this warning?

There are three main sets of reasoning behind bringing this to the users attention.

New yum features require yum being "the" packaging API

There are now a few features in yum, requested by users of the package management system, that require yum is aware of all package actions on the system. Here a few of the current ones:

The most obvious example is "yum history", which records when packages were installed, when and by whom. If yum is not involved in installing/updating/removing/etc. some packages then a lot of the benefits of "yum history" are gone. For instance there is no useful audit trail anymore, you can't use "yum history list blah" and know you have all the instances where something happened to "blah".

Yum now has it's own database, for package information it wants to record but has no corresponding entry in the rpmdb, the obvious example is "the id of the repository that this package was installed from" but there are quite a few pieces of info. now.

Following on from the previous point, rpmdb versions are a significant feature for managing many machines by yum. They require information from the yumdb, so installing something via. yum on one machine but via. rpm on another would give the machines different "rpmdb versions".

This is not a complete list, and as more package management features are implemented they are much more likely to be implemented at the yum layer than at the rpm layer. Not because rpm is bad, but for the same reasons that the above features were implemented in yum, it's much easier and faster to implement them there.

Rpm is often abused, when used directly

We, the yum developers, often find that people using rpm directly only do so to solve problems in a way that just creates more problems (and often looks like yum is at fault). For instance using any of --force, --nodeps, --justdb is pretty much outright lying to yum, and is pretty much guaranteed to confuse yum vs. rpm. For instance over the last 3 months, over 10% of the bugs we've had against yum have been due to this kind of problem (and probably very close to 100% of bugs about "rpm doesn't like the depsolving solution yum did").

There is also the problem of users accidentally using "rpm --install" which is very different from what yum thinks of as an "install" operation. Which, again, just creates problems and even if yum works around these it is unlikely to do so in a way the user wished.

In addition to the problems it creates there are no known reasons to use rpm instead of yum. Over the last year or so yum has added features like the downgrade and reinstall commands (which obsolete using rpm --oldpackage) and are generally much easier for the user (in this case esp. combined with yum history undo/redo). Recent versions of yum will even allow you to do things like "yum install http://example.com/blah.rpm" which was the last use case where you could easily do something with rpm and not with yum (and even then we also changed yum so we could add the --releasever, which is a much easier solution to the common reason why people want http installs). But if you can think of something that is easier to do with rpm, we'll almost certainly add a feature to yum to solve that problem.

We do extra work, and so inform the user

We now do a non-trivial amount of work when we detect that something has happened to the rpmdb, including a full "yum check" run, so that yum can show problems as close to their source as possible. Just stopping yum for a significant amount of time, or outputting problem reports, with no context would not be a good way to interact with the user. We are also able to cache rpm data, and thus. have yum run faster, when yum is aware of all rpmdb operations (so again, we'd want some way to let the user know: yum is running slower, and this is why).

What can I do to make it stop?

The most obvious solution is the one we recommend: Only use yum, or tools that use yum APIs, to install/update/remove/etc. packages on your system. If you don't like yum, and prefer to use some other packaging system that's fine but we'd assume you'd want to use that system all the time (and thus. it doesn't matter what yum does or doesn't do, as you won't be using it).

If you think you really need to use yum some/most of the time but something else the rest of the time, then there is a way to currently turn all this checking off. In your yum.conf set "history_record = false", and yum will not record any history (the "yum history" command will be useless) but yum will also not be able to tell when the rpmdb has changed. This will break other features, like the yumdb, but it will stop the warnings.

Known problems

Old versions of the "remove-with-leaves" yum plugin called an API which would corrupt the state yum thought the rpmdb was in. Older versions of the "keys" plugin may also have been affected. This meant that the warning was produced all the time, and yum history would not be happy. Newer versions of yum fixed the APIs affected.

The akmod packages seem to work by calling rpm directly, and thus. operating without yum's knowledge, they need to be fixed.

Multiple problems with multiple versions

2009-10-09T13:42:21Z

I've now seen a few requests for yum which can be generalized as "work better with multiple versions of packages". So I decided to write this, instead of replying N times to all the different requests.

The simple feature: just do what rpm does

Since it's inception rpm differentiated between an "install" and an "update", allowing you to install X-1-1 and X-2-1 as long as neither package provided the same filename as the other with different content. At the rpm layer this is mostly an easy problem to solve, if the user wants to install instead of update you just check all the files in both packages and as long as there isn't a conflict you install the new package but don't remove the old one.

As I said this mostly works well, you can update some packages and install/remove the other. However one problem you'll quickly run into is that if you have both X-1-1 and X-2-1 installed, but these now need to be updated by X-1-2 and X-2-2. But there's no easy way to do this, as there's no good way to tell rpm "I update just X-1-* (or X-2-*) packages". This is a core problem for yum, if you have multiple versions of a package installed what does "yum update" do.

Kernel: The special case

The kernel is treated differently from every other package, mainly because you can't move to the new version without a reboot, the running version needs to be able to access it's modules and if you only have one version and it breaks the machine doesn't boot anymore. So yum has infrastructure setup to install multiple versions of packages, which is used/tested just by the kernel (although you can give that ability to any package).

However the kernel has a few positive features that are in it's favour. The two main ones being that each kernel package has files that are 100% unique to it and that no other package "uses" the data within the kernel package (so nothing needs to answer the question "which of the N versions available should I use").

This works fine for the kernel, and has an obvious answer to the question "what does yum update do" ... if there is a newer version of the package than the one installed, we install it, and if the installonly_limit has been reached for this package we remove the oldest one (that isn't the running kernel, and hasn't been manually marked to be kept in the yumdb).

This simple policy wouldn't work for anything other packages though (like the example above), so we either need to change this policy (and make sure the kernel package keeps working) or try something else...

Embed the version in the package name

This is the next obvious solution, and is used quite frequently. You still need to make sure neither package has files which conflict with the other, and there are usability problems to do with the fact the package X is now called X1 and/or X2. But this, at least, can work and allows you to have X1-1-1 be upgraded by X1-1-2 and X2-2-1 be upgraded by X2-2-2.

The problems with doing this usually stem from the fact the packages are not isolated, and they will often provide some of the same things (so even though their names are different, they can be referred to by a common provide name). This is esp. a problem with rpm autoprovides. It then becomes problematic to test, because instead of one test case (install package X) to see if it works, you now have three testcases (install X1, install X2 and install X1 and X2). This testing complexity rises as you have more versioned packages, X3 is 7, X4 is 15!

The good thing about this method (from a yum developers point of view) is that nothing needs to be done on the package manager side. Each package is completely independent, so if normal packages install then versioned packages install.

Taking a look an example, python2 and python3 packages

For a concrete example consider having python2 and python3 (as of 2009 that is desirable because of the backwards incompatibility snafu with python3), we can "easily" create a python3 package that is installable alongside the current default of python (== python-2.6/python-2.7/whatever). There are some auto-provides, but we could either remove/change them on the python3 package or make sure nothing else is using them. Users can install both and run either /usr/bin/python or /usr/bin/python3, all seems good.

If this was enough, you can now rest ... until the first time some random package decides it'd like to use python3 instead of python, and you get a huge number of complaints that users now have two versions of python wasting disk/bandwidth/etc.

However core python is very rarely enough, for any GUI application you'll need python3 bindings to gtk and for anything a bit more complex (like yum) you'll need bindings to lots of C APIs (rpm, xml, gpg and curl to name a few of yum's requirements). This means you've gone to a lot of work for something that isn't very useful, or you have to create versioned packages for a significant part of the packages that touch python ... and then you have to test that (which, remember, is 3 times the work).

And you have to maintain that, testing updates and doing security errata (3x the work), and deal with bugs/complaints from people saying that package FOO is only available in version A and they need it in the other version ... or logging bugs against python-blah instead of python-blah3. Of course humans are lazy, so to get around doing some of this work someone will decide it's a good idea to have a single python-blah package that works with either/both versions of python ... and then you'll have more problems.

And then at the end you still have the usability problems where the user installs "python-blah" but really wanted to install "python3-blah" not understanding why things don't work.

Lies, damn lies, and benchmarks

2009-07-31T19:11:15Z

Or why I hate "quick benchmarks"...

Recently I've started to see a lot more of what I'd call "quick benchmarks", often it's to do with yum but mainly I think that's because those tend to get sent to me by multiple methods. So I thought I'd try and write down why I react so negatively/dismissively to them, how people can spot the underlying problems that annoy me and even better some advise on how you can go about doing some real benchmarks if that kind of thing interests you (but it's much more work than quick benchmarks).

The summary of the problem is that quick software benchmarking often involves taking a huge amount of differences between two applications and have a single number result. Then you compare just the numbers, and come to a conclusion. So X gets 3 and Y gets 5 for problem ABCD ... therefore Y is 66% worse than X at ABCD. Except that might be a highly misleading (or worse) conclusion, for a number of reasons:

Benchmarking Bias

This is often the route cause of all problems in quick benchmarks, and because of it's pervasiveness in human nature I would say that a significant portion of the work in doing a good benchmark is making sure you've tried to reduce the effect of your own bias. The bias happens in many forms, from the simple fact that when testing X and Y you might have a deeper understanding of one and so test it's strengths; make less configuration errors; or even assume favourable results are correct but unfavourable ones are incorrect.

An old example

Probably the first experience I had of this was at the end of the 1990s, I can find a link to the original "discussions" now but I can relate the main points. Linux had supported SMP (running on 2 or more CPUs at once) for a couple of years and FreeBSD was just starting to look at it seriously. While doing the Linux work the kernel developers had created a simple benchmark of rebuilding the kernel with different "make -j" configurations (controlling how parallel make), probably because it's simple tests a bunch of things at once and affects kernel developers personally. Obviously it was natural for the FreeBSD developers to do the same kind of thing, when they were doing the same kinds of work. I found what I think is the original FreeBSD report that someone posted on this FreeBSD SMP - kernel compilation bench.

Note that both of the above benchmarks were meant to be tested against themselves (ie. the developer would run the benchmark, make some changes and then run the benchmark again on the new kernel with the changes and see the difference). And, again, they were great for that ... pretty much all developers have similar kinds of test runs that they run. However someone then decided to take "the result for Linux" and compare it to "the result for FreeBSD", and came to the conclusion that because the FreeBSD number was smaller than the Linux number then that meant "FreeBSD was already faster at SMP than Linux". This conclusion was repeated by a number of knowledgable FreeBSD developers before a Linux developer pointed out the obvious fact that the tests didn't just have differenet kernels, but different source trees were being compiled with different compilation toolchains.

Again, I don't think the FreeBSD developers intentionally meant to create or use bogus data just that given some random numbers implying that FreeBSD was better than Linux they were biased to believe them and so didn't think about it too much.

A couple of yum examples

I've seen a lot of people test "yum makecache" vs. "apt-get update" or "smart update" or "zypper refresh". This makes some sense from the point of view of an apt developer/user because this is an operation that the user has to run before any set of operations to make sure the database is upto date, so any improvement (relative to older versions of your self) is a significant win. However a yum user is unlikely to ever run this command because the default mode of operation is for yum to manage database synchronization.

Another common problem is to compare something like "apt-cache search" / "apt-file search" against "yum search" / "yum provides", the problem here is that apt-cache / apt-files just do a simple grep on the cache of available packages ... yum does it's searches against what yum can see use (so, for example, versionlock'ing a package will affect the results you get in yum ... as will installing packages).

These problems should be obvious after even a moments thought about how yum is used, but again most of the people publishing this kind of results either don't use yum or are expecting it to be slower and so don't think about results which confirm that expectation.

Measuring the slow thing

Many developers are aware that their applications have a "fast path" and a "slow path", which normally align with "unexpected cases work, but are slow" and "expected, normal, cases work quickly". Benchmarkers often do not know these distinctions. I've seen numerous cases where someone benchmarks something that wouldn't be done in real life. Alas. this is hard to spot for the normal user, and is one of the reasons that it's wise to speak with the developers of whatever you are benchmarking.

Benchmarking 1 point and then concluding about N

The obvious yum example

By far the most common problem here is people run a simple benchmark like "time (echo n | yum update)" compared to the same operation in apt or zypp and then concluding X is x% faster at updating than Y. The problem here is that updating actually has a number of operations (breaking them down roughly):

read repo. configuration, cmd line options etc.
check if repo. metadata is current.
download repo. metadata if not current.
merge configured repo. metadata into "available".
read current rpm metadata.
depsolve updates+obsoletes+etc. from rpm and repo. metadata.
output changes.
confirm changes.
download updates.
install updates.

However even though there are 10 operations above, the "simple update benchmark" only really tests "6. depsolve ...". Which is fine on it's own, it's worth knowing what the depsolve time is (and I run this benchmark myself with yum to find out that info.) but depsolve time is not the same as update time. In fact most of the time it's in the noise for the update operation as a whole (obviously saying your update takes 25% of the time is much more fun than saying it takes 98% of the time).

I've even seen problems here where they'll test apt on Debian vs. yum on Fedora, which much like the FreeBSD vs. Linux test changes so many different things that you can't conclude anything about just the package managers. There have also been cases where someone tested "apt upgrade" vs. "yum upgrade" but the yum operation also download repo. metadata, as it wasn't current/available.

Benchmarking N points and then concluding about 1

The FreeBSD vs. Linux point

As I said above the technical problem with the FreeBSD make kernel vs. Linux make kernel problem was that it was testing 3 different sets of things (buildtools, source code, kernel) and drawing a conclusion about only 1 of them.

Phoronix has many examples here

Phoronix has many examples of all the different problems you can get into here. From things like their "apache benchmark", which turned out to have nothing to do with Apache on either platform they tested it on, and mp3 encoding "filesystem benchmarks". To their usual "conclusions" which take 10-20 completely separate points, with completely separate error rates, and come up with an "average".

The obvious yum example

The usual way this happens in package management "benchmarking" is that a random number of commands will be tested, like install; update; remove; whatprovides; search; list ... and then a final score will be given. This completely ignores the fact some of those commands are being run much more often than others, and in different situations.

Standard deviation is always significant

The simple way to think of this is that multiple runs of anything you test need to be performed, and if you don't have results where the answer is "X and Y perform the same" (even though the absolute numbers are never going to be identical) ... the benchmarking has probably screwed up the standard deviation.

Phoronix has many examples here

This is often combined with the previous problem, if you test X vs. Y and get the results of 1000 and 1200 then there's a huge difference between a SD (standard deviation) of 500 and an SD of 1 on those results. And this is esp. important when you have another result which is 10 vs. 20 and an SD that could be 0.1 or 5.

The obligatory yum point

I never see benchmarks of package management operations that include a standard deviation value, and this can be esp. important when different values and/or different configurations can affect the performance significantly.

What to do, if you want to create a real benchmark

All of the above is not to try to discourage real benchmarking, as this is a time consuming and unrewarding task (IMO), but often bad benchmarking is worse than none at all. If you aren't up for that then feel free to suggest usecases that seem difficult/slow/etc., even if you need to explain it as "currently I run X like this, and it takes N time, how do I do something similar with Y in a similar amount of time" ... this is not the same thing, because the answer might well be "run Y like this instead". But if you are up for the challenge then...

As I said earlier, if you are benchmarking X and Y and comparing them then you need to be very familiar with both X and Y, which means investing a significant amount of time working with both X and Y. This very likely means that you want to speak to developers of both X and Y, both about the measurements you are trying to take and the results you are getting. I am always very suspicious if the benchmarking seems to be one sided (in that more knowledge was available about X and Y, at the time of testing), or more to the point I'm very suspicious if you are benchmarking yum but I've no idea who you are.

Also, even if you create a good benchmark which shows X is 999x faster than Y for operation FOO, and FOO is a useful feature that people will want to take advantage of. It is likely that the developers for Y will be able to change Y (sometimes trivially, as it might be an assumed edge case or regression noone had hit, yet) which reduces the difference significantly. This also often means that doing benchmarks properly "just" makes both X and Y better, with the published results showing they are within 5% or whatever and so "nobody cares" about the benchmarks. Which makes running the benchmarks pretty unrewarding, but hey you want to be the benchmarker not me... :)

BugZilla feature of the year -- recent (commented) history

2009-04-29T22:38:09Z

Something I've wanted for a long time is "show me the BugZilla tickets which I've looked at "recently" (last day, week, etc.). Like almost everyone else I speak to I get by saving all my bugzilla email and then doing seartches in evolution.

However today ajaxxx found/used advanced bugzilla search options to give a page which has the recent most history of comments you've done. This isn't quite the same as being able to see everything that you've looked at, but it's pretty close.

The url and the saved search

The search works due to two "advanced features: 1. Selecting yourself as the commentator: <field0-1-0=commenter>, <type0-1-0=equals>, and <value0-1-0=%user%> and 2. Selecting comment changes that happened after 8 days ago <field0-0-0=longdesc>, <type0-0-0=changedafter>, and <value0-0-0=8d>. The "8d" part can obviously be changed to reflect different history. To experience the feature a full working URL is: here

I also created a saved search called Commented in last 8 days. You may also want to change your column layout to include the changed time (alas. I don't know of any way to create a custom view for just a single search, *sigh*).

Bonus: search for bad attachment types

Another problem I hit a lot is that python code tracebacks in plain text, but users often leave the "crash" as the default application/octet-stream ... which means it can't be viewed. Really I'd like BugZilla to notice this, and fix it. But after seeing the above search I created: YUM* weird attachments, which searches for any BZ (belonging to the yum group) that has an attachment of type application-octet/stream that isn't obsoleted. Then it's a simple matter of manually fixing the problem bugs.

Update: 2009-04-30 --

Thanks to mcepl, I've now fixed the "shared" links so the should work for everyone != me :).

Why trusted third party repos. will always be a bad idea

2009-04-03T19:04:20Z

Why not make third party repos. first class

Every now and again, someone takes a look at apt/yum/zypper/smart/PK/whatever and decides that although they have support for third party repos. it's "too annoying" for third parties to get users or for users to use them and so this is a problem which needs to be fixed. Another way this is presented is that the package managers should support "One Click Install". I will hopefully explain (once and for all) why this isn't a problem, and what third parties can do to get what they actually want (to make their users lives easier).

This is not a new problem

A long time ago now, I used Debian-2.2 on my desktop+server and it was good. But then the desktop seemed old and luckily for me Ximian came out with a GNOME desktop for Debian-2.2. So trusting that Ximian had tested everything, and worked to get the latest nice desktop bits into my stable Debian-2.2 I added their repo. and upgraded. I noted they they were replacing large parts of the desktop, and not just adding packages, but this was pretty much what I expected so it didn't bother me. And life was good, I now had a stable system and a new desktop.

Then came the point where I wanted to "distro. upgrade" from Debian-2.2 to Debian-3.0, so I did what everyone does "apt-get update && apt-get dist-upgrade" and I expected that the Ximian stuff was probably going to get lost, as this is what happened on all Red Hat CD updates I'd ever done. What actually happened though was apt-get tried to resolve dependencies for a while, and then said it couldn't do it and gave up. Life was not good.

The core problem is distributed database synchronization

The core problem is that "package management" is actually "database management", where moving from pkgA-1 to pkgA-2 is more about database synchronization than anything else. So when you add a "third party repo." you now have "distributed database management/synchronization". In simple terms. this means that you can test that Debian-X to Debian-Y works, or Fedora-X to Fedora-Y but this testing will not apply to Debian-X1 or Fedora-X1, and while you could expand testing to cover those *-X1 cases (at great cost) noone knows how to make it work for all of -X1, -X2, -X3, ... -XN.

Ignoring the DB we still have the trust problem

Even if we could magically solve the distributed database problem there are significant problems with splitting your database beyond a certain point. At the inevitiable endgame, let's say that all the "large" applications have their own repository (openoffice, evolution, gnome, firefox, kde, apache-httpd, postgresql, mysql, etc. etc.) Now whenever you want to update your view of this distributed database you have to contact N different repositories instead of the 2 you have for Fedora. Making this usable for 10 repos. is a significant amount of work, making it usable for 100 or even 1,000 repos. is a huge amount of work.

Also the quality control of the packages that can get onto your system is now distributed, because the quality of any package on the system is the minimum of that applied on all the repositories you have available. This is also true of the reliability/security, so instead of a single point of failure for most users you have N single points of failure.

But what about "simple" packages and "semi-trusted third party repos"

One solution to the giant distributed database problem is to have "trusted third party repos." not participate in the database. They say something like "I need at least LSB-blah" but have no other dependencies. In theory this is workable, but to do this someone has to write a lot of code in all the packager managers that handle these semi-trusted repos. And even after these code changes to sandbox the packages from the semi-trusted repos. you still have a significant portion of the trust problems to do with managing a lot of repos. and dealing with the network etc.

But IMO the most damning problem with this approach is the number of uses it could be put to, because the packages within these semi-trusted repos. will have much less features than first class packages used in trusted repos. This means that each application in these repos. would have to have it's own copy of everything outside of LSB, so you might end up with 10 copies of FOO until it gets into the next LSB. And the semi-trusted repo. would have to deal with security updates for all of those things itself (and you have to trust it to keep doing so, in a timely manner).

What about if I just ignore all of the above

Even if you ignored or solved all of the above problems, a core point remains that you need a chain of trust starting from your main distribution. This could be Fedora installing the *-release file for the repo. you want, or some large amount of new code to do basically the same thing. The problem is that Fedora doesn't want to provide that chain of trust, for legal and other non-technical reasons, so you are back at the same place you've started from.

So why all the code to support multiple repos.

There are a number of cases where multiple repos. work well, and so is a useful feature to have. However in these cases the extra repos. should rarely be classed as "third party". For instance Fedora has a "release" repo. and an "updates" repo., to cut down on metadata, and an "updates-testing" repo. so users can easily turn that on or off. RHEL also contains many extension repos. for specific applications like clustering, or updated MySQL ... but again any problems here can be solved by changes to the "main" repo. and these specified sets of repos. are tested together.

Something that isn't obviuosly an extension repo. is the rpmfusion repos. for Fedora. But in reality they try and act as much like an extension repo., for US legally problematic packages, as possible. For instance while they are controlled outside of Fedora (and so, from the Fedora infrastructure POV are third party and untrusted) some of the people who control them are part of the Fedora community and so do the integration testing and can help fix any problems caused by using them with Fedora. They also have similar package review guidelines to Fedora, so the quality aspect is maintained as much as possible. However even with these constraints there are still some problems due to the database being distributed and not synchronized.

It's also somewhat common for specific companies/groups to internally have repos., either for custom builds of distro. applications or addon applications they wrote/bought. However these act exactly like extension repos. in that someone is charged with doing all the integration testing and they have the trust problem solved due to it being controlled by them. These repos. can also easily be integrated into the installation, so they don't have the problems people are trying to work around with third party repos.

Possible solution for third parties

This does imply a possible solution for random third party repo. providers that want this problem solved, pool your resources and join the upstream community (to help with integration). An obvious choice is to join the work the rpmfusion community is doing (and also join the Fedora community). This way 100 or 1,000 third party package providers could all provide automatic updates etc. but with some implied level of QA/trust/etc. so that users could tell the good third party from the bad. This would significantly improve the user experience of getting packages from that third party, helping both sides. Another useful property of this is that nothing has to change in any of the package managers.

A quick explanation of package sizes in yum and rpm

2008-09-02T21:54:20Z

It's pretty common to think that a specific thing always has a specific size, and people tend to think of an "rpm package" as a single object thus. the it's common to ask what is "the size of an rpm". However if you have a 1MB text file, and gzip compresses it to 50KB which you then upload to a HTTP server you now have at least 3 different sizes: text size; compressed size and upload size (includes HTTP headers etc.) and asking for the size. So it is with rpm packages, and their many sizes.

The three common sizes of an rpm package

I'm going to use the yum package object notation to explain the common different sizes, as those are the easiest to see/use (see my previous post on how to simply get package objects from yum):

An .rpm file package has pkg.archivesize (rpm TAG RPMTAG_ARCHIVESIZE), pkg.installedsize (rpm TAG RPMTAG_SIZE) and (for yum available package objects) pkg.packagesize (a stat of the .rpm file).
Installed rpm packages have pkg.archivesize (rpm TAG RPMTAG_ARCHIVESIZE) and pkg.packagesize (rpm TAG RPMTAG_SIZE). pkg.installedsize doesn't exist, because an installed package can't be installed.

How are those values calculated?

rpm calculates the RPMTAG_SIZE value as the simple summation of the sizes of all the files within the rpm. rpm calculates the RPMTAG_ARCHIVESIZE value as the value of the cpio archive within the rpm, after decompression. These values are often very close. As I said above the pkg.packagesize value for .rpm files is just that value returned by stat(2).

So, what is pkg.size and why does it change?

Within yum there is a pkg.size, which maps to pkg.packagesize, which is used in all UI code that just wants to know "the size" of a package. This value has the property that the value you get for "need to download X bytes to install" and "will free up X bytes on removal" is correct, which is what most people want to see most of the time. However using this value does mean that the "the size of the rpm package" changes after you install an rpm, so it can be confusing if you try and compare pkg.size values between installed and available packages (either via. yum info, or looking at the values presented when installing a new kernel and removing an old one, say).

So, what's the best way to compare the size of an rpm package?

As you can see above, archivesize is the same for an available package and an installed one. So if you install the yum-lsit-data plugin, you can use "yum --showduplicates info-archive-sizes <pkgs>". Also you can always compare with just the available packages (and not installed ones), for instance "repoquery --show-duplicates --qf '%{nevra} %{archivesize} %{size}\n' <pkgs>"

Programming with Yum in 5 minutes, or so

2008-08-30T06:12:03Z

There are a lot of lines of code in yum, and it can be somewhat intimidating at first glace. However a significant amount of effort has been made to make simple things easy, and the hard things not so hard. The start of any code using yum, will almost certainly have these four lines (and always the first and last one :).

    1 #! /usr/bin/python -tt
    2 
    3 import os
    4 import sys
    5 import yum

Those lines just tell python, you'd you'd like to be able to use the yum code, and some stuff for the OS. Next the first bit of real code, and something which is also in almost every piece of code using yum:

    7 yb = yum.YumBase()

This creates a yum instance, that you can work with. Then one more piece, that is very useful:

    9 yb.conf.cache = os.geteuid() != 0

This just tells the yum instance not to try and update any of it's data, as the caller of the script probably hasn't got the permissions to do so.

Now we have a real yum object that we can do things with, the three most useful parts to access are pkgSack, rpmdb and repos. The first two basically act the same, but rpmdb performs queries based on the installed packages on the local machine and pkgSack performs them against all the enabled (normally remote) repositories. The repos attribute is almost always used for one of three things, calling repos.enableRepo(), repos.disableRepo() and less often repos.listEnabled(). The latter for if you need to set/override some specific configuration for the repos.

The pkgSack and rpmdb attributes have a fairly large number of functions you can call, most of which return "package objects" these are the main things you work with most in yum code. Probably the most useful functions to get those package objects are: searchNevra(), returnPackages() and searchPrimaryFields(). There are also some optimized varients like, searchNames() and returnNewestByNameArch(). Some examples would be:

Simple version of "yum list" command

   11 # Get the repository package objects matching the passed arguments
   12 pkgs = yb.pkgSack.returnNewestByNameArch(patterns=sys.argv[1:])
   13 
   14 for pkg in pkgs:
   15     print "%s: %s" % (pkg, pkg.summary)

Simple stats. gathering from installed packages

   17 # Find the ten biggest installed packages
   18 pkgs = yb.rpmdb.returnPackages()
   19 pkgs.sort(key=lambda x: x.size, reverse=True)
   20 print "Top ten installed packages:"
   21 done = set()
   22 for pkg in pkgs:
   23     if pkg.name in done:
   24         continue
   25     done.add(pkg.name)
   26     print "%s: %sMB" % (pkg, pkg.size / (1024 * 1024))
   27     if len(done) >= 10:
   28         break

Slightly more advanced topics

After playing with yum code for a little bit, you'll probably experience a function which you might think would return a "package object" but doesn't returning a tuple of data instead. The two common tuples within yum are the "package tuple" and the "dependency tuple" which are:</b>

   30 # Pacakge tuple:
   31 (pkg.name, pkg.arch, pkg.epoch, pkg.version, pkg.release)
   32 # Dependency (or the Provides/Requires/Conflicts/Obsoletes (PRCO)) tuple:
   33 (pkg.name, 'EQ', (pkg.epoch, pkg.version, pkg.release))

After that you'll probably start playing with the "transaction info" (yb.tsInfo), and the "install", "update" and "remove" functions of YumBase. So you can change the system as well as query it. Then you'll want to present your information in a way that looks as good as a normal yum command (using the "internal" output module), although that is less unsupported. A somewhat useful example might be:

"Clever" way to manually update almost all the metadata for any usable repos. installed

    1 #! /usr/bin/python -tt
    2 
    3 import os
    4 import sys
    5 import yum
    6 
    7 yb = yum.YumBase()
    8 
    9 yb.conf.cache = os.geteuid() != 0
   10 
   11 from urlgrabber.progress import TextMeter
   12 
   13 # Use the "internal" output mode of yum's cli
   14 sys.path.insert(0, '/usr/share/yum-cli')
   15 import output
   16 
   17 # Try not to be annoying if run from cron etc.
   18 if sys.stdout.isatty():
   19     yb.repos.setProgressBar(TextMeter(fo=sys.stdout))
   20     yb.repos.callback = output.CacheProgressCallback()
   21     yumout = output.YumOutput()
   22     freport = ( yumout.failureReport, (), {} )
   23     yb.repos.setFailureCallback( freport )
   24 
   25 # Enable all the repos. a user might want to use and sync. the metadata.
   26 # Note this needs to be done before the repositories are used.
   27 for name in ('updates-testing', 'rawhide', 'livna', 'adobe-linux-i386',
   28              'brew', 'rhts', 'koji-static'):
   29     yb.repos.enableRepo(name + ',')
   30 for repo in yb.repos.listEnabled():
   31     yb.repos.enableRepo(repo.id + '-source'    + ',')
   32     yb.repos.enableRepo(repo.id + '-debuginfo' + ',')
   33 yb.repos.doSetup()
   34 for repo in yb.repos.listEnabled():
   35     repo.mdpolicy        = 'group:main'
   36     repo.metadata_expire = 0
   37     repo.repoXML
   38 # This is somehwat "magic", it unpacks the metadata making it usable.
   39 yb.repos.populateSack(mdtype='metadata', cacheonly=1)
   40 yb.repos.populateSack(mdtype='filelists', cacheonly=1)

Hopefully that will go some way to giving you an overview of how you can use the yum API to perform queries or tasks that would otherwise be very difficult. For further information you can use the help feature of ipython to look at docstrings for the variuos components, and even use the TAB complete feature of ipython to see all the available attributes of packages, repos, pkgSack or the YumBase itself.

Abusing the depsolver for fun/Sudoku

2008-08-25T03:58:19Z

After seeing the article on sudoku in apt/dpkg, I naturally thought "I wonder how you can do that in rpm". The big differences (from what I read/understood of the article) being that rpm doesn't mind if two packages with the same virtual provide are installed at once where dpkg does (and their "game" relies on that).

The natural solution seemed to be to use provides and conflicts, although I didn't think of other forms for long before just trying to implement that one :). The basic premise is to have 9 * 9 * 9 packages, each providing cell_1.1.1 through cell_9.9.9 (which is the row, column and value) and cell_value_1.1 through cell_value_9.9 (just the row and column). Then require all 82 cell_values, with any specific cells. You then just add conflicts in each package obeying the rules for block, row and column. Simple.

I was pretty sure yum would fail at solving the sudoku puzzles, because it basically requires that when you get multiple answers to a provides question (what package provides cell_1.1) you need to repeatedly narrow it down based on future information. This is kind of on the longer term. road map for yum so that it'll get better results, in certain edge cases, for real packages in real repositories.

Read on for scripts/repos. and results (spoiler, yum does fail and smart succeeds).

I've made the scripts and repositories for the game and puzzles available. So it's as simple as possible to try it out.

The puzzle_complete_* packages are complete puzles (all 82 cells prefilled, and the "game" included). The puzzle_(medium|hard)_* packages are just random games I copied from gnome-sudoku on medium/hard setting. Note that you can't actually do the "install" as the packages don't exist in anything other than metadata, but that's probably a good thing.

Results are:

yum-3.2.18 just picks some "random" values for the missing cells, and then complains about the resulting conflicts.
apt-0.5.15lorg3.94-3.fc9.x86_64 seems to decide a bunch of the cells are bad, and gives an "unmet dependencies" error.
smart-0:0.52-54.fc9.x86_64 solves the medium puzzles! And it hasn't given up on one of the non-guessing hard puzzles yet, although it's been over 20 minutes.

Update: smart solves hard_1 (no guessing required) and hard_2 (which requires guessing), taking only 3-4 minutes for hard_2 although hard_1 takes it just over 30 minutes. Interesting.

Not putting all your pkgs in one repo. (yum edition)

2008-08-13T01:43:58Z

As you create packages for private use, the question will eventually come up "where do I put these". The choice is obvious for the first package, just create a repo. (using createrepo -d) and distribute the my.repo file. However as you create more packages the answer should expand to having multiple repos. Different package managers have a different ideas about what should go inside a single repository, and the corollary to that is if you take the "best practice" for some other package manager and apply it to yum the results might not be all they could be. This posting will try and lay out the best practises for yum (3.2.z), and some of the reasons for them.

Probably the most common problem I see is people who think "yum-priorities" and/or "yum-versionlock" is a good solution to their problem, it is not. Another warning sign is the use of exclude and/or includepkgs, which are just a faster version of the above. While those plugins adds certain features that make certain things possible, needing those features is always a sign that you've got the separation of packages into repositories wrong (or something is just plain broken).

One of the least worst problems is they are not scalable, the basic implementation gets all the "good" packages and then excludes all the "bad" packages. However the real problems start due to the shortcuts they take so they can be faster, for instance if pkgA depends on pkgB and only pkgB is a "good" / "bad" package then pkgA will now either having a missing dependency or will end up running with something it wasn't built with/for. As more packages are added to the repositories it's more likely problems will arise.

The correct solution to problems that suggest yum-priorities, having packages that are an extension of an upstream and packages that override others in the upstream, is to split your one repository into more than one. Here is a quick list of reasons you'd want more than one repository:

debuginfo:
Debuginfo packages are large, and rarely needed. Also the yum-utils command, debuginfo-install, will automatically install them correctly if they are in a repo. with the same base repoid but with "-debuginfo" appended.
source:
Source packages also tend to be large, and also rarely needed. In this case the yum-utils command, yumdownloader --source, will automatically download them correctly if they are in a repo. with the same base repoid but with "-source" appended.
Binary architectures:
Putting .i386 and .ppc packages in the same repository is a little more convenient on the server side, but it means every action on the client side has to filter out all the packages of the wrong architecture. Also, it means you have almost no control over multilib issues on the client (.i386 and .x86_64 on a x86_64 machine, for example).
stability or freshness:
While yum can install old packages, and move to newer packages that aren't the latest versions, it's support for this is much weaker than the more expected case of installing or moving to the latest available package (for instance dependencies of installed packages will currently always move to the latest available). So it makes sense to have myrepo-release, myrepo-stable, myrepo-rc, myrepo-testing, myrepo-development type repositories.

Yum's ability to allow you to easily temporarily add a repository with --enablerepo=myrepo-testing (or use yum-aliases to create a synonym) or even use yum-tmprepo, makes this a much more viable alternative than it otherwise would be. As this means you can have several repositories that are configured to be disabled but then you temporarily enable them (to install a single package etc.).
enhances or overrides:
Sometimes you want extra packages because they are not available elsewhere and sometimes you want custom "upstream" packages (for specific local patches etc.). You should not mix these two types of packages, this also makes it easier to answer questions like "Do our extension packages rely on any of our custom upstream packages".

One of the main things that isn't listed above is "security updates" vs. "bugfix updates", this type of update is often split in other package managers. However in yum you should just create (or generate) updateinfo.xml data for the repository, and then use the yum-security plugin to select security updates (or fixes for specific CVE/BZs/etc.)

Understanding groups in yum

2008-08-08T00:15:26Z

Every now and again someone will will ask a question about groups in yum that amounts to:

If I do "groupinstall xyz" and then I do "groupremove xyz" why do I not end up where I started.

The main problem here is thinking of "groups" as objects that are installed and/or removed by yum, in fact currently the only way yum stores data on your computer is:

via. it's caches of network data (can be removed at any time)
via. it's log file (can be removed at any time)
via. it's transaction log (can be removed at any time, although yum won't be able to recover from transaction errors)
via. rpm

...and given that rpm only installs pacakges and that doesn't include extra package data like which "groups" the package is in, it's easy to realize that yum cannot perform the groupinstall/groupremove operations using that model (even if that seems like the "correct" thing to do).

What really happens then?

The simple way of thinking about it is that each group is a collection of package names, and on a groupinstall/groupremove yum collects all the package names in the group and tries to install each one (or remove each one). This works exactly as if you had run groupinfo, put all the names in a text file and run that through the "yum shell" command, there is very little magic in how this works and is a simple model (once you forget the "obvious" model as above). One way of thinking about it is that groups are more like "tags" in del.icio.us or livejournal etc.

There is a little more complication in that each group actually has four lists of package names, but groupinfo also displays the different lists in the groups so again the model is the same as the text file example.

So as you might expect from the above: if you have x, y and z installed; then groupinstall "foo" which contains a, b and y; then groupremove "foo" -- you'll end up with x and z.

But what about if we just add some magic?

The next question people ask (usually without fully understanding the above) is something like ok so groupremove will remove things I had installed before a groupinstall ... but if I do "groupinstall GRP1 GRP2" and then "groupremove GRP1" it should be easy to just keep any packages in GRP1 and GRP2, no?

And this does sound easy to implement, just only remove files that are only in GRP1. Except that packages can be in any number of groups. So consider the first example again, the question implicitly assumes that "x" getting remove should be based on whether "x" happens to be in another group or not. This little bit of magic in yum would then become very magic for the user, and it would be very hard to tell what a groupremove command is going to do.

But, but, usability! Do what the users expect!

In my opinion most of the problem here is with the way applications present the concept of "group" to the user, including yum itself (although noone would like the new command name for the cmd line client). GUI package management applications that use yum group definitions shouldn't present the user with an option to "install group X" instead the operation should be presented to the user as "install/remove all packages in group X" with the option to (de)select only parts of the list.

The other thing to do is for people to fix their groups so that "common" applications aren't in weird groups, so that groupremove on those groups is a useful operation again.

Cool, groups are interesting but I can't control the groups from Fedora/etc.

Actually you can, the full list of groups in yum is taken from all the enabled repositories. This means that if you create an empty repository with a group file in it, you can create your own groups! Just like tagging your own URLs in del.icio.us

Understanding why YUM seems "slow", and some instant solutions

2008-05-21T07:39:18Z

The previous entry was more of a general explanation about YUM speed and that if used in it's normal environment how current YUM is often more than fast enough, this entry is going to be a companion to it but focus on specific things that might make YUM appear slow but would better be described as using it suboptimally.

Note that I'm still not going to directly compare to other tools as I'm not as familiar with them and, as I said in the previous article, the tools are designed so differently that they don't lend themselves to comparisons. Also, as I also said before, if something is fine if it takes less than 10 seconds if two tools take 6 seconds and 4 seconds it doesn't really matter if yum is the faster (again, see the previous post, this should not be the end goal IMO).

Benchmarking "makecache":
If you are benchmarking the makecache command in yum, you have utterly failed. This command should be entirely a network test. If you have repos. that you care about which are shipping with only .xml data and not also .sqlite data (generated from: createrepo -d) then complain to whoever is generating the data. Nothing YUM can do will make this faster than 0 seconds (it also tends to be smaller, and so comes down the network sooner).
Metadata checks slowing you down:

Yes, YUM will automatically check that your metadata is current every 90 minutes, you can alter this using the metadata_expire configuration variable but the better thing to do is run some non-interactive process which will do the check every hour (yum-updatesd can be configured to just do this, but "yum list updates" in a cron job will also work).
One big repo. hurts performance:

As I've said YUM is mostly tested with Fedora/CentOS/etc. all of which use $basearch variables to have different architecture packages in different places, and also to have the set of three repositories "foo, foo-source and foo-debuginfo" instead of a single "foo" repository. This not only means that YUM doesn't have to inspect the metadata for the source packages in normal operations but that it doesn't even need to download the metadata for them over the network. The YUM related tools, like yumdownloader --source and debuginfo-install will enable the relevant repos. if you keep to the above naming convention.
Some exotic features are just painful:

There are still some corner cases, for instance using the "includepkgs" configuration option still requires that YUM load all of the repo. data into memory whereas using "exclude" configuration option only requires inspecting the metadata for packages which match. This may get better over time, but is something to be aware of if performance is not what you'd hoped for.
Using a newer YUM in Fedora 7/8:

If you want to try the latest YUM in Fedora 7 or 8, for the speed gains or for the extra features, you can do the following:
```
% yum install pygpgme
% yum install --enablerepo=development yum yum-utils python-urlgrabber
```
...you should still review what the second command above wants to do, just to make sure it doesn't want to install 100s of packages in the later case, however I've used the above on multiple machines, on multiple architectures and on Fedora 7 and 8 ... without any problems.

YUM resource usage, an accurate assessment

2008-05-21T06:58:33Z

As a YUM developer it's hard not to notice the number of complaints/attacks on YUM about it being slow and/or using too much memory, the most common thing about almost all of these complaints though is how inaccurate they are. To be clear, YUM did used to take a lot more time than was required to do simple operations and there are very likely some more improvements that can be made for certain situations BUT yum is currently "not slow" at most normal operations for Fedora users and RHEL/CentOS users should be getting a much newer/faster variant than they currently have in the very near future.

The most common misleading piece of data is to directly compare YUM to APT, smart or Zypper. The most obvious problem is that the commands/interfaces do not directly map between applications, and so hard to compare. For instance, at the simplest level, "yum install" accepts package names, "provides" or filenames contained within a package all of which can have wildcards and/or some of epoch/version/release/arch. But the problem goes even deeper than that, as the core assumptions each application makes are very different and so hard to compare. For instance, yum will automatically update it's metadata for the configured repositories but smart and apt both require the user does this manually. Or that yum/rpm is assumed to be used in an environment that has dependencies which use a combination of package names, versioned PRCO (Provides, Requires, Conflicts and Obsoletes) and explicit file dependencies.

In short it's fair to say that each package management tool has a close relationship with the main distribution(s) it is developed for, if for no other reason than that's where the developers tend to come from but also in more than one instance in the last 6 months specific cases of how new YUM features would work were defined by Fedora and not just the YUM developers.

Which brings us to YUM resources usage within Fedora/etc. which, as I said initially, in older versions has been a real and valid complaint against YUM. However the situation has been much improved over the last year. The use cases for which current YUM can still be "significantly" slower than we'd like is vanishingly small. Obviously Fedora 7 and 8, as well as RHEL/CentOS 5 will be slower to pickup the speed enhancements that have been made since 3.2.8, however this is not a failing of YUM more a consequence of the longer release periods associated with those distributions.

As a final note I'd say that the biggest reason a lot of these recent complaints annoy me is not just that they are inaccurate but that they tend to grossly mislead the reader into thinking that "package management" is a solved problem and the biggest obstacle to solve is whether "install foo" takes 3 seconds or 6 seconds. In my opinion this could not be further from the truth, managing 2-6 machines is an ugly problem in all the current solutions and managing 100-10,000 is basically not done. Then there are things like the 10x10 problem, where we are only starting to see ideas like KOPERS which might help solve the problem (but will require changes in the way package management is assumed to work). And that's completely ignoring the problems that have had attempted solutions (that failed) multiple times, like "rollback" support.

Python uses too much memory?

2008-05-21T05:25:48Z

There's a common attack leveled against Python applications that they take up too much memory, by people who understand the language difference (against, say, C) and by people just looking at their process list. This is often esp. evident on the newer x86_64 computers.

So as the Fedora Python maintainer, and a yum developer (an application written mostly using python), I figured it was probably worth investigating what the difference really was.

First I wrote a simple program which just created new yum.YumBase() objects and appended them to a list (numbers got from parsing /proc/self/status) which gave the following results:

.x86_64     0 peak 219.90MB size 219.90MB rss  13.30MB
.x86_64     1 peak 219.90MB size 219.90MB rss  13.33MB
.x86_64 90001 peak 610.46MB size 610.46MB rss 403.75MB

.i386       0 peak  20.65MB size  20.65MB rss   9.61MB
.i386       1 peak  20.65MB size  20.65MB rss   9.63MB
.i386   90001 peak 212.77MB size 212.77MB rss 201.82MB

...which seems pretty damning of python on .x86_64 and/or yum, 2x for RSS and much more for VSZ (10x to start with above, which is obviously a lot). So then I added a "pmap" call right at the end, to find out where that allocated memory was going, the most interesting pieces of data being:

0000000000601000 449696K rw---    [ anon ]
[...]
00002aaaaab5a000  76136K r----  /usr/lib/locale/locale-archive
[...]
00002aaaafa8d000     20K r-x--  /usr/lib64/python2.5/lib-dynload/stropmodule.so
00002aaaafa92000   2044K -----  /usr/lib64/python2.5/lib-dynload/stropmodule.so
00002aaaafc91000      8K rw---  /usr/lib64/python2.5/lib-dynload/stropmodule.so

...on .x86_64, and taking single shared object as an example vs on .i386:

00c58000             16K r-x--  /usr/lib/python2.5/lib-dynload/stropmodule.so
00c5c000              8K rwx--  /usr/lib/python2.5/lib-dynload/stropmodule.so
[...]
09290000         222296K rwx--    [ anon ]
[...]
b7d23000           2048K r----  /usr/lib/locale/locale-archive

...as you can see the shared library has a 2MB hole in the middle of it, which is counted towards it's VSZ even though it is not writable, executable or readable (and so I'd assume is not using any real memory). This basically means that VSZ is worthless on .x86_64, and is just even more worthless for python programs because they tend to load more shared objects.

The locale archive being 38 times bigger is explained by these lines from glibc/locale/loadarchive.c:

      /* Map an initial window probably large enough to cover the header
         and the first locale's data.  With a large address space, we can
         just map the whole file and be sure everything is covered.  */

      mapsize = (sizeof (void *) > 4 ? archive_stat.st_size
                 : MIN (archive_stat.st_size, ARCHIVE_MAPPING_WINDOW));

      result = __mmap64 (NULL, mapsize, PROT_READ, MAP_FILE|MAP_COPY, fd, 0);

...which means any program that uses the C locale functions gets an extra ~73MB of VSZ at startup on .x86_64.

The next interesting part of the data from pmap is that there are roughly 24 "anonymous" mappings for .x86_64 and only 20 for .i386, a little investigation shows that glibc is again the reason as the default value for M_MMAP_THRESHOLD (basically when glibc creates new entries for data, instead of reusing old ones) doesn't expand with size_t/time_t/etc. (which are twice as big). You can see this by setting MALLOC_MMAP_MAX_=0 in the environment, before running your application and that will produce the same number of "anonymous" mappings on x86_64.

And after taking into account all of the above, which is completely the domain of glibc and not python, the memory numbers add up as "simple doubling" as you go from 4 byte size_t/time_t/intptr_t/etc. to 8 bytes for the same.

I called it, Obama for democratic nominee/president

2008-01-08T00:26:18Z

Feel free to pick me up and jiggle me about before asking other questions obviously answered by crappy human nature:

Obama to win over Hillary, March 1st, 2007 05:07 pm

The myth of the "simple String API", that isn't really needed

2007-10-04T05:12:55Z

I saw three things recently which just confirmed that noone learns anything:

The "we don't need a real string API" on the GIT mailing list, followed by almost a month now of struggling to implement another one (and wishing they had some of the ustr features).
GLibc's proposed fix for C/POSIX not having a usable string API. Yes, it's the old let's all pretend IO (FILE *) is a good string abstraction ... you need to add, search/etc, and then add some more? Well multiple copies were always good fun, pretty much guaranteeing people will do a bunch of workarounds if they use it at all.
The Linux kernel is again looking for yet another almost a worthwhile string API, pretending it'll only be useful for this specific IO type ... because seq_printf() wasn't unique enough.

Best quote by far, IMO (from the GIT thread):

"Please, no. Let's not pull in a dependency for something as simple as a string library." -- Kristian Høgsberg

As a minor good note, ustr is in Fedora and so pretty much everyone in Fedora land has access to a usable string API with a single -l ... and the newer versions of libsemanage (SELinux) use ustr. I'm not going to hold my breath for mass sanity to occur though.

Software packaging, and the 10 × 10 problem

2007-06-20T06:16:59Z

First off, I hate all current software packaging for Linux. It's one step up from manually downloading tarballs, which I was doing 10 years ago, and isn't even a real superset of that functionality. Yes, it's better than doing that and yes it's better than what Windows has to offer. But it's still crap, and it annoys me every day, however it seems to be one of those problems that noone seems to want to fix properly and yet they keep thinking up stupid bandaids to work around the fact it doesn't work well. Specifically the major circular argument that is what I've come to think of as the "10 × 10 problem"…

Repeatedly over the last 5+ years I've tried to tell people about how we should be solving this problem, and I've mostly got dismissed … so here it is for posterity.

One of the currently major problems with software packaging and distribution is updating it. All software has bugs, or missing features and those affect different people in different ways. Say you have two people, one who is writting a new web application in python using postgresql and one who is just browsing the web, sharing some files via. the web server and running his own blog. It probably seems obvious to most sane people that the updates these two people want will vary wildly for the five packages: firefox, python, ipython, postgresql, apache-httpd. That is, unless you are in the business of distributing software packages.

With software packaging/release/whatever the arugment is most often about who gets screwed over least, in the above example a common "solution" is to do very minor updates for firefox and maybe ipython but nothing else … want firefox-2, or the latest postgresql/python? Too bad. Wait for the next "major" release and get those with everything else. A common painful package is the Linux kernel, mainly just because everyone uses it, and so all the software packagers argue about how often it should/shouldn't be updated. This is roughly the same as arguing what single human language would be best for the entire OS, if everyone should use python or C to develop all applications or what font everyone should use for all text. There is no correct answer, because the entire question is completely insane.

When I first thought about this, I was visiting/speaking with quite a few different Linux customers and I came up with the general idea of the "10 × 10 problem". The general statement of the problem is that you have 10 customers, each of which have 10 different "changes" they want for the next release. The current "solution" to the problem involves picking a number between 0 and 100, doing that amount of changes and giving the result to everyone so that you have a very small number of "releases" which can be managed by hand. This rarely makes anyone happy, and is basically what Microsoft; Sun and SCO have done for the last 10 years.

Obviously in the real world things are more complicated than the above, one of the most common differences is that each of the 10 customers has changes that are important enough they don't want to wait for "the next release" and they'll also want some of the changes you've done for the other people but also have desires like "apart those changes, I don't want anything else to change". This also destroys any hope of there ever being a magic number of changes you can do, that will actually make everyone happy. And yet, still we get arguments about what the best magic number is.

The "solution" to this problem is to admit defeat and stop managing software packaging and distribution like it's 1995, imagine for a moment that each "customer" had their own private software packaging and distribution team and used that instead of paying someone outside to do the work for them. In this senario it's very likely that all the high priority changes would be done immediately, and that all 10 changes would make it into "their" next release with no other changes. Now imagine that all those private teams communicated openly with each other, this would result in a similar outcome with the added benefit that significant changes done by one team that are desired by others would be merged into their releases.

The common complaint about why this "can't be done" by an external entity is that managing so many releases by hand is "hard" and would cost too much money. In my so humble opinion this is a bit like saying that managing more than one version of a C source file is hard and time consuming, as against implementing some decent SCM tools so that you can manage 100s of 1,000s of different version sets. The next common complaint is that the entire OS needs to be tested as one unit, which is mostly untrue and esp. so when you are talking about changes the average customer desires … and again, any truth in this is mostly a lack of decent tools.

The reason this could never have happened over the last 20 years is that the code itself was hidden from the customers, so they had no choice except to take what they were given by their software packaging and distribution company. This is not the case now, and I've already started to see the emergance of people implementing the good solution privately because they can't buy it from anyone. I'm just waiting for someone to give people what they want, and charge for it ... and I hope I work for the company that does it.

As a final optimistic note, I have helped the real solution come a tiny bit closer to reality. With Fedora 7 you can now install the yum-security package, and do things like "yum --security update -y" to get just security updates or "yum --bz 1234 update -y" to get just the updates which fix bugzilla.redhat.com#1234.

Fedora Kernel update causes: mkrootdev expected fs options

2007-04-24T00:19:06Z

I'm mainly dropping this somewhere so other people can find it, I recently did a kernel update and when I rebooted I got:

mkrootdev expected fs options
mount: missing mount point
setuproot: moving /dev/failed no such file or directory
setuproot: error mounting /proc
setuproot: error mounting /sys
switchroot: mount failed No such file or directory
kernel panic

...it turns out this was because I'd somehow got two / entries in /etc/fstab, as I found from this posting. The only thing you need to be aware of is that you also have a copy of /etc/fstab in the initrd, so after fixing /etc/fstab you'll want to re-run the postinstall scriptlet (rpm -q --scripts kernel), Ie. /sbin/new-kernel-pkg --package kernel --mkinitrd --depmod --install <kernel version>

lighttpd's "AIO sendfile"

2006-11-14T22:01:22Z

I keep track of a couple of the other web servers out there, even though I don't think much of them ... for various reasons. But when I saw that lighttpd was advertising AIO sendfile for their 1.5.x I was pretty impressed, while I hadn't been following it closely I didn't think the Linux AIO sendfile code had gone in yet (I'd been planning on trying to use the splice() API with IO helpers to do the same thing ... but I keep not getting around to it).

Alas. looking deeper into it, the release note is very misleading. They are actually doing AIO reads into allocated memory, and then using non-AIO sendfile() on the fd associated with the memory. Of course, this means:

It copies the entire file into memory (in sections of half a MB).
It requires copying an entire file section before anything is sent.
There is no sharing! So if you have 10,000 connections requesting the same 2MB file you get 5GB of memory used.
The real page cache readahead is going to have to be that much more intelligent to do the right thing.
There is no guarantee that the memory isn't swapped! So you can still have all the blocking artifacts of normal non-AIO sendfile().

...basically this is a lot of work to reimplement the page cache, badly.

I also seriously question whether the same "improvement" could have been got by just over allocating on the process to CPU mapping (something that I understand lighttpd can do as well as and-httpd). It also implies that posix_fadvise() should be doing more work (it is supposed to solve this exact problem), but maybe sprinkling explicit readhead() calls would also work around the problem (without screwing everyone else on the box).

Also why lighttpd uses sendfile() instead of just write() for the last piece, I'm not sure ... it's like doing a mmap() and then calling sendfile() on the mmap'd fd.

Re(tar)ddit.com, newegg and others ignoring HTTP/HTML

2006-10-21T05:24:14Z

So I tried reddit.com about a year ago, mainly due to the whole Y combinator thing. It seemed like it might be cute, but at the time didn't seem to contain any decent content on the front page and it required JavaScript to do anything interactive. So, I figured I'd come back a bit later when they've had a chance to polish it and take another look.

As you might guess, it's still JS only and they almost seem proud of it. Like they get more Web-2.0 juice or something. Hello 1995 called to let you know links are possible without JS. They even have "buttons" that you can attach to your site, so people can vote up/down your content easily ... but the "button" is just a script tag. Yeh, so not only do I apparently want to leave all my non-JS enabled agents out in the cold ... I want to actively mix-in JS from a third party? It's like the blind being led by the retards.

I could almost understand, the "JS is almost everywhere" so we'll just ignore everyone who wants security/stability/speed kind of argument for reddit.com ... if the rest of their usage of HTTP/HTML didn't seem like it was written by a 13yr old. For instance http://reddit.com/static/reddit.js is their "main" blob of imported JS functions. Now, given it says static in the URL, you might imagine that this library code would be heavily optimized with all the latest HTTP bits possible (in fact I was surprised there wasn't a version/date in the URL). But no, there's no Expires/Cache-Control headers. At least it has an ETag and does Content-Encoding (although lighttpd is broken and ignores it for HEAD requests).

But, reddit.com is still somewhat easy to dismiss as crack smokin' 13yr olds who still need a few years to grow up. After all it's not a real business, and probably doesn't make worthwhile amounts of money. But, then, newegg.com does ... and there almost everything works without JS, the notable exception being actually doing the sale. Sure, let me look at stuff and even do searches with a secure browser ... but when I need to type in my credit card, that's the time to rely on JS to do 1998 style forms.

Unsurprisingly amazon.com, as the largest online retailer, have consistently had the best UI for non-JS user-agents. And even though tags, as their latest tweak, started off as JS only then moved to working perfectly without JS within a couple of weeks. But, I guess maybe it's got less 2.0 juice now.

HTTP for desktop applications

2006-02-07T23:41:11Z

So to continue the HTTP theme, Miguel's idea that desktop applications communicate via. HTTP seems completely insane. I can only presume Miguel was not involved at all in the Mono HTTP API implementations. As part of writing my webserver, I've already written about how terrible the HTTP spec. is ... hell apache-httpd blatantly ignores significant parts of it.

It's not like there's even a hope that by "HTTP" Miguel meant "METHOD <path> <version>\r\nHeaders: value\r\n\r\n" ... much like people often do when they talk about "XML" but really mean "something in readable text deliniated by angle brackets". Because he want's to be able to "reuse" existing implementations. So, here's my explanations of the list of "benefits" to using HTTP:

HTTP is a well known protocol: HTTP is a well mis-understood protocol, it has a huge array of parsing pitfalls. And that doesn't even include the number of things people screw up, that you should work around if you want to be nice. Basically all HTTPDs have had security bugs in their HTTP parsers, and-httpd is a big exception here ... and even then I'd never suggest I have implemented HTTP bug free.
There are plenty of HTTP client and server implementations that can be employed: There are plenty of broken client/server libraries, and there isn't even a well defined way to communicate "here's a list of stuff" without resorting to complete insanity like XMLRPC or WebDAV. Also hooking anything into more than one server basically means writing CGI (or, possibly FastCGI) or doing multiple implementations.
The protocol can be as simple or as complex as needed by the applications. From passing all the arguments on a REST header as arguments to the tool-aware SOAP: Riiight. Show me a single HTTP client that does SOAP requests which degrade to REST. You basically have to pick one, so you go from clients needing TCP+MYPROTO to TCP+HTTP+REST+MYPROTO ... and this is better? ... for something 99.999% of the time is going to be machine local?
HTTP works well with large volumes of data and large numbers of clients: hahahaha. Apache dies easily with small numbers of clients, and has only got LFS support in 2.2.0.
Scaling HTTP is a well understood problem: Maybe ... but it's not easy. And the scaling currently is all to do with "browser HTTP", Ie. GET is basically the only method used ... if GNOME introduces random methods for different things in the desktop most of the current scaling knowledge goes --> that way.
Users can start with a REST API, but can easily move to SOAP if they need to: This is a repeat of problem/benefit #3 ... again implying that having random crap at the end of the HTTP transport is any more useful than having a MS word file containing a single element with base64 encoded data in it.
HTTP includes format negotiations: clients can request a number of different formats and the server can send the best possible match, it is already part of the protocol. This colors the request with a second dimension if they choose to.: Sure, and how many HTTP servers/services currently implement this? I've not seen a CGI that does, lighttpd can't do it, apache-httpd has .var files (which are unmaintianable IMO) and "multiviews" for normal people (which you shouldn't enable as then performance becomes horrible). There's also the problem that most of the browsers don't implement it well either (Ie. firefox "prefers" XML and XHTML over HTML, even though it renders HTML better (and faster) ... hell, it doesn't even render link tags in XML) so it's not exactly been tested a lot in the field.
Servers can take advantage of HTTP frameworks to implement their functionality on the desktop: I think this is a "free remote desktop" play, now with caching and goes through firewalls! Because that's the only reason people aren't running remote apps. on their home machine, at work.
It is not another dependency on an obscure Linux library: Yeh, I'm sure most people know what libcurl or libneon are ... and anything else that actually solves the problem is going to be shipped by everyone that cares. Plus you'll need something better than "just use HTTP and link with curl" ... which will be in it's own "obscure Linux library".
The possibility of easily capturing, proxying and caching any requests: Doing captures, is a benefit ... and I can almost imagine designing a protocol over HTTP for that (of course nothing will decode the bit over HTTP ... but it gets you some way there). Proxying is just another battle in the users vs. IT dept. war ... and I can't see the users winning that one. Caching is laughable, most web apps. are terrible at caching ... and most of the simple "HTTP clients" like robots and Atom readers aren't implementing Accept-Encoding/If-None-Match/If-Unmodified-Since (in spite of god knowns how many words on the subject).
Redirects could be used to redirect request to remote hosts transparently and securily: Maybe, although I don't see what is secure about it. And, again, most of the clients already have problems differentiating between 301, 302, 303 and 307 ... AFAICS firefox does the same thing for all of them.

Design by committee

2006-02-07T22:44:33Z

So Nat says:

Today Dan Winship wrote a wonderful mail about the perils of designing software by community process.

So, I have to wonder, does Nat/Dan/Miguel think that Apache-httpd is "bad"? My feeling is that it's a brilliant example of how committee designed software is terrible, hell the fact the config. parser let's each module parse it's own syntax seems like letting each gedit plugin have it's own GUI theme. And the way the module stuff is done reaks of "vision by committee", in that everyone just does their thing in their module so they don't have to speak to everyone else.

I also think this is somewhat interesting, as it also supports something I've believed for a long time ... Ie. design/security/quality isn't all it's espoused to be. It's nice, and everyone is happy to say so but very few are willing to pay for it (either with money, or even in time/work to move from something else). Compatability, speed and dancing monkeys all pay a greater roll.

And to bring the argument back to the desktop, I even feel the same way ... I don't care how much quality design they've put into GNOME, it's all (and more) canceled out when they break focus follows mouse or infinite space on my panel buttons.

And-httpd and security

2005-09-22T17:21:55Z

Well after using, what is now, And-httpd personally for many months, I've finally released an official version, seperate from Vstr. I've also backed it up with a $500 "security guarantee", and I'm not sure if that's stupid or "insightful", I guess time will tell. There are scarily huge amount of web servers at freshmeat ... which makes me wonder why apache-httpd is still so popular, but I guess quality is different from quantity ... and most Linux distributions are reluctant to ship more than one of anything, and given you have to ship apache-httpd that kind of settles it. *sigh*.

Anyway, after releasing promises of money in return for security bugs (which I'm assuming I won't have to pay, but then I think professional poker players assume they'll win too) I'm going for a long weekend away from a computer. So don't feel too depsondent when I don't respond to messages that I'm stupid, and almost certainly $500 poorer :).

James Antill

XML grep

A few things you might not know about RHEL-6.1+ yum

Time to look at a few features of yum in RHEL-6.1 now that it's released

Search is more userfriendly

The updateinfo command

The versionlock command

Manage your own .repo variables

yum has it's own DB

Additional data in "yum history"

"yum install" is now fully kickstart compatible

Easier to change yum configuration

Working towards managing 10 machines easily

Yum "update" from F-11 to F-13, with some help from distro-sync

The "simple" start.

First problem.

distro-sync

Explaining: "Warning: RPMDB altered outside of yum."

What does it mean?

Why has yum started to emit this warning?

New yum features require yum being "the" packaging API

Rpm is often abused, when used directly

We do extra work, and so inform the user

What can I do to make it stop?

Known problems

Multiple problems with multiple versions

The simple feature: just do what rpm does

Kernel: The special case

Embed the version in the package name

Taking a look an example, python2 and python3 packages

Lies, damn lies, and benchmarks

Or why I hate "quick benchmarks"...

Benchmarking Bias

An old example

A couple of yum examples

Measuring the slow thing

Benchmarking 1 point and then concluding about N

The obvious yum example

Benchmarking N points and then concluding about 1

The FreeBSD vs. Linux point

Phoronix has many examples here

The obvious yum example

Standard deviation is always significant

Phoronix has many examples here

The obligatory yum point

What to do, if you want to create a real benchmark

BugZilla feature of the year -- recent (commented) history

The url and the saved search

Bonus: search for bad attachment types

Update: 2009-04-30 --

Why trusted third party repos. will always be a bad idea

Why not make third party repos. first class

This is not a new problem

The core problem is distributed database synchronization

Ignoring the DB we still have the trust problem

But what about "simple" packages and "semi-trusted third party repos"

What about if I just ignore all of the above

So why all the code to support multiple repos.

Possible solution for third parties

A quick explanation of package sizes in yum and rpm

The three common sizes of an rpm package

How are those values calculated?

So, what is pkg.size and why does it change?

So, what's the best way to compare the size of an rpm package?

Programming with Yum in 5 minutes, or so

Simple version of "yum list" command

Simple stats. gathering from installed packages

Slightly more advanced topics

"Clever" way to manually update almost all the metadata for any usable repos. installed

Abusing the depsolver for fun/Sudoku

Not putting all your pkgs in one repo. (yum edition)

Understanding groups in yum

What really happens then?

But what about if we just add some magic?

But, but, usability! Do what the users expect!

Cool, groups are interesting but I can't control the groups from Fedora/etc.

Understanding why YUM seems "slow", and some instant solutions

Benchmarking "makecache":

Metadata checks slowing you down:

One big repo. hurts performance: