05:15 pm - Multiple problems with multiple versions

I've now seen a few requests for yum which can be generalized as "work better with multiple versions of packages". So I decided to write this, instead of replying N times to all the different requests.

Current Mood: pensive

06:37 pm - Why trusted third party repos. will always be a bad idea

Why not make third party repos. first class

Every now and again, someone takes a look at apt/yum/zypper/smart/PK/whatever and decides that although they have support for third party repos. it's "too annoying" for third parties to get users or for users to use them and so this is a problem which needs to be fixed. Another way this is presented is that the package managers should support "One Click Install". I will hopefully explain (once and for all) why this isn't a problem, and what third parties can do to get what they actually want (to make their users lives easier).

10:06 pm - A quick explanation of package sizes in yum and rpm

It's pretty common to think that a specific thing always has a specific size, and people tend to think of an "rpm package" as a single object thus. the it's common to ask what is "the size of an rpm". However if you have a 1MB text file, and gzip compresses it to 50KB which you then upload to a HTTP server you now have at least 3 different sizes: text size; compressed size and upload size (includes HTTP headers etc.) and asking for the size. So it is with rpm packages, and their many sizes.

The three common sizes of an rpm package

I'm going to use the yum package object notation to explain the common different sizes, as those are the easiest to see/use (see my previous post on how to simply get package objects from yum):

• An .rpm file package has pkg.archivesize (rpm TAG RPMTAG_ARCHIVESIZE), pkg.installedsize (rpm TAG RPMTAG_SIZE) and (for yum available package objects) pkg.packagesize (a stat of the .rpm file).

• Installed rpm packages have pkg.archivesize (rpm TAG RPMTAG_ARCHIVESIZE) and pkg.packagesize (rpm TAG RPMTAG_SIZE). pkg.installedsize doesn't exist, because an installed package can't be installed.

How are those values calculated?

rpm calculates the RPMTAG_SIZE value as the simple summation of the sizes of all the files within the rpm. rpm calculates the RPMTAG_ARCHIVESIZE value as the value of the cpio archive within the rpm, after decompression. These values are often very close. As I said above the pkg.packagesize value for .rpm files is just that value returned by stat(2).

So, what is pkg.size and why does it change?

Within yum there is a pkg.size, which maps to pkg.packagesize, which is used in all UI code that just wants to know "the size" of a package. This value has the property that the value you get for "need to download X bytes to install" and "will free up X bytes on removal" is correct, which is what most people want to see most of the time. However using this value does mean that the "the size of the rpm package" changes after you install an rpm, so it can be confusing if you try and compare pkg.size values between installed and available packages (either via. yum info, or looking at the values presented when installing a new kernel and removing an old one, say).

So, what's the best way to compare the size of an rpm package?

As you can see above, archivesize is the same for an available package and an installed one. So if you install the yum-lsit-data plugin, you can use "yum --showduplicates info-archive-sizes <pkgs>". Also you can always compare with just the available packages (and not installed ones), for instance "repoquery --show-duplicates --qf '%{nevra} %{archivesize} %{size}\n' <pkgs>"

08:49 pm - Programming with Yum in 5 minutes, or so

There are a lot of lines of code in yum, and it can be somewhat intimidating at first glace. However a significant amount of effort has been made to make simple things easy, and the hard things not so hard. The start of any code using yum, will almost certainly have these four lines (and always the first and last one :).

    1 #! /usr/bin/python -tt
    2 
    3 import os
    4 import sys
    5 import yum

Those lines just tell python, you'd you'd like to be able to use the yum code, and some stuff for the OS. Next the first bit of real code, and something which is also in almost every piece of code using yum:

    7 yb = yum.YumBase()

This creates a yum instance, that you can work with. Then one more piece, that is very useful:

    9 yb.conf.cache = os.geteuid() != 0

This just tells the yum instance not to try and update any of it's data, as the caller of the script probably hasn't got the permissions to do so.

11:58 pm - Abusing the depsolver for fun/Sudoku

After seeing the article on sudoku in apt/dpkg, I naturally thought "I wonder how you can do that in rpm". The big differences (from what I read/understood of the article) being that rpm doesn't mind if two packages with the same virtual provide are installed at once where dpkg does (and their "game" relies on that).

The natural solution seemed to be to use provides and conflicts, although I didn't think of other forms for long before just trying to implement that one :). The basic premise is to have 9 * 9 * 9 packages, each providing cell_1.1.1 through cell_9.9.9 (which is the row, column and value) and cell_value_1.1 through cell_value_9.9 (just the row and column). Then require all 82 cell_values, with any specific cells. You then just add conflicts in each package obeying the rules for block, row and column. Simple.

I was pretty sure yum would fail at solving the sudoku puzzles, because it basically requires that when you get multiple answers to a provides question (what package provides cell_1.1) you need to repeatedly narrow it down based on future information. This is kind of on the longer term. road map for yum so that it'll get better results, in certain edge cases, for real packages in real repositories.

Read on for scripts/repos. and results (spoiler, yum does fail and smart succeeds).

07:02 am - Understanding why YUM seems "slow", and some instant solutions

The previous entry was more of a general explanation about YUM speed and that if used in it's normal environment how current YUM is often more than fast enough, this entry is going to be a companion to it but focus on specific things that might make YUM appear slow but would better be described as using it suboptimally.

Note that I'm still not going to directly compare to other tools as I'm not as familiar with them and, as I said in the previous article, the tools are designed so differently that they don't lend themselves to comparisons. Also, as I also said before, if something is fine if it takes less than 10 seconds if two tools take 6 seconds and 4 seconds it doesn't really matter if yum is the faster (again, see the previous post, this should not be the end goal IMO).

10:00 pm - YUM resource usage, an accurate assessment

As a YUM developer it's hard not to notice the number of complaints/attacks on YUM about it being slow and/or using too much memory, the most common thing about almost all of these complaints though is how inaccurate they are. To be clear, YUM did used to take a lot more time than was required to do simple operations and there are very likely some more improvements that can be made for certain situations BUT yum is currently "not slow" at most normal operations for Fedora users and RHEL/CentOS users should be getting a much newer/faster variant than they currently have in the very near future.

As a final note I'd say that the biggest reason a lot of these recent complaints annoy me is not just that they are inaccurate but that they tend to grossly mislead the reader into thinking that "package management" is a solved problem and the biggest obstacle to solve is whether "install foo" takes 3 seconds or 6 seconds. In my opinion this could not be further from the truth, managing 2-6 machines is an ugly problem in all the current solutions and managing 100-10,000 is basically not done. Then there are things like the 10x10 problem, where we are only starting to see ideas like KOPERS which might help solve the problem (but will require changes in the way package management is assumed to work). And that's completely ignoring the problems that have had attempted solutions (that failed) multiple times, like "rollback" support.

02:22 am - Python uses too much memory?

There's a common attack leveled against Python applications that they take up too much memory, by people who understand the language difference (against, say, C) and by people just looking at their process list. This is often esp. evident on the newer x86_64 computers.

So as the Fedora Python maintainer, and a yum developer (an application written mostly using python), I figured it was probably worth investigating what the difference really was.