05:15 pm - Multiple problems with multiple versions

I've now seen a few requests for yum which can be generalized as "work better with multiple versions of packages". So I decided to write this, instead of replying N times to all the different requests.

Current Mood: pensive

02:47 pm - Lies, damn lies, and benchmarks

Or why I hate "quick benchmarks"...

Recently I've started to see a lot more of what I'd call "quick benchmarks", often it's to do with yum but mainly I think that's because those tend to get sent to me by multiple methods. So I thought I'd try and write down why I react so negatively/dismissively to them, how people can spot the underlying problems that annoy me and even better some advise on how you can go about doing some real benchmarks if that kind of thing interests you (but it's much more work than quick benchmarks).

The summary of the problem is that quick software benchmarking often involves taking a huge amount of differences between two applications and have a single number result. Then you compare just the numbers, and come to a conclusion. So X gets 3 and Y gets 5 for problem ABCD ... therefore Y is 66% worse than X at ABCD. Except that might be a highly misleading (or worse) conclusion, for a number of reasons:

Benchmarking Bias

This is often the route cause of all problems in quick benchmarks, and because of it's pervasiveness in human nature I would say that a significant portion of the work in doing a good benchmark is making sure you've tried to reduce the effect of your own bias. The bias happens in many forms, from the simple fact that when testing X and Y you might have a deeper understanding of one and so test it's strengths; make less configuration errors; or even assume favourable results are correct but unfavourable ones are incorrect.

An old example

Probably the first experience I had of this was at the end of the 1990s, I can find a link to the original "discussions" now but I can relate the main points. Linux had supported SMP (running on 2 or more CPUs at once) for a couple of years and FreeBSD was just starting to look at it seriously. While doing the Linux work the kernel developers had created a simple benchmark of rebuilding the kernel with different "make -j" configurations (controlling how parallel make), probably because it's simple tests a bunch of things at once and affects kernel developers personally. Obviously it was natural for the FreeBSD developers to do the same kind of thing, when they were doing the same kinds of work. I found what I think is the original FreeBSD report that someone posted on this FreeBSD SMP - kernel compilation bench.

Note that both of the above benchmarks were meant to be tested against themselves (ie. the developer would run the benchmark, make some changes and then run the benchmark again on the new kernel with the changes and see the difference). And, again, they were great for that ... pretty much all developers have similar kinds of test runs that they run. However someone then decided to take "the result for Linux" and compare it to "the result for FreeBSD", and came to the conclusion that because the FreeBSD number was smaller than the Linux number then that meant "FreeBSD was already faster at SMP than Linux". This conclusion was repeated by a number of knowledgable FreeBSD developers before a Linux developer pointed out the obvious fact that the tests didn't just have differenet kernels, but different source trees were being compiled with different compilation toolchains.

Again, I don't think the FreeBSD developers intentionally meant to create or use bogus data just that given some random numbers implying that FreeBSD was better than Linux they were biased to believe them and so didn't think about it too much.

A couple of yum examples

I've seen a lot of people test "yum makecache" vs. "apt-get update" or "smart update" or "zypper refresh". This makes some sense from the point of view of an apt developer/user because this is an operation that the user has to run before any set of operations to make sure the database is upto date, so any improvement (relative to older versions of your self) is a significant win. However a yum user is unlikely to ever run this command because the default mode of operation is for yum to manage database synchronization.

Another common problem is to compare something like "apt-cache search" / "apt-file search" against "yum search" / "yum provides", the problem here is that apt-cache / apt-files just do a simple grep on the cache of available packages ... yum does it's searches against what yum can see use (so, for example, versionlock'ing a package will affect the results you get in yum ... as will installing packages).

These problems should be obvious after even a moments thought about how yum is used, but again most of the people publishing this kind of results either don't use yum or are expecting it to be slower and so don't think about results which confirm that expectation.

Measuring the slow thing

Many developers are aware that their applications have a "fast path" and a "slow path", which normally align with "unexpected cases work, but are slow" and "expected, normal, cases work quickly". Benchmarkers often do not know these distinctions. I've seen numerous cases where someone benchmarks something that wouldn't be done in real life. Alas. this is hard to spot for the normal user, and is one of the reasons that it's wise to speak with the developers of whatever you are benchmarking.

Benchmarking 1 point and then concluding about N

The obvious yum example

By far the most common problem here is people run a simple benchmark like "time (echo n | yum update)" compared to the same operation in apt or zypp and then concluding X is x% faster at updating than Y. The problem here is that updating actually has a number of operations (breaking them down roughly):

1. read repo. configuration, cmd line options etc.
2. check if repo. metadata is current.
3. download repo. metadata if not current.
4. merge configured repo. metadata into "available".
5. read current rpm metadata.
6. depsolve updates+obsoletes+etc. from rpm and repo. metadata.
7. output changes.
8. confirm changes.
9. download updates.
10. install updates.

However even though there are 10 operations above, the "simple update benchmark" only really tests "6. depsolve ...". Which is fine on it's own, it's worth knowing what the depsolve time is (and I run this benchmark myself with yum to find out that info.) but depsolve time is not the same as update time. In fact most of the time it's in the noise for the update operation as a whole (obviously saying your update takes 25% of the time is much more fun than saying it takes 98% of the time).

I've even seen problems here where they'll test apt on Debian vs. yum on Fedora, which much like the FreeBSD vs. Linux test changes so many different things that you can't conclude anything about just the package managers. There have also been cases where someone tested "apt upgrade" vs. "yum upgrade" but the yum operation also download repo. metadata, as it wasn't current/available.

Benchmarking N points and then concluding about 1

The FreeBSD vs. Linux point

As I said above the technical problem with the FreeBSD make kernel vs. Linux make kernel problem was that it was testing 3 different sets of things (buildtools, source code, kernel) and drawing a conclusion about only 1 of them.

Phoronix has many examples here

Phoronix has many examples of all the different problems you can get into here. From things like their "apache benchmark", which turned out to have nothing to do with Apache on either platform they tested it on, and mp3 encoding "filesystem benchmarks". To their usual "conclusions" which take 10-20 completely separate points, with completely separate error rates, and come up with an "average".

The obvious yum example

The usual way this happens in package management "benchmarking" is that a random number of commands will be tested, like install; update; remove; whatprovides; search; list ... and then a final score will be given. This completely ignores the fact some of those commands are being run much more often than others, and in different situations.

Standard deviation is always significant

The simple way to think of this is that multiple runs of anything you test need to be performed, and if you don't have results where the answer is "X and Y perform the same" (even though the absolute numbers are never going to be identical) ... the benchmarking has probably screwed up the standard deviation.

Phoronix has many examples here

This is often combined with the previous problem, if you test X vs. Y and get the results of 1000 and 1200 then there's a huge difference between a SD (standard deviation) of 500 and an SD of 1 on those results. And this is esp. important when you have another result which is 10 vs. 20 and an SD that could be 0.1 or 5.

The obligatory yum point

I never see benchmarks of package management operations that include a standard deviation value, and this can be esp. important when different values and/or different configurations can affect the performance significantly.

What to do, if you want to create a real benchmark

All of the above is not to try to discourage real benchmarking, as this is a time consuming and unrewarding task (IMO), but often bad benchmarking is worse than none at all. If you aren't up for that then feel free to suggest usecases that seem difficult/slow/etc., even if you need to explain it as "currently I run X like this, and it takes N time, how do I do something similar with Y in a similar amount of time" ... this is not the same thing, because the answer might well be "run Y like this instead". But if you are up for the challenge then...

As I said earlier, if you are benchmarking X and Y and comparing them then you need to be very familiar with both X and Y, which means investing a significant amount of time working with both X and Y. This very likely means that you want to speak to developers of both X and Y, both about the measurements you are trying to take and the results you are getting. I am always very suspicious if the benchmarking seems to be one sided (in that more knowledge was available about X and Y, at the time of testing), or more to the point I'm very suspicious if you are benchmarking yum but I've no idea who you are.

Also, even if you create a good benchmark which shows X is 999x faster than Y for operation FOO, and FOO is a useful feature that people will want to take advantage of. It is likely that the developers for Y will be able to change Y (sometimes trivially, as it might be an assumed edge case or regression noone had hit, yet) which reduces the difference significantly. This also often means that doing benchmarks properly "just" makes both X and Y better, with the published results showing they are within 5% or whatever and so "nobody cares" about the benchmarks. Which makes running the benchmarks pretty unrewarding, but hey you want to be the benchmarker not me... :)

06:57 pm - BugZilla feature of the year -- recent (commented) history

Something I've wanted for a long time is "show me the BugZilla tickets which I've looked at "recently" (last day, week, etc.). Like almost everyone else I speak to I get by saving all my bugzilla email and then doing seartches in evolution.

However today ajaxxx found/used advanced bugzilla search options to give a page which has the recent most history of comments you've done. This isn't quite the same as being able to see everything that you've looked at, but it's pretty close.

The url and the saved search

The search works due to two "advanced features: 1. Selecting yourself as the commentator: <field0-1-0=commenter>, <type0-1-0=equals>, and <value0-1-0=%user%> and 2. Selecting comment changes that happened after 8 days ago <field0-0-0=longdesc>, <type0-0-0=changedafter>, and <value0-0-0=8d>. The "8d" part can obviously be changed to reflect different history. To experience the feature a full working URL is: here

I also created a saved search called Commented in last 8 days. You may also want to change your column layout to include the changed time (alas. I don't know of any way to create a custom view for just a single search, *sigh*).

Bonus: search for bad attachment types

Another problem I hit a lot is that python code tracebacks in plain text, but users often leave the "crash" as the default application/octet-stream ... which means it can't be viewed. Really I'd like BugZilla to notice this, and fix it. But after seeing the above search I created: YUM* weird attachments, which searches for any BZ (belonging to the yum group) that has an attachment of type application-octet/stream that isn't obsoleted. Then it's a simple matter of manually fixing the problem bugs.

Update: 2009-04-30 --

06:37 pm - Why trusted third party repos. will always be a bad idea

Why not make third party repos. first class

Every now and again, someone takes a look at apt/yum/zypper/smart/PK/whatever and decides that although they have support for third party repos. it's "too annoying" for third parties to get users or for users to use them and so this is a problem which needs to be fixed. Another way this is presented is that the package managers should support "One Click Install". I will hopefully explain (once and for all) why this isn't a problem, and what third parties can do to get what they actually want (to make their users lives easier).

10:06 pm - A quick explanation of package sizes in yum and rpm

It's pretty common to think that a specific thing always has a specific size, and people tend to think of an "rpm package" as a single object thus. the it's common to ask what is "the size of an rpm". However if you have a 1MB text file, and gzip compresses it to 50KB which you then upload to a HTTP server you now have at least 3 different sizes: text size; compressed size and upload size (includes HTTP headers etc.) and asking for the size. So it is with rpm packages, and their many sizes.

The three common sizes of an rpm package

I'm going to use the yum package object notation to explain the common different sizes, as those are the easiest to see/use (see my previous post on how to simply get package objects from yum):

• An .rpm file package has pkg.archivesize (rpm TAG RPMTAG_ARCHIVESIZE), pkg.installedsize (rpm TAG RPMTAG_SIZE) and (for yum available package objects) pkg.packagesize (a stat of the .rpm file).

• Installed rpm packages have pkg.archivesize (rpm TAG RPMTAG_ARCHIVESIZE) and pkg.packagesize (rpm TAG RPMTAG_SIZE). pkg.installedsize doesn't exist, because an installed package can't be installed.

How are those values calculated?

rpm calculates the RPMTAG_SIZE value as the simple summation of the sizes of all the files within the rpm. rpm calculates the RPMTAG_ARCHIVESIZE value as the value of the cpio archive within the rpm, after decompression. These values are often very close. As I said above the pkg.packagesize value for .rpm files is just that value returned by stat(2).

So, what is pkg.size and why does it change?

Within yum there is a pkg.size, which maps to pkg.packagesize, which is used in all UI code that just wants to know "the size" of a package. This value has the property that the value you get for "need to download X bytes to install" and "will free up X bytes on removal" is correct, which is what most people want to see most of the time. However using this value does mean that the "the size of the rpm package" changes after you install an rpm, so it can be confusing if you try and compare pkg.size values between installed and available packages (either via. yum info, or looking at the values presented when installing a new kernel and removing an old one, say).

So, what's the best way to compare the size of an rpm package?

As you can see above, archivesize is the same for an available package and an installed one. So if you install the yum-lsit-data plugin, you can use "yum --showduplicates info-archive-sizes <pkgs>". Also you can always compare with just the available packages (and not installed ones), for instance "repoquery --show-duplicates --qf '%{nevra} %{archivesize} %{size}\n' <pkgs>"

08:49 pm - Programming with Yum in 5 minutes, or so

There are a lot of lines of code in yum, and it can be somewhat intimidating at first glace. However a significant amount of effort has been made to make simple things easy, and the hard things not so hard. The start of any code using yum, will almost certainly have these four lines (and always the first and last one :).

    1 #! /usr/bin/python -tt
    2 
    3 import os
    4 import sys
    5 import yum

Those lines just tell python, you'd you'd like to be able to use the yum code, and some stuff for the OS. Next the first bit of real code, and something which is also in almost every piece of code using yum:

    7 yb = yum.YumBase()

This creates a yum instance, that you can work with. Then one more piece, that is very useful:

    9 yb.conf.cache = os.geteuid() != 0

This just tells the yum instance not to try and update any of it's data, as the caller of the script probably hasn't got the permissions to do so.

11:58 pm - Abusing the depsolver for fun/Sudoku

After seeing the article on sudoku in apt/dpkg, I naturally thought "I wonder how you can do that in rpm". The big differences (from what I read/understood of the article) being that rpm doesn't mind if two packages with the same virtual provide are installed at once where dpkg does (and their "game" relies on that).

The natural solution seemed to be to use provides and conflicts, although I didn't think of other forms for long before just trying to implement that one :). The basic premise is to have 9 * 9 * 9 packages, each providing cell_1.1.1 through cell_9.9.9 (which is the row, column and value) and cell_value_1.1 through cell_value_9.9 (just the row and column). Then require all 82 cell_values, with any specific cells. You then just add conflicts in each package obeying the rules for block, row and column. Simple.

I was pretty sure yum would fail at solving the sudoku puzzles, because it basically requires that when you get multiple answers to a provides question (what package provides cell_1.1) you need to repeatedly narrow it down based on future information. This is kind of on the longer term. road map for yum so that it'll get better results, in certain edge cases, for real packages in real repositories.

Read on for scripts/repos. and results (spoiler, yum does fail and smart succeeds).

11:03 pm - Not putting all your pkgs in one repo. (yum edition)

As you create packages for private use, the question will eventually come up "where do I put these". The choice is obvious for the first package, just create a repo. (using createrepo -d) and distribute the my.repo file. However as you create more packages the answer should expand to having multiple repos. Different package managers have a different ideas about what should go inside a single repository, and the corollary to that is if you take the "best practice" for some other package manager and apply it to yum the results might not be all they could be. This posting will try and lay out the best practises for yum (3.2.z), and some of the reasons for them.

11:40 pm - Understanding groups in yum

Every now and again someone will will ask a question about groups in yum that amounts to:

If I do "groupinstall xyz" and then I do "groupremove xyz" why do I not end up where I started.

07:02 am - Understanding why YUM seems "slow", and some instant solutions

The previous entry was more of a general explanation about YUM speed and that if used in it's normal environment how current YUM is often more than fast enough, this entry is going to be a companion to it but focus on specific things that might make YUM appear slow but would better be described as using it suboptimally.

Note that I'm still not going to directly compare to other tools as I'm not as familiar with them and, as I said in the previous article, the tools are designed so differently that they don't lend themselves to comparisons. Also, as I also said before, if something is fine if it takes less than 10 seconds if two tools take 6 seconds and 4 seconds it doesn't really matter if yum is the faster (again, see the previous post, this should not be the end goal IMO).

10:00 pm - YUM resource usage, an accurate assessment

As a YUM developer it's hard not to notice the number of complaints/attacks on YUM about it being slow and/or using too much memory, the most common thing about almost all of these complaints though is how inaccurate they are. To be clear, YUM did used to take a lot more time than was required to do simple operations and there are very likely some more improvements that can be made for certain situations BUT yum is currently "not slow" at most normal operations for Fedora users and RHEL/CentOS users should be getting a much newer/faster variant than they currently have in the very near future.

As a final note I'd say that the biggest reason a lot of these recent complaints annoy me is not just that they are inaccurate but that they tend to grossly mislead the reader into thinking that "package management" is a solved problem and the biggest obstacle to solve is whether "install foo" takes 3 seconds or 6 seconds. In my opinion this could not be further from the truth, managing 2-6 machines is an ugly problem in all the current solutions and managing 100-10,000 is basically not done. Then there are things like the 10x10 problem, where we are only starting to see ideas like KOPERS which might help solve the problem (but will require changes in the way package management is assumed to work). And that's completely ignoring the problems that have had attempted solutions (that failed) multiple times, like "rollback" support.

02:22 am - Python uses too much memory?

There's a common attack leveled against Python applications that they take up too much memory, by people who understand the language difference (against, say, C) and by people just looking at their process list. This is often esp. evident on the newer x86_64 computers.

So as the Fedora Python maintainer, and a yum developer (an application written mostly using python), I figured it was probably worth investigating what the difference really was.

12:23 am - I called it, Obama for democratic nominee/president

Feel free to pick me up and jiggle me about before asking other questions obviously answered by crappy human nature:

Obama to win over Hillary, March 1st, 2007 05:07 pm

Current Mood: bored

01:00 am - The myth of the "simple String API", that isn't really needed

I saw three things recently which just confirmed that noone learns anything:

• The "we don't need a real string API" on the GIT mailing list, followed by almost a month now of struggling to implement another one (and wishing they had some of the ustr features).
• GLibc's proposed fix for C/POSIX not having a usable string API. Yes, it's the old let's all pretend IO (FILE *) is a good string abstraction ... you need to add, search/etc, and then add some more? Well multiple copies were always good fun, pretty much guaranteeing people will do a bunch of workarounds if they use it at all.
• The Linux kernel is again looking for yet another almost a worthwhile string API, pretending it'll only be useful for this specific IO type ... because seq_printf() wasn't unique enough.

Best quote by far, IMO (from the GIT thread):

"Please, no. Let's not pull in a dependency for something as simple as a string library." -- Kristian Høgsberg

As a minor good note, ustr is in Fedora and so pretty much everyone in Fedora land has access to a usable string API with a single -l ... and the newer versions of libsemanage (SELinux) use ustr. I'm not going to hold my breath for mass sanity to occur though.

02:00 am - The last 10%

ustr a rough timeline, notes on the last 10%

Common software development wisdom says "the last 10%" is often a significant time consumer, if not 50%, of the work. So when I recently created ustr, I thought I had a pretty good candidate to get some real numbers. For a start, in my favour, I'd already created Vstr and so knew most of the interfaces I'd want and roughly how to write them.

05:09 am - Software packaging, and the 10 × 10 problem

First off, I hate all current software packaging for Linux. It's one step up from manually downloading tarballs, which I was doing 10 years ago, and isn't even a real superset of that functionality. Yes, it's better than doing that and yes it's better than what Windows has to offer. But it's still crap, and it annoys me every day, however it seems to be one of those problems that noone seems to want to fix properly and yet they keep thinking up stupid bandaids to work around the fact it doesn't work well. Specifically the major circular argument that is what I've come to think of as the "10 × 10 problem"…

Current Mood: disappointed

12:14 am - Fedora Kernel update causes: mkrootdev expected fs options

I'm mainly dropping this somewhere so other people can find it, I recently did a kernel update and when I rebooted I got:

mkrootdev expected fs options
mount: missing mount point
setuproot: moving /dev/failed no such file or directory
setuproot: error mounting /proc
setuproot: error mounting /sys
switchroot: mount failed No such file or directory
kernel panic

...it turns out this was because I'd somehow got two / entries in /etc/fstab, as I found from this posting. The only thing you need to be aware of is that you also have a copy of /etc/fstab in the initrd, so after fixing /etc/fstab you'll want to re-run the postinstall scriptlet (rpm -q --scripts kernel), Ie. /sbin/new-kernel-pkg --package kernel --mkinitrd --depmod --install <kernel version>

10:01 pm - lighttpd's "AIO sendfile"

I keep track of a couple of the other web servers out there, even though I don't think much of them ... for various reasons. But when I saw that lighttpd was advertising AIO sendfile for their 1.5.x I was pretty impressed, while I hadn't been following it closely I didn't think the Linux AIO sendfile code had gone in yet (I'd been planning on trying to use the splice() API with IO helpers to do the same thing ... but I keep not getting around to it).

Alas. looking deeper into it, the release note is very misleading. They are actually doing AIO reads into allocated memory, and then using non-AIO sendfile() on the fd associated with the memory. Of course, this means:

• It copies the entire file into memory (in sections of half a MB).
• It requires copying an entire file section before anything is sent.
• There is no sharing! So if you have 10,000 connections requesting the same 2MB file you get 5GB of memory used.
• The real page cache readahead is going to have to be that much more intelligent to do the right thing.
• There is no guarantee that the memory isn't swapped! So you can still have all the blocking artifacts of normal non-AIO sendfile().

...basically this is a lot of work to reimplement the page cache, badly.

I also seriously question whether the same "improvement" could have been got by just over allocating on the process to CPU mapping (something that I understand lighttpd can do as well as and-httpd). It also implies that posix_fadvise() should be doing more work (it is supposed to solve this exact problem), but maybe sprinkling explicit readhead() calls would also work around the problem (without screwing everyone else on the box).

Also why lighttpd uses sendfile() instead of just write() for the last piece, I'm not sure ... it's like doing a mmap() and then calling sendfile() on the mmap'd fd.

Current Mood: sad

04:41 am - Re(tar)ddit.com, newegg and others ignoring HTTP/HTML

So I tried reddit.com about a year ago, mainly due to the whole Y combinator thing. It seemed like it might be cute, but at the time didn't seem to contain any decent content on the front page and it required JavaScript to do anything interactive. So, I figured I'd come back a bit later when they've had a chance to polish it and take another look.

As you might guess, it's still JS only and they almost seem proud of it. Like they get more Web-2.0 juice or something. Hello 1995 called to let you know links are possible without JS. They even have "buttons" that you can attach to your site, so people can vote up/down your content easily ... but the "button" is just a script tag. Yeh, so not only do I apparently want to leave all my non-JS enabled agents out in the cold ... I want to actively mix-in JS from a third party? It's like the blind being led by the retards.

I could almost understand, the "JS is almost everywhere" so we'll just ignore everyone who wants security/stability/speed kind of argument for reddit.com ... if the rest of their usage of HTTP/HTML didn't seem like it was written by a 13yr old. For instance http://reddit.com/static/reddit.js is their "main" blob of imported JS functions. Now, given it says static in the URL, you might imagine that this library code would be heavily optimized with all the latest HTTP bits possible (in fact I was surprised there wasn't a version/date in the URL). But no, there's no Expires/Cache-Control headers. At least it has an ETag and does Content-Encoding (although lighttpd is broken and ignores it for HEAD requests).

But, reddit.com is still somewhat easy to dismiss as crack smokin' 13yr olds who still need a few years to grow up. After all it's not a real business, and probably doesn't make worthwhile amounts of money. But, then, newegg.com does ... and there almost everything works without JS, the notable exception being actually doing the sale. Sure, let me look at stuff and even do searches with a secure browser ... but when I need to type in my credit card, that's the time to rely on JS to do 1998 style forms.

Unsurprisingly amazon.com, as the largest online retailer, have consistently had the best UI for non-JS user-agents. And even though tags, as their latest tweak, started off as JS only then moved to working perfectly without JS within a couple of weeks. But, I guess maybe it's got less 2.0 juice now.

Current Mood: shocked

10:46 pm - HTTP for desktop applications

So to continue the HTTP theme, Miguel's idea that desktop applications communicate via. HTTP seems completely insane. I can only presume Miguel was not involved at all in the Mono HTTP API implementations. As part of writing my webserver, I've already written about how terrible the HTTP spec. is ... hell apache-httpd blatantly ignores significant parts of it.

Current Mood: scared