06:57 pm - BugZilla feature of the year -- recent (commented) history

Something I've wanted for a long time is "show me the BugZilla tickets which I've looked at "recently" (last day, week, etc.). Like almost everyone else I speak to I get by saving all my bugzilla email and then doing seartches in evolution.

However today ajaxxx found/used advanced bugzilla search options to give a page which has the recent most history of comments you've done. This isn't quite the same as being able to see everything that you've looked at, but it's pretty close.

The url and the saved search

The search works due to two "advanced features: 1. Selecting yourself as the commentator: <field0-1-0=commenter>, <type0-1-0=equals>, and <value0-1-0=%user%> and 2. Selecting comment changes that happened after 8 days ago <field0-0-0=longdesc>, <type0-0-0=changedafter>, and <value0-0-0=8d>. The "8d" part can obviously be changed to reflect different history. To experience the feature a full working URL is: here

I also created a saved search called Commented in last 8 days. You may also want to change your column layout to include the changed time (alas. I don't know of any way to create a custom view for just a single search, *sigh*).

Bonus: search for bad attachment types

Another problem I hit a lot is that python code tracebacks in plain text, but users often leave the "crash" as the default application/octet-stream ... which means it can't be viewed. Really I'd like BugZilla to notice this, and fix it. But after seeing the above search I created: YUM* weird attachments, which searches for any BZ (belonging to the yum group) that has an attachment of type application-octet/stream that isn't obsoleted. Then it's a simple matter of manually fixing the problem bugs.

Update: 2009-04-30 --

06:37 pm - Why trusted third party repos. will always be a bad idea

Why not make third party repos. first class

Every now and again, someone takes a look at apt/yum/zypper/smart/PK/whatever and decides that although they have support for third party repos. it's "too annoying" for third parties to get users or for users to use them and so this is a problem which needs to be fixed. Another way this is presented is that the package managers should support "One Click Install". I will hopefully explain (once and for all) why this isn't a problem, and what third parties can do to get what they actually want (to make their users lives easier).

10:06 pm - A quick explanation of package sizes in yum and rpm

It's pretty common to think that a specific thing always has a specific size, and people tend to think of an "rpm package" as a single object thus. the it's common to ask what is "the size of an rpm". However if you have a 1MB text file, and gzip compresses it to 50KB which you then upload to a HTTP server you now have at least 3 different sizes: text size; compressed size and upload size (includes HTTP headers etc.) and asking for the size. So it is with rpm packages, and their many sizes.

The three common sizes of an rpm package

I'm going to use the yum package object notation to explain the common different sizes, as those are the easiest to see/use (see my previous post on how to simply get package objects from yum):

• An .rpm file package has pkg.archivesize (rpm TAG RPMTAG_ARCHIVESIZE), pkg.installedsize (rpm TAG RPMTAG_SIZE) and (for yum available package objects) pkg.packagesize (a stat of the .rpm file).

• Installed rpm packages have pkg.archivesize (rpm TAG RPMTAG_ARCHIVESIZE) and pkg.packagesize (rpm TAG RPMTAG_SIZE). pkg.installedsize doesn't exist, because an installed package can't be installed.

How are those values calculated?

rpm calculates the RPMTAG_SIZE value as the simple summation of the sizes of all the files within the rpm. rpm calculates the RPMTAG_ARCHIVESIZE value as the value of the cpio archive within the rpm, after decompression. These values are often very close. As I said above the pkg.packagesize value for .rpm files is just that value returned by stat(2).

So, what is pkg.size and why does it change?

Within yum there is a pkg.size, which maps to pkg.packagesize, which is used in all UI code that just wants to know "the size" of a package. This value has the property that the value you get for "need to download X bytes to install" and "will free up X bytes on removal" is correct, which is what most people want to see most of the time. However using this value does mean that the "the size of the rpm package" changes after you install an rpm, so it can be confusing if you try and compare pkg.size values between installed and available packages (either via. yum info, or looking at the values presented when installing a new kernel and removing an old one, say).

So, what's the best way to compare the size of an rpm package?

As you can see above, archivesize is the same for an available package and an installed one. So if you install the yum-lsit-data plugin, you can use "yum --showduplicates info-archive-sizes <pkgs>". Also you can always compare with just the available packages (and not installed ones), for instance "repoquery --show-duplicates --qf '%{nevra} %{archivesize} %{size}\n' <pkgs>"

08:49 pm - Programming with Yum in 5 minutes, or so

There are a lot of lines of code in yum, and it can be somewhat intimidating at first glace. However a significant amount of effort has been made to make simple things easy, and the hard things not so hard. The start of any code using yum, will almost certainly have these four lines (and always the first and last one :).

    1 #! /usr/bin/python -tt
    2 
    3 import os
    4 import sys
    5 import yum

Those lines just tell python, you'd you'd like to be able to use the yum code, and some stuff for the OS. Next the first bit of real code, and something which is also in almost every piece of code using yum:

    7 yb = yum.YumBase()

This creates a yum instance, that you can work with. Then one more piece, that is very useful:

    9 yb.conf.cache = os.geteuid() != 0

This just tells the yum instance not to try and update any of it's data, as the caller of the script probably hasn't got the permissions to do so.

11:58 pm - Abusing the depsolver for fun/Sudoku

After seeing the article on sudoku in apt/dpkg, I naturally thought "I wonder how you can do that in rpm". The big differences (from what I read/understood of the article) being that rpm doesn't mind if two packages with the same virtual provide are installed at once where dpkg does (and their "game" relies on that).

The natural solution seemed to be to use provides and conflicts, although I didn't think of other forms for long before just trying to implement that one :). The basic premise is to have 9 * 9 * 9 packages, each providing cell_1.1.1 through cell_9.9.9 (which is the row, column and value) and cell_value_1.1 through cell_value_9.9 (just the row and column). Then require all 82 cell_values, with any specific cells. You then just add conflicts in each package obeying the rules for block, row and column. Simple.

I was pretty sure yum would fail at solving the sudoku puzzles, because it basically requires that when you get multiple answers to a provides question (what package provides cell_1.1) you need to repeatedly narrow it down based on future information. This is kind of on the longer term. road map for yum so that it'll get better results, in certain edge cases, for real packages in real repositories.

Read on for scripts/repos. and results (spoiler, yum does fail and smart succeeds).

11:03 pm - Not putting all your pkgs in one repo. (yum edition)

As you create packages for private use, the question will eventually come up "where do I put these". The choice is obvious for the first package, just create a repo. (using createrepo -d) and distribute the my.repo file. However as you create more packages the answer should expand to having multiple repos. Different package managers have a different ideas about what should go inside a single repository, and the corollary to that is if you take the "best practice" for some other package manager and apply it to yum the results might not be all they could be. This posting will try and lay out the best practises for yum (3.2.z), and some of the reasons for them.

11:40 pm - Understanding groups in yum

Every now and again someone will will ask a question about groups in yum that amounts to:

If I do "groupinstall xyz" and then I do "groupremove xyz" why do I not end up where I started.

07:02 am - Understanding why YUM seems "slow", and some instant solutions

The previous entry was more of a general explanation about YUM speed and that if used in it's normal environment how current YUM is often more than fast enough, this entry is going to be a companion to it but focus on specific things that might make YUM appear slow but would better be described as using it suboptimally.

Note that I'm still not going to directly compare to other tools as I'm not as familiar with them and, as I said in the previous article, the tools are designed so differently that they don't lend themselves to comparisons. Also, as I also said before, if something is fine if it takes less than 10 seconds if two tools take 6 seconds and 4 seconds it doesn't really matter if yum is the faster (again, see the previous post, this should not be the end goal IMO).

10:00 pm - YUM resource usage, an accurate assessment

As a YUM developer it's hard not to notice the number of complaints/attacks on YUM about it being slow and/or using too much memory, the most common thing about almost all of these complaints though is how inaccurate they are. To be clear, YUM did used to take a lot more time than was required to do simple operations and there are very likely some more improvements that can be made for certain situations BUT yum is currently "not slow" at most normal operations for Fedora users and RHEL/CentOS users should be getting a much newer/faster variant than they currently have in the very near future.

As a final note I'd say that the biggest reason a lot of these recent complaints annoy me is not just that they are inaccurate but that they tend to grossly mislead the reader into thinking that "package management" is a solved problem and the biggest obstacle to solve is whether "install foo" takes 3 seconds or 6 seconds. In my opinion this could not be further from the truth, managing 2-6 machines is an ugly problem in all the current solutions and managing 100-10,000 is basically not done. Then there are things like the 10x10 problem, where we are only starting to see ideas like KOPERS which might help solve the problem (but will require changes in the way package management is assumed to work). And that's completely ignoring the problems that have had attempted solutions (that failed) multiple times, like "rollback" support.

02:22 am - Python uses too much memory?

There's a common attack leveled against Python applications that they take up too much memory, by people who understand the language difference (against, say, C) and by people just looking at their process list. This is often esp. evident on the newer x86_64 computers.

So as the Fedora Python maintainer, and a yum developer (an application written mostly using python), I figured it was probably worth investigating what the difference really was.

12:23 am - I called it, Obama for democratic nominee/president

Feel free to pick me up and jiggle me about before asking other questions obviously answered by crappy human nature:

Obama to win over Hillary, March 1st, 2007 05:07 pm

Current Mood: bored

01:00 am - The myth of the "simple String API", that isn't really needed

I saw three things recently which just confirmed that noone learns anything:

• The "we don't need a real string API" on the GIT mailing list, followed by almost a month now of struggling to implement another one (and wishing they had some of the ustr features).
• GLibc's proposed fix for C/POSIX not having a usable string API. Yes, it's the old let's all pretend IO (FILE *) is a good string abstraction ... you need to add, search/etc, and then add some more? Well multiple copies were always good fun, pretty much guaranteeing people will do a bunch of workarounds if they use it at all.
• The Linux kernel is again looking for yet another almost a worthwhile string API, pretending it'll only be useful for this specific IO type ... because seq_printf() wasn't unique enough.

Best quote by far, IMO (from the GIT thread):

"Please, no. Let's not pull in a dependency for something as simple as a string library." -- Kristian Høgsberg

As a minor good note, ustr is in Fedora and so pretty much everyone in Fedora land has access to a usable string API with a single -l ... and the newer versions of libsemanage (SELinux) use ustr. I'm not going to hold my breath for mass sanity to occur though.

02:00 am - The last 10%

ustr a rough timeline, notes on the last 10%

Common software development wisdom says "the last 10%" is often a significant time consumer, if not 50%, of the work. So when I recently created ustr, I thought I had a pretty good candidate to get some real numbers. For a start, in my favour, I'd already created Vstr and so knew most of the interfaces I'd want and roughly how to write them.

05:09 am - Software packaging, and the 10 × 10 problem

First off, I hate all current software packaging for Linux. It's one step up from manually downloading tarballs, which I was doing 10 years ago, and isn't even a real superset of that functionality. Yes, it's better than doing that and yes it's better than what Windows has to offer. But it's still crap, and it annoys me every day, however it seems to be one of those problems that noone seems to want to fix properly and yet they keep thinking up stupid bandaids to work around the fact it doesn't work well. Specifically the major circular argument that is what I've come to think of as the "10 × 10 problem"…

Current Mood: disappointed

12:14 am - Fedora Kernel update causes: mkrootdev expected fs options

I'm mainly dropping this somewhere so other people can find it, I recently did a kernel update and when I rebooted I got:

mkrootdev expected fs options
mount: missing mount point
setuproot: moving /dev/failed no such file or directory
setuproot: error mounting /proc
setuproot: error mounting /sys
switchroot: mount failed No such file or directory
kernel panic

...it turns out this was because I'd somehow got two / entries in /etc/fstab, as I found from this posting. The only thing you need to be aware of is that you also have a copy of /etc/fstab in the initrd, so after fixing /etc/fstab you'll want to re-run the postinstall scriptlet (rpm -q --scripts kernel), Ie. /sbin/new-kernel-pkg --package kernel --mkinitrd --depmod --install <kernel version>

10:01 pm - lighttpd's "AIO sendfile"

I keep track of a couple of the other web servers out there, even though I don't think much of them ... for various reasons. But when I saw that lighttpd was advertising AIO sendfile for their 1.5.x I was pretty impressed, while I hadn't been following it closely I didn't think the Linux AIO sendfile code had gone in yet (I'd been planning on trying to use the splice() API with IO helpers to do the same thing ... but I keep not getting around to it).

Alas. looking deeper into it, the release note is very misleading. They are actually doing AIO reads into allocated memory, and then using non-AIO sendfile() on the fd associated with the memory. Of course, this means:

• It copies the entire file into memory (in sections of half a MB).
• It requires copying an entire file section before anything is sent.
• There is no sharing! So if you have 10,000 connections requesting the same 2MB file you get 5GB of memory used.
• The real page cache readahead is going to have to be that much more intelligent to do the right thing.
• There is no guarantee that the memory isn't swapped! So you can still have all the blocking artifacts of normal non-AIO sendfile().

...basically this is a lot of work to reimplement the page cache, badly.

I also seriously question whether the same "improvement" could have been got by just over allocating on the process to CPU mapping (something that I understand lighttpd can do as well as and-httpd). It also implies that posix_fadvise() should be doing more work (it is supposed to solve this exact problem), but maybe sprinkling explicit readhead() calls would also work around the problem (without screwing everyone else on the box).

Also why lighttpd uses sendfile() instead of just write() for the last piece, I'm not sure ... it's like doing a mmap() and then calling sendfile() on the mmap'd fd.

Current Mood: sad

04:41 am - Re(tar)ddit.com, newegg and others ignoring HTTP/HTML

So I tried reddit.com about a year ago, mainly due to the whole Y combinator thing. It seemed like it might be cute, but at the time didn't seem to contain any decent content on the front page and it required JavaScript to do anything interactive. So, I figured I'd come back a bit later when they've had a chance to polish it and take another look.

As you might guess, it's still JS only and they almost seem proud of it. Like they get more Web-2.0 juice or something. Hello 1995 called to let you know links are possible without JS. They even have "buttons" that you can attach to your site, so people can vote up/down your content easily ... but the "button" is just a script tag. Yeh, so not only do I apparently want to leave all my non-JS enabled agents out in the cold ... I want to actively mix-in JS from a third party? It's like the blind being led by the retards.

I could almost understand, the "JS is almost everywhere" so we'll just ignore everyone who wants security/stability/speed kind of argument for reddit.com ... if the rest of their usage of HTTP/HTML didn't seem like it was written by a 13yr old. For instance http://reddit.com/static/reddit.js is their "main" blob of imported JS functions. Now, given it says static in the URL, you might imagine that this library code would be heavily optimized with all the latest HTTP bits possible (in fact I was surprised there wasn't a version/date in the URL). But no, there's no Expires/Cache-Control headers. At least it has an ETag and does Content-Encoding (although lighttpd is broken and ignores it for HEAD requests).

But, reddit.com is still somewhat easy to dismiss as crack smokin' 13yr olds who still need a few years to grow up. After all it's not a real business, and probably doesn't make worthwhile amounts of money. But, then, newegg.com does ... and there almost everything works without JS, the notable exception being actually doing the sale. Sure, let me look at stuff and even do searches with a secure browser ... but when I need to type in my credit card, that's the time to rely on JS to do 1998 style forms.

Unsurprisingly amazon.com, as the largest online retailer, have consistently had the best UI for non-JS user-agents. And even though tags, as their latest tweak, started off as JS only then moved to working perfectly without JS within a couple of weeks. But, I guess maybe it's got less 2.0 juice now.

Current Mood: shocked

10:46 pm - HTTP for desktop applications

So to continue the HTTP theme, Miguel's idea that desktop applications communicate via. HTTP seems completely insane. I can only presume Miguel was not involved at all in the Mono HTTP API implementations. As part of writing my webserver, I've already written about how terrible the HTTP spec. is ... hell apache-httpd blatantly ignores significant parts of it.

Current Mood: scared

10:23 pm - Design by committee

So Nat says:

Today Dan Winship wrote a wonderful mail about the perils of designing software by community process.

So, I have to wonder, does Nat/Dan/Miguel think that Apache-httpd is "bad"? My feeling is that it's a brilliant example of how committee designed software is terrible, hell the fact the config. parser let's each module parse it's own syntax seems like letting each gedit plugin have it's own GUI theme. And the way the module stuff is done reaks of "vision by committee", in that everyone just does their thing in their module so they don't have to speak to everyone else.

I also think this is somewhat interesting, as it also supports something I've believed for a long time ... Ie. design/security/quality isn't all it's espoused to be. It's nice, and everyone is happy to say so but very few are willing to pay for it (either with money, or even in time/work to move from something else). Compatability, speed and dancing monkeys all pay a greater roll.

And to bring the argument back to the desktop, I even feel the same way ... I don't care how much quality design they've put into GNOME, it's all (and more) canceled out when they break focus follows mouse or infinite space on my panel buttons.

Current Mood: thoughtful

12:59 pm - And-httpd and security

Well after using, what is now, And-httpd personally for many months, I've finally released an official version, seperate from Vstr. I've also backed it up with a $500 "security guarantee", and I'm not sure if that's stupid or "insightful", I guess time will tell. There are scarily huge amount of web servers at freshmeat ... which makes me wonder why apache-httpd is still so popular, but I guess quality is different from quantity ... and most Linux distributions are reluctant to ship more than one of anything, and given you have to ship apache-httpd that kind of settles it. *sigh*.

Anyway, after releasing promises of money in return for security bugs (which I'm assuming I won't have to pay, but then I think professional poker players assume they'll win too) I'm going for a long weekend away from a computer. So don't feel too depsondent when I don't respond to messages that I'm stupid, and almost certainly $500 poorer :).