James Antill - Software packaging, and the 10 × 10 problem
Jun. 20th, 2007
05:09 am - Software packaging, and the 10 × 10 problem
First off, I hate all current software packaging for Linux. It's one step up from manually downloading tarballs, which I was doing 10 years ago, and isn't even a real superset of that functionality. Yes, it's better than doing that and yes it's better than what Windows has to offer. But it's still crap, and it annoys me every day, however it seems to be one of those problems that noone seems to want to fix properly and yet they keep thinking up stupid bandaids to work around the fact it doesn't work well. Specifically the major circular argument that is what I've come to think of as the "10 × 10 problem"…
Repeatedly over the last 5+ years I've tried to tell people about how we should be solving this problem, and I've mostly got dismissed … so here it is for posterity.
One of the currently major problems with software packaging and distribution is updating it. All software has bugs, or missing features and those affect different people in different ways. Say you have two people, one who is writting a new web application in python using postgresql and one who is just browsing the web, sharing some files via. the web server and running his own blog. It probably seems obvious to most sane people that the updates these two people want will vary wildly for the five packages: firefox, python, ipython, postgresql, apache-httpd. That is, unless you are in the business of distributing software packages.
With software packaging/release/whatever the arugment is most often about who gets screwed over least, in the above example a common "solution" is to do very minor updates for firefox and maybe ipython but nothing else … want firefox-2, or the latest postgresql/python? Too bad. Wait for the next "major" release and get those with everything else. A common painful package is the Linux kernel, mainly just because everyone uses it, and so all the software packagers argue about how often it should/shouldn't be updated. This is roughly the same as arguing what single human language would be best for the entire OS, if everyone should use python or C to develop all applications or what font everyone should use for all text. There is no correct answer, because the entire question is completely insane.
When I first thought about this, I was visiting/speaking with quite a few different Linux customers and I came up with the general idea of the "10 × 10 problem". The general statement of the problem is that you have 10 customers, each of which have 10 different "changes" they want for the next release. The current "solution" to the problem involves picking a number between 0 and 100, doing that amount of changes and giving the result to everyone so that you have a very small number of "releases" which can be managed by hand. This rarely makes anyone happy, and is basically what Microsoft; Sun and SCO have done for the last 10 years.
Obviously in the real world things are more complicated than the above, one of the most common differences is that each of the 10 customers has changes that are important enough they don't want to wait for "the next release" and they'll also want some of the changes you've done for the other people but also have desires like "apart those changes, I don't want anything else to change". This also destroys any hope of there ever being a magic number of changes you can do, that will actually make everyone happy. And yet, still we get arguments about what the best magic number is.
The "solution" to this problem is to admit defeat and stop managing software packaging and distribution like it's 1995, imagine for a moment that each "customer" had their own private software packaging and distribution team and used that instead of paying someone outside to do the work for them. In this senario it's very likely that all the high priority changes would be done immediately, and that all 10 changes would make it into "their" next release with no other changes. Now imagine that all those private teams communicated openly with each other, this would result in a similar outcome with the added benefit that significant changes done by one team that are desired by others would be merged into their releases.
The common complaint about why this "can't be done" by an external entity is that managing so many releases by hand is "hard" and would cost too much money. In my so humble opinion this is a bit like saying that managing more than one version of a C source file is hard and time consuming, as against implementing some decent SCM tools so that you can manage 100s of 1,000s of different version sets. The next common complaint is that the entire OS needs to be tested as one unit, which is mostly untrue and esp. so when you are talking about changes the average customer desires … and again, any truth in this is mostly a lack of decent tools.
The reason this could never have happened over the last 20 years is that the code itself was hidden from the customers, so they had no choice except to take what they were given by their software packaging and distribution company. This is not the case now, and I've already started to see the emergance of people implementing the good solution privately because they can't buy it from anyone. I'm just waiting for someone to give people what they want, and charge for it ... and I hope I work for the company that does it.
As a final optimistic note, I have helped the real solution come a tiny bit closer to reality. With Fedora 7 you can now install the yum-security package, and do things like "yum --security update -y" to get just security updates or "yum --bz 1234 update -y" to get just the updates which fix bugzilla.redhat.com#1234.