You are viewing illiterat

James Antill - Software packaging, and the 10 × 10 problem

Jun. 20th, 2007

05:09 am - Software packaging, and the 10 × 10 problem

Previous Entry Add to Memories Share Next Entry

Comments:

[User Picture]
From:fanf
Date:June 20th, 2007 07:25 am (UTC)
(Link)
A big problem is cascading dependencies, where upgrading app A pulls in an upgrade to library B which is not reliably backwards-compatible, so you have to upgrade app C to a version compiled against the new B - but you didn't want C to change.

The solution to this is to allow multiple parallel installations of packages, so that the two versions of B are not forced to conflict. This is useful for applications as well as libraries. If I can install the new version of A alongside the old version, then it's really easy to revert the upgrade - I just have to change a PATH or a symlink and restart it. I can also test the upgraded A (at least a little bit) before committing to running it. Package deinstallation then becomes a garbage collection job, instead of being crucial for the correctness of the package management tool.

OpenPKG sort-of does this, but it does so by allowing multiple installations of the package manager, so it's rather coarse-grained which means you lose sharing between apps whose version dependencies you want to decouple.

Nix claims to do exactly what I want, but I have not investigated it in detail...
[User Picture]
From:illiterat
Date:June 20th, 2007 02:12 pm (UTC)

Dependancies aren't as big a problem as assumed

(Link)
A big problem is cascading dependencies, where upgrading app A pulls in an upgrade to library B which is not reliably backwards-compatible, so you have to upgrade app C to a version compiled against the new B - but you didn't want C to change.

Kind of, there are certain packages which tend to depend on newer core libraries but it's not obvious how much you'll just be able to sign off on the deps. too. For instance large GUI based ones, like evolution, often depend on newer libs. ... but I think in the real world it'd be a pretty strange requirement that you have the latest evolution but libgtk etc. can't change.

Much more likely is that you treat applications as groups with most or all of their deps. The easiest cases are things like "I've install fish, am on the mailing list and so want the latest version of that" ... or the "web developer" problem where you don't want a 2-5 year old PostgreSQL/Apache-httpd on an OS that you'd otherwise classify as "very stable".

I'm prepared to be proven wrong though :), and I do think solving the multiple versions problem (while hard) is doable … and the solution is very likely to add useful features to the package manager/distribution layer. But I might not start out trying to solve that problem first :).

I haven't heard of Nix, although I have played with rPath a bit and that's a significant step forward. I'd looked at OpenPKG before and the "pipe everything into sh" part scares me a bit, also while it seems like it has a couple of nice things for solving some of the problem it doesn't seem to attack the problem itself (so managing N slight variations of Fedora, say, wouldn't be significantly easier for the consumer or producer).

[User Picture]
From:fanf
Date:June 20th, 2007 02:40 pm (UTC)

Re: Dependancies aren't as big a problem as assumed

(Link)
Right, treating apps and their deps as a group is basically what I'd like to achieve. It's easy to install a new app and pull in its dependencies automatically if you can be certain that this will not disrupt a running system. If each dependency is a separate package then you get sharing where possible.

Regarding your Evolution/gtk example, I'd say that if you are a heavy user of a gtk application then you don't want your work disrupted by a forced upgrade to the latest version just because Evolution wanted a new gtk. So the problem is not that gtk can't change, it's that you need to keep the old version around to support the important application.

The main difficulty with solving the multiple versions problem is that autoconf and libtool are built on the assumption that there's one of each thing on the system installed in a "usual" place. Fixing build scripts so that they work when you break this assumption is a tedious and soul-destroying occupation.
From:(Anonymous)
Date:January 10th, 2008 10:56 pm (UTC)

Re: Dependancies aren't as big a problem as assumed

(Link)
You've identified the problem correctly - libraries aren't backwards-compatible. The solution you propose is not so good. Why not "fix the problem"? Come up with an API/policy to FORCE libraries to be backwards compatible? Essentially, if a new function needs to be added, or an existing function must be modified, create a new function name and add it to the API. Errors in existing API functions can be corrected without renaming. Or somesuch policy.
[User Picture]
From:illiterat
Date:January 11th, 2008 05:53 am (UTC)

Re: Dependancies aren't as big a problem as assumed

(Link)
Why not "fix the problem"? Come up with an API/policy to FORCE libraries to be backwards compatible?

Even if that was "just" very hard, how would you force libraries to do it? It's like saying the solution to a problem is to get all web pages to use white text on a black background. And a lot of libraries are backwards compatible at an interface API/ABI level, but that doesn't help as sometimes the only way to be 100% backwards compatible is to not change.

Essentially, if a new function needs to be added, or an existing function must be modified, create a new function name and add it to the API. Errors in existing API functions can be corrected without renaming. Or somesuch policy.

And what do you do when the function is called from 6 other functions, that you also export ... do you now have to create 7 new functions to do a 1 line change? And then what do you do for the applications that call the function which has changed behaviour ... do they call the new function or the old one? What about for the two scripts calling the applications, one of which wants old and one new? Pretending you can solve all the problems with a single result just doesn't work.