Log in

Lies, damn lies, and benchmarks - James Antill

Jul. 31st, 2009

02:47 pm - Lies, damn lies, and benchmarks

Previous Entry Share Next Entry


[User Picture]
Date:August 2nd, 2009 04:50 am (UTC)

I agree with you

app X does foo slower than app Y does foo. For the user this means that X is worse than Y, the details about the implementation that causes this is irrelevant here (Y uses a cache while X does not, or X performs this additional steps).

If I said that I didn't mean to, by all means complain if "doing operation FOO" is significantly different. When I spoke about "yum update" the point there was that taking a small subset of the overall operation, measuring that, and then complaining about the full operation based on that small difference is misleading at best.

For example say a full operation of "yum update" takes 26 seconds, and using the same data (and getting the same result) "foo update" takes 22 seconds ... now let's say that the depsolving part of each takes 6 seconds and 2 seconds respectively. Now it's fair to say that the difference in time is likely due to the depsolver, and that yum is slower, but those 4 seconds are worth significantly less when you include the 20 other seconds needed for the operation.

Related to that, if you use something with manual synchronization like apt but don't include the "apt-get update" part then that is "unfair", and shouldn't be passed off as merely implementation. Because real users need to synchronize before doing an operation.

(Reply) (Parent) (Thread)