You are viewing illiterat

James Antill - Lies, damn lies, and benchmarks

Jul. 31st, 2009

02:47 pm - Lies, damn lies, and benchmarks

Previous Entry Add to Memories Share Next Entry

Comments:

From:(Anonymous)
Date:August 1st, 2009 11:04 pm (UTC)
(Link)
Well one point you missed here is this:

app X does foo slower than app Y does foo.

For the user this means that X is worse than Y, the details about the implementation that causes this is irrelevant here (Y uses a cache while X does not, or X performs this additional steps).

Such benchmarks are not really helpful in finding and fixing issues, but the fact that X is slower when doing foo _is_ a problem (be it inefficient code or bad/worse design decision does not matter for the user).

Take this as an example: app A starts 20% faster than app B, so people will complain that B is worse here, B might be just doing more tasks on startup, but the user does not care about this the end result is that B _is_ slower at startup.

(well ok startup is not really a "task" but you should get the point).
[User Picture]
From:illiterat
Date:August 2nd, 2009 04:50 am (UTC)

I agree with you

(Link)
app X does foo slower than app Y does foo. For the user this means that X is worse than Y, the details about the implementation that causes this is irrelevant here (Y uses a cache while X does not, or X performs this additional steps).

If I said that I didn't mean to, by all means complain if "doing operation FOO" is significantly different. When I spoke about "yum update" the point there was that taking a small subset of the overall operation, measuring that, and then complaining about the full operation based on that small difference is misleading at best.

For example say a full operation of "yum update" takes 26 seconds, and using the same data (and getting the same result) "foo update" takes 22 seconds ... now let's say that the depsolving part of each takes 6 seconds and 2 seconds respectively. Now it's fair to say that the difference in time is likely due to the depsolver, and that yum is slower, but those 4 seconds are worth significantly less when you include the 20 other seconds needed for the operation.

Related to that, if you use something with manual synchronization like apt but don't include the "apt-get update" part then that is "unfair", and shouldn't be passed off as merely implementation. Because real users need to synchronize before doing an operation.