You are viewing illiterat

James Antill - Testing Vstr, and 100% code coverage

Aug. 30th, 2004

06:08 pm - Testing Vstr, and 100% code coverage

Previous Entry Share Next Entry

Well, I've finally done it. Vstr has 100% code coverage, even though bugs in GCC makes it say it's 3 lines off. It took a lot more time than I expected at the beginning and less time than I'd feared a few months ago. And it certainly found a bunch of bugs over and above just coverage testing for all the functions and flags.

I still imagine there's this huge bug that's so obvious and frequently used by everyone else, but just isn't triggered in any of the code paths I use. But, in practice, I know that if something goes wrong with code that uses Vstr, "it's not a bug in the library", which is nice.

It's kind of interesting to look back at the time line. Some of the ideas for Vstr started in the printf like function for Twocan in 1996-ish. Then there was socket_com in late 1999, which was a generalization of that idea. After a few months I realized that I needed a real string interface, and then could have a very small layer on top of that for doing network IO. So basically scrapped socket_com, although I think I continued to put work into making the printf() ISO 9899 compliant as I knew that was coming to the new project.

So Vstr actually started early 2001 (at least the mtime's say me so :), the idea was to "simply" take the socket_com code, change the namespace and make everything look like a string instead of data from the network that has random string like properties. So instead of a magic "get data from network, but only do a callback when a line comes back or if this many bytes have been received" it was more of "get data from network" + "search for data").

It's interesting to look at the numbers now, for this "simple" change...

Date Version SLOC src SLOC exmaples SLOC tst SLOC total
Oct 14th 2001 0.1.1 5,938 1,844 0 7,782
Jan 20th 2003 1.0.0 15,505 1,572 4,827 21,904
Aug 30th 2004 1.0.13 19,352 9,697 11,159 40,208

Probably most interesting is that it took less time to go from 0.1.1 (concept, basically working) to 1.0.0 (unit test and documentation for every symbol) than it did to then go to 1.0.13 (100% unit test coverage). It's possible I was working on it harder, or just more focused, between 0.1.1 and 1.0.0 (I was very surprised to find that 1.0.12 had been released in 7 months before 1.0.13). I also did a lot of testing during 0.x and a lot of features (esp. example programs) during 1.0.x, so it isn't clear cut. However it seems reasonable to believe that they are at least close complexity wise (all the features vs. all the testing).

While I'm hesitant to say it like this I'm not certain, from a linear numbers point of view, that the code coverage was worth it. Although it's hard to measure, my gut feeling is that the number of bugs found by unit tests wasn't as big post 1.0.0 (excepting new functions -- and this seems obvious as you are talking about testing the last 15% so it should only have 17% of the bugs in the first 85%). However from an emotional point of view the argument seems much stronger, in theory a bug can be anywhere in your code and you'll never find it if you don't test it ... but if at least all lines are run at least once, you know it's not going to be a completely stupid one like:

x = NULL; *x = foo;

and the chance of any bug at all is much reduced, or maybe that's just confidence ... which seems rational for confidence to go up in proportion to the amount of effort. So going from 33% to 66%, isn't going to significantly boost your confidence of the whole but going from 66% to 99% is going to boost it a lot.

I've been thinking about writing more, in my free time, specifically about how you can test your code in C. But I wonder if anyone would actually read it, and implement it. So, as always, I've thought about finishing my own secure httpd and dnsd (httpd is in the examples for Vstr), an edge trigger IO event framework has also been on the wish list for years now. All of which would benefit me directly, based on what I did :).

Current Mood: pleasedpleased

Comments:

From:(Anonymous)
Date:July 29th, 2005 02:39 pm (UTC)
(Link)
FYI, 100% code coverage does not mean you can be sure a problem is not caused by a bug in your code. For one thing, it's possible to "cover" a piece of code without actually covering all of the contexts in which it gets called. The very same pointer dereference on the very same line can be fine one time and a crash the next, and that's still a bug even if you tested the first case. Someone could write code that completely omitted any kind of error checking, and get 100% coverage by just exercising the case where everything's in an ideal state. It's very common for the same mental blind spot that led to there being a bug in the first place to show up as a missing or incorrect test case as well, if the two were written by the same programmer. Higher coverage is better than low, but it's no guarantee that code is bug-free.
(Reply) (Thread)
[User Picture]
From:illiterat
Date:July 29th, 2005 03:05 pm (UTC)

Re: Still not right...

(Link)
Wow ... I'm impressed, not only do I specifically say that on the Vstr page but if you read all of the above (specifically the end of the third to last paragraph) I find it hard to believe you could think I don't know that.
(Reply) (Thread)