James Antill - Testing Vstr, and 100% code coverage
Aug. 30th, 2004
06:08 pm - Testing Vstr, and 100% code coverage
Well, I've finally done it. Vstr has 100% code coverage, even though bugs in GCC makes it say it's 3 lines off. It took a lot more time than I expected at the beginning and less time than I'd feared a few months ago. And it certainly found a bunch of bugs over and above just coverage testing for all the functions and flags.
I still imagine there's this huge bug that's so obvious and frequently used by everyone else, but just isn't triggered in any of the code paths I use. But, in practice, I know that if something goes wrong with code that uses Vstr, "it's not a bug in the library", which is nice.
It's kind of interesting to look back at the time line. Some of the ideas for Vstr started in the printf like function for Twocan in 1996-ish. Then there was socket_com in late 1999, which was a generalization of that idea. After a few months I realized that I needed a real string interface, and then could have a very small layer on top of that for doing network IO. So basically scrapped socket_com, although I think I continued to put work into making the printf() ISO 9899 compliant as I knew that was coming to the new project.
So Vstr actually started early 2001 (at least the mtime's say me so :), the idea was to "simply" take the socket_com code, change the namespace and make everything look like a string instead of data from the network that has random string like properties. So instead of a magic "get data from network, but only do a callback when a line comes back or if this many bytes have been received" it was more of "get data from network" + "search for data").
It's interesting to look at the numbers now, for this "simple" change...
|Date||Version||SLOC src||SLOC exmaples||SLOC tst||SLOC total|
|Oct 14th 2001||0.1.1||5,938||1,844||0||7,782|
|Jan 20th 2003||1.0.0||15,505||1,572||4,827||21,904|
|Aug 30th 2004||1.0.13||19,352||9,697||11,159||40,208|
Probably most interesting is that it took less time to go from 0.1.1 (concept, basically working) to 1.0.0 (unit test and documentation for every symbol) than it did to then go to 1.0.13 (100% unit test coverage). It's possible I was working on it harder, or just more focused, between 0.1.1 and 1.0.0 (I was very surprised to find that 1.0.12 had been released in 7 months before 1.0.13). I also did a lot of testing during 0.x and a lot of features (esp. example programs) during 1.0.x, so it isn't clear cut. However it seems reasonable to believe that they are at least close complexity wise (all the features vs. all the testing).
While I'm hesitant to say it like this I'm not certain, from a linear numbers point of view, that the code coverage was worth it. Although it's hard to measure, my gut feeling is that the number of bugs found by unit tests wasn't as big post 1.0.0 (excepting new functions -- and this seems obvious as you are talking about testing the last 15% so it should only have 17% of the bugs in the first 85%). However from an emotional point of view the argument seems much stronger, in theory a bug can be anywhere in your code and you'll never find it if you don't test it ... but if at least all lines are run at least once, you know it's not going to be a completely stupid one like:
x = NULL; *x = foo;
and the chance of any bug at all is much reduced, or maybe that's just confidence ... which seems rational for confidence to go up in proportion to the amount of effort. So going from 33% to 66%, isn't going to significantly boost your confidence of the whole but going from 66% to 99% is going to boost it a lot.
I've been thinking about writing more, in my free time, specifically about how you can test your code in C. But I wonder if anyone would actually read it, and implement it. So, as always, I've thought about finishing my own secure httpd and dnsd (httpd is in the examples for Vstr), an edge trigger IO event framework has also been on the wish list for years now. All of which would benefit me directly, based on what I did :).