Re: [Lit.] Buffer overruns
From: infobahn (infobahn_at_btinternet.com)
Date: 02/02/05
- Next message: infobahn: "Re: [Lit.] Buffer overruns"
- Previous message: David Wagner: "Re: [Lit.] Buffer overruns"
- In reply to: David Wagner: "Re: [Lit.] Buffer overruns"
- Next in thread: Trevor L. Jackson, III: "Re: [Lit.] Buffer overruns"
- Reply: Trevor L. Jackson, III: "Re: [Lit.] Buffer overruns"
- Reply: Paul Rubin: "Re: [Lit.] Buffer overruns"
- Reply: Douglas A. Gwyn: "Re: [Lit.] Buffer overruns"
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] [ attachment ]
Date: Wed, 2 Feb 2005 10:26:01 +0000 (UTC)
David Wagner wrote:
>
> So, it sounds like your secret sauce for catching inadvertent bugs is in
> the N-person code review stage.
No. There is no secret sauce. There is no silver bullet. The simple
truth is that you will continue to get bugs in your code for as
long as you're prepared to put up with them. If you don't insist
on people who know what they're doing, you'll continue to get bugs
in your code. Hire good people, let them redesign your core development
processes to place the emphasis on quality (and I don't mean all
that ISO 9000 nonsense where any old rubbish goes as long as it's
documented; I mean "knowing what you're going to do, knowing how
to do it right, and then actually doing it right, and checking that
you've done it right"). Give them the time they need to do the work
properly. The bug rate will fall through the floor.
> Code reviews are good, but I don't know
> how to be sure they're going to catch all buffer overruns. I don't have
> any data, so at the end of the day perhaps this comes down to experience.
Experience and knowledge, yes.
> I can only speak about my own experience. I've done many code reviews,
Not of C code, I trust. With the greatest respect, I don't think
you can do that effectively. Not yet, anyway, although I'm
doing what I can to hasten the day when you can!
> so I know how easy it is to miss a buffer overrun bug, and I find it
> all too easy to imagine buffer overruns getting missed by multiple
> reviewers. I can tell one story of a 100-line program that we *knew*
> had a bug somewhere (it was producing the wrong output) and that
> three of us stared at for hours without being able to see the bug.
> Even knowing it was producing the wrong output, we looked at the code
> and were certain the code had to be right. Turns out our certainty
> was misplaced, and the code did have a subtle bug that was missed by
> all three of us. That was not a buffer overrun, but it illustrates
> my fears of how correlated the process of finding bugs seems to be.
> It seems like some bugs are overlooked despite many eyeballs.
It seems to me that your anecdote merely outlines the value of
knowing the rules of the language you're using.
> Maybe you have lots of experience with 3-person code reviews and can
> say something like "Nope, after 50,000 lines of code reviewed, I have
> never seen a buffer overrun bug get past all 3 of our code reviewers".
> Can you say something like that?
Easily (for random values of 3 and 50,000), because it's trivially
true. :-) Obviously, if I /had/ seen a buffer overrun bug get past
me, it wouldn't have got past me.
> Can you put some numbers on it?
> And can you say how you can tell that no buffer overrun was missed?
Maybe you have lots of experience with cryptography, and can say
something like "Nope, after 50,000 secret messages encrypted and
sent over the Net, I have never found evidence that a message was
decrypted without the key."
How can you tell that no message was decrypted without the key?
Does this mean we needn't bother to use well-crafted algorithms,
since we can't prove they do any good? Does it mean we must never
send encrypted data over the Net, just in case Eve manages to
decrypt it? Of course not.
> Quantifiable experience with N-person ought to trump my intuition about
> the limits of code reviews and my extrapolations from my experience with
> 1-person code reviews.
In my experience, a successful code review has the following
characteristics:
1) it takes place within the context of a development environment
in which the specs are clear and the interfaces well-defined;
2) the programmer is willing to follow the intelligently-written
in-house style guide (no, we're not talking about bracing style
here!);
3) the programmer and the reviewers are all knowledgeable about
the C language and standard library, and any other libraries,
in-house or otherwise, and documentation for all of these is
readily available - *including* the language specification;
4) the programmer and the reviewers all recognise the importance
of good quality code;
5) the reviewers are given copies of the functional spec for
the unit (function, module, whatever) under review, the code,
the test plan, and any localised test harness, well in advance
of the review meeting;
6) ample time is reserved for the meeting;
7) as well as the programmer, the review should comprise at
least four other people:
* an arbiter, whose knowledge of C should be sufficiently
encyclopaedic that any decisions he has to make can be
accepted by all;
* a note-taker, whose notes will be typed up and circulated
after the meeting;
* at least two other reviewers. More than four others is
probably counter-productive, for what I hope are obvious
reasons.
8) the arbiter's decision, in those rare cases where he must
decide because there is no consensus, is final, so he has
to be given sufficient authority to make that decision
stick.
9) the amount of code to be reviewed in one sitting should not
be so much that the meeting goes on interminably. People
have work to do. Review a little code, often.
This is NOT a complete guide to perfect code reviews. It's just a
potted guide, and whilst those who conduct satisfactory code reviews
within their own organisations will, I think, recognise most or
all of the points I've made above, they will undoubtedly note at
least one or two other points that I have perhaps overlooked in
this summary.
> But if you don't have quantifiable experience like this, how can you
> be so confident that N-person code reviews suffice to eliminate all
> buffer overruns? How do you know?
How do you know nobody has cracked Twofish? Either it has been
cracked or it hasn't. If it has, maybe you're aware of it and
maybe you're not. If you're not aware that it has, how do you
know that it hasn't? And if you don't know that it hasn't, how
can you entrust any data to it?
Um, that reads rather confusingly.
DW is aware DW is not aware
+-------------------+------------------+
Twofish HAS | DW won't trust | Does DW trust |
been cracked | TF until it's | TwoFish? |
| been fixed. | |
+-------------------+------------------+
Twofish has | | Does DW trust |
NOT been | Unprovable. | TwoFish? |
cracked | | |
+-------------------+------------------+
So: do you trust TwoFish? Even though it may have been cracked without
your knowledge?
One answer I would consider to be worthwhile is this: "Twofish was
designed by a bunch of security experts who collaborated to produce
the best algorithm they could devise; it was thoroughly analysed by
experts, and no practical attack was found. We can't guarantee it's
secure (because, no matter what the mathematical pedants think, we
can't prove this particular negative), but our considered expert
opinion is that it is trustworthy. Caveat downloador, but FWIW we
trust it ourselves."
Sound familiar?
- Next message: infobahn: "Re: [Lit.] Buffer overruns"
- Previous message: David Wagner: "Re: [Lit.] Buffer overruns"
- In reply to: David Wagner: "Re: [Lit.] Buffer overruns"
- Next in thread: Trevor L. Jackson, III: "Re: [Lit.] Buffer overruns"
- Reply: Trevor L. Jackson, III: "Re: [Lit.] Buffer overruns"
- Reply: Paul Rubin: "Re: [Lit.] Buffer overruns"
- Reply: Douglas A. Gwyn: "Re: [Lit.] Buffer overruns"
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] [ attachment ]
Relevant Pages
|
|
