Re: [Lit.] Buffer overruns

blmblm_at_myrealbox.com
Date: 02/12/05 

Date: 12 Feb 2005 22:26:19 GMT

In article <28WdnVVipbvqo5PfRVn-vw@rcn.net>, <jmfbahciv@aol.com> wrote:
>In article <374769F5855kiU1@individual.net>, blmblm@myrealbox.com wrote:
>>In article <ycSdnaQz08ktW5HfRVn-rQ@rcn.net>, <jmfbahciv@aol.com> wrote:

[ snip ]

>>>Ok. I'll try to watch for different meanings. So far, I hadn't
>>>detected any.
>>
>>Hm! Because I often have that "two people divided by a common
>>vocabulary" feeling in reading your posts.
>
>The probability of that happening is 100%. Morten has whacked
>me over the head trying to figure out what my words mean. I am
>usually unaware that a term I've used all my working life (a.k.a
>DEC) doesn't exist now. It is also a fact of the computing biz
>that meanings of words change about every five years, especially
>the cybercurd.

Probably true. I have to do a quick mental shuffle every time
you mention "monitor" (I know what you mean, more or less, but
since this is a word I usually interpret differently -- and no,
not always to mean something with a screen ....).

>>>Note that there are many ways to "allocate" many
>>>flavors of space.
>>
>>True enough -- I was using "allocate" to talk about how a program
>>manages its address space (and in that context one can talk about
>>static versus dynamic allocation, stack versus heap, etc.), but
>>of course that's the application-program view, and there's also
>>the OS view ....
>
>And a general purpose timesharing OS can be at exec level (which
>is the level you think the OS is at) and/or user level.

Are you talking about -- hm, the best example I know is IBM's
VM operating system, where one OS (MVS, CMS, many instances of
either/both) can run as "user-level" code under the control of another
(VM)?

[ snip ]

>>I'm having a hard time imagining a situation in which the OS isn't
>>ultimately responsible for which processes have access to which parts
>>of physical memory.
>
>Any real time system will cause the OS to keep hands off. TOPS-10
>had an I/O method that allowed a program to tell the OS to keep
>its hands off. I can tell TOPS-10 to put my program at physical
>location nn. I can't think why I, an application, would want to
>do this, but there was code out in the that world that did.

So what happens if some other program is already using physical
location nn?

>> .. But maybe there are circumstances in which non-OS
>>processes can be trusted to do that themselves. Most of what I know
>>about operating systems comes from undergrad-level textbooks, plus
>>my vague recollections of dabbling in mainframe OS code many many
>>years ago. The principles seem not too difficult to understand,
>>but perhaps "the devil is in the details", as they say.
>
>TW would say, "A small matter of programming."
>
>Have you ever looked at SIMH emulator and played with all the
>OSes it can run?

No. Sounds interesting. I'll put it on the (long long) mental
list of stuff to read up on if you can provide a bit more info
to assist a Web search?

>> .. (I think
>>there's an example of just such a detail below.)
>
>Other people are posting more examples that sound more recent.
>Look for them.

Yeah. Got 'em.

[ snip ]

>>>>I'm not quite sure how deadly embraces come into this discussion --
>>>>seems like that's more of a problem in managing concurrency -- but
>>>>okay.
>>>
>>>I, an app program, request 100K core from the monitor.
>>>In order for it to give me that much core it has to swap
>>>out a job that will free up 100K. The only job that can
>>>be swapped out to make enough core available to me is my job.
>>
>>*Oh* .... <light bulb>
>
>Yep. Good. It can get worse. Think of a 100 of these jobs
>on the same system demanding all the same things at the same
>time.
>
>>
>>>Now put your OS hat on: How do you, as an OS, deal with that?
>>
>>Are there any choices other than "tell the requesting application
>>'no way'"?
>
>Yup. Segments is an old way. Others have posted
>other ways. The -20 implemented demand paging.

Oh. Sure. I guess I assumed from your original description that
memory-management strategies other than swapping whole jobs
in/out weren't an option.

Of course, this just shifts the problem to a different granularity,
right? so that instead of choosing a job (I keep wanting to say
"process" here, but presumably it's more or less the same thing in
this context) to swap out you're choosing a job to steal pages from,
and it's probably a good idea to be sure you don't try to steal
from yourself ....

[ snip ]

>>I still don't get what you're saying -- I mean, from the application
>>program point of view does it really matter *why* the OS won't give
>>you the memory you requested (none available, limits set by you,
>>limits set by OS, etc.)? shouldn't you mostly be concerned with
>>just checking, after you ask for some memory, that you actually got
>>it? which of course carelessly written programs don't always do.
>
>Knowing why is very important. If you don't know that, you,
>as the programmmer, cannot decide what to do without messing up.

Well .... Hm, I suppose so. Options might include asking the OS
to raise your limit (if that's allowed), or waiting and hoping
more memory will be available later, or just failing as gracefully
as possible?

>That's why our monitor would return error codes. Error codes told
>me why the monitor couldn't service me.

But of course if sloppy programmers ignore those carefully-provided
error codes .... :-)

>>However, I'm remembering that in some other branch of this thread
>>there was mention of operating systems that "overcommit", meaning
>>(if I understand right) that they sometimes grant requests for
>>memory in such a way that when the application program tries to
>>access the memory, it's not there.
>
>A request for resources that causes an overcommit is merely
>a warning from the app to the OS saying, "Be prepared."
>In an IBM data processing shop, the OS would delay scheduling
>the job until all the resources it needed was available.

Sure, I worked in such shops in a previous life. :-)

But that doesn't seem to be what the other posters mean by "overcommit"
-- they seem to be saying that the OS says to the program "here's your
memory, go ahead" in circumstances such that if the program actually
*does* go ahead and use the memory, it fails in some way the programmer
would not expect unless he/she knew about this overcommit thing.

>>Now *THAT* would be a situation in which even a carefully-written
>>program could come to grief.
>>
>>(I have to say that I find this rather shocking behavior on the
>>part of the OS. But perhaps I lack an appreciation for how in the
>>real world sometimes performance matters more than "correctness"?
>
>It's perfectly reasonable behaviour for an OS to do this.
>Think about OS developers who have a teensy little system but
>wish to service users as if it were a GREAT BIG SYSTEM.
>Overcommitting is an obvious tradeoff.

I think at least one of us is not understanding what "overcommit"
means in this context.

If you have a limited amount of physical memory but want to service
users as if you had a lot of memory, well .... You implement virtual
memory, no? so when applications request memory, their requests don't
necessarily have to be satisfied out of physical memory available
right then, but IMO the OS should be making sure it has the swap-space
resources to deal with the requesting process. I vaguely remember
that the MVS (old IBM mainframe OS -- you may know this) systems I
used would refuse to start a job unless there was enough swap space
for it. What seems wrong to me is to go ahead and start the job,
and then let it get into nasty trouble when it actually needs the
swap space that isn't there. "YMMV", maybe.

(I'm sure we both know that a teensy little system pretending to be
a GREAT BIG SYSTEM can, um, encounter performance problems.)

>>My idea is that it doesn't matter how fast something runs if it
>>doesn't give the right answers, but that may be naive.)
>
>It depends on how you want the computer to be used.

Yeah. As you said, there have been some other posts giving
examples ....

I'm apt to focus on the motto "first make it work, then make it
fast" -- but I'm dimly aware that there is also a competing motto,
"sometimes late answers are wrong answers."

Engineering, including software engineering (assuming that term
means anything at all), is full of trade-offs.

[ snip ]

>>>>Well, with a language that doesn't do bounds checking, you aren't
>>>>going to get a nice obvious error message like "buffer overflow,
>>>>you idiot!" Instead you're going to get either a possibly-cryptic
>>>>error message/code from the OS telling you the program tried to
>>>>access memory it didn't own (well, with a reasonably sophisticated
>>>>OS -- I once found out the hard way that MS-DOS in its early form
>>>>didn't try to defend its own data areas from application programs,
>>>>and was *that* ever a learning experience, after years in the
>>>>mainframe world), or .... Well, when you overflow a buffer, but
>>>>not so badly as to go outside your program's assigned memory,
>>>>the result is likely to be the trashing of some data area unrelated
>>>>to what you thought you were doing, and the symptoms can be pretty
>>>>much anything, including the security problems that were the initial
>>>>subject of this thread.
>>>
>>>Yup. My methods of defending against that on these toys I use,
>>>is to have the keyboard code where any overwriting will cause
>>>keyboard freezure.
>>
>>Say whaaa ....? I have no idea what you're talking about here.
>
>Oh, I use a Windows 3.1 under DOS 6.0. If I am writing up
>an email or a post, and the buffer where the text is getting stored
>grows too large, the first thing the fucking OS will over write
>is my keyboard device driver which freezes the screen. This
>avoids the OS deciding there is something wrong, hurry up and
>write the FAT (which just got clobbered), then display itself.
>This always causes me to have to rebuild the file system.

"!@#$ing OS" indeed. I started out in the mainframe world and
was blissfully unaware for a long time that PC OS's didn't
necessarily defend themselves from users as vigorously as the
mainframe OS's I was used to. The experiences that taught me
otherwise stil make good stories. (Or so I think, anyway.
The audiences on which I inflict them may think otherwise.)

Anyway it sounds like Windows 3.1 is in the doesn't-defend-itself
category. How annoying.

>>Is this something you do as a user running programs that might
>>overflow buffers, or something you can in coding the programs, or
>>what?
>
>It's how I accidently laid out the device drivers on this DOS
>machine and discovered that making the keyboard die was less
>painful than making the code servicing the file system die.

So, you came up with a creative way to protect yourself from the
worst consequences of the OS's undesirable behavior. Cool!

>>Sure. But as I said above, it seems overwhelmingly likely that
>>most students will at some point write a program that fails with
>>the cryptic error message, and -- well, yeah, many won't associate
>>that with some lecture they heard weeks ago, but they'll ask
>>the instructor or their friends or someone, and *then* they'll
>>make the association. And the more time and frustration is
>>involved in getting from the cryptic message to the source of
>>the problem, the better they'll remember. :-)
>
>And when they discover that the programmer they've been swearing
>at is themselves, the embarassment will train them.

That's the hope! "What idiot wrote *this* co .... Oh."

>>Unfortunately, even then they're apt to be sloppy about putting in
>>good error checking. Hm, maybe that means that eventually the memory
>>fades of how they spent hours researching the cryptic error message,
>>and it turned out to be an out-of-bounds access .... Maybe it takes
>>several of those experiences to really get the point across.
>
>The best technique to cycle from maintenance to developement
>and back.

Probably. That also teaches the value of making your code
understandable to human readers and not just the compiler.

>> ..For
>>a while I worked as a maintenance programmer on a moderately big
>>and complicated program, and when you spend enough hours tracking
>>down bugs in other people's code that turn out to be out-of-bounds
>>access ....
>
>And that was the top bandaid you took off. :-)

"The top band-aid?" By the time I quit that job, I was pretty good
at finding such bugs and patching them. Extensive code reviews and
rewrites, though, weren't really in my job description, nor likely
within my abilities.

>>>Warning: Thread knot. To answer a question you asked some
>>>time ago but I didn't want to answer until I'd looked it up
>>>because I knew I'd *** it up if I answered from memory...
>>>
>>>m^2c^4 = sqrt( e^2 -p^2c^2)
>>
>>Well, now I'm going to have to go back and find out exactly what it was
>>I asked -- I vaguely remember that there *was* a question related to
>>the formula E = m c^2,
>
>IIRC, I told you E-mc^2 was a special case. The whole formula
>is one I just wrote.

Ah yes, I remember that exchange now. Thanks.

>And there's something wrong with what I wrote but I have no idea what.

I think so too (that there's something wrong) -- my first thought
on reading your formula was how it related to the E = m c^2, and
that maybe it reduced to the familiar one if p=0. (What is p here,
by the way?) But the exponents are wrong. Are you sure that "sqrt"
needs to be there? If it's not, then this formula reduces to the
familiar one when p=0. ?? 

-- 
| B. L. Massingill
| ObDisclaimer:  I don't speak for my employers; they return the favor.
Quantcast