Re: Linux as resilient as BSD, resource hogging?

From: Kasper Dupont (kasperd@daimi.au.dk)
Date: 08/03/02 

From: Kasper Dupont <kasperd@daimi.au.dk>
Date: Sat, 03 Aug 2002 01:55:10 +0200

"John S. Dyson" wrote:
>
> "Kasper Dupont" <kasperd@daimi.au.dk> wrote in message news:3D4AE1DB.4813C5B3@daimi.au.dk...
> > Jem Berkes wrote:
> > >
> > > I know that there are several ways that a regular user can, on a Linux
> > > system, disable/crash the system by doing things that use up all available
> > > resources. For instance, $0&$0 (using bash), or perl fork bomb -- and
> > > ulimit certainly doesn't stop everything.
> > >
> > > I'm guessing that BSD can withstand such things -- it's more popular among
> > > ISPs who have to deal with many, potentially untrustworthy users. Why can't
> > > Linux handle these resource hogging issues? What mechanisms are there in
> > > BSD that we can add to Linux?
> >
> > I don't know what BSD does, but I could point out a few things that
> > could be done better in Linux. The following ideas is not aimed at
> > one particular resource, most of them can be used for all resources.
> >
> > Ulimit is good, but it does not solve all problems like some people
> > tend to claim. We have a limited amount of a resource, and we try
> > to manage it by assigning a maximum to each user.
> >
> One data point: I did fully implement a memory controlling ulimit (admittedly
> on a per-process basis) on FreeBSD. This made sure that the resident set
> was truly limited, and replaced pages on a working-set model, rather than
> global. I ran numerous tests of various algorithms (from complex to simplistic),
> and the general result was that adding a hard memory limit (unlike the soft
> limit as currently implemented) tended to increase disk traffic enough to cause
> an aggregate performance decrease. It is likely possible to mitigate the effect
> of the additional memory traffic, but I didn't persue that direction any further.
> My goal, over 5yrs ago, in the FreeBSD VM work was to maintain good traditional
> UNIX-like behavior, but if the goal would have been biased more towards a
> system where resource mgmt would have been needed, it would have taken
> ALOT more than a simple VM resident set limit to make the system run better
> as an aggregate.
>
> IMO, when the goal of the algorithm (in the sense of memory mgmt) is to
> minimize I/O (while supporting the continued runnability of processes), then
> the system wide global performance will be maximized. Fairness WRT memory
> hogs shouldn't be *solely* managed by a memory limit policy, if that policy
> causes additional paging. This MIGHT be a case where a 'swapping' or higher
> level scheduling concept could be worthwhile, when the memory hog that naturally
> increases the system wide paging rate (either with a memory limit or not) thereby
> seriously negatively impacting the performance of other processes, then the
> *hog* process would be suspended. The performance would become choppy,
> but the paging device resource (response time, transaction rate, transfer rate)
> does become problematical.
>
> On FreeBSD, it really does fairly well achieve a stable and apparently near
> minimal paging behavior for 'normal' programs and program mixes. However,
> it really does have the same kind of natural, pathological behavior, common
> to many other OSes, when a single process can demand and force lots of paging
> in the system. I tend to think of it like this: if a process is a VM memory hog,
> and it forces alot of paging activity (which causes system wide resource
> contention), then it is not generally worthwhile to force more paging by limiting
> the memory resources available to that process. There might be some cases
> where the process can be 'bottled up' and limit the system-wide impact, but
> I found that it is pretty clear that a hard memory limit isn't the right answer alone.
>
> The problems with a paging hog need to be considered WRT to paging latencies
> as well as the memory hog program itself. Paging latencies can be helped a little
> bit by allowing fewer (perhaps one) program to be runnable and competitive for
> memory at any given time. This can make the system very choppy, and is (of course)
> not a general solution. Penalizing the 'hog' process by allowing it to run effectively
> as a background MEMORY priority, might also be a solution, unless the hog has
> realtime constraints of it's own.
>
> (This is just meant as a data point for one aspect of the resource mgmt issue.)
>
> John

OK, perhaps I didn't make my statements clear enough.

In the scenario you describe there are actually at least
three different resources involved:

1) Virtual memory/swap
2) Physical RAM
3) Disk I/O bandwidth

All three resources must be properly managed, but due to
their different nature they should not be managed exactly
identical. In my previous posting I properly didn't make
the distinction between 1 and 2 clear enough.

Resource number 1 virtual memory should be carefully
limited, because we cannot withdraw it when we have first
given it to the process.

Resource number 2 is very tricky, we can come back to that.

Resource number 3 is fairly simple to manage, this is just
hardware access for a periode of time, it can like CPU time
be withdrawn right away at the end of the current sector
transfer. The only tricky thing about this resource is that
fairness and efficiency does not play well together. An
elevator algorithm could achieve good efficiency, but will
sometimes tend to be unfair and give one process all the
bandwidth until it finishes all it's data. So in addition
to that, some fairness must be ensured.

The physical memory is tricky, we can withdraw it from a
usermode process. But it cannot be withdrawn for free. We
have to swap data out and later in again. And there are
so many different uses for memory that all have to be
considered. Limiting the amount of memory a process can
use can be good for other processes, but the limited
process can get hurt badly. The fact that it starts
trashing badly should not hurt other processes performance
too much, if we just manage disk bandwidth cleverly.

But still in some cases we should allow a single process
to use almost all our physical memory for short periods of
time. For some computations having the full working set
in memory is very critical to performance. The difference
between 99% of the working set in RAM and 100% of the
working set in RAM can literally change the speed at which
the computation completes by a factor 10000. So if we have
256MB of RAM and a process with a working set of 230MB
we'd better give it that much memory once in a while and
swap it completely out once in a while. It would complete
a lot faster that way. Unfortunately it could easilly
take a minute swapping such a large amount of data out
and other data in, so we'd need very long timeslices. I
do think that the current 2.4 kernels handle this quite
well, but I also do think that in some extreme cases the
2.2 kernel would actually do better. I think it is a lot
easier to get 2.2 remove all libraries from RAM than it
is with 2.4. In some cases removing the libraries for
short time can be good, but in most cases it is bad.

I think the major problem with the way Linux currently
uses physical RAM is, that data that has only been
accessed once or twice gets to stay too long in memory.
Try reading a 1GB file sequentially, you will find that
almost all your memory gets used for keeping part of
this file in cache. That is not good, the file contents
should rather be removed quickly again, because they are
not going to be used again soon. This does not only
apply to reading files, but also to mmaped files, and
scanning of directory structures. 

-- 
Kasper Dupont -- der bruger for meget tid på usenet.
For sending spam use mailto:razrep@daimi.au.dk
or mailto:mcxumhvenwblvtl@skrammel.yaboo.dk

Relevant Pages

  • Re: Linux as resilient as BSD, resource hogging?
    ... > one particular resource, most of them can be used for all resources. ... I did fully implement a memory controlling ulimit (admittedly ... causes additional paging. ... *hog* process would be suspended. ...
    (comp.os.linux.security)
  • Re: C++ sucks for games
    ... Just a typo that can be horrendous to ... but if your memory management is scattered and out of control, ... >>implement raii, and i think that's a serious flaw. ... resource - and resource can be anything at all. ...
    (comp.lang.lisp)
  • RFC: Memory Controller
    ... We've seen a lot of discussion lately on the memory controller. ... limit the usage of the resource to the specified value. ... with the specified guarantee of resource availability. ...
    (Linux-Kernel)
  • Re: RAM-CPU Singularity
    ... resource to compensate the urgent needs of another kind of resource. ... That's why the original thinkers came up with the cache stuff: ... It's the same with processors and memory. ... I guess you called "virtual memory" the swap space on disk, ...
    (comp.theory)
  • Re: [ckrm-tech] [RFC][PATCH] UBC: user resource beancounters
    ... kernel memory and virtual memory. ... fair CPU scheduler is not tied to UBC in any regard. ... the infrastructure (for resource management) we ... it is also more logical to have them separate since they ...
    (Linux-Kernel)