Re: use ipchains to block all ports > 60,000

From: erik (erik_at_geenspam.vanwesten.net)
Date: 08/28/03 

Date: Thu, 28 Aug 2003 01:46:23 +0200

Neil Sandow wrote:

>
>
> Tim Haynes wrote:
>
> > Neil Sandow <rx@rxlist.com> writes:
> >
> >
> >>> Next question: does that persist on subsequent runs? Does the
> >>> number vary, if so, by how much?
> >>
> >>
> >> Pretty much the same every time, might vary by 1 or 2
> >
> >
> >
> > OK. So there's a consistent difference, quite a gross one. A
> variation of 1
> > or 2 is experimental error; in a false-positive report, I'd expect
> > chkrootkit not to be able to discover between 0-2 pids, once every
> > quite-a-few runs, if the box is heavily fork()ing processes left
> > right & centre. A near-as-dammit-constant large-number like 27
> > going AWOL isn't cool.
>
>
> Yes, but running ./chkrootkit -x lkm shows all the process id's and
> they
> are no different than what ps shows. It's not like there are hidden
> processes going on that shouldn't be running (as far as I can tell)

That does not mean anything... yet.

>
> >
> >
> >>>> When I run ./chkrootkit -x lkm it shows me the processes and
> >>>> when
> I check
> >>>> the output of ps (ps -edf or ps aux) I see all the same
> >>>> processes
> id's.
> >>>> I've read that you can get false positives for short lived
> processes that
> >>>> may have finished while chkrootkit is running but these are all
> >>>> stable running processes such as init, sshd, mingetty, etc. I'm
> >>>> not sure
> why I'm
> >>>> getting this error since ps and chkproc both show identical
> process id's.
> >>>
> >>>
> >>> How about `ls /proc/[0-9]* | wc -l' and compare that with `ps
> auxww | wc
> >>> -l'
> >>> several times over?
> >>>
> >>
> >> Hmmm very different.
> >>
> >> `ls /proc/[0-9]* | wc -l' returns 377 (every time)
> >>
> >>
> >> `ps auxww | wc -l' returns 28 (every time)
> >
> >
> >
> > OK, I cocked-up on the first of those. Woops I should've requested:
> >
> > ls -1d /proc/[0-9]* | wc -l
> >
> > instead - a simple count of how many second-level directories
> > comprised solely of pid#s there are, contrasted with how many
> > processes are
> running.
> >
> > Only 28 processes on the box? What is its role?
>
>
> It's an LVS (director) for a web site (load balancer) As the LVS is
> compiled into the kernel it doesn't even show as a process. Nothing
> else going on here except sshd which allows me to log in and monitor
> the
> load. That's all it does.

Ok.

>
>
> >
> > Give us a `whereis ps` output as well.
> >
>
> [root@www chkrootkit]# whereis ps
> ps: /bin/ps /usr/share/man/man1/ps.1.bz2
>
> >
> >>> Do both a netstat -plant | grep LISTEN
> >>
> >>
> >> I get:
> >> tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
> 480/sshd
> >
> >
> >
> > Just sshd? That's possibly a good thing. Now what version of ssh is
> > that? (Telnet into port 22 and give us the banner string.)
> >
>
> Telnet not running but here's the ouput of ssh -V and sshd -V
>
> OpenSSH_3.0.1p1, SSH protocols 1.5/2.0, OpenSSL 0x0090581f
> sshd version OpenSSH_3.0.1p1

That is _bad_ news. There are very serious exploits floating around for
a version this old. Please, please, please update. Recheck your box.
Boot off unsuspected media. Check with checkrootkit again from this
unsuspected media.

Put the suggested hub between the box and the internet, and monitor the
traffic. Do a full range portscan, not the standard limited one in
nmap. Check udp. Do not trust the output from the box when it is booted
normally.

>
> >
> >>> and nmap your box remotely - all ports - and compare what it
> >>> thinks are listening.
> >>
> >>
> >> I'm not sure how to do that....
> >
> >
> >
> > On a remote box with nmap,
> >
> > nmap -sS -P0 -p1-65535 myboxip -O -o somelogfile.nmap
> >
> > (syn/fin scan, ignoring failure to respond to pings, all ports,
> > your
> box's
> > externally-visible IP#, with OS-detection, outputs to file as well
> > as stdout).
> >
> OK, ran that from an external box and it showed open ports 22, 80,
> plus the masqueraded ports which have been set up to allow ssh into
> the
> realservers behind the LVS. Should I be concerned that netstat
> -plant
> | grep listen only returned port 22?

Not necessarily. But it is not conclusive. That is the problem. It might
be that ports are triggered only after receiving a properly formed icmp
for example.

>
> >
> >>> Also you need to monitor traffic going out - preferably by
> >>> inserting a hub just adjacent to the machine with another box
> >>> listening in promiscuous mode to what's going past. Also, snort
> >>> would be an idea. If you can't place a sniffing box in the way,
> >>> run snort on the box itself and configure it to look for
> >>> port-scans and dodgy outgoing traffic.
> >>
> >>
> >> Now I'm way over my head. Drowning......
> >
> >
> >
> >
> > Your box -----------ethernet--------> outside world
> >
> > right?
> >
> > Now consider:
> >
> > Your box ----HUB------ethernet--------> outside world
> > |
> > |
> > `-------ethernet---> sniffing host
> >
> > By temporarily breaking the network connection and inserting a hub
> > there, you can plug another box into that hub that'll see all the
> > traffic
> coming &
> > going with tcpdump, ethereal or snort, as long as its network
> > device
> is in
> > promiscuous mode.
> >
> Would I have to get off my couch? :>
>
> >
> > One idea you might want to consider: get pen and paper, and list
> > all the functions and services this box provides, all the critical
> > data stored on it, all the tiresome configuration-tweaks present on
> > it. List the
> cron-jobs
> > it runs. List the users on it and how it gets them.
> > Now, in the background, you can start work on building a
> > replacement box with clean, uptodate, trusted CDs offline, apply
> > distribution's
> errata, and
> > be vaguely confident of being able to switch the two in the
> > eventuality that you discover (a) a crack has been perpetrated or
> > (b) you never get certain knowlege[0] but want a clean slate.
> >
>
> That certainly sounds reasonable!
>
> > [0] Finding a running pid that shouldn't be, an LKM, a network
> > [socket
> open
> > & listening that netstat doesn't show, evidence of massive scans
> emanating
> > from that machine, evidence of users you know not of appearing on
> > the
> box,
> > - these things are concrete. Without one or more of these, or
> > something very similar, you're more or less clean. People, please
> > provide more
> clues
> > to look for, by all means.
> >
> > ~Tim
>
>
> My ISP looked for evidence of massive scans emanating from my ip
> address
> and found none. No unknown users, no unknown pid's. I'm reluctant to
> take heroic measures without, as you say, concrete evidence of a real
> problem. -Neil

The only, but serious problem lies in the fact that you run a very
vulnerable service. The versions of ssh on the freebsd boxes behind
this machine are also this old?

What you could have done when designing your network layout was
investing a little bit extra in a machine to catch just these events: a
remote/secure loghost. IMHO that _should_ be done for any (commercial)
website with a layout as yours. The RTI can be very short with events
like this.

EJ 

-- 
Remove the obvious part (including the dot) for my email address