Re: sendmail compromised - Somebody help me!

• Previous message: Robert Glueck: "Re: Firewall hits passing through a NAT router - How does that work?"
• In reply to: Ohmster: "Re: sendmail compromised - Somebody help me!"
• Next in thread: Barton L. Phillips: "Re: sendmail compromised - Somebody help me!"
• Reply: Barton L. Phillips: "Re: sendmail compromised - Somebody help me!"
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] [ attachment ]

Date: Wed, 27 Apr 2005 03:30:39 -0400

On Wed, 27 Apr 2005 01:41:23 +0000, Ohmster wrote:

> "Bev A. Kupf" <bevakupf@myhome.net> wrote in
> news:slrnd6tn8p.i7a.bevakupf@myhome.net:
>
>> On Tue, 26 Apr 2005 23:45:16 GMT,
>> Ohmster (notareal@emailaddress.com) wrote:
>>
>> There are two ways for you to take advice from professionals in the
>> field. The first is to consider that the opinion offered _may_ just be
>> a well considered opinion, especially if it comes from more than one
>> source.
>
> I got quite a bit of good advice. Searching through the apache logs for
> files that can email and are being used more often than should be. I got
> a great tip from Jem to use lsof to snap a list of open files when the
> incident occurs. Got advice about tripwire and it is installed, but I am
> not sure if that would be of much help now. The barn door and all.

If you haven't seen a (tripwire) report in a couple of years then it's not
really doing you any good, installed or not. Forget about tripwire for
now, unless of course you think it might be useful to know why it stopped
sending you those long e-mails... Maybe your box was cracked in 2003?

> Mike
> sent me a detailed pdf file, analysis of the security, some of it pretty
> scary, some quite minor. I appreciate and an working on "the list".

You should have been scanning your sites yourself regularly right along,
and you might have been able to take care of those "scary" things long
ago. You set up what seems to be an audacious site, with mail and web
servers and dog knows what else, all connected to the public; by
comparison my own local systems are very simple and minimal, with _no_
_servers_, a good and tested firewall, etc, etc, etc, and I still scan my
externals regularly. My mails and domains are hosted by external
providers with their own staffs of dedicated, knowledgeable professionals.
 How much did you save by putting all your (essential) stuff on a box in
 your house? $10/month, $20, $30, what??? And are all the rest of us in
 the world supposed to take our lumps from your box because you wouldn't
 hire someone who knew what s/he was doing, and who would do all the
 routine but necessary things to keep it running and safe, the same things
 that you obviously didn't know enough about or care enough about to learn
 how to do?

Look Ohmster, I really don't want to be unnecessarily harsh or unkind. You
continue to answer in mostly good ways, and it appears you are a serious
and in many ways responsible person. I have read and understood what you
have written. In several ways you sound like a man who is looking for
good alternatives. I know that you want to be convinced that all your
troubles are just because of a bad formmail script, and that is even still
possible. But from what you have written, you wouldn't know if your box
were vectoring an attack on a nuclear power station, or on the FBI or NSA.
 Honestly and truly, your refusal to disconnect your server is just simply
not responsible, is just simply not acceptable. Read below.

HERE IS A GOOD SUGGESTION:

Outsource your servers to a qualified hosting concern, and _THEN_
disconnect and rebuild your own home-based server. There are hundreds, if
not thousands of good, reasonably priced hosting companies easily found
with common, free search engines. It shouldn't take more than a day or
two at most for the DNS changes to be reflected worldwide. You can just
FTP your web pages to their servers, set up you e-mail with them, and then
you can take you time to deal with your wife's rattling tailpipe. Doesn't
that sound like a good alternative?

To make it easy for you (this is not a "plug", dog knows they don't need
any more "problem customers"), here is one that I know is good. They have
excellent e-mail support (no telephone support) during their US Pacific
Coast working hours.

        https://www.tigertech.net/

Starts at $6.25 per month per domain. Doesn't that sound like a cheap and
easy way out of your immediate problems? And you can almost immediately
save all the rest of us in the world from the assaults that your
home-based server might spew. Hope you take this advice immediately.
Really, really do.

> Got
> tips about phpbb2 from Michael from bugtraq. All really good advice, all
> helpful, either now or for future consideration. There were a few that
> insisted that the box come offline, without any reasonable expectation
> of getting it back online again.

Would you drive a car knowing it had bad brakes and might kill someone?
Would it be the other motorist's concern that you didn't know how long it
would be before you could see your way clear to fixing your brakes? Get
real. Park it. Take a cab.

> That really was not helpful. I use this server every day for work and
> cannot simply take it down like that.

You certainly can, and simply, and without disrupting anything essential.

Hire a remote server. It's cheap and easy. You can get lots of GOOD help
here and elsewhere at no cost to you. But if you don't take the good
advice just because it is free (even though it comes the same from
multiple known good sources) then you can vent and complain all you want,
the good advice won't help you. People are trying to make a living
providing exactly the services that you need, and for affordable prices,
and you should stop complaining and listen. Then hire some of those good
and competent people to do what you need done. Throw your bias and your
hubris on the trash, not your server hardware. Would you know the
difference?

>> The second is to throw a hissy-fit. The facts are that _your_
>> incompetence led to _your_ box possibly being compromised. As a
>> sysadmin, its _your_ job to keep your box current with whatever patches
>> are offered for your system.
>
> Hissy fit? Ma'am, I made it clear that I could not take the box offline

You certainly can and you certainly should.

> for an extended period, but could do things now like shutdown sendmail,
> as that seems to be a threat to others and myself.

You don't know clue #1 what you machine is doing.

> I will listen to any
> other reasonable suggestions that do not bring down the servers,

See the above

> even
> removing the formmail pages.

So, who cares about your formmail pages except you. We care about
compromised machines being connected to the public, and about intransigent
know-nothings who insist on keeping bad, unmaintained systems connected.

> How many times must one stress the same point, about taking down the
> servers, only to have it ignored and to request that the box be taken
> offline indefinitely and immediately?

You have no right to keep a compromised machine connected. It is way,
_WAY_ worse than rude to suggest otherwise.

> I
> was frustrated, that is all. The "hissy fit" was directed at no one in
> particular and I made it very clear that it was not directed at you,
> Bev, unless you misunderstood "Not helpful, not appreciated, not wanted,
> save your breath." to be a personal attack. It was not. I meant "save
> your breath for something important like breathing" other than to repeat
> the same advice that I cannot take

As outlined above, there is absolutely no reason you should not take the
good advice given you.

> in it's entirety and had already made quite clear, several times.

It has been made quite clear to you that it is not acceptable to leave a
compromised system connected. Are we clear enough, yet? Or not?

>> Let's say that your box is used to launch attacks on several other
>> boxes. Is it right that someone else has to spend their time
>> (minimally) redesigning a firewall, because _you_ were incompetent?
>
> Of course not, that is why I shutdown sendmail and stopped it from
> starting at boot time. The "attacks" of spam are a result of some
> security issue with a formmail exploit of apache.

I really do not want to be unnecessarily unkind. But you are acting in a
really dense manner. As loose as your system is, was and has been, you
really don't have any credibility to say what (else) has or has not been
compromised. The only safe assumption is that your box has been trashed,
raped and plundered. See my above good advice, and then disconnect your
machine.

> I cannot shut down
> apache or take the box down for days, weeks, or longer,

Yes you can.

> but I can stop
> sendmail and the spam will cease because of it, until I can find the
> real root of the formmail exploits.

We all have SPAM filters. SPAM is bad, but it is not what we are thinking
about most. If your box is cracked, it could be vectored to blow up a
NUKE. Smaaten up; U wouldn't know if it was or not. Disconnect it!

> I even ran a chkrootkit to be sure
> as was suggested from the professionals. ...again with the "incompetent"
> word?

If you had been competent, you would not have needed to ask the
"professionals" in the first place

>> And btw, I've cut most of your rant out, but no one has suggested that
>> you throw the box out. Now a sensible sysadmin would have a backup of
>> all the data. And it would take less than a day to reload a _secure_
>> operating system, and then restore the data from backup (we are after
>> all talking about a single box here). But everything that you've
>> displayed of yourself here indicates that sense is something that
>> doesn't come easily for you. So, off course you don't have any
>> backups.
>
> Uh, yeah, I have the data backed up on the original hard disks.

If you are determined to run such an audacious web presence in your home,
get yourself a CD writer (if you don't already have one) and do your
backups to removable media. If you box is cracked, all the data on your
hard drives has been parsed, and changed to suit the crackers' desires and
whims. Your backups should be designed for easy automated restores, so
that you can, in fact restore your entire system in a few hours,
automatically, except for switching CD's.

> The
> /home disk and the /, swap, and /boot partitions are on a second disk,
> this was just done recently and the data is there. The data is not so
> "mission critical" that I need daily backups of it. It took me a long
> time to get this system running like it is, I could not install and
> configure everything for a new system in a day.

Find out where all this wonderful stuff that you treasure is saved and
back it all up every time you tweak it, and in a way that makes it easy,
fast and automatic to restore. That's what a system administrator does,
among other things. If you don't know how to do this, hire a hosting
company. They are competent, you are not. Your machines are a hazard to
everyone else in the world.

> There is firewall and
> NAT, there are a few mysql databases, there is the web server and
> virtual hosts, there is samba and shares, there are personal and custom
> tweaks. I could install a new distro and get it running in a day though,
> but I would have to re-install and re-configure everything to work as it
> did for a new distro and that could not be done in a day, at least by
> me, I am not a professional at this, but I do enjoy it.

You may enjoy it better when you know how to do it properly.

> It is difficult
> to be "incompetent" when one claims no excellence in the first place.

You claimed "excellence in the first place" when you put your audacious
servers onto the internet. If that is not what you intended to do, then
just plain take them down.

> Yes, I have the backup that I need.

The backups that you have aren't what you need if you cannot restore your
system over a weekend. Fact, Jack.

>> You came here for advice. Whether you take what is offered to you or
>> not is your choice. Too bad you don't like tough medicine if that's
>> what is called for.
>
> Tough medicine? Look, I came here to root out an apache formmail
> exploit, you were the one that actually directed me here. I got good
> medicine. That kind of "tough medicine" might be called for in your
> book, had I discovered a serious security flaw in the ckrootkit, I would
> have taken the box down. I won't run a compromised box where I know that
> root access has been compromised, that would be just plain foolish.
> There is no indication of that level of compromise. Just a
> non-privileged daemon mailing out spam from a formmail exploit. Yeah it
> bites the big one and all but it can be contained until something can be
> done to make it more secure.

Well, maybe you are correct that only one script has been penetrated, and
then again maybe not. You are clearly trying, but with the level of
knowledge and expertise and care that you have demonstrated, you have no
reason, right or business to be running public servers. Take your box down
and hire some professionals. It doesn't cost a lot. They (may) know what
they are doing; you certainly do not.

>> You've been given other advice. Find out which scripts are being
>> accessed repeatedly from Apache's access_log. Did you do that?
>
> Sure, searching through the logs for files accessible to apache that can
> email was good advice. But there is over a hundred megs of text logs.
> That is a lot of searching. I tried but got bleary eyed trying. Dave
> showed me how to use regex in grep to help with the searching. Maybe I
> am not searching properly but I am working on refining the search. It
> will take a while though. Sendmail is shut down in the meantime.

D-uh. If it takes you a couple of years to notice that tripwire isn't
sending you daily e-mails, how serious are you about the tools that you
claim to have working?

>> Heck, no. Put plainly, you're _incompetent_. And stupidity is its own
>> reward.
>
> Again with that word. If I came here claiming to be a professional
> sysadmin, then you could probably say "incompetent" and you would be
> right. I never said that. I am a casual linux user and I learn more as
> time passes. I setup and use the servers on my linux machine and I need
> them. I am very sad that there is a formmail exploit on my system
> because I really do need the servers. I never insulted you personally,
> Bev, and took the time to say this in my "hissy fit". I did not come
> here to exchange insults, I just wanted some tips or pointers in the
> right direction. I got some, by some very well meaning, professional
> individuals. The problem is not solved yet but I have things to work on
> now. I do believe that you were the one to point out that this is a
> formmail exploit in the first place. What is the point in swapping
> insults? I am sure you are competent enough to do your job and I
> appreciate your time.
>
>> Beverly

You got some tips and pointers. Get a professional hosting service and
most of these problems will disappear. Disconnect your box until you know
more and our concerns will level right off. Quid pro quo.

Best wishes.

ps. Hope you appreciate how much time and effort it has taken us to get a
very simple result.

• Previous message: Robert Glueck: "Re: Firewall hits passing through a NAT router - How does that work?"
• In reply to: Ohmster: "Re: sendmail compromised - Somebody help me!"
• Next in thread: Barton L. Phillips: "Re: sendmail compromised - Somebody help me!"
• Reply: Barton L. Phillips: "Re: sendmail compromised - Somebody help me!"
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] [ attachment ]