apache compromised to send spam, need way to check file access

From: Ohmster (notareal_at_emailaddress.com)
Date: 04/25/05 

Date: Mon, 25 Apr 2005 04:05:30 GMT

First of all, I am not a linux expert. I have done linux for a few years
and have managed to get a successful server/gateway/firewall with samba
networking for my home LAN. I an not a true "newbie" but do not understand
all of the tools at my disposal and would appreciate some tolerance and
real help and guidance now, I really need your help. If you could please be
"helpful and nice", this would go a long way, thank you very much. I can
take a little criticism but pure criticism and no actual help would be
rather discouraging. <quite sincere here>

I started this conversation in comp.mail.sendmail and after finding out
that this is apache spamming the world at large via sendmail, it was
suggested that I take my security question here by Susan. My original
thread can be found as "sendmail compromised - Somebody help me!" in the
newsgroup, comp.mail.sendmail, if anyone needs further information. Sue's
summary of my post:

<short summary: OP runs webserver on a DSL-connection. Recently spam
injected by his webserver is clogging his mailqueue. Distribution is
RedHat 9 and no, he has not heard about fedoralegacy>

I have a redhat 9 linux server on a 24/7 ADSL connection that acts as a
server/gateway/firewall for my small, 3 PC home LAN. The redhat box has 3
FQDNs on it with 3 virtual hosts that are hosted in the user_dirs. One site
is my personal site with virtually no content, I use for an http file
server for friends. The second site is a family website that runs phpbb
2.0.6, the entire site is behind an .htacccess file that passwords it. I
also run coppermine photo gallery 1.2.1 on it. Coppermine and phpbb can
both send mail. The third site is an Outlook Express stationery website for
a friend and it now has a guestbook, openbook version 1.2.2. The guestbook
can send mail when a user makes an entry, it will send mail to myself and
the webmaster. My ISP blocks port 25 traffic so the server cannot receive
any email from the world but it can send mail out by using the smarthost
feature of the sendmail.mc file. I have enabled this feature in order to
send mail and not be rejected by the dnsbl lists that reject all mail from
anyone on a DUN list. Apparently anyone running a mail server on an IP
range that falls into a cable, dial up, or DSL connection is black listed
so I used the smarthost to send mail through my ISP's mail server, back in
the days before port 25 was truly "stealthed" by my ISP for everyone with
an Internet account. Cannot send mail on port 25 now, not even to another
mail server mail account, only through my ISP's mail server.

I know that redhat 9 is old now and should be replaced with a newer distro,
perhaps FC3 or so. I cannot do this now, I have years of installs, setups,
and tweaks on this server and cannot bring it down for weeks or months
while I install a new distro, but I am seriously considering it and will
get to it, someday this year, I hope. For now, the redhat 9 box has to stay
and something must be done to stop the spam emails coming from apache.

I believe the spam to be coming from apache as this email that I got from
my redhat box via pop3 showed up in my inbox:

---------------------------------------------------------------------

Return-Path: <apache@ohmster.com>
Received: from ohmster.com (localhost.localdomain [127.0.0.1])
        by ohmster.com (8.12.8/8.12.8) with ESMTP id j3J0bADa030038
        for <root@ohmster.com>; Mon, 18 Apr 2005 20:37:17 -0400
Received: (from apache@localhost)
        by ohmster.com (8.12.8/8.12.8/Submit) id j3J0b8wP030036
        for root; Mon, 18 Apr 2005 20:37:08 -0400
Date: Mon, 18 Apr 2005 20:37:08 -0400
From: Apache <apache@ohmster.com>
Message-Id: <200504190037.j3J0b8wP030036@ohmster.com>
To: root@ohmster.com
Subject: Account compromised
Status: O
X-SpamSubtract-Analysis: other: not to or cc me
X-SpamSubtract-Analysis: user moved message to inbox.

This account has been compromised, please clean it

---------------------------------------------------------------------

Looking through the maillogs shows that apache has sent a large amount of
spam email all over the world. This is very troubling and has to be
stopped. What I believe and most everyone in the sendmail newsgroup
believes is that a php, pl, or cgi file that can send mail has been
exploited and is being used to spam the world at large via apache. Some
version info on my setup:

[root@ohmster mail]# uname -a
Linux ohmster.com 2.4.20-31.9 #1 Tue Apr 13 18:04:23 EDT 2004 i686 i686
i386 GNU/Linux
[root@ohmster mail]# rpm -q sendmail
sendmail-8.12.8-9.90
[root@ohmster mail]# rpm -q httpd
httpd-2.0.40-21.17.legacy
[root@ohmster mail]#

I had an incident today and in the past, where the redhat box has slowed
way down to a crawl and by ssh'ing over to the box from my desktop
computer, I ran top and discovered a runaway perl process, owned by apache,
that was chowing down on 99% of the CPU usage of the machine. This activity
continued until I killed the perl process with a kill -9. Here is the very
top line of the top screen:

16313 apache 25 0 2092 468 240 R 99.7 0.0 1884m 0 perl

I could not even restart httpd until this perl process was killed, see
results:

[root@ohmster mail]# service httpd restart
Stopping httpd: [ OK ]
Starting httpd: (98)Address already in use: make_sock: could not bind to
address 0.0.0.0:443
no listening sockets available, shutting down
[FAILED]
[root@ohmster mail]# kill -9 16313 (The runaway perl process)
[root@ohmster mail]# service httpd restart
Stopping httpd: [FAILED]
Starting httpd: [ OK ]
[root@ohmster mail]#

Something had took over the socket(?) and was pretty much running things
over here on my end with a perl process, a web mail exploit, perhaps? Would
there be any logs of use to investigate this incident a little further,
perhaps?

What I desperately need is to find a way to see what web directory files
are being unusually accessed a lot, with regard to the rest of the site.
Since all three web sites are such low traffic, anyone using a mail capable
file would stick out like a sore thumb. How can I check when files in a web
directory are accessed in a short period of time, let's say a few days or a
week? I cannot find a way to make ls do this. I am not sure it can be done
with ls. Is there any kind of script that I can run with cron to watch all
of the files in a directory, and it's subdirectories, to see when they are
accessed and report a date and time for each incident of access, and log it
to a file that can be examined? There has to be a way to monitor these
public_html directories for file access for a short time and see which
files are being used to spam the world with apache. I need to do this for a
short time to find the culprit so that I can put a stop to it.

Sue is right, this is now a security question so I have to seek help from
the linux security community on this so that is why I am here. Please help
me to find a way to find out which perl files are being used in this
exploit so that it can be stopped. I will watch this thread closely and
will respond to any requests for further information and tests. Thank you,
I really need your help now. 

-- 
~Ohmster
ohmster at newsguy dot com

Relevant Pages