Re: Still after the apache spammer, more info

From: Me Here (Me_at_here.com)
Date: 04/26/05 

Date: Tue, 26 Apr 2005 09:27:08 -0400

Ohmster;

Just a quick top-posted note to say you should seriously consider the
worst has happened and prepare for a re-build. You can not be paranoid
enough when it comes to an unknown exploit, even when it seems it's
limited to apache.

Good luck!

Me.

Ohmster wrote:
> For anyone following along, the redhat 9 machine that sends tons of spam
> email to the world at large, all by itself. Here is more information,
> hoping that it will prove to be useful in tracking down the unsecured
> files. Quick history, up to date:
>
> Redhat 9 machine on 24/7 ADSL connection, using rp-pppoe to connect and
> give the machine "a real IP address". Machine is a
> server/gateway/firewall for home LAN of 2 XP Pro machines. Machine has 3
> FQDNs, running apache with 3 virtual hosts for the domains. Running phpbb
> 2.0.6 (Can email and coppermine photo gallery 1.2.1 (Can email on one
> virtual host, behind .htaccess directory for user/pass, basic auth. This
> is a family website for web board and photos. Passworded to keep private
> family info more or less private. 2nd vhost is my personal domain,
> nothing of interest, used mostly as an http file server for friends when
> given an direct URL to a file, no directory browsing. 3rd domain a small
> Outlook Express stationery website for a friend, public. Runs openbook
> guestbook 1.2.2 (Can email) for a guestbook. Web root is in /var/www/html
> contains little of interest, just a go away message and phpMyAdmin-2.5.0
> also resides there.
>
> For the most part, the machine is secure and good. All is well. There
> have been episodes of mass spamming of the world at large by apache on
> this machine. All spam emails are sent by apache and accepted by sendmail
> it happens as relay=apache@localhost.
>
> When these episodes of mass spamming occur, the machine will slow to a
> crawl and top reveals a perl process, owned by apache, eating 99% CPU and
> this will continue until you kill the process or it just "finishes", how
> long that takes, I don't know. The top program reveals something like
> this when it happens:
>
> 16313 apache 25 0 2092 468 240 R 99.7 0.0 1884m 0 perl
>
> I can grep through the maillogs and find this PID and this reveals that
> the spam is taking place. This runaway process is sending out the spam.
> But it seems like during this incident, the apache daemon is on a "spam
> spree", you can see from the maillog where this PID was found and the
> surrounding logs (Space around line with PID 16313 to make it easy to
> find):
>
> ---------------------------------------------------------------------
> Apr 19 08:06:36 ohmster sendmail[16302]: j3JC6ZDa016300: to=
> <srip@fullnet.com>, ctladdr=<apache@ohmster.com> (48/48), delay=00:00:01,
> xdelay=00:00:00, mailer=relay, pri=30329, relay=mail.bellsouth.net.
> [205.152.59.17], dsn=2.0.0, stat=Sent (Message received:
> 20050419120636.ZFVE18830.imf20aec.mail.bellsouth.net@ohmster.com)
> Apr 19 08:06:37 ohmster sendmail[16303]: j3JC6b5p016303: from=apache,
> size=3183, class=0, nrcpts=1, msgid=<1113912397.13381.update@visa.com>,
> relay=apache@localhost
> Apr 19 08:06:38 ohmster sendmail[16305]: j3JC6bDa016305: from=
> <apache@ohmster.com>, size=3365, class=0, nrcpts=1, msgid=
> <1113912397.13381.update@visa.com>, proto=ESMTP, daemon=MTA,
> relay=localhost.localdomain [127.0.0.1]
> Apr 19 08:06:38 ohmster sendmail[16303]: j3JC6b5p016303:
> to=glynwill@world-net.net, ctladdr=apache (48/48), delay=00:00:01,
> xdelay=00:00:01, mailer=relay, pri=30167, relay=[127.0.0.1] [127.0.0.1],
> dsn=2.0.0, stat=Sent (j3JC6bDa016305 Message accepted for delivery)
> Apr 19 08:06:39 ohmster sendmail[16307]: j3JC6bDa016305: to=
> <glynwill@world-net.net>, ctladdr=<apache@ohmster.com> (48/48), delay=
> 00:00:02, xdelay=00:00:01, mailer=relay, pri=30341,
> relay=mail.bellsouth.net. [205.152.59.17], dsn=2.0.0, stat=Sent (Message
> received:
> 20050419120638.ZFWB18830.imf20aec.mail.bellsouth.net@ohmster.com)
> Apr 19 08:06:39 ohmster sendmail[16308]: j3JC6dRt016308: from=apache,
> size=3180, class=0, nrcpts=1, msgid=<1113912399.13382.update@visa.com>,
> relay=apache@localhost
> Apr 19 08:06:40 ohmster sendmail[16310]: j3JC6eDa016310: from=
> <apache@ohmster.com>, size=3359, class=0, nrcpts=1, msgid=
> <1113912399.13382.update@visa.com>, proto=ESMTP, daemon=MTA,
> relay=localhost.localdomain [127.0.0.1]
> Apr 19 08:06:40 ohmster sendmail[16308]: j3JC6dRt016308:
> to=celiaf@netscape.net, ctladdr=apache (48/48), delay=00:00:01, xdelay=
> 00:00:01, mailer=relay, pri=30164, relay=[127.0.0.1] [127.0.0.1], dsn=
> 2.0.0, stat=Sent (j3JC6eDa016310 Message accepted for delivery)
> Apr 19 08:06:41 ohmster sendmail[16312]: j3JC6eDa016310: to=
> <celiaf@netscape.net>, ctladdr=<apache@ohmster.com> (48/48), delay=
> 00:00:01, xdelay=00:00:01, mailer=relay, pri=30335,
> relay=mail.bellsouth.net. [205.152.59.17], dsn=2.0.0, stat=Sent (Message
> received:
> 20050419120641.ZFWV18830.imf20aec.mail.bellsouth.net@ohmster.com)
>
> Apr 19 08:06:41 ohmster sendmail[16313]: j3JC6fK4016313: from=apache,
> size=3181, class=0, nrcpts=1, msgid=<1113912401.13383.update@visa.com>,
> relay=apache@localhost
>
> Apr 19 08:06:42 ohmster sendmail[16315]: j3JC6gDa016315: from=
> <apache@ohmster.com>, size=3361, class=0, nrcpts=1, msgid=
> <1113912401.13383.update@visa.com>, proto=ESMTP, daemon=MTA,
> relay=localhost.localdomain [127.0.0.1]
> Apr 19 08:06:42 ohmster sendmail[16313]: j3JC6fK4016313:
> to=jlaguirre@atwebo.com, ctladdr=apache (48/48), delay=00:00:01, xdelay=
> 00:00:00, mailer=relay, pri=30165, relay=[127.0.0.1] [127.0.0.1], dsn=
> 2.0.0, stat=Sent (j3JC6gDa016315 Message accepted for delivery)
> Apr 19 08:06:43 ohmster sendmail[16318]: j3JC6hDp016318: from=apache,
> size=3178, class=0, nrcpts=1, msgid=<1113912403.13384.update@visa.com>,
> relay=apache@localhost
> Apr 19 08:06:43 ohmster sendmail[16317]: j3JC6gDa016315: to=
> <jlaguirre@atwebo.com>, ctladdr=<apache@ohmster.com> (48/48), delay=
> 00:00:01, xdelay=00:00:01, mailer=relay, pri=30337,
> relay=mail.bellsouth.net. [205.152.59.17], dsn=2.0.0, stat=Sent (Message
> received:
> 20050419120643.ZFXF18830.imf20aec.mail.bellsouth.net@ohmster.com)
> Apr 19 08:06:44 ohmster sendmail[16320]: j3JC6hDa016320: from=
> <apache@ohmster.com>, size=3355, class=0, nrcpts=1, msgid=
> <1113912403.13384.update@visa.com>, proto=ESMTP, daemon=MTA,
> relay=localhost.localdomain [127.0.0.1]
> Apr 19 08:06:44 ohmster sendmail[16318]: j3JC6hDp016318:
> to=celiafg@yahoo.com, ctladdr=apache (48/48), delay=00:00:01, xdelay=
> 00:00:01, mailer=relay, pri=30162, relay=[127.0.0.1] [127.0.0.1], dsn=
> 2.0.0, stat=Sent (j3JC6hDa016320 Message accepted for delivery)
> Apr 19 08:06:45 ohmster sendmail[16322]: j3JC6hDa016320: to=
> <celiafg@yahoo.com>, ctladdr=<apache@ohmster.com> (48/48), delay=
> 00:00:01, xdelay=00:00:01, mailer=relay, pri=30331,
> relay=mail.bellsouth.net. [205.152.59.17], dsn=2.0.0, stat=Sent (Message
> received:
> 20050419120645.ZFXN18830.imf20aec.mail.bellsouth.net@ohmster.com)
> Apr 19 08:06:46 ohmster sendmail[16323]: j3JC6j2c016323: from=apache,
> size=3186, class=0, nrcpts=1, msgid=<1113912405.13385.update@visa.com>,
> relay=apache@localhost
> Apr 19 08:06:47 ohmster sendmail[16325]: j3JC6kDa016325: from=
> <apache@ohmster.com>, size=3371, class=0, nrcpts=1, msgid=
> <1113912405.13385.update@visa.com>, proto=ESMTP, daemon=MTA,
> relay=localhost.localdomain [127.0.0.1]
> Apr 19 08:06:47 ohmster sendmail[16323]: j3JC6j2c016323:
> to=donny@virtualmalaysia.com, ctladdr=apache (48/48), delay=00:00:02,
> xdelay=00:00:01, mailer=relay, pri=30170, relay=[127.0.0.1] [127.0.0.1],
> dsn=2.0.0, stat=Sent (j3JC6kDa016325 Message accepted for delivery)
>
> ---------------------------------------------------------------------
> Darned maillog wraps pretty badly in here. I put a space to separate out
> PID 16313 to show where it is in the maillog and you can see that it is
> smack dab in the middle of a massive spam party. This is only a very
> small portion of the maillog, there is tons of it in there for this log.
> The other logs are empty, this only "happens when it happens" and I find
> out about it when I get "message cannot be delivered for 5 days" messages
> in my email or some other email error tip off.
>
> These spam attacks only occur once in a while, it happened right when I
> sent my first post for help and I killed the runaway perl process with a
> kill -9. I have the maillog, I have the top report showing the PID, but I
> don't have a lsof to a file from the time. I did not know about lsof and
> had I caught an lsof to file at that time, I could have examined it and
> grepped for files that apache would use to send mail; like pl, cgi, or
> php.
>
> Now there is no more spam, since I caught the machine doing it and killed
> the process, it has not occurred since. I can watch the machine and watch
> top and when it happens again, snag the maillog, the top output, and
> capture the lsof at the time and would have what I need to hunt this
> down. I have to wait for it to happen again and this could be weeks or
> who knows how long.
>
> It seems that someone, somewhere, has found a file on the machine,
> accessible by apache, and is exploiting the hell out of it. These "spam
> soirees" produce massive spam, so much that this must be another machine
> somewhere that is running a script or some other bulk spam list against
> my apache server. Once the spam party is over, it is over and there is no
> more trace of it.
>
> So, other than wait for it to happen again, catch the process PID with
> top, catch the lsof output to a file, and save the maillog from the time,
> is there anything else I can do? What do you folks suggest? Is there any
> sort of monitor that I can setup for this condition to catch it when it
> happens? cpu usage goes through the roof, way over 90%. Bandwidth is
> being used up for sending spam. It would be ideal to have a script
> running that would monitor CPU usage and take a snapshot of lsof, some
> sort of top output, and the maillog at the time this happens. Dream on?
> What other tools are available to fight something like this?
>
> I do use samba sometimes to work on web stuff because it is easy to open
> files with Dreamweaver when I have a samba share and can copy files back
> and forth to my windows machine. The downside of this is that samba makes
> every file it copies or writes to the linux machine executable. I may
> have files set as executable all over the place that need not be. I catch
> them and chmod them back when I find them. Going back over all of the web
> roots and cleaning that up now.
>
> I need some help with this as this is a very frustrating and elusive
> problem to catch. I am answering the replies and forthcoming with more
> information as I get it to help track this thing down. So close but so
> far...
>
> Thank you for your time and your insight.
>

Relevant Pages

  • Still after the apache spammer, more info
    ... the redhat 9 machine that sends tons of spam ... running apache with 3 virtual hosts for the domains. ... Darned maillog wraps pretty badly in here. ... don't have a lsof to a file from the time. ...
    (comp.os.linux.security)
  • Re: where does lsof get its info?
    ... the info I can get from a simple lsof. ... My thinking is that at the moment some php/cgi script calls the mail function (through apache), apache is still reading that same script. ... Also I have increased the logging content of apache but there is still nothing which links an item of spam to that entry in access_log. ...
    (comp.os.linux.misc)
  • Re: Logs and how to read them
    ... > internal machines are mostly Windows and I did check for viruses and worms. ... What tells you that these two independent maillog entries were relay ... As advised by Peter you better ask your ISP for details of the SPAM ...
    (Fedora)
  • Re: Still after the apache spammer, more info
    ... running apache with 3 virtual hosts for the domains. ... > have been episodes of mass spamming of the world at large by apache on ... All spam emails are sent by apache and accepted by sendmail ...
    (comp.os.linux.security)
  • RE: Attempts to push spam through apache
    ... Attempts to push spam through apache ... Obviously this is an effort to pump spam through my server to 208.17.33.40. ... Are these just random spammer attempts to find an open proxy? ...
    (Focus-Linux)