Re: Symantec AV signature corruption

• Previous message: Russ: "Re: Symantec AV signature corruption"
• Maybe in reply to: Ben Reardon: "Symantec AV signature corruption"
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] [ attachment ]

Date:         Tue, 18 Feb 2003 19:20:46 +1000
From: Ben Reardon <ben.reardon@BIGPOND.COM.AU>
To: NTBUGTRAQ@LISTSERV.NTBUGTRAQ.COM

I've received a lot of feedback off list with this issue. (Thanks to the
contributors for suggestions and extra info) It seems a reasonable
number of companies were adversely affected by this problem.

The vendor response on this issue may have been a bit light on detail
and some clarification of the issue and some experiences may be valuable
to the NTBugtraq readership.

There were some good suggestions and observations that I thought would
benefit Symantec users on the list. Some of this information may be
inaccurate, and I take no responsibility for it damaging your systems.

INTERESTING TIDBITS:
The behaviour of the workstations affected by this problem was that the
NAV service was unable to start or an error message whilst starting the
NAV application or whilst attempting to view the virus list.

According to Symantec, the signature version in question was the
following update. It affected the HTTP based version for about 2 hours
apparently. The FTP based one was ok (I'm told, but I had similar
problems with a pilot where I am ftp-ing navup8.exe files - may have
been coincidence)
20030211-004-i32.exe

This has been discussed with some Symantec contribution here (URL will
probably wrap):

http://servicenews.symantec.com/cgi-bin/displayArticle.cgi?article=<1030
112231622.9497909659@servicenews.symantec.com>&group=symantec.support.ne
twork.nortonantivirusce76.general&tpre=ent

There was also an interesting article on detecting sig corruption:

http://service1.symantec.com/support/ent-security.nsf/docid/200208070859
4148?Open&src=w

WHAT HAPPENED
It looks like our case is bad luck, bad planning and perhaps some
non-optimal practices by the vendor. More on this later...

We have 2 environments for various reasons. One is NAV 8. One is NAV 4

NAV 4 (sigs updated via NAVUP32.EXE in the users login script)
Sequence of events
- Download dodgy signatures to distribution server.
- Extract sigs and place in directories for replication to be run with
login scripts using navupd32.exe
- Users logged in in the morning and applied dodgy sigs.
- Problem detected.
- Autoprotect service failed, would not start again.
- Had to write script to manually rollback to last signature, and delete
the corrupt one.
- Allow users to sign in again to do new sig update process
Note this is in contrast to Symantec's reply post regarding the state of
AV protection after receiving the bad sigs.

NAV 7.61/8 (Sigs updated via VDTM via a central server)
This was the sequence of events. These may be peculiar to our
environment, but fairly coincidental. If not directly related to the
problem at hand would be valuable experience in similar circumstances.

- Download dodgy signatures to server (signature version 4)
- Dodgy sigs pushed to clients via VDTM (signature version 4).
- Client accepted sigs silently, and effectively had zero viruses in the
inspect list.
- New fixed "signature version 4" comes out and I install on my server.
- Clients don't get pushed the new version 4 because they are already on
version 4! (And that's if the client service hasn't fallen over yet)
- Upon reboot, the client service fails to start because the environment
is incorrect.
- You can no longer see the clients from the Console and have to
backdate by manual methods.
- The workaround is client by client, and while it can be scripted,
takes considerable time and is not pretty.
See document ID 1999063010361948 on Symantec's website for help with
manual backdate of signatures.

POST-MORTEM - What went wrong?
Bad Luck:
- We downloaded the signatures from the affected source in the 2 hours
that were seen as a problem.
Bad Planning:
- We did not have a process to check MD5 hash values for this signature
file.
Non-optimal vendor practices:
- Even if the md5 checksum didn't match, the exe still ran to produce
corrupt signatures. Perhaps they could be made more robust.
- Alerting, there was no effective alerting in my opinion. In the end,
it may be in everyone's interest that the vendor confesses to its
customers that there "may be a chance that your AV isn't working due to
a corrupt signature available for download for 2 hours this morning"...

LESSONS LEARNT
1. Check md5 checksums. If you are using scripted Intelligent Updater
downloads, don't forget to integrate them into your script. The symantec
supplied scripts provide for no checking - so Ddo it yourself.
2. Consider testing of the sigs even if checksums match (High cost for
us. Symantec should be doing "sig is fundamentally broken" checking
anyway).
3. Consider a waiting period so that bugs in sigs are worked out.
(Problematic if automated system)
4. The vendor should consider self checks in Intelligent Updater and any
other signature distribution technologies (and no, md5 doesn't count).
Perhaps they could consider cryptographically signing of sigs (PGP or
X.509) and perform automatic cert validation at the client. Something
like this should be a "must have" in large corporate environment. I
think Liveupdate has some form of signing in the form of a file called
liveupdt.sig, but according to the vendor, Intelligent Updater is of
more benefit for corporate users, and I can't see any signing
technologies being used for it.
5. Better internal alerting to detect problems with AV services. You
shouldn't have to rely on users or vendors to make you aware of an
issue.
6. Consider where automation of virus sig downloads can let you down and
build in better controls (manual or otherwise) to limit these problems.
7. In the event of issues like this, have a backup methodology to get
signatures or modifications out to clients. Our NAV 8 clients were
physically unable to receive new sigs as the service could not start
after a reboot (again, the NAV8 problem may have been down to other
factors that I cannot determine)

Thanks for your responses. I've certainly learnt a lot in this exercise.
If I have any technical details wrong I apologise, but regardless I
think the incident has bought into consideration some important concepts
(regardless of vendor).

Cheers
Ben Reardon

oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
Delivery co-sponsored by TruSecure Corporation
oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
TICSA - Anniversary Special - Limited Time

Become TICSA certified for just $221.25 US when you register before 3/31/03
with PROMO "TS0103" at www.2test.com. NO membership fees, certification
good for 2 years. Price for international delivery just $296.25 US, with
this offer. Offer cannot be combined with any other special and expires
3/31/03. Visit www.trusecure.com/ticsa for full details.

oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo

• Next message: Paul Szabo: "Revised: MS03-004"
• Previous message: Russ: "Re: Symantec AV signature corruption"
• Maybe in reply to: Ben Reardon: "Symantec AV signature corruption"
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] [ attachment ]