Known-Good MD5 Database

bgp4 writes "Have you ever examined a system you thought was broken into but you weren't sure? If only you had run an integrity verification program like osiris or Tripwire first you could have figured out what programs had been changed. In an effort to help out in the instances when you can't answer the question "what was this like before?" we've constructed a searchable database of MD5 and SHA-1 hashes for files in many standard operating systems. You can search using the filename or the checksum and see if you have a trojaned binary or an overactive imagination. Currently at knowngoods.org we have many FreeBSD, OS X, Linux, and Solaris installations checksummed and cataloged. If you have other programs or distributions you would like to see in the database, please let us know."

Checksum

[Anonymous Coward]

checksum [reference.com]:

<storage, communications> A computed value which depends on the contents of a block of data and which is transmitted or stored along with the data in order to detect corruption of the data. The receiving system recomputes the checksum based upon the received data and compares this value with the one sent with the data. If the two values are the same, the receiver has some confidence that the data was received correctly.

The checksum may be 8 bits (modulo 256 sum), 16, 32, or some other size. It is computed by summing the bytes or words of the data block ignoring overflow. The checksum may be negated so that the total of the data words plus the checksum is zero.

Internet packets use a 32-bit checksum.

Compromised /bin/md5

[Cadre]

What they don't say and what a lot of security folks forget to do is that they can't check your checksums of binaries on the same box. You need to copy the files to another box and check the checksums there with a known good version of your checksumming binary. The local version of your checksumming binary could have been compromised.

Heck, the utilities you used to pull the binary off the machine in question could have been compromised and may not be actually copying the binary in question, but a good version of the binary. The only way to check this would be to mount the drive on another machine and check it there... And if people aren't doing that (which it's a pain in the ass) all this website is going to do is give people a false sense of security.

Polymorphic files

[cperciva]

There is one problem with this: Some files are going to be different every time they are compiled. In particular, quite a few files include time stamps.

A few months ago I put together a list of the "polymorphic" files in FreeBSD 4.6:

/kernel, /boot/loader, and /boot/pxeboot all contain user, host, time, and date stamps, as expected.

All .a files (126 in /usr/lib, one in /usr/libdata/perl/5.00503/mach/auto/DynaLoader) contain indices of .o files, including seconds-since-epoch stamps

User, host, time, and date stamps are found in /etc/mail/freebsd.cf /usr/sbin/named /usr/libexec/named-xfer

Time and date stamps are found in: /usr/bin/suidperl /usr/bin/ntpq /usr/sbin/ntp(d|date|dc|timeset|trace) /usr/sbin/isdn(d|debug|monitor|phone|telctl) /usr/libdata/perl/5.00503/mach/perllocal.pod

Date stamps are found in: /usr/sbin/ppp /var/db/port.mkversion /usr/share/doc/usd/(07.mail|13.viref|18.msdiffs|19 .memacros|20.meref)/paper.ascii.gz (once you ungzip them) /usr/share/perl/man/man3/(Config|DynaLoader).3.gz (once you ungzip them)

Files which are always the same size, but have randomized contents: /usr/share/games/fortune/*.dat /var/games/phantasia/void

These files are always going to set off alarms if you've upgraded-by-source. (On the other hand, if a file *not* on this list has a different checksum, it probably just means that you've applied a security patch.)

md5sum Binary Might Be Trojaned

[John Hasler]

Boot from a known good floppy or CD to check your md5sums.

Debian / debsums

[zsazsa]

Debian has this built into the OS with debsums [debian.org].

It does require a legit dpkg database (and md5sum, and the debsums program itself...) but it's a nice tool.

Use BITZI too!

[aminorex]

I'd rather see everyone using bitzi.com, since it's
goal is to gather metadata for *every* file in the
universe, and keep the data free, supported by a
related business model (and a viable, sustainable
support mechanism is GOOD), but I support this
project too, because choice and freedom are goods.
Therefore, I urge everyone to submit metadata
to both projects.

If you only submit to one, however, please submit
to bitzi, because it provides an automation API,
and uses better hashes.

Note that I have no affiliation with the Bitzi company.

Re:What about source builds?

[Anonymous Coward]

You're wrong. If you compile from source you can be sure of what you're getting. You do realize, don't you, that replacing utilities like ls would is a key part of any rootkit?

Another Resource

[Taim]

NIST (The National Institute of Standards and Technology) currently has a program to provide this service, though largely focused on Microsoft OSes and associated apps. It may be found here: National Software Reference Library [nist.gov]

The complete list of software they've checksummed can be found here: Software Listing [nist.gov] or you can use their search engine if you're looking for a specific application here: Search Engine [nist.gov]

Bleah

[digitaltraveller]

NIST [nist.gov] does this too. For a different reason though. To help forensic examiners eliminate non-important data in a suspect's computer. They use 4 different hash algorithms (MD5, SHA-1, CRC32, and one other), so good luck finding a collision for all 4. They were giving out copies of the CD-hashdb at an InfoSec conference I was at recently.

Re:Something's wrong here

[ShmuelP]

A simpler way would be to just have a big, signed list of md5's in some file that tripwire checks against. Tripwire would check the signature on the file before believing the md5's in it.

Note: this is exactly what tripwire already does. Except that it also stores other file attributes as well.

Er, rpm -V?

[SlashChick]

For Red Hat-based systems, at least, rpm -V will do pretty much exactly what you're looking for.

From the man page for rpm:

The general form of an rpm verify command is

rpm {-V|--verify} [select-options] [--nodeps] [--nofiles] [--nomd5] [--noscripts]

Verifying a package compares information about the installed files in the package with
information about the files taken from the package metadata stored in the rpm database. Among other things, verifying compares the size, MD5 sum, permissions, type, owner and group of each file. Any discrepencies are displayed. ... The (mnemonically emBoldened) character denotes failure of the corresponding --verify test:

S file Size differs

M Mode differs (includes permissions and file type)

5 MD5 sum differs

D Device major/minor number mis-match

L readLink(2) path mis-match

U User ownership differs

G Group ownership differs

T mTime differs


So while that's a bit cryptic, a shell script run once every x days (30? 14?) should tell you what files have changed. All you would have to do is run rpm -qa to grab a list of the packages in your system, and then loop through the list and run rpm -V for each RPM returned.

For instance, running rpm -V on two common packages, Apache and PHP, shows me the following:
# rpm -V php
S.5....T c /etc/php.ini
(php.ini has changed... which in this case means I've tweaked some of PHP's default settings.)

# rpm -V apache
S.5....T c /etc/httpd/conf/httpd.conf
missing /var/www/html/index.html
missing /var/www/html/poweredby.png
(Okay, I've changed httpd.conf, again pretty much a given, and I've removed a couple of the default files.)

I guess this website seems pretty unneeded to me. Granted, the above is just for RPM-based systems, but I'm sure Debian and ports have similar options. And to the people who have installed from source and say "What about me?", I say, first, never underestimate the power of a package management system, and second, check out CheckInstall [asic-linux.com.mx], which allows you to create an RPM or DEB just by saying "checkinstall" instead of "make install". If you feel you must compile from source, checkinstall is a necessity! Using checkinstall gives you all the benefits of a package management system while still allowing for the flexibility that compiling from source provides.

Between checkinstall and up2date, I'm a very happy Red Hat customer. I just wish more people knew about some of the truly powerful things in package management systems (such as the verify command detailed above.) Package management systems are there for a reason. Use them! :)

Re:What about source builds?

[pVoid]

Yes, the Win32 PE format (portable executable) has a checksum field which is 'normally' not used.

It *is* checked for *some* critical system images however... I know for sure that some files in /system32 (so called 'KnownDlls') are among this list.

Note though, that this checksum is to prevent accidental data corruption and not maintain system security integrity; since the checksum field is actually in the file itself, it can be updated after a virus/haxxor has patched the target file.

Solaris Fingerprint Database

[nbvb]

Sun already provides this for Solaris.

http://sunsolve.Sun.COM/pub-cgi/fileFingerprints .p l

It contains information for:

Operating Systems

Solaris SPARC - 2.0, 2.1, 2.2, 2.3, 2.4, 2.5, 2.5.1, 2.6, Solaris 7 and Solaris 8
Solaris x86 - 2.1, 2.4, 2.5, 2.5.1, 2.6, Solaris 7 and Solaris 8
Solaris PPC - 2.5.1
Trusted Solaris SPARC - 2.5, 2.5.1 and 7
Trusted Solaris 7 x86
Most CDs bundled with Solaris 2.6 and later.

Patches

Nearly all released Solaris patches, including all SunSolve CDs to date. (4.0.11)
All Solaris 2.6/7 Maintenance updates.
All patches available from SunSolve.

Unbundled Products

Around 150 CDs with unbundled products are included. If you are missing any particular product, please feel free to send email and we will try to include it as soon as possible.

Re:how does this work?

[Anonymous Coward]

you're a retard.

how about you search /dev/null, and get the good md5 then compare to yours, yours is diff, you've been hacked

Re:What about source builds?

[deranged unix nut]

Unless your compiler, linker, assembler, libraries, or source code have been modified.

Sheesh, dosen't anyone read old ACM articles?

http://www.acm.org/classics/sep95/

At some point, unless you build your system from scratch, cross compile on multiple systems, burn your own BIOS ROM, and write the microcode for your NIC and all other interface devices, you are trusting *SOMEONE ELSE* for the security of your system.

Re:This is one of those things...

[Anonymous Coward]

It WAS done sooner. Sun have a fingerprints
database for Solaris binaries.

Re:Something's wrong here

[Nailer]

Every package in the current Red Hat Linux is signed using GPG, and IIRC this has been the case for a few years now. Most other Linux distros also sign their packages.

Re:Er, rpm -V?

[D0wnsp0ut]

For Red Hat-based systems, at least, rpm -V will do pretty much exactly what you're looking for.

For those who pointed out that the RPM database is a local database, remember, you can get those MD5 hashes from 2 other sources:

• your installation medium (CD)
• from Red Hat's web site itself

Of course, using the web site is going to be a lot more labor intensive, but it is available. Writing a script to compare hashes computed using RPMs from the CD image and comparing them to your installed binaries should be a piece of cake (using the --dump option to -ql).

Re:Something's wrong here

[wirelessbuzzers]

MAC == Message Authentication Code. It's basically a hash with a secret key. Some good ones (in addition to algorithms written as MACs) are Encryption_Key(Hash(File)) and Hash(Key1,File,Key2)