Binaries spontaneously changed on CentOS

Question

This has my slightly worried: I am using afick to monitor a CentOS 6 server for changes to files & directories. I want to detect changes to binary files, PHP scripts that are being smuggled onto the server, configiration files that change, etc. This runs daily and I get an email with detected changes. Usually it only contains log files and changes after I updated my webcode or installed new software. Today I seemed to have hit jackpot, but I'm not sure.

I received an email where the MD5 checksum of hundreds of files have changed, but not their timestamp or size. This includes executables like /bin/gawk but also libraries as /lib/libasound.so.2.0.0. This all happened between 4:00 January 1 and 4:00 January 2 (afick runs at 4:00).

As a test, I restored /bin/gawk from backup and ran a manual md5 checksum; indeed the file has changed. But a diff between the two binaries is somewhat non-conclusive:

--- old.gawk.hex        2017-01-02 15:56:06.000000000 +0100
+++ new.gawk.hex        2017-01-02 15:56:14.000000000 +0100
@@ -881,12 +881,12 @@
 00003700  a6 03 00 00 00 00 00 00  d1 04 00 00 12 00 0d 00  |................|
 00003710  f0 6d 42 00 00 00 00 00  2a 10 00 00 00 00 00 00  |.mB.....*.......|
 00003720  01 00 00 00 b0 6b 5a 56  65 fd 1b 6d 00 00 00 00  |.....kZVe..m....|
-00003730  00 00 00 00 44 00 00 00  b0 6b 5a 56 b2 04 c4 e2  |....D....kZV....|
+00003730  00 00 00 00 44 00 00 00  56 e5 5d 58 82 a0 c7 cf  |....D...V.]X....|
 00003740  00 00 00 00 00 00 00 00  62 00 00 00 b0 6b 5a 56  |........b....kZV|
 00003750  58 97 65 11 00 00 00 00  00 00 00 00 97 10 00 00  |X.e.............|
 00003760  b0 6b 5a 56 30 fb 60 86  00 00 00 00 00 00 00 00  |.kZV0.`.........|
 00003770  b0 2f 40 83 34 00 00 00  01 00 00 00 00 00 00 00  |./@.4...........|
-00003780  e0 08 65 00 00 00 00 00  e0 1f c8 83 34 00 00 00  |..e.........4...|
+00003780  e0 08 65 00 00 00 00 00  e0 1f 88 0a 35 00 00 00  |..e.........5...|
 00003790  01 00 00 00 00 00 00 00  08 09 65 00 00 00 00 00  |..........e.....|
 000037a0  50 2d 15 83 34 00 00 00  01 00 00 00 00 00 00 00  |P-..4...........|
 000037b0  d0 ff ff ff ff ff ff ff  58 2d 15 83 34 00 00 00  |........X-..4...|
@@ -19806,13 +19806,13 @@
 *
 000501e0  28 00 65 00 00 00 00 00  1e 59 40 00 00 00 00 00  |(.e......Y@.....|
 000501f0  00 00 00 00 00 00 00 00  00 b1 e8 82 34 00 00 00  |............4...|
-00050200  10 cd ec 82 34 00 00 00  50 32 a2 83 34 00 00 00  |....4...P2..4...|
-00050210  80 79 e6 82 34 00 00 00  e0 2f a2 83 34 00 00 00  |.y..4..../..4...|
+00050200  10 cd ec 82 34 00 00 00  50 32 62 0a 35 00 00 00  |....4...P2b.5...|
+00050210  80 79 e6 82 34 00 00 00  e0 2f 62 0a 35 00 00 00  |.y..4..../b.5...|
 00050220  20 87 e7 82 34 00 00 00  20 bc e8 82 34 00 00 00  | ...4... ...4...|
 00050230  20 9f e7 82 34 00 00 00  b0 05 e8 82 34 00 00 00  | ...4.......4...|
 00050240  d0 af e9 82 34 00 00 00  20 5e ed 82 34 00 00 00  |....4... ^..4...|
 00050250  40 7e ee 82 34 00 00 00  40 71 ec 82 34 00 00 00  |@~..4...@q..4...|
-00050260  10 9d e9 82 34 00 00 00  30 6f a1 83 34 00 00 00  |....4...0o..4...|
+00050260  10 9d e9 82 34 00 00 00  30 6f 61 0a 35 00 00 00  |....4...0oa.5...|
 00050270  f0 d7 ec 82 34 00 00 00  60 19 e3 82 34 00 00 00  |....4...`...4...|
 00050280  e0 b1 e9 82 34 00 00 00  10 85 ee 82 34 00 00 00  |....4.......4...|
 00050290  30 84 ec 82 34 00 00 00  40 20 e6 82 34 00 00 00  |0...4...@ ..4...|
(etc)

Of course, my first thought was hacking but seeing the diff makes me wonder. No real code seems to have changed; I'm no expert on ELF binaries but I think these are only relocation offset tables for shared libraries.

So what do you think really happened? Apart from hacking the only other possibility I can think of is a 'security' measure where shared library offsets are randomized and the linked-to binaries must be updated as well. But why now? The last time I installed some software was on December 23rd and nothing odd showed up in between. The only cronjob that might be related is /etc/cron.daily/prelink, but if so why now?

Answer 1

The difference in binary checksums you have described is probably due to prelinking. RHEL-based Linux distributions like CentOS and Fedora have prelinking enabled by default. Here's how a 2009 LWN.net article explains the concept behind prelink:

Linux programs typically consist of a binary executable file that refers to multiple shared libraries. These libraries are loaded into memory once and shared by multiple executables. In order to make that happen, the dynamic linker (i.e. ld.so) needs to change the binary in memory such that any addresses of library objects point to the right place in memory. For applications with many shared libraries—GUI programs for example—that process can take some time.

The idea behind pre-linking is fairly simple: reduce the amount of time the dynamic linker needs to spend doing these address relocations by doing it in advance and storing the results. The prelink program processes ELF binaries and shared libraries in much the same way that ld.so would, and then adds special ELF sections to the files describing the relocations. When ld.so loads a pre-linked binary or library, it checks these sections and, if the libraries are loaded at the expected location and the library hasn't changed, it can do its job much more quickly.

However, if libraries are always loaded at the same memory location, attackers can try to target these locations with malicious code, which is why Redhat distro's run prelink regularly with the -R option (to achieve Address Space Layout Randomization). One consequence of changing memory locations is that the checksum of binary executable files will change. Therefore if you are using a file integrity checker such as AIDE, you will be alerted about a "changed" binary when in fact the only change that has occurred is ASLR via prelink -R.

References: https://en.wikipedia.org/wiki/Prelink#Linux