Encrypting Apache Logs

by J. Edward Durrett

One of the features of commercial logging solutions is encrypted storage of log data [1]. This is necessary since
regulatory compliance and standards (PCI, HIPPA, etc) for an organization usually mandates the logs are kept and
that sensitive data be encrypted when at rest. Log data, even when there is seemingly no personally identifying
information, can be extremely revealing and even dangerous in the wrong hands.

Really, seems like a lot of work . . . why?

It is easy to explain why we would want to encrypt financial or health data, but why would an NGO, for example,
want to bother with that level of security for just log data? It is simple, if an adversary can get their hands on
your log data, they know who has gone to your website and when. Even if they don’t have the ip addresses to
identify individuals, the data can be used to form a bigger picture. A spike in traffic corresponding to a time
zone is inductive of the region/country your site is drawing visitors. This alone could be enough to give your
organization tougher focus by intelligence agencies. That is just one of multiple ways one can manipulate and
massage information out of logs.

The solution I use is taking advantage of the logging to pipe feature in Apache. Log data is piped to gpg and the
encrypted data is written to disk. When I first proposed this to a colleague, she immediately asked about CPU
impact. As I am sure that is one of the most common questions asked, the answer is: yes, there is a slight impact.
The graph compares cpu usage with and without gpg logging. This was done on using ab (apache benchmarking utility)
simulating 1000 requests with a concurrency of 100:

The rest of this post is a technical explanation on how to secure your log data with asymmetrical encryption using
freely available tools. In the simplest form, sending log data to gpg is accomplished by modifying Apache’s
configuration to point to a basic script to handle the data:

CustomLog "|$/path_to _script/pipe_to_gpg.sh" combined

The contents of the script are:

while read logdata; do 
   echo $logdata | gpg -ear [key] --batch >> /path/encrypted.log 

Here the options to be used for gpg can be tweaked. --homedir is a helpful option when getting the configuration
correct for your environment. The advantage to calling a script is you can also add a signature, include a time
stamp, or pipe the entire process across a network to be done on another machine.

There is a problem with doing this that way. Namely, when Apache calls an external program it does that with the
original user that started Apache, usually root, not the lower privileged www-data. This means the
logging/encrypting script above is running as root.

Following the principle that programs should have the least privileges possible, the logging/encrypting process
needs to have some restrictions. One way to do this is to create a chroot for running the process. Here is a
script to create a gpg chroot:

#Set the path for the chroot here - use the full path! 
mkdir $gnupg_chroot mkdir $gnupg_chroot/etc touch $gnupg_chroot/etc/passwd touch $gnupg_chroot/etc/group touch $gnupg_chroot/etc/master.passwd touch $gnupg_chroot/etc/pwd.db touch $gnupg_chroot/etc/spwd.db mkdir $gnupg_chroot/libexec cp /libexec/ld-elf.so.1 $gnupg_chroot/libexec/ld-elf.so.1 mkdir $gnupg_chroot/lib cp /lib/libz.so.6 $gnupg_chroot/lib/libz.so.6 cp /lib/libreadline.so.8 $gnupg_chroot/lib/libreadline.so.8 cp /lib/libc.so.7 $gnupg_chroot/lib/libc.so.7 cp /lib/libthr.so.3 $gnupg_chroot/lib/libthr.so.3 cp /lib/libncurses.so.8 $gnupg_chroot/lib/libncurses.so.8 cp /lib/libedit.so.7 $gnupg_chroot/lib/libedit.so.7 mkdir $gnupg_chroot/usr mkdir $gnupg_chroot/usr/lib cp /usr/lib/libbz2.so.4 $gnupg_chroot/usr/lib/libbz2.so.4 mkdir $gnupg_chroot/bin cp /bin/sh $gnupg_chroot/bin/sh mkdir $gnupg_chroot/dev pkg --rootdir $gnupg_chroot install gnupg

Before starting the chroot, /dev needs to be populated. Using the rule set for jails works well for this, although
limiting /dev to even less devices should be considered:

mount -t devfs devfs /path_to_chroot/dev 
devfs -m /path_to_chroot//dev rule -s 4 applyset

By telling Apache to call chroot, the process is now limited to a restricted area of the file system:

CustomLog "|$/usr/sbin/chroot /path_in_chroot/pipe_to_gpg.sh" combined 

Further restrictions on the process can be added by using the -u option to chroot to run the process with lower
privileges. This works well with the –homedir option to gpg.

On another machine with the private key, the logs can be decrypted:

gpg --allow-multiple-messages -q encrypted.log 
xx.xxx.xxx.xx - - [18/Mar/2017:18:29:30 +0000] "GET /index.php HTTP/1.1" 200 3402 "https://jedwarddurrett.com/index.php" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0" gpg: WARNING: multiple plaintexts seen
xx.xxx.xxx.xx - - [18/Mar/2017:18:31:49 +0000] "GET /contact.php HTTP/1.1" 200 1445 "https://jedwarddurrett.com/index.php" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"

[1] https://www.balabit.com/network-security/syslog-ng/log-server-appliance/features
[2] https://httpd.apache.org/docs/2.4/logs.html#piped

Copyright (c) 2019, Jason Edward Durrett - All content on this site, unless otherwise noted, is subject to this license.

Please contact me if any errors, such as erroneous / misleading content or missing / incomplete attribution, are found.