# Audit Logging

# Aggregate System Log Stream and Secure Transportation

For auditing purposes, the system journal, kernel audit messages, and container runtime output are collected into a single stream of log messages, then optionally transported to an aggregator via the syslog protocol in real time, under the protection of TLS.

# Overview

The aggregated system log stream is collected from two independent sources in real time. The majority of the messages from the stream originate from system daemons and the kernel; few of them come from container output at runtime. All sources emit messages asynchronously. Therefore, the stream does not have and cannot have a concept of "logging context" that establishes any sort of relationship among the messages, not even a "happen-before" relationship.

The log stream is handled by a syslog-compatible daemon (rsyslog). Output from system daemons that are typically managed by systemId is handled by journald which is instructed to forward a copy of its content to the log stream. Kernel auditing monitors file deletion and process creation among other events, and once triggered journald forwards the output audit messages to the log stream. Container runtime output as well as container daemon operations are also monitored by journald and are subsequently forwarded to the log stream.

Forwarding of container runtime output to the system journal also leverages the advantage of automated disk usage management provided by the system journal, that prevents container output from occupying an indefinite amount of disk space in the long run.

The log stream is then optionally transported to a syslog-compatible server for aggregation, under the protection of TLS. By design, the log stream always receives a copy of log messages rather than making exclusive claim of them, enabling developers to inspect the original log messages using conventional tools such as journalctl.

As a result, important system events are captured in the log stream, such as:

File rename and deletion by unprivileged user

audispd: node=houzuo-guo-ability-edge-vm type=SYSCALL msg=audit(1536826671.592:947): arch=c000003e syscall=263 success=yes exit=0 a0=ffffff9c a1=564439138190 a2=0 a3=100 items=2 ppid=14238 pid=14463 auid=1000 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts5 ses=11 comm="rm" exe="/bin/rm" key="abb-ability-edge-audit-delete-trigger"

System logon attempts

audispd: node=houzuo-guo-ability-edge-vm type=USER_START msg=audit(1536826695.872:959): pid=14508 uid=0 auid=1000 ses=189 msg='op=PAM:session_open acct="howard" exe="/usr/sbin/sshd" hostname=127.0.0.1 addr=127.0.0.1 terminal=ssh res=success'

System user management actions

audispd: node=houzuo-guo-ability-edge-vm type=USER_CHAUTHTOK msg=audit(1536826760.382:970): pid=14630 uid=1000 auid=1000 ses=189 msg='op=PAM:chauthtok acct="howard" exe="/usr/bin/passwd" hostname=houzuo-guo-ability-edge-vm addr=? terminal=pts/6 res=failed'

Alteration to system firewall

audispd: node=houzuo-guo-ability-edge-vm type=NETFILTER_CFG msg=audit(1536826829.891:1077): table=filter family=2 entries=137

Process creation

type=SYSCALL msg=audit(1536827398.506:1732): arch=c000003e syscall=59 success=yes exit=0 a0=7ffc709d61b0 a1=560a7fb18d60 a2=560a7fb1d460 a3=8 items=2 ppid=14591 pid=15928 auid=1000 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts6 ses=189 comm="true" exe="/bin/true" key="abb-ability-edge-audit-create-process"

System startup and shutdown

# Container Daemon Operations

Spawning new containers

dockerd[17749]: time="2018-09-13T08:41:07.945223698Z" level=debug msg="Calling POST /v1.35/containers/create?name=abbbbbbb"

Inspect digest of container image

dockerd[17749]: time="2018-09-13T08:39:39.156640364Z" level=debug msg="found match for linux/amd64 with media type application/vnd.docker.distribution.manifest.v2+json, digest sha256:02892826401a9d18f0ea01f8a2f35d328ef039db4e1edcc45c630314a0457d5b"

Image download attempts

dockerd[17749]: time="2018-09-13T08:39:39.156564062Z" level=debug msg="Pulling ref from V2 registry: alpine:latest"

Individual container output

dockerd[19033]: This message shows that your installation appears to be working correctly.

# Setup

As a prerequisite, the cooperate network should have X509 infrastructure in place to generate the TLS certificate for both the aggregator server and the Edge host. Once these are verified, alter the TLS parameters defined in edge.env (or edge.env.dps). The path to the PEM-encoded CA certificate anchor must be readable by everyone (chmod 644) on the Edge host.

For development scenarios, use the X509 CA certificate sys-trail/dummy-cert.pem and key sys-trail/dummy-key.pem on both the log aggregator server and the Edge host. As TLS settings demand strong verification of the server name, place the following line inside /etc/hosts and substitute AGGREGATOR_IP with the server IP address:

AGGREGATOR_IP abb-ability-edge-development-sys-log-dummy-certificate

Copy sys-trail/dummy-cert.pem to the Edge host /etc/rsyslog-ca.pem file and grant read permission chmod 644 /etc/rsyslog-ca.pem.

With TLS settings and the CA anchor file in place, invoke the installer program abb-iot-edge-setup and the log stream transport will be set up automatically. If your installation does not require an aggregator, the setup will do no harm, and a copy of the most recent audit log entries and system journal entries will remain on the system's transient storage for local diagnosis.

# Operation

A copy of most recent system journal that includes container output and audit entries is kept for diagnosis and inspection by operating system. The available entries among the journal can be inspected by root user via the journalctl command, for example:

Read latest couple of entries and follow all updates: journalctl -f
Read entries between system boot and now, do not follow updates: journalctl --boot
Read latest couple of entries and follow all updates of a system service such as SSH daemon: journalctl -f -u ssh
Read entries of a system service such as SSH daemon between system boot and now, do not follow updates: journalctl --boot -u ssh
Read latest couple of entries and follow all updates of a container journalctl -f CONTAINER_ID=#####
Read entries of a container between the earliest available point-in-time and now: journalctl CONTAINER_ID=#####

# Kiwi Syslog Server

# Example Aggregator

Kiwi syslog server is a proprietary implementation of the syslog protocol for the Windows platform. It works well as an aggregator of log streams from an arbitrary number of Edge hosts. The software is available in both trial and full versions. Only the full version supports TLS transport.

To set up:

Register the installation of the Kiwi syslog server using the full version license key.
Import the server certificate, including the certificate chain and private key, as a bundled PKCS12 file into the "Local Machine" certificate store. For development scenarios, use the PKCS 12 file sys-trail/dummy-cert-with-key-empty-password.pfx.
In the Kiwi syslog server, under File → Setup, navigate to the Secure TCP tree item. Check the Listen for secure TCP syslog messages checkbox and select the imported certificate. Then, set up the remainder of the items, as follows:

auditlogging

Click OK and the log stream from the Edge host will appear shortly.

# Diagnosis

If the log stream does not arrive at the aggregator server, run the command systemctl status rsyslogd from the Edge host shell to determine the cause.

← Edge Secrets Device ID Management →