Collect logs from Linux systems

Linux distributions commonly use a "syslog" system logging agent to retrieve events from the kernel (/proc/kmsg) and accept log messages from user-space applications (/dev/log). Initially, this logger was syslogd; later, syslog‑ng added additional features; finally, Rsyslog is the logger in everyday use today. For more information about Linux syslog, see syslog.

Many modern Linux distributions also use the systemd init system, which includes a journal component for handling log messages. All logs generated by systemd-controlled processes are sent to the journal. The journal also handles logs written to /dev/log. The journal stores logs in a binary format, either in memory or on disk; the logs can be accessed with the journalctl tool. Systemd can also be configured to forward logs via a socket to a local logger like Rsyslog or NXLog Agent.

NXLog Agent can be configured to collect Linux system logs by being a substitute for Rsyslog and collecting logs in its place, through forwarding logs via a socket from Rsyslog, via reading log files written by Rsyslog, and lastly, by reading systemd journal logs directly.

Replacing Rsyslog with NXLog Agent

Follow these steps to disable Rsyslog and configure NXLog Agent to collect logs in its place.

Configure NXLog Agent to collect kernel logs, the systemd journal, and the /dev/log socket. See the example below.
Disable and stop Rsyslog by running the following commands as root:
```
# systemctl disable rsyslog
```
```
# systemctl stop rsyslog
```

Reload the journald configuration:

# systemctl force-reload systemd-journald

Restart NXLog Agent:
```
# systemctl restart nxlog
```

Example 1. Replacing Rsyslog with NXLog Agent

This example configures NXLog Agent to read kernel logs with the im_kernel module, read daemon logs from the systemd journal socket with the im_systemd module, and accept other user-space logs from the /dev/log socket with im_uds. In the om_tcp output module instance, all of the logs are converted to JSON format, BSD syslog headers are added, and the logs are forwarded to another host via TCP.

nxlog.conf

<Extension json>
    Module         xm_json
</Extension>

<Extension syslog>
    Module         xm_syslog
</Extension>

<Input kernel>
    Module         im_kernel
    Exec           parse_syslog_bsd();
</Input>

<Input systemd>
    Module         im_systemd
</Input>

<Input devlog>
    Module         im_uds
    UDS            /dev/log
    FlowControl    FALSE
    Exec           $raw_event =~ s/\s+$//; (1)
    Exec           parse_syslog_bsd();
</Input>

<Output out>
    Module         om_tcp
    Host           192.168.1.1:1514
    Exec           $Message = to_json(); to_syslog_bsd();
</Output>

<Route r>
    Path           kernel, systemd, devlog => out
</Route>

1	Some local syslog sources add a trailing newline (`\n`) to each log message. This line automatically removes newlines and any other trailing whitespace before processing the log.

Forwarding logs via socket from Rsyslog to NXLog Agent

By adding a small configuration file, Rsyslog can be configured to forward logs to NXLog Agent via a Unix domain socket. This is the least intrusive of the options documented here.

By default, SELinux blocks communication via Unix domain sockets in CentOS 7. To enable socket communication, run the following commands:

# audit2allow -i /var/log/messages -M nxlog-fix

and then:

# semodule -i nxlog-fix.pp

The description below contains steps for configuring Rsyslog to work with NXLog Agent.

Configure NXLog Agent to accept log messages from Rsyslog via a socket. See the example below.

Configure Rsyslog to write to the socket by adding the following configuration file. See the Rsyslog documentation for more information about configuring what is forwarded to NXLog Agent.

/etc/rsyslog.d/nxlog.conf

# Load omuxsock module
$ModLoad omuxsock

# Set socket path
$OMUxSockSocket /opt/nxlog/var/spool/nxlog/rsyslog_sock

# Configure template to preserve PRI part (must be on a single line)
$template SyslogWithPRI,"<%PRI%>%timegenerated% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%"

# Forward all log messages
*.* :omuxsock:;SyslogWithPRI

# Only forward log messages of "notice" priority and higher
#*.notice :omuxsock:;SyslogWithPRI

Restart NXLog Agent and Rsyslog in that order to create and use the socket (NXLog Agent must create the socket before Rsyslog will write to it). Run systemctl restart nxlog and systemctl restart rsyslog.

Example 2. Collecting logs via socket from Rsyslog

With this example configuration, NXLog Agent will create the socket and accept log messages from Rsyslog through the socket. The logs will then be parsed as syslog, converted to JSON format, prefixed with a BSD syslog header, and forwarded to another host via TLS.

nxlog.conf

<Extension json>
    Module         xm_json
</Extension>

<Extension syslog>
    Module         xm_syslog
</Extension>

<Input in>
    Module         im_uds
    UDS            /opt/nxlog/var/spool/nxlog/rsyslog_sock
    Exec           parse_syslog();
</Input>

<Output out>
    Module         om_ssl
    Host           192.168.1.1:6514
    CAFile         %CERTDIR%/ca.pem
    CertFile       %CERTDIR%/client-cert.pem
    CertKeyFile    %CERTDIR%/client-key.pem
    Exec           $Message = to_json(); to_syslog_bsd();
</Output>

Reading logs written to file by Rsyslog

NXLog Agent can be configured to read log messages written to file by Rsyslog, for example /var/log/messages. This is a slightly more intrusive option than the steps given in [forwarding-logs-via-socket-from-rsyslog-to-nxlog].

NXLog Agent will not have access to the facility and severity codes because Rsyslog, by default, follows the BSD syslog convention of not writing the PRI code to the /var/log/messages file.

By default, NXLog Agent runs as user nxlog and does not have permission to read files in /var/log. The simplest solution for this is to run NXLog Agent as root by omitting the User option, but it is more secure to provide the necessary permissions explicitly.

Check the user or group ownership of the files in /var/log and configure them if necessary. Some distributions use a group for the log files by default. On Debian/Ubuntu, for example, Rsyslog is configured to use the adm group. Otherwise, modify the Rsyslog configuration to use different ownership for log files as shown below.
/etc/rsyslog.conf or /etc/rsyslog.d/nxlog.conf
```
$FileOwner root
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022

# Default on Debian/Ubuntu
$FileGroup adm

# Or use the "nxlog" group directly
#$FileGroup nxlog
```
Run NXLog Agent under a user or group that has permission to read the log files. Either use a user or group directly with the User or Group option in nxlog.conf or add the nxlog user to a group that has permission. For example, on Debian/Ubuntu add the nxlog user to the adm group by running usermod -a -G adm nxlog.
If necessary, fix permissions for any files NXLog Agent will be reading from that already exist (use the correct group for your system).
```
# chgrp adm /var/log/messages
# chmod g+r /var/log/messages
```
Configure NXLog Agent to read from the required file(s) (see the example below), then restart NXLog Agent.
If the Rsyslog configuration has been modified, restart Rsyslog (systemctl restart rsyslog).

Example 3. Reading Rsyslog log files

With the following configuration, NXLog Agent will read logs from /var/log/messages, parse the events as syslog, convert them to JSON, and forward the plain JSON to another host via TCP.

nxlog.conf

<Extension json>
    Module    xm_json
</Extension>

<Extension syslog>
    Module    xm_syslog
</Extension>

<Input in>
    Module    im_file
    File      '/var/log/messages'
    Exec      parse_syslog();
</Input>

<Output out>
    Module    om_tcp
    Host      192.168.1.1:1514
    Exec      $raw_event = to_json();
</Output>

Reading systemd journal logs

NXLog Agent can be configured to read systemd journal logs directly with the im_systemd input module. This module is available with the distribution-specific Linux packages.

Example 4. Reading systemd journal logs with im_systemd

The following configuration uses the im_systemd input module to read systemd journal logs. It uses the xm_json extension to convert logs to JSON format and forwards them to another host via TCP.

nxlog.conf

<Extension json>
    Module    xm_json
</Extension>

<Input systemd>
    Module    im_systemd
    Exec      to_json();
</Input>

<Output out>
    Module    om_tcp
    Host      192.168.1.1:1514
</Output>

The user running NXLog Agent must be added to the systemd-journal group to be able to read systemd journal logs.

Output sample

The following is a systemd journal log in JSON format after it was processed by NXLog Agent.

{
  "Severity": "warning",
  "SeverityValue": 4,
  "Message": "(==) Max clients allowed: 256, resource mask: 0x1fffff",
  "SourceName": "/usr/lib/gdm3/gdm-x-session",
  "ProcessID": 2172,
  "User": "jdoe",
  "Group": "jdoe",
  "ProcessName": "Xorg",
  "ProcessExecutable": "/usr/lib/xorg/Xorg",
  "ProcessCmdLine": "/usr/lib/xorg/Xorg vt2 -displayfd 3 -auth /run/user/1000/gdm/Xauthority -background none -noreset -keeptty -verbose 3",
  "Capabilities": "0",
  "AuditSession": 2,
  "AuditUID": "jdoe",
  "SystemdCGroup": "/user.slice/user-1000.slice/session-2.scope",
  "SystemdSession": "2",
  "SystemdUnit": "session-2.scope",
  "SystemdOwnerUID": "jdoe",
  "SystemdSlice": "user-1000.slice",
  "SelinuxContext": "unconfined\n",
  "BootID": "9eeef5226ff441e0a3c8bd990ded3af0",
  "MachineID": "7994bd4cc9744ab78846106097c30ff6",
  "SysInvID": "d4a984306a31454ba5bbca0f23e9fc06",
  "Hostname": "Ubuntu-1",
  "Transport": "stdout",
  "EventReceivedTime": "2021-05-12T09:27:52.554654+02:00",
  "SourceModuleName": "systemd",
  "SourceModuleType": "im_systemd"
}

For generic Linux packages that do not include the im_systemd module, a similar solution can be achieved with a script that executes the journalctl tool to query the systemd journal.

Example 5. Reading systemd journal with a helper script

This script uses the journalctl tool to query the systemd journal. It specifies the following arguments:

-o json: Sets the output format to standard JSON written one record per line.
--cursor-file: Sets the path to the cursor file. If the file exists, it starts reading from the position stored in the cursor file. When stopped it will write the cursor to the last entry it read in this file.
--no-pager: Pipe output to the standard output instead of the default pager.

systemd-tail.sh

#!/bin/bash
CRS=/opt/nxlog/var/spool/nxlog/systemd-tail.pos
SAMPLE_TIME=1

while sleep $SAMPLE_TIME; do
  journalctl -o json --cursor-file=${CRS} --no-pager
done

The following configuration uses the im_exec input module to execute the helper script and forwards logs to another host via TCP. The xm_json extension can be used to parse log data into fields for further processing.

nxlog.conf

<Extension json>
    Module     xm_json
</Extension>

<Input systemd>
    Module     im_exec
    Command    /usr/local/bin/systemd-tail.sh
    <Exec>
        parse_json();

        # Drop debug messages
        if ($PRIORITY == "7")
            drop();
    </Exec>
</Input>

<Output out>
    Module     om_tcp
    Host       192.168.1.1:1514
</Output>

The user running NXLog Agent must have permission to read and execute the script.

Output sample

The following is a systemd journal log in JSON format after it was processed by NXLog Agent.

{
  "_SYSTEMD_INVOCATION_ID": "22b0df526f7e4630a3d1ce2af8e9e50a",
  "_GID": "1000",
  "_CAP_EFFECTIVE": "0",
  "MESSAGE": "(==) Max clients allowed: 256, resource mask: 0x1fffff",
  "_PID": "1315",
  "_COMM": "Xorg",
  "_HOSTNAME": "Ubuntu-1",
  "_EXE": "/usr/lib/xorg/Xorg",
  "_MACHINE_ID": "7994bd4cc9744ab78846106097c30ff6",
  "_BOOT_ID": "7afc2c1a7e154687af0865c1398ba630",
  "_SYSTEMD_SLICE": "user-1000.slice",
  "_SYSTEMD_CGROUP": "/user.slice/user-1000.slice/session-2.scope",
  "__CURSOR": "s=78878570db18458e882cf0825126cdf2;i=cba63;b=7afc2c1a7e154687af0865c1398ba630;m=1ab0c60;t=5c1cd3c03d8f3;x=aa5241a02d97b76b",
  "_SYSTEMD_UNIT": "session-2.scope",
  "SYSLOG_IDENTIFIER": "/usr/lib/gdm3/gdm-x-session",
  "_SELINUX_CONTEXT": "unconfined\n",
  "_UID": "1000",
  "_AUDIT_LOGINUID": "1000",
  "__MONOTONIC_TIMESTAMP": "27987040",
  "_CMDLINE": "/usr/lib/xorg/Xorg vt2 -displayfd 3 -auth /run/user/1000/gdm/Xauthority -background none -noreset -keeptty -verbose 3",
  "_AUDIT_SESSION": "2",
  "__REALTIME_TIMESTAMP": "1620462102894835",
  "_TRANSPORT": "stdout",
  "_STREAM_ID": "0554d01d26664c979d5eff46ae4e4f41",
  "_SYSTEMD_OWNER_UID": "1000",
  "_SYSTEMD_USER_SLICE": "-.slice",
  "PRIORITY": "4",
  "_SYSTEMD_SESSION": "2"
}

Disclaimer

While we endeavor to keep the information in our guides up to date and correct, NXLog makes no representations or warranties of any kind, express or implied about the completeness, accuracy, reliability, suitability, or availability of the content represented here. We update our screenshots and instructions on a best-effort basis.

The accurateness of the content was tested and proved to be working in our lab environment at the time of the last revision with the following software versions:

NXLog Agent version 5.7.7898
Ubuntu 20.04.4 LTS

Last revision: 29 March 2023