Linux system logs
NXLog can be used to collect and process logs from a Linux system.
Linux distributions normally use a "Syslog" system logging agent to retrieve
events from the kernel (/proc/kmsg
) and accept log messages from user-space
applications (/dev/log
). Originally, this logger was syslogd; later
syslog‑ng added additional features, and finally Rsyslog is the logger in
common use today. For more information about Linux syslog, see Syslog.
Many modern Linux distributions also use the systemd init system, which
includes a journal component for handling log messages. All messages generated
by systemd-controlled processes are sent to the journal. The journal also
handles messages written to /dev/log
. The journal stores logs in a binary
format, either in memory or on disk; the logs can be accessed with the
journalctl tool. systemd can also be configured to forward logs via a socket
to a local logger like Rsyslog or NXLog.
There are several ways that NXLog can be configured to collect Linux logs. See Replacing Rsyslog for details about replacing Rsyslog altogether, handling all logs with NXLog instead. See Forwarding messages via socket for a simple way to forward all logs to NXLog without disabling Rsyslog (this is the least intrusive option). It is also possible to read the log files written by Rsyslog, see Reading Rsyslog log files. Finally, NXLog can read the systemd journal directly, see Reading the systemd journal.
Replacing Rsyslog
Follow these steps to disable Rsyslog and configure NXLog to collect logs in its place.
-
Configure NXLog to collect events from the kernel, the systemd journal, and the
/dev/log
socket. See the example below. -
Disable and stop Rsyslog by running the following commands as root:
# systemctl disable rsyslog
# systemctl stop rsyslog
-
Reload the journald configuration:
# systemctl force-reload systemd-journald
-
Restart NXLog:
# systemctl restart nxlog
This example configures NXLog to read kernel events with the im_kernel module, read daemon messages from the systemd journal socket with the im_systemd module, and accept other user-space messages from the /dev/log
socket with im_uds.
In the om_tcp output module instance, all of the logs are converted to JSON format, BSD syslog headers are added, and the logs are forwarded to another host via TCP.
<Extension _json>
Module xm_json
</Extension>
<Extension _syslog>
Module xm_syslog
</Extension>
<Input kernel>
Module im_kernel
Exec parse_syslog_bsd();
</Input>
<Input systemd>
Module im_systemd
</Input>
<Input devlog>
Module im_uds
UDS /dev/log
FlowControl FALSE
Exec $raw_event =~ s/\s+$//; (1)
Exec parse_syslog_bsd();
</Input>
<Output out>
Module om_tcp
Host 192.168.1.1:1514
Exec $Message = to_json(); to_syslog_bsd();
</Output>
<Route r>
Path kernel, systemd, devlog => out
</Route>
1 | Some local syslog sources add a trailing newline (\n ) to each log message.
This line automatically removes newlines and any other trailing whitespace before processing the message. |
Forwarding messages via socket
By adding a short configuration file, Rsyslog can be configured to forward messages to NXLog via a Unix domain socket. This is the least intrusive of the options documented here.
By default, SELinux blocks communication via Unix domain sockets in CentOS7. To enable socket communication, run the following commands.
and then
|
The description below contains steps for configuring Rsyslog to work with NXLog.
-
Configure NXLog to accept log messages from Rsyslog via a socket. See the example below.
-
Configure Rsyslog to write to the socket by adding the following configuration file. See the Rsyslog documentation for more information about configuring what is forwarded to NXLog.
/etc/rsyslog.d/nxlog.conf# Load omuxsock module $ModLoad omuxsock # Set socket path $OMUxSockSocket /opt/nxlog/var/spool/nxlog/rsyslog_sock # Configure template to preserve PRI part (must be on a single line) $template SyslogWithPRI,"<%PRI%>%timegenerated% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%" # Forward all log messages *.* :omuxsock:;SyslogWithPRI # Only forward log messages of "notice" priority and higher #*.notice :omuxsock:;SyslogWithPRI
-
Restart NXLog and Rsyslog in that order to create and use the socket (NXLog must create the socket before Rsyslog will write to it). Run
systemctl restart nxlog
andsystemctl restart rsyslog
.
With this example configuration, NXLog will create the socket and accept log messages from Rsyslog through the socket. The messages will then be parsed as syslog, converted to JSON format, prefixed with a BSD syslog header, and forwarded to another host via TLS.
<Extension _json>
Module xm_json
</Extension>
<Extension _syslog>
Module xm_syslog
</Extension>
<Input in>
Module im_uds
UDS /opt/nxlog/var/spool/nxlog/rsyslog_sock
Exec parse_syslog();
</Input>
<Output out>
Module om_ssl
Host 192.168.1.1:6514
CAFile %CERTDIR%/ca.pem
CertFile %CERTDIR%/client-cert.pem
CertKeyFile %CERTDIR%/client-key.pem
Exec $Message = to_json(); to_syslog_bsd();
</Output>
Reading Rsyslog log files
NXLog can be configured to read log messages written to file by Rsyslog,
/var/log/messages
for example. This is a slightly more intrusive option than
the steps given in Forwarding messages via socket.
NXLog will not have access to the facility and
severity codes because Rsyslog, by default, follows the BSD
syslog convention of not writing the PRI code to the
/var/log/messages file.
|
By default, NXLog runs as user nxlog
and does not have permission to
read files in /var/log
. The simplest solution for this is to run NXLog
as root by omitting the User
option, but it is more secure to provide the
necessary permissions explicitly.
-
Check the user or group ownership of the files in
/var/log
and configure if necessary. Some distributions use a group for the log files by default. On Debian/Ubuntu, for example, Rsyslog is configured to use theadm
group. Otherwise, modify the Rsyslog configuration to use different ownership for log files as shown below./etc/rsyslog.conf or /etc/rsyslog.d/nxlog.conf$FileOwner root $FileCreateMode 0640 $DirCreateMode 0755 $Umask 0022 # Default on Debian/Ubuntu $FileGroup adm # Or use the "nxlog" group directly #$FileGroup nxlog
-
Run NXLog under a user or group that has permission to read the log files. Either use a user or group directly with the
User
orGroup
option innxlog.conf
, or add thenxlog
user to a group that has permission. For example, on Debian/Ubuntu add thenxlog
user to theadm
group by runningusermod -a -G adm nxlog
. -
If necessary, fix permissions for any files NXLog will be reading from that already exist (use the correct group for your system).
# chgrp adm /var/log/messages # chmod g+r /var/log/messages
-
Configure NXLog to read from the required file(s) (see the example below), then restart NXLog.
-
If the Rsyslog configuration has been modified, restart Rsyslog (
systemctl restart rsyslog
).
With the following configuration, NXLog will read logs from
/var/log/messages
, parse the events as syslog, convert them to JSON, and
forward the plain JSON to another host via TCP.
<Extension _json>
Module xm_json
</Extension>
<Extension _syslog>
Module xm_syslog
</Extension>
<Input in>
Module im_file
File '/var/log/messages'
Exec parse_syslog();
</Input>
<Output out>
Module om_tcp
Host 192.168.1.1:1514
Exec $raw_event = to_json();
</Output>
Reading the systemd journal
NXLog can be configured to read systemd journal messages directly with the im_systemd input module. This module is available with the distribution-specific Linux packages. See the list of installer packages for the supported platforms.
The following configuration uses the im_systemd input module to read systemd journal messages. It uses the xm_json extension to convert log records to JSON format and forwards them to another host via TCP.
<Extension _json>
Module xm_json
</Extension>
<Input systemd>
Module im_systemd
Exec to_json();
</Input>
<Output out>
Module om_tcp
Host 192.168.1.1:1514
</Output>
The user running NXLog must be added to the systemd-journal
group to be able to read systemd journal logs.
|
The following is a systemd journal log record in JSON format after it was processed by NXLog.
{
"Severity": "warning",
"SeverityValue": 4,
"Message": "(==) Max clients allowed: 256, resource mask: 0x1fffff",
"SourceName": "/usr/lib/gdm3/gdm-x-session",
"ProcessID": 2172,
"User": "jdoe",
"Group": "jdoe",
"ProcessName": "Xorg",
"ProcessExecutable": "/usr/lib/xorg/Xorg",
"ProcessCmdLine": "/usr/lib/xorg/Xorg vt2 -displayfd 3 -auth /run/user/1000/gdm/Xauthority -background none -noreset -keeptty -verbose 3",
"Capabilities": "0",
"AuditSession": 2,
"AuditUID": "jdoe",
"SystemdCGroup": "/user.slice/user-1000.slice/session-2.scope",
"SystemdSession": "2",
"SystemdUnit": "session-2.scope",
"SystemdOwnerUID": "jdoe",
"SystemdSlice": "user-1000.slice",
"SelinuxContext": "unconfined\n",
"BootID": "9eeef5226ff441e0a3c8bd990ded3af0",
"MachineID": "7994bd4cc9744ab78846106097c30ff6",
"SysInvID": "d4a984306a31454ba5bbca0f23e9fc06",
"Hostname": "Ubuntu-1",
"Transport": "stdout",
"EventReceivedTime": "2021-05-12T09:27:52.554654+02:00",
"SourceModuleName": "systemd",
"SourceModuleType": "im_systemd"
}
For generic Linux packages that do not include the im_systemd module, a similar solution can be achieved with a script that executes the journalctl tool to query the systemd journal.
This script uses the journalctl tool to query the systemd journal. It specifies the following arguments:
- -o json
-
Sets the output format to standard JSON written one record per line.
- --cursor-file
-
Sets the path to the cursor file. If the file exists, it starts reading from the position stored in the cursor file. When stopped it will write the cursor to the last entry it read in this file.
- --no-pager
-
Pipe output to the standard output instead of the default pager.
#!/bin/bash
CRS=/opt/nxlog/var/spool/nxlog/systemd-tail.pos
SAMPLE_TIME=1
while sleep $SAMPLE_TIME; do
journalctl -o json --cursor-file=${CRS} --no-pager
done
The following configuration uses the im_exec input module to execute the helper script and forwards logs to another host via TCP. The xm_json extension can be used to parse log data into fields for further processing.
<Extension _json>
Module xm_json
</Extension>
<Input systemd>
Module im_exec
Command /usr/local/bin/systemd-tail.sh
<Exec>
parse_json();
# Drop debug messages
if ($PRIORITY == "7")
drop();
</Exec>
</Input>
<Output out>
Module om_tcp
Host 192.168.1.1:1514
</Output>
The user running NXLog must have permission to read and execute the script. |
The following is a systemd jounral log record in JSON format after it was processed by NXLog.
{
"_SYSTEMD_INVOCATION_ID": "22b0df526f7e4630a3d1ce2af8e9e50a",
"_GID": "1000",
"_CAP_EFFECTIVE": "0",
"MESSAGE": "(==) Max clients allowed: 256, resource mask: 0x1fffff",
"_PID": "1315",
"_COMM": "Xorg",
"_HOSTNAME": "Ubuntu-1",
"_EXE": "/usr/lib/xorg/Xorg",
"_MACHINE_ID": "7994bd4cc9744ab78846106097c30ff6",
"_BOOT_ID": "7afc2c1a7e154687af0865c1398ba630",
"_SYSTEMD_SLICE": "user-1000.slice",
"_SYSTEMD_CGROUP": "/user.slice/user-1000.slice/session-2.scope",
"__CURSOR": "s=78878570db18458e882cf0825126cdf2;i=cba63;b=7afc2c1a7e154687af0865c1398ba630;m=1ab0c60;t=5c1cd3c03d8f3;x=aa5241a02d97b76b",
"_SYSTEMD_UNIT": "session-2.scope",
"SYSLOG_IDENTIFIER": "/usr/lib/gdm3/gdm-x-session",
"_SELINUX_CONTEXT": "unconfined\n",
"_UID": "1000",
"_AUDIT_LOGINUID": "1000",
"__MONOTONIC_TIMESTAMP": "27987040",
"_CMDLINE": "/usr/lib/xorg/Xorg vt2 -displayfd 3 -auth /run/user/1000/gdm/Xauthority -background none -noreset -keeptty -verbose 3",
"_AUDIT_SESSION": "2",
"__REALTIME_TIMESTAMP": "1620462102894835",
"_TRANSPORT": "stdout",
"_STREAM_ID": "0554d01d26664c979d5eff46ae4e4f41",
"_SYSTEMD_OWNER_UID": "1000",
"_SYSTEMD_USER_SLICE": "-.slice",
"PRIORITY": "4",
"_SYSTEMD_SESSION": "2"
}