Collecting, parsing, and forwarding syslog messages
Syslog is a standard that network devices, operating systems, and applications use to log system events and messages. You can send syslog-formatted logs to a central log server for analysis, troubleshooting, auditing, or storage. Syslog messages typically include information about the message source, the event severity level, and a timestamp.
NXLog Agent can collect, generate, and forward syslog messages in various formats. Its Syslog extension parses syslog messages into log records and generates syslog messages from them. You can use NXLog Agent to consolidate syslog messages from Windows and Linux sources into a single collection point. It can also function as a syslog server, receiving and storing syslog messages sent by other network devices and applications.
This guide describes the different syslog protocols and discusses how to use them with NXLog Agent.
BSD syslog (RFC 3164)
The original syslog daemon was written for the Sendmail email server in the 1980s and became the standard logging system for Unix-style systems. Because the BSD syslog protocol was never formally standardized, implementations diverged, and in 2001 RFC 3164 eventually documented their "observed behavior". This explains why many variations of BSD syslog exist today.
BSD syslog defines both the log message format and the transport.
BSD syslog format
The legacy format is relatively simple and has a loosely defined structure, with the following components. While this is the common and recommended structure, there are no strict requirements. A device may send messages containing only free-form text, without PRI or HEADER parts.
| Component | Description |
|---|---|
|
The priority value, which encodes both the facility and severity level.
Calculate it by multiplying the facility code by eight and adding the severity code.
RFC 3164 defines the facility and severity codes.
For example, Many syslog daemons do not write the priority value to file. In that case, each log message begins with the timestamp instead. |
|
The local date and time when the message was generated, in the format |
|
The name or IP address of the machine that generated the message.
For example, |
|
Typically contains the name of the program or process generating the message, and is often followed by the process ID in brackets and a colon.
For example, |
|
Free-form text containing the log message, spanning until the EOL.
It can contain only ASCII printable characters (32-126).
However, the payload itself can be JSON or another structured data format.
For example, |
BSD syslog transport
According to RFC 3164, the BSD syslog protocol uses UDP as its transport layer. Each UDP packet carries a single log message. BSD syslog implementations often also support TCP, with or without TLS encryption, though RFC 3164 does not cover either.
Disadvantages of BSD syslog
The BSD syslog protocol has several disadvantages.
-
The transport defined by RFC 3164 uses UDP and provides no mechanism to ensure reliable log delivery, integrity, or confidentiality of log messages.
-
Many undocumented variations exist among implementations.
-
The timestamp indicates neither the year nor the timezone and does not provide precision greater than the second.
-
Many syslog daemons drop the priority field when writing to log files, so the facility and severity codes are lost.
-
BSD syslog limits the log message to 1024 bytes.
-
BSD syslog allows only ASCII characters 32-126, with no Unicode or line breaks.
IETF syslog (RFC 5424-5426)
In 2009, the IETF released RFC 5424, 5425, and 5426 as "Proposed Standards" intended to replace the "legacy" BSD syslog:
-
RFC 5424 defines the base message format, including a timestamp with year, timezone, and fractional seconds, a structured data field for key-value pairs, and UTF-8 encoding.
-
RFC 5425 defines the use of TLS and supports multiline log messages. RFC 9662 updated the recommended cipher suites in 2024.
-
RFC 5426 describes the use of UDP transport.
IETF syslog format
The IETF format is more structured than the legacy BSD format, with the following components.
| Component | Description |
|---|---|
|
The priority value, which encodes both the facility and severity level.
Calculate it by multiplying the facility code by eight and adding the severity code.
RFC 5424 defines the facility and severity codes.
For example, |
|
The syslog protocol version number.
It is always |
|
The timestamp when the message was generated, including the year, time, milliseconds, and timezone.
It follows a format derived from RFC 3339.
For example, |
|
The Fully Qualified Domain Name (FQDN), IP address, hostname of the machine, or |
|
The name of the device or application that generated the message, similar to the tag in the BSD format. |
|
The ID of the process that generated the message, or |
|
The ID of the message type, or |
|
Optional key-value pairs enclosed in square brackets and prefixed with an SD-ID, or |
|
Optional, free-form, single-line text containing the log message. A UTF-8-encoded message must start with a Unicode byte order mark (BOM). |
IETF syslog transport
IETF syslog can use UDP or TCP with or without TLS. UDP transport is described by RFC 5426, and TLS encryption is described by RFC 5425.
RFC 5425 also documents an octet framing method, in which each message is prefixed with its length. Octet framing is required whenever a message may contain embedded newline characters, since a plain newline record separator could otherwise be mistaken for the end of the message. Octet framing can be used with TCP with or without TLS, as shown in the following example of the raw data sent over the wire.
101 <13>1 2026-10-09T14:23:10.873750+00:00 myhost - - - [NXLOG@14506 TestField="test value"] test message
When messages are guaranteed to be single-line, octet framing isn’t necessary.
IETF syslog is then commonly transferred over TCP, with or without TLS, using the newline (\n) character as the record separator instead, similar to BSD syslog.
Collecting and parsing syslog
You can configure NXLog Agent to collect syslog messages by:
-
Reading syslog files written by local syslog daemons.
-
Collecting syslog via the local
/dev/logUnix domain socket. -
Receiving syslog over the network using UDP or TCP with optional TLS.
Reading syslog files
Configuring NXLog Agent to read syslog from a file allows other local syslog daemons to continue logging as before. In this scenario, it’s likely that NXLog Agent cannot access the facility and severity codes, because many syslog daemons do not write the PRI field to log files by default.
Ensure NXLog Agent has permission to read log files in /var/log.
See Reading logs written to file by rsyslog for more information.
This configuration reads log messages from files and parses them using the parse_syslog() procedure.
<Extension syslog>
Module xm_syslog
</Extension>
<Input system_logs>
Module im_file
File '/var/log/syslog'
Exec parse_syslog(); (1)
</Input>
| 1 | Parses the log message as either BSD or IETF format using the parse_syslog() procedure. To parse a specific format, use the parse_syslog_bsd() or parse_syslog_ietf() procedure instead. |
Collecting syslog via /dev/log
Many applications support logging by sending messages to the /dev/log Unix domain socket.
The system logger, such as rsyslog or systemd-journald, accepts these messages and stores them as configured.
NXLog Agent can read from this socket in place of the system logger, so applications keep logging as before.
-
Configure NXLog Agent (see the example below).
-
Free
/dev/log, because only one program can listen on the socket. For example, runsystemctl disable --now rsyslog, or removemodule(load="imuxsock")from/etc/rsyslog.confand restart rsyslog. See also Replacing rsyslog with NXLog Agent. -
Restart NXLog Agent.
On systemd-based distributions, such as Ubuntu 24.04, /dev/log is a symbolic link to the systemd-journald socket, so stopping rsyslog is not enough to free it.
To collect these logs, read the systemd journal with the Systemd input module instead.
See Reading systemd journal logs.
|
This configuration uses the Unix Domain Sockets input module to read messages from /dev/log and the parse_syslog() procedure to parse them.
<Extension syslog>
Module xm_syslog
</Extension>
<Input system_logs>
Module im_uds
UDS /dev/log
FlowControl FALSE (1)
Exec parse_syslog();
</Input>
| 1 | We recommend deactivating FlowControl when collecting from /dev/log.
Otherwise, the syslog() system call blocks when the output queue becomes full, and programs that write to the system log may become unresponsive. |
Collecting syslog over UDP or TCP
NXLog Agent can listen on a network port to receive syslog over UDP or TCP.
-
Configure NXLog Agent with the UDP, TCP, or TLS/SSL input module (see the examples below).
-
If a syslog daemon is already listening on the same port, free the port, because only one program can listen on each port. For example, run
systemctl disable --now rsyslog, or removeinput(type="imudp" port="514")from/etc/rsyslog.confand restart rsyslog. -
Restart NXLog Agent.
This configuration accepts either BSD or IETF syslog over UDP from the local system only.
<Extension syslog>
Module xm_syslog
</Extension>
<Input udp_listen>
Module im_udp
ListenAddr localhost:514 (1)
Exec parse_syslog();
</Input>
| 1 | Listens on localhost only.
Because UDP can drop messages, we do not recommend it for receiving logs over the network. |
This configuration accepts either BSD or IETF syslog over TCP without octet framing on all available network interfaces.
<Extension syslog>
Module xm_syslog
</Extension>
<Input tcp_listen>
Module im_tcp
ListenAddr 0.0.0.0:1514
Exec parse_syslog();
</Input>
This configuration accepts IETF syslog over TCP, with support for octet framing.
<Extension syslog>
Module xm_syslog
</Extension>
<Input tcp_listen>
Module im_tcp
ListenAddr 0.0.0.0:1514
InputType Syslog_TLS (1)
Exec parse_syslog_ietf();
</Input>
| 1 | The Syslog_TLS input reader parses octet-framed messages, as described by RFC 5425. Despite its name, it works with or without TLS. |
This configuration accepts IETF syslog over TCP with TLS encryption and supports octet framing. For more information about how NXLog Agent secures data transfer with certificates, see TLS/SSL data transfer.
<Extension syslog>
Module xm_syslog
</Extension>
<Input tcp_listen>
Module im_ssl
ListenAddr 0.0.0.0:6514
CAFile %CERTDIR%/ca.pem (1)
CertFile %CERTDIR%/agent-cert.pem (2)
CertKeyFile %CERTDIR%/agent-key.pem (3)
InputType Syslog_TLS (4)
Exec parse_syslog_ietf();
</Input>
| 1 | The CAFile directive specifies the path of the Certificate Authority (CA) certificate used to verify the certificate presented by the remote host. |
| 2 | The CertFile directive specifies the path of the local NXLog Agent server’s certificate. |
| 3 | The CertKeyFile directive specifies the path of the NXLog Agent server’s certificate private key. |
| 4 | The Syslog_TLS input reader parses octet-framed messages, as described by RFC 5425. |
Filtering syslog
Filtering keeps or discards syslog messages based on their contents. Use the Syslog extension to parse syslog messages into fields. You can then use conditional statements to test the values of those fields and the drop() procedure to discard unwanted messages. For more information about fields, see NXLog Agent records and fields.
This configuration converts syslog messages with higher severity to JSON and drops the rest.
<Extension syslog>
Module xm_syslog
</Extension>
<Extension json>
Module xm_json
</Extension>
<Input system_logs>
Module im_uds
UDS /dev/log
<Exec>
parse_syslog_bsd(); (1)
if ($SyslogSeverityValue < 6) { (2)
to_json(); (3)
}
else {
drop();
}
</Exec>
</Input>
| 1 | Parses the message into fields using the parse_syslog_bsd() procedure of the Syslog extension. |
| 2 | Keeps only messages with $SyslogSeverityValue below 6.
Lower values are more severe, from 0 (emergency) to 7 (debug). |
| 3 | Converts the kept messages to JSON using the to_json() procedure of the JSON extension. |
You can also filter messages using the values of a group of fields.
This configuration converts syslog messages that match a severity, facility, or source name to JSON and drops the rest.
<Extension syslog>
Module xm_syslog
</Extension>
<Extension json>
Module xm_json
</Extension>
<Input system_logs>
Module im_uds
UDS /dev/log
<Exec>
parse_syslog_bsd(); (1)
if (($SyslogSeverityValue < 6) OR (2)
($SyslogFacility IN ('AUTHPRIV', 'AUTH', 'MAIL', 'CRON')) OR
($SourceName IN ('apt', 'nxlog', 'osqueryd')))
{
to_json(); (3)
}
else {
drop();
}
</Exec>
</Input>
| 1 | Parses the message into fields using the parse_syslog_bsd() procedure of the Syslog extension. |
| 2 | Keeps messages that meet any of the conditions on $SyslogSeverityValue, $SyslogFacility, or $SourceName. |
| 3 | Converts the kept messages to JSON using the to_json() procedure of the JSON extension. |
You can also filter syslog messages by the $Message field using regular expressions.
This configuration converts kernel messages that contain a specific string to JSON and drops the rest.
<Extension syslog>
Module xm_syslog
</Extension>
<Extension json>
Module xm_json
</Extension>
<Input kernel_logs>
Module im_kernel (1)
<Exec>
parse_syslog_bsd(); (2)
if $Message =~ /mount options/ { (3)
to_json(); (4)
}
else {
drop();
}
</Exec>
</Input>
| 1 | Reads messages from the Linux kernel using the Kernel input module. |
| 2 | Parses the message into fields using the parse_syslog_bsd() procedure of the Syslog extension. |
| 3 | Keeps only messages that include the string mount options in the $Message field, matched using a regular expression. |
| 4 | Converts the kept messages to JSON using the to_json() procedure of the JSON extension. |
Generating syslog messages
You can configure NXLog Agent to generate BSD or IETF syslog and:
-
Write messages to file.
-
Send messages to the system logger via the
/dev/logUnix domain socket. -
Forward messages to another destination over the network using UDP or TCP with optional TLS.
In each case, use the to_syslog_bsd() or to_syslog_ietf() procedure to generate the $raw_event field from the corresponding syslog fields in the event record.
Writing syslog messages to file
Use the File output module to write logs to a file.
This configuration writes BSD syslog messages to a file.
<Extension syslog>
Module xm_syslog
</Extension>
<Output syslog_file>
Module om_file
File '/tmp/syslog-bsd.log' (1)
Exec to_syslog_bsd(); (2)
</Output>
| 1 | The File directive specifies the path of the output file. NXLog Agent must have permission to write to this location. |
| 2 | Formats the message as BSD syslog using the to_syslog_bsd() procedure of the Syslog extension. |
You can write BSD syslog to a file without the PRI part, emulating traditional syslog implementations.
This configuration writes BSD syslog messages to a file after removing the PRI part.
<Extension syslog>
Module xm_syslog
</Extension>
<Output syslog_file>
Module om_file
File '/tmp/syslog-bsd-no-pri.log'
<Exec>
to_syslog_bsd();
$raw_event =~ s/^\<\d+\>//; (1)
</Exec>
</Output>
| 1 | Uses a regular expression to remove the PRI part from the $raw_event field generated by the to_syslog_bsd() procedure. |
Sending syslog messages to the system logger via /dev/log
You can use the Unix Domain Sockets output module to send logs to the system logger, such as rsyslog or systemd-journald.
This configuration sends BSD syslog messages to the system logger via /dev/log.
<Extension syslog>
Module xm_syslog
</Extension>
<Output uds>
Module om_uds
UDS /dev/log (1)
Exec to_syslog_bsd(); (2)
</Output>
| 1 | The UDS directive specifies the socket to write to. |
| 2 | Formats the message as BSD syslog using the to_syslog_bsd() procedure of the Syslog extension. |
Forwarding syslog messages to a remote logger over UDP or TCP
This configuration sends BSD syslog over TCP.
<Extension syslog>
Module xm_syslog
</Extension>
<Output tcp>
Module om_tcp
Host 192.168.1.1:1514 (1)
Exec to_syslog_bsd(); (2)
</Output>
| 1 | The Host directive specifies the IP address and port of the remote host. |
| 2 | Formats the message as BSD syslog using the to_syslog_bsd() procedure of the Syslog extension. |
This configuration sends IETF syslog messages over TCP with TLS and uses octet framing.
<Extension syslog>
Module xm_syslog
</Extension>
<Output tcp_tls>
Module om_ssl
Host 192.168.1.1:6514 (1)
CAFile %CERTDIR%/ca.pem (2)
CertFile %CERTDIR%/agent-cert.pem (3)
CertKeyFile %CERTDIR%/agent-key.pem (4)
OutputType Syslog_TLS (5)
Exec to_syslog_ietf(); (6)
</Output>
| 1 | The Host directive specifies the IP address and port of the remote host. |
| 2 | The CAFile directive specifies the path of the Certificate Authority (CA) certificate used to verify the certificate presented by the remote host. |
| 3 | The CertFile directive specifies the path of the NXLog Agent client certificate. |
| 4 | The CertKeyFile directive specifies the path of the NXLog Agent client certificate private key. |
| 5 | The Syslog_TLS output writer writes octet-framed messages, as described by RFC 5425. |
| 6 | Formats the message as IETF syslog using the to_syslog_ietf() procedure of the Syslog extension. |
Extending syslog
IETF syslog has a dedicated structured data field for key-value pairs. BSD syslog has only a free-form message field, so you must encode key-value pairs in the message itself. This section describes how to implement structured data with NXLog Agent in either syslog format.
IETF syslog structured data
This section documents how NXLog Agent parses and generates the structured data part of the IETF syslog format. NXLog Agent can parse IETF syslog with the parse_syslog() procedure provided by the Syslog extension.
This configuration collects IETF syslog messages from a file, parses and converts them to JSON, and outputs the result to another file.
Structured data parameters become fields prefixed with their SD-ID (for example, MYAPP.Source), unless the SD-ID is NXLOG, as in this sample.
<Extension syslog>
Module xm_syslog
</Extension>
<Extension json>
Module xm_json
</Extension>
<Input ietf_syslog>
Module im_file
File '/var/log/ietf-syslog'
Exec parse_syslog();
</Input>
<Output json_file>
Module om_file
File '/tmp/syslog-ietf.json'
Exec to_json();
</Output>
<Route r>
Path ietf_syslog => json_file
</Route>
<13>1 2026-10-09T14:23:10.000000+00:00 myserver - - - [NXLOG@14506 Purpose="test"] This is a test message.
{
"EventReceivedTime": "2026-10-09T14:23:11.071221+00:00",
"SourceModuleName": "ietf_syslog",
"SourceModuleType": "im_file",
"Hostname": "myserver",
"SyslogFacilityValue": 1,
"SyslogFacility": "USER",
"SyslogSeverityValue": 5,
"SyslogSeverity": "NOTICE",
"SeverityValue": 2,
"Severity": "INFO",
"EventTime": "2026-10-09T14:23:10.000000+00:00",
"Purpose": "test",
"Message": "This is a test message."
}
NXLog Agent can also generate IETF syslog with a structured data part using the to_syslog_ietf() procedure provided by the Syslog extension.
This configuration parses the input JSON, converts it to IETF syslog format, and outputs the result to a file. NXLog Agent writes fields that are not part of the message header as IETF syslog structured data.
<Extension syslog>
Module xm_syslog
</Extension>
<Extension json>
Module xm_json
</Extension>
<Input json_logs>
Module im_file
File '/var/log/json'
Exec parse_json();
</Input>
<Output syslog_file>
Module om_file
File '/tmp/syslog-ietf.log'
Exec to_syslog_ietf();
</Output>
<Route r>
Path json_logs => syslog_file
</Route>
{
"EventTime": "2026-10-09 14:23:10",
"Hostname": "myserver",
"Purpose": "test",
"Message": "This is a test message."
}
<13>1 2026-10-09T14:23:10.000000+00:00 myserver - - - [NXLOG@14506 EventReceivedTime="2026-10-09 14:23:11" SourceModuleName="json_logs" SourceModuleType="im_file" Purpose="test"] This is a test message.
JSON over syslog
A common way to transfer structured data with syslog is to encode it as JSON in the free-form message field. The syslog header stays unchanged, so the messages remain compatible with syslog devices and servers.
This configuration sends Windows events as JSON in syslog messages over UDP.
<Extension syslog>
Module xm_syslog
</Extension>
<Extension json>
Module xm_json
</Extension>
<Input windows_events>
Module im_msvistalog (1)
</Input>
<Output udp>
Module om_udp
Host 192.168.1.1:514
<Exec>
$Message = to_json(); (2)
to_syslog_bsd(); (3)
</Exec>
</Output>
<Route r>
Path windows_events => udp
</Route>
| 1 | Reads events from Windows Event Log using the Event Log for Windows input module. |
| 2 | Converts the event fields to JSON using the to_json() function of the JSON extension, and stores the result in the $Message field. |
| 3 | Adds a BSD syslog header to the message using the to_syslog_bsd() procedure of the Syslog extension. |
<11>Oct 9 14:23:10 WIN-SERVER Microsoft-Windows-Security-Auditing[0x3990]: {"EventTime":"2026-10-09T14:23:10.000000+00:00","Hostname":"WIN-SERVER","Keywords":"9227875636482146304","EventType":"AUDIT_FAILURE","SeverityValue":4,"Severity":"ERROR","EventID":4625,"SourceName":"Microsoft-Windows-Security-Auditing","ProviderGuid":"{54849625-5478-4994-A5BA-3E3B0328C30D}","Version":0,"TaskValue":12544,"OpcodeValue":0,"RecordNumber":456703,"ActivityID":"{A9B5A229-9B0A-0008-7AA2-B5A90A9BDA01}","ExecutionProcessID":1124,"ExecutionThreadID":7052,"Channel":"Security","Message":"An account failed to log on.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-5-21-3603782361-128455402-3223720292-1002\r\n\tAccount Name:\t\tjdoe\r\n\tAccount Domain:\t\tNXLOG\r\n\tLogon ID:\t\t0x9d84141\r\n\r\nLogon Type:\t\t\t2\r\n\r\nAccount For Which Logon Failed:\r\n\tSecurity ID:\t\tS-1-0-0\r\n\tAccount Name:\t\tjdoe\r\n\tAccount Domain:\t\tNXLOG\r\n\r\nFailure Information:\r\n\tFailure Reason:\t\tUnknown user name or bad password.\r\n\tStatus:\t\t\t0xc000006e\r\n\tSub Status:\t\t0xc000006e\r\n\r\nProcess Information:\r\n\tCaller Process ID:\t0x3990\r\n\tCaller Process Name:\tC:\\Program Files\\Google\\Chrome\\Application\\chrome.exe\r\n\r\nNetwork Information:\r\n\tWorkstation Name:\tWIN-SERVER\r\n\tSource Network Address:\t-\r\n\tSource Port:\t\t-\r\n\r\nDetailed Authentication Information:\r\n\tLogon Process:\t\tAdvapi\r\n\r\n\tAuthentication Package:\tNegotiate\r\n\tTransited Services:\t-\r\n\tPackage Name (NTLM only):\t-\r\n\tKey Length:\t\t0\r\n\r\nThis event is generated when a logon request fails. It is generated on the computer where access was attempted.\r\n\r\nThe Subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.\r\n\r\nThe Logon Type field indicates the kind of logon that was requested. The most common types are 2 (interactive) and 3 (network).\r\n\r\nThe Process Information fields indicate which account and process on the system requested the logon.\r\n\r\nThe Network Information fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.\r\n\r\nThe authentication information fields provide detailed information about this specific logon request.\r\n\t- Transited services indicate which intermediate services have participated in this logon request.\r\n\t- Package name indicates which sub-protocol was used among the NTLM protocols.\r\n\t- Key length indicates the length of the generated session key. This will be 0 if no session key was requested.","Category":"Logon","Opcode":"Info","SubjectUserSid":"S-1-5-21-3603782361-128455402-3223720292-1002","SubjectUserName":"jdoe","SubjectDomainName":"NXLOG","SubjectLogonId":"0x9d84141","TargetUserSid":"S-1-0-0","TargetUserName":"jdoe","TargetDomainName":"NXLOG","Status":"0xc000006e","FailureReason":"%%2313","SubStatus":"0xc000006e","LogonType":"2","LogonProcessName":"Advapi","AuthenticationPackageName":"Negotiate","WorkstationName":"WIN-SERVER","TransmittedServices":"-","LmPackageName":"-","KeyLength":"0","ProcessId":"0x3990","ProcessName":"C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe","IpAddress":"-","IpPort":"-","EventReceivedTime":"2026-10-09T14:23:11.427302+00:00","SourceModuleName":"windows_events","SourceModuleType":"im_msvistalog"}