NXLog search
Let's talk Start free
Let's talk Start free
1.03

Convert log formats

SIEMs and log analytics platforms usually require logs in a specific format, such as syslog or JSON. However, your log sources might not support producing logs in the required format. NXLog Agent supports converting logs to several formats. See also the closely related topics of parsing unstructured logs and parsing standard log formats.

Below, we provide examples of converting logs to syslog, JSON, GELF, and the NXLog Binary format.

Generate syslog messages

Syslog is a popular log format, and many SIEMs accept logs in this format. NXLog Agent includes a syslog extension that supports generating logs in the BSD (RFC 3164), newer IETF (RFC 5424), and Snare syslog formats. See our syslog integration guide for more information and examples.

Example 1. Converting unstructured logs to syslog

This configuration reads Apaches HTTP Server logs with the im_file input module. This module populates the $raw_event field. It then uses the to_syslog_ietf() procedure of the xm_syslog module to convert the log record to an IETF syslog message.

nxlog.conf
<Extension syslog>
    Module    xm_syslog
</Extension>

<Input apache>
    Module    im_file
    Path      '/var/log/httpd/*'
    Exec      to_syslog_ietf();
</Input>

The following is an Apache HTTP Server access log sample.

Input sample
192.168.3.20 - - [16/Apr/2024] "GET /cgi-bin/try/ HTTP/1.0" 200 3395

When the NXLog Agent configuration above processes this log event, it transforms it into the following syslog message.

Output sample
<13>1 2024-04-16T16:16:37.647610+02:00 WEB-SRV - - - [NXLOG@14506 EventReceivedTime="2024-04-16 16:16:37" SourceModuleName="apache" SourceModuleType="im_file"] 192.168.3.20 - - [16/Apr/2024] "GET /cgi-bin/try/ HTTP/1.0" 200 3395

Snare syslog is a specialized syslog format used by the Snare Central SIEM, Snare Agent, and other third-party tools that have adopted it. See Generating Snare logs in our Snare integration guide for more detailed information.

Example 2. Converting Windows events to Snare syslog format

This configuration collects Windows events with the im_msvistalog input module. This module automatically parses log records into fields. It then uses the to_syslog_snare() procedure of the xm_syslog module to convert the log record to Snare syslog format.

nxlog.conf
<Extension syslog>
    Module    xm_syslog
</Extension>

<Input windows_event_log>
    Module    im_msvistalog
    Exec      to_syslog_snare();
</Input>

The following is a Windows security event ID 4798 sample, shown in XML format.

Input sample
A user's local group membership was enumerated.

Subject:
  Security ID:       EXAMPLE\admin
  Account Name:      admin
  Account Domain:    EXAMPLE
  Logon ID:          0x9259AD7

User:
  Security ID:       PC1\Administrator
  Account Name:      Administrator
  Account Domain:    PC1

Process Information:
  Process ID:        0x38e8
  Process Name:      C:\Windows\explorer.exe

Log name:    Security
Source:      Microsoft Windows security    Logged:        16/04/2024 18:38:13
Event ID:    4798                          Task Category: User Account Management
Level:       Information                   Keywords:      Audit Success
User:        N/A                           Computer:      PC1.example.com
OpCode:      Info

When the NXLog Agent configuration above processes this event, it transforms it into the following Snare syslog message.

Output sample
<14>Apr 16 18:38:13 PC1.example.com MSWinEventLog	1	Security	45	Tue Apr 16 18:38:13 2024	4798	Microsoft-Windows-Security-Auditing	N/A	N/A	Success Audit	PC1.examle.com	User Account Management		A user's local group membership was enumerated.    Subject:   Security ID:  S-1-5-21-3288775215-2077974584-1458381936-1001   Account Name:  admin   Account Domain:  EXAMPLE   Logon ID:  0x9259ad7    User:   Security ID:  S-1-5-21-1694160624-234216347-2203645164-500   Account Name:  Administrator   Account Domain:  PC1    Process Information:   Process ID:  0x38e8   Process Name:  C:\Windows\explorer.exe	2052854

Generate JSON logs

JSON is among the most popular data interchange formats for web-based services and APIs. NXLog Agent includes a JSON extension that can easily convert log records to JSON.

Example 3. Converting unstructured logs to JSON

This configuration listens for log events with the im_tcp input module, which populates the $raw_event field. It then uses the to_json() procedure of the xm_json module to convert the log record to JSON format.

nxlog.conf
<Extension json>
    Module        xm_json
</Extension>

<Input tcp>
    Module        im_tcp
    ListenAddr    0.0.0.0:1514
    <Exec>
        $Message = $raw_event;  (1)
        to_json();
    </Exec>
</Input>
1 The to_json() procedure does not include the $raw_event field when converting the log record to JSON. Therefore, you must parse $raw_event into fields or, as in this example, create a new field with the value of $raw_event.

The following is a syslog message collected from a Linux host.

Input sample
<38>Apr 16 15:45:43 myhost sshd[8459]: Failed password for invalid user linda from 192.168.1.60 port 38176 ssh2

When the NXLog Agent configuration above processes this log event, it transforms it into the following JSON object comprising the $Message field we added in the configuration, the $MessageSourceAddress added by the im_tcp module, and the NXLog Agent core fields.

Output sample
{
  "MessageSourceAddress": "192.168.1.115",
  "EventReceivedTime": "2024-04-16T15:46:10.166035+02:00",
  "SourceModuleName": "tcp",
  "SourceModuleType": "im_tcp",
  "Hostname": "LOG-SRV",
  "Message": "<38>Apr 16 15:45:43 myhost sshd[8459]: Failed password for invalid user linda from 192.168.1.60 port 38176 ssh2"
}
Example 4. Converting Windows events to JSON

This configuration collects Windows events with the im_msvistalog input module. This module automatically parses log records into fields. It then uses the to_json() procedure of the xm_json module to convert the log record to JSON format.

nxlog.conf
<Extension json>
    Module    xm_json
</Extension>

<Input windows_event_log>
    Module    im_msvistalog
    Exec      to_json();
</Input>

The following is a Windows security event ID 4798 sample.

Input sample
A user's local group membership was enumerated.

Subject:
  Security ID:       EXAMPLE\admin
  Account Name:      admin
  Account Domain:    EXAMPLE
  Logon ID:          0x9259AD7

User:
  Security ID:       PC1\Administrator
  Account Name:      Administrator
  Account Domain:    PC1

Process Information:
  Process ID:        0x38e8
  Process Name:      C:\Windows\explorer.exe

Log name:    Security
Source:      Microsoft Windows security    Logged:        16/04/2024 18:38:13
Event ID:    4798                          Task Category: User Account Management
Level:       Information                   Keywords:      Audit Success
User:        N/A                           Computer:      PC1.example.com
OpCode:      Info

When the NXLog Agent configuration above processes this log event, it transforms it into the following JSON object.

Output sample
{
  "EventTime": "2024-04-16T18:38:13.5142877+02:00",
  "Hostname": "PC1.example.com",
  "Keywords": "0x8020000000000000",
  "LevelValue": 0,
  "EventType": "AUDIT_SUCCESS",
  "SeverityValue": 2,
  "Severity": "INFO",
  "EventID": 4798,
  "SourceName": "Microsoft-Windows-Security-Auditing",
  "ProviderGuid": "{54849625-5478-4994-A5BA-3E3B0328C30D}",
  "Version": 0,
  "TaskValue": 13824,
  "OpcodeValue": 0,
  "RecordNumber": 2055435,
  "ActivityID": "{FE8FAE58-8CEC-0002-E6AE-8FFEEC8CDA01}",
  "ExecutionProcessID": 1048,
  "ExecutionThreadID": 2468,
  "Channel": "Security",
  "Message": "A user's local group membership was enumerated.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-5-21-3288775215-2077974584-1458381936-1001\r\n\tAccount Name:\t\tadmin\r\n\tAccount Domain:\t\tEXAMPLE\r\n\tLogon ID:\t\t0x9259ad7\r\n\r\nUser:\r\n\tSecurity ID:\t\tS-1-5-21-1694160624-234216347-2203645164-500\r\n\tAccount Name:\t\tAdministrator\r\n\tAccount Domain:\t\tPC1\r\n\r\nProcess Information:\r\n\tProcess ID:\t\t0x38e8\r\n\tProcess Name:\t\tC:\\Windows\\explorer.exe",
  "Category": "User Account Management",
  "Opcode": "Info",
  "Level": "Information",
  "TargetUserName": "Administrator",
  "TargetDomainName": "PC1",
  "TargetSid": "S-1-5-21-1694160624-234216347-2203645164-500",
  "SubjectUserSid": "S-1-5-21-3288775215-2077974584-1458381936-1001",
  "SubjectUserName": "admin",
  "SubjectDomainName": "EXAMPLE",
  "SubjectLogonId": "0x9259ad7",
  "CallerProcessId": "0x38e8",
  "CallerProcessName": "C:\\Windows\\explorer.exe",
  "EventReceivedTime": "2024-04-16T18:38:13.388661+02:00",
  "SourceModuleName": "windows_event_log",
  "SourceModuleType": "im_msvistalog"
}

Generate GELF logs

Graylog Extended Log Format (GELF) is a JSON-based, structured log format created by Graylog. You can use NXLog Agent’s xm_gelf extension to generate GELF output and send it over UDP or TCP. See our Graylog integration guide for more information on sending logs to Graylog.

Example 5. Converting unstructured logs to GELF

This configuration collects syslog messages with the im_file input module. This module populates the $raw_event field. It then forwards log records to a remote server in GELF over UDP.

nxlog.conf
<Extension gelf>
    Module        xm_gelf
</Extension>

<Input system_messages>
    Module        im_file
    Path          '/var/log/syslog'
</Input>

<Output udp>
    Module        om_udp
    Host          192.168.1.123:12201
    OutputType    GELF_UDP  (1)
</Output>
1 Sets the OutputType directive to GELF_UDP, an output writer function provided by xm_gelf.

The following is a syslog message collected from a Linux host.

Input sample
<38>Apr 16 15:45:43 myhost sshd[8459]: Failed password for invalid user linda from 192.168.1.60 port 38176 ssh2

NXLog Agent forwards this log event via UDP with the following JSON payload.

Output sample
{
  "version": "1.1",
  "EventReceivedTime": "2024-04-16 15:45:56",
  "SourceModuleName": "system_messages",
  "SourceModuleType": "im_file",
  "Hostname": "LOG-SRV",
  "ShortMessage": "<38>Apr 16 15:45:43 myhost sshd[8459]: Failed password for inval",
  "FullMessage": "<38>Apr 16 15:45:43 myhost sshd[8459]: Failed password for invalid user linda from 192.168.1.60 port 38176 ssh2",
  "EventTime": "2024-04-16T15:45:56.556581+02:00",
  "SeverityValue": 6,
  "MessageSourceAddress": "192.168.1.115"
}

NXLog Binary format

The Binary format is a proprietary format that preserves all event fields when sending logs to another NXLog Agent instance or NXLog Platform. The NXLog Transport input and output modules use the Binary format by default. You can also use this format with other stream-oriented modules by specifying the InputType and OutputType directives.

Example 6. Sending logs to a relay agent in the NXLog Binary format

This configuration collects syslog messages with the im_file input module and parses records with the xm_syslog module. It then forwards log records to another NXLog Agent instance configured to receive logs with im_batchcompress.

nxlog.conf
<Extension syslog>
    Module    xm_syslog
</Extension>

<Input system_messages>
    Module    im_file
    File      '/var/log/syslog'
    Exec      parse_syslog();  (1)
</Input>

<Output relay_agent>
    Module    om_batchcompress
    Host      192.168.1.101:2514
</Output>
1 Calls the parse_syslog() procedure to parse the record into structured data.

The following is a syslog message collected from a Linux host.

Input sample
<38>Apr 16 15:45:43 myhost sshd[8459]: Failed password for invalid user linda from 192.168.1.60 port 38176 ssh2

When NXLog Agent forwards this message, it will send the following fields to the relay agent. Note that although the timestamp of the syslog message above does not contain the year, the procedure adds it to the $EventTime field to make it a valid timestamp.

Field Value

$EventReceivedTime

2024-04-16 15:45:56

$EventTime

2024-04-16 15:45:43

$Hostname

myhost

$Message

Failed password for invalid user linda from 192.168.1.60 port 38176 ssh2

$ProcessID

8459

$raw_event

<38>Apr 16 15:45:43 myhost sshd[8459]: Failed password for invalid user linda from 192.168.1.60 port 38176 ssh2

$Severity

INFO

$SeverityValue

2

$SourceModuleName

system_messages

$SourceModuleType

im_file

$SourceName

sshd

$SyslogFacility

AUTH

$SyslogFacilityValue

4

$SyslogSeverity

INFO

$SyslogSeverityValue

6

© Copyright 2024 NXLog Ltd.

PRIVACY POLICY GENERAL TERMS OF BUSINESS