Convert data formats

SIEMs and log analytics platforms usually require data in a specific format, such as syslog or JSON. However, your telemetry data sources might not support producing events or metrics in the required format. NXLog Agent supports converting data to several formats. See also the closely related topics of parsing unstructured events and parsing common event formats.

Below, we provide examples of converting logs to syslog, JSON, GELF, and the NXLog Binary format.

Generate syslog messages

Syslog is a popular event format, and many SIEMs accept data in this format. NXLog Agent includes a syslog extension that supports generating events in the BSD (RFC 3164), newer IETF (RFC 5424), and Snare syslog formats. See our syslog integration guide for more information and examples.

Example 1. Converting unstructured events to syslog

This configuration reads Apaches HTTP Server logs with the im_file input module. This module populates the $raw_event field. It then uses the to_syslog_ietf() procedure of the xm_syslog module to convert the event to an IETF syslog message.

nxlog.conf
<Extension syslog>
    Module    xm_syslog
</Extension>

<Input apache>
    Module    im_file
    File      '/var/log/httpd/*'
    Exec      to_syslog_ietf();
</Input>

The following is an Apache HTTP Server access event sample.

Input sample
192.168.3.20 - - [16/Apr/2024] "GET /cgi-bin/try/ HTTP/1.0" 200 3395

When the NXLog Agent configuration above processes this event, it transforms it into the following syslog message.

Output sample
<13>1 2024-04-16T16:16:37.647610+02:00 WEB-SRV - - - [NXLOG@14506 EventReceivedTime="2024-04-16 16:16:37" SourceModuleName="apache" SourceModuleType="im_file"] 192.168.3.20 - - [16/Apr/2024] "GET /cgi-bin/try/ HTTP/1.0" 200 3395

Snare syslog is a specialized syslog format used by the Snare Central SIEM, Snare Agent, and other third-party tools that have adopted it. See Generating Snare logs in our Snare integration guide for more detailed information.

Example 2. Converting Windows events to Snare syslog format

This configuration collects Windows events with the im_msvistalog input module. This module automatically parses events into fields. It then uses the to_syslog_snare() procedure of the xm_syslog module to convert the event to Snare syslog format.

nxlog.conf
<Extension syslog>
    Module    xm_syslog
</Extension>

<Input windows_event_log>
    Module    im_msvistalog
    Exec      to_syslog_snare();
</Input>

The following is a Windows security event ID 4798 sample, shown in XML format.

Input sample
A user's local group membership was enumerated.

Subject:
  Security ID:       EXAMPLE\admin
  Account Name:      admin
  Account Domain:    EXAMPLE
  Logon ID:          0x9259AD7

User:
  Security ID:       PC1\Administrator
  Account Name:      Administrator
  Account Domain:    PC1

Process Information:
  Process ID:        0x38e8
  Process Name:      C:\Windows\explorer.exe

Log name:    Security
Source:      Microsoft Windows security    Logged:        16/04/2024 18:38:13
Event ID:    4798                          Task Category: User Account Management
Level:       Information                   Keywords:      Audit Success
User:        N/A                           Computer:      PC1.example.com
OpCode:      Info

When the NXLog Agent configuration above processes this event, it transforms it into the following Snare syslog message.

Output sample
<14>Apr 16 18:38:13 PC1.example.com MSWinEventLog	1	Security	45	Tue Apr 16 18:38:13 2024	4798	Microsoft-Windows-Security-Auditing	N/A	N/A	Success Audit	PC1.examle.com	User Account Management		A user's local group membership was enumerated.    Subject:   Security ID:  S-1-5-21-3288775215-2077974584-1458381936-1001   Account Name:  admin   Account Domain:  EXAMPLE   Logon ID:  0x9259ad7    User:   Security ID:  S-1-5-21-1694160624-234216347-2203645164-500   Account Name:  Administrator   Account Domain:  PC1    Process Information:   Process ID:  0x38e8   Process Name:  C:\Windows\explorer.exe	2052854

Generate JSON logs

JSON is among the most popular data interchange formats for web-based services and APIs. NXLog Agent includes a JSON extension that can easily convert telemetry data to JSON.

Example 3. Converting unstructured events to JSON

This configuration listens for events with the im_tcp input module, which populates the $raw_event field. It then uses the to_json() procedure of the xm_json module to convert the event to JSON format.

nxlog.conf
<Extension json>
    Module        xm_json
</Extension>

<Input tcp>
    Module        im_tcp
    ListenAddr    0.0.0.0:1514
    <Exec>
        $Message = $raw_event;  (1)
        to_json();
    </Exec>
</Input>
1 The to_json() procedure does not include the $raw_event field when converting the event to JSON. Therefore, you must parse $raw_event into fields or, as in this example, create a new field with the value of $raw_event.

The following is a syslog message collected from a Linux host.

Input sample
<38>Apr 16 15:45:43 SERVER1 sshd[8459]: Failed password for invalid user jdoe from 192.168.1.60 port 38176 ssh2

When the NXLog Agent configuration above processes this event, it transforms it into the following JSON object comprising the $Message field we added in the configuration, the $MessageSourceAddress added by the im_tcp module, and the NXLog Agent core fields.

Output sample
{
  "MessageSourceAddress": "192.168.1.115",
  "EventReceivedTime": "2024-04-16T15:46:10.166035+02:00",
  "SourceModuleName": "tcp",
  "SourceModuleType": "im_tcp",
  "Hostname": "LOG-SRV",
  "Message": "<38>Apr 16 15:45:43 myhost sshd[8459]: Failed password for invalid user linda from 192.168.1.60 port 38176 ssh2"
}
Example 4. Converting Windows events to JSON

This configuration collects Windows events with the im_msvistalog input module. This module automatically parses incoming events into fields. It then uses the to_json() procedure of the xm_json module to convert the event to JSON format.

nxlog.conf
<Extension json>
    Module    xm_json
</Extension>

<Input windows_event_log>
    Module    im_msvistalog
    Exec      to_json();
</Input>

The following is a Windows security event ID 4798 sample.

Input sample
A user's local group membership was enumerated.

Subject:
  Security ID:       EXAMPLE\admin
  Account Name:      admin
  Account Domain:    EXAMPLE
  Logon ID:          0x9259AD7

User:
  Security ID:       PC1\Administrator
  Account Name:      Administrator
  Account Domain:    PC1

Process Information:
  Process ID:        0x38e8
  Process Name:      C:\Windows\explorer.exe

Log name:    Security
Source:      Microsoft Windows security    Logged:        16/04/2024 18:38:13
Event ID:    4798                          Task Category: User Account Management
Level:       Information                   Keywords:      Audit Success
User:        N/A                           Computer:      PC1.example.com
OpCode:      Info

When the NXLog Agent configuration above processes this event, it transforms it into the following JSON object.

Output sample
{
  "EventTime": "2024-04-16T18:38:13.5142877+02:00",
  "Hostname": "PC1.example.com",
  "Keywords": "0x8020000000000000",
  "LevelValue": 0,
  "EventType": "AUDIT_SUCCESS",
  "SeverityValue": 2,
  "Severity": "INFO",
  "EventID": 4798,
  "SourceName": "Microsoft-Windows-Security-Auditing",
  "ProviderGuid": "{54849625-5478-4994-A5BA-3E3B0328C30D}",
  "Version": 0,
  "TaskValue": 13824,
  "OpcodeValue": 0,
  "RecordNumber": 2055435,
  "ActivityID": "{FE8FAE58-8CEC-0002-E6AE-8FFEEC8CDA01}",
  "ExecutionProcessID": 1048,
  "ExecutionThreadID": 2468,
  "Channel": "Security",
  "Message": "A user's local group membership was enumerated.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-5-21-3288775215-2077974584-1458381936-1001\r\n\tAccount Name:\t\tadmin\r\n\tAccount Domain:\t\tEXAMPLE\r\n\tLogon ID:\t\t0x9259ad7\r\n\r\nUser:\r\n\tSecurity ID:\t\tS-1-5-21-1694160624-234216347-2203645164-500\r\n\tAccount Name:\t\tAdministrator\r\n\tAccount Domain:\t\tPC1\r\n\r\nProcess Information:\r\n\tProcess ID:\t\t0x38e8\r\n\tProcess Name:\t\tC:\\Windows\\explorer.exe",
  "Category": "User Account Management",
  "Opcode": "Info",
  "Level": "Information",
  "TargetUserName": "Administrator",
  "TargetDomainName": "PC1",
  "TargetSid": "S-1-5-21-1694160624-234216347-2203645164-500",
  "SubjectUserSid": "S-1-5-21-3288775215-2077974584-1458381936-1001",
  "SubjectUserName": "admin",
  "SubjectDomainName": "EXAMPLE",
  "SubjectLogonId": "0x9259ad7",
  "CallerProcessId": "0x38e8",
  "CallerProcessName": "C:\\Windows\\explorer.exe",
  "EventReceivedTime": "2024-04-16T18:38:13.388661+02:00",
  "SourceModuleName": "windows_event_log",
  "SourceModuleType": "im_msvistalog"
}

Generate GELF events

Graylog Extended Log Format (GELF) is a JSON-based, structured log event format created by Graylog. You can use NXLog Agent’s xm_gelf extension to generate GELF output and send it over UDP or TCP. See our Graylog integration guide for more information on sending telemetry data to Graylog.

Example 5. Converting unstructured events to GELF

This configuration collects syslog messages with the im_file input module. This module populates the $raw_event field. It then forwards events to a remote server in GELF over UDP.

nxlog.conf
<Extension gelf>
    Module        xm_gelf
</Extension>

<Input system_messages>
    Module        im_file
    File          '/var/log/syslog'
</Input>

<Output udp>
    Module        om_udp
    Host          192.168.1.123:12201
    OutputType    GELF_UDP  (1)
</Output>
1 Sets the OutputType directive to GELF_UDP, an output writer function provided by xm_gelf.

The following is a syslog message collected from a Linux host.

Input sample
<38>Apr 16 15:45:43 SERVER1 sshd[8459]: Failed password for invalid user jdoe from 192.168.1.60 port 38176 ssh2

NXLog Agent forwards this event via UDP with the following JSON payload.

Output sample
{
  "version": "1.1",
  "EventReceivedTime": "2024-04-16 15:45:56",
  "SourceModuleName": "system_messages",
  "SourceModuleType": "im_file",
  "Hostname": "LOG-SRV",
  "ShortMessage": "<38>Apr 16 15:45:43 myhost sshd[8459]: Failed password for inval",
  "FullMessage": "<38>Apr 16 15:45:43 myhost sshd[8459]: Failed password for invalid user linda from 192.168.1.60 port 38176 ssh2",
  "EventTime": "2024-04-16T15:45:56.556581+02:00",
  "SeverityValue": 6,
  "MessageSourceAddress": "192.168.1.115"
}

NXLog Binary format

The Binary format is a proprietary format that preserves all event fields when sending them to another NXLog Agent instance or NXLog Platform. The NXLog Transport input and output modules use the Binary format by default. You can also use this format with other stream-oriented modules by specifying the InputType and OutputType directives.

See Log records and fields for more information on how NXLog Agent handles data.

Example 6. Sending events to a relay agent in the NXLog Binary format

This configuration collects syslog messages with the im_file input module and parses them with the xm_syslog module. It then forwards the events to another NXLog Agent instance configured to receive data with im_batchcompress.

nxlog.conf
<Extension syslog>
    Module    xm_syslog
</Extension>

<Input system_messages>
    Module    im_file
    File      '/var/log/syslog'
    Exec      parse_syslog();  (1)
</Input>

<Output relay_agent>
    Module    om_batchcompress
    Host      192.168.1.101:2514
</Output>
1 Calls the parse_syslog() procedure to parse the event into structured data.

The following is a syslog message collected from a Linux host.

Input sample
<38>Apr 16 15:45:43 SERVER1 sshd[8459]: Failed password for invalid user jdoe from 192.168.1.60 port 38176 ssh2

When NXLog Agent forwards this message, it will send the following fields to the relay agent. Note that although the timestamp of the syslog message above does not contain the year, the procedure adds it to the $EventTime field to make it a valid timestamp.

Field Value

$EventReceivedTime

2024-04-16 15:45:56

$EventTime

2024-04-16 15:45:43

$Hostname

SERVER1

$Message

Failed password for invalid user jdoe from 192.168.1.60 port 38176 ssh2

$ProcessID

8459

$raw_event

<38>Apr 16 15:45:43 SERVER1 sshd[8459]: Failed password for invalid user jdoe from 192.168.1.60 port 38176 ssh2

$Severity

INFO

$SeverityValue

2

$SourceModuleName

system_messages

$SourceModuleType

im_file

$SourceName

sshd

$SyslogFacility

AUTH

$SyslogFacilityValue

4

$SyslogSeverity

INFO

$SyslogSeverityValue

6