Convert data formats
SIEMs and log analytics platforms usually require data in a specific format, such as syslog or JSON. However, your telemetry data sources might not support producing events or metrics in the required format. NXLog Agent supports converting data to several formats. See also the closely related topics of parsing unstructured events and parsing common event formats.
Below, we provide examples of converting logs to syslog, JSON, GELF, and the NXLog Binary format.
Generate syslog messages
Syslog is a popular event format, and many SIEMs accept data in this format. NXLog Agent includes a syslog extension that supports generating events in the BSD (RFC 3164), newer IETF (RFC 5424), and Snare syslog formats. See our syslog integration guide for more information and examples.
This configuration reads Apaches HTTP Server logs with the im_file input module.
This module populates the $raw_event field.
It then uses the to_syslog_ietf() procedure of the xm_syslog module to convert the event to an IETF syslog message.
<Extension syslog>
Module xm_syslog
</Extension>
<Input apache>
Module im_file
File '/var/log/httpd/*'
Exec to_syslog_ietf();
</Input>The following is an Apache HTTP Server access event sample.
192.168.3.20 - - [16/Apr/2024] "GET /cgi-bin/try/ HTTP/1.0" 200 3395When the NXLog Agent configuration above processes this event, it transforms it into the following syslog message.
<13>1 2024-04-16T16:16:37.647610+02:00 WEB-SRV - - - [NXLOG@14506 EventReceivedTime="2024-04-16 16:16:37" SourceModuleName="apache" SourceModuleType="im_file"] 192.168.3.20 - - [16/Apr/2024] "GET /cgi-bin/try/ HTTP/1.0" 200 3395Snare syslog is a specialized syslog format used by the Snare Central SIEM, Snare Agent, and other third-party tools that have adopted it. See Generating Snare logs in our Snare integration guide for more detailed information.
This configuration collects Windows events with the im_msvistalog input module. This module automatically parses events into fields. It then uses the to_syslog_snare() procedure of the xm_syslog module to convert the event to Snare syslog format.
<Extension syslog>
Module xm_syslog
</Extension>
<Input windows_event_log>
Module im_msvistalog
Exec to_syslog_snare();
</Input>The following is a Windows security event ID 4798 sample, shown in XML format.
A user's local group membership was enumerated.
Subject:
Security ID: EXAMPLE\admin
Account Name: admin
Account Domain: EXAMPLE
Logon ID: 0x9259AD7
User:
Security ID: PC1\Administrator
Account Name: Administrator
Account Domain: PC1
Process Information:
Process ID: 0x38e8
Process Name: C:\Windows\explorer.exe
Log name: Security
Source: Microsoft Windows security Logged: 16/04/2024 18:38:13
Event ID: 4798 Task Category: User Account Management
Level: Information Keywords: Audit Success
User: N/A Computer: PC1.example.com
OpCode: InfoWhen the NXLog Agent configuration above processes this event, it transforms it into the following Snare syslog message.
<14>Apr 16 18:38:13 PC1.example.com MSWinEventLog 1 Security 45 Tue Apr 16 18:38:13 2024 4798 Microsoft-Windows-Security-Auditing N/A N/A Success Audit PC1.examle.com User Account Management A user's local group membership was enumerated. Subject: Security ID: S-1-5-21-3288775215-2077974584-1458381936-1001 Account Name: admin Account Domain: EXAMPLE Logon ID: 0x9259ad7 User: Security ID: S-1-5-21-1694160624-234216347-2203645164-500 Account Name: Administrator Account Domain: PC1 Process Information: Process ID: 0x38e8 Process Name: C:\Windows\explorer.exe 2052854Generate JSON logs
JSON is among the most popular data interchange formats for web-based services and APIs. NXLog Agent includes a JSON extension that can easily convert telemetry data to JSON.
This configuration listens for events with the im_tcp input module, which populates the $raw_event field.
It then uses the to_json() procedure of the xm_json module to convert the event to JSON format.
<Extension json>
Module xm_json
</Extension>
<Input tcp>
Module im_tcp
ListenAddr 0.0.0.0:1514
<Exec>
$Message = $raw_event; (1)
to_json();
</Exec>
</Input>| 1 | The to_json() procedure does not include the $raw_event field when converting the event to JSON.
Therefore, you must parse $raw_event into fields or, as in this example, create a new field with the value of $raw_event. |
The following is a syslog message collected from a Linux host.
<38>Apr 16 15:45:43 SERVER1 sshd[8459]: Failed password for invalid user jdoe from 192.168.1.60 port 38176 ssh2When the NXLog Agent configuration above processes this event, it transforms it into the following JSON object comprising the $Message field we added in the configuration, the $MessageSourceAddress added by the im_tcp module, and the NXLog Agent core fields.
{
"MessageSourceAddress": "192.168.1.115",
"EventReceivedTime": "2024-04-16T15:46:10.166035+02:00",
"SourceModuleName": "tcp",
"SourceModuleType": "im_tcp",
"Hostname": "LOG-SRV",
"Message": "<38>Apr 16 15:45:43 myhost sshd[8459]: Failed password for invalid user linda from 192.168.1.60 port 38176 ssh2"
}This configuration collects Windows events with the im_msvistalog input module. This module automatically parses incoming events into fields. It then uses the to_json() procedure of the xm_json module to convert the event to JSON format.
<Extension json>
Module xm_json
</Extension>
<Input windows_event_log>
Module im_msvistalog
Exec to_json();
</Input>The following is a Windows security event ID 4798 sample.
A user's local group membership was enumerated.
Subject:
Security ID: EXAMPLE\admin
Account Name: admin
Account Domain: EXAMPLE
Logon ID: 0x9259AD7
User:
Security ID: PC1\Administrator
Account Name: Administrator
Account Domain: PC1
Process Information:
Process ID: 0x38e8
Process Name: C:\Windows\explorer.exe
Log name: Security
Source: Microsoft Windows security Logged: 16/04/2024 18:38:13
Event ID: 4798 Task Category: User Account Management
Level: Information Keywords: Audit Success
User: N/A Computer: PC1.example.com
OpCode: InfoWhen the NXLog Agent configuration above processes this event, it transforms it into the following JSON object.
{
"EventTime": "2024-04-16T18:38:13.5142877+02:00",
"Hostname": "PC1.example.com",
"Keywords": "0x8020000000000000",
"LevelValue": 0,
"EventType": "AUDIT_SUCCESS",
"SeverityValue": 2,
"Severity": "INFO",
"EventID": 4798,
"SourceName": "Microsoft-Windows-Security-Auditing",
"ProviderGuid": "{54849625-5478-4994-A5BA-3E3B0328C30D}",
"Version": 0,
"TaskValue": 13824,
"OpcodeValue": 0,
"RecordNumber": 2055435,
"ActivityID": "{FE8FAE58-8CEC-0002-E6AE-8FFEEC8CDA01}",
"ExecutionProcessID": 1048,
"ExecutionThreadID": 2468,
"Channel": "Security",
"Message": "A user's local group membership was enumerated.\r\n\r\nSubject:\r\n\tSecurity ID:\t\tS-1-5-21-3288775215-2077974584-1458381936-1001\r\n\tAccount Name:\t\tadmin\r\n\tAccount Domain:\t\tEXAMPLE\r\n\tLogon ID:\t\t0x9259ad7\r\n\r\nUser:\r\n\tSecurity ID:\t\tS-1-5-21-1694160624-234216347-2203645164-500\r\n\tAccount Name:\t\tAdministrator\r\n\tAccount Domain:\t\tPC1\r\n\r\nProcess Information:\r\n\tProcess ID:\t\t0x38e8\r\n\tProcess Name:\t\tC:\\Windows\\explorer.exe",
"Category": "User Account Management",
"Opcode": "Info",
"Level": "Information",
"TargetUserName": "Administrator",
"TargetDomainName": "PC1",
"TargetSid": "S-1-5-21-1694160624-234216347-2203645164-500",
"SubjectUserSid": "S-1-5-21-3288775215-2077974584-1458381936-1001",
"SubjectUserName": "admin",
"SubjectDomainName": "EXAMPLE",
"SubjectLogonId": "0x9259ad7",
"CallerProcessId": "0x38e8",
"CallerProcessName": "C:\\Windows\\explorer.exe",
"EventReceivedTime": "2024-04-16T18:38:13.388661+02:00",
"SourceModuleName": "windows_event_log",
"SourceModuleType": "im_msvistalog"
}Generate GELF events
Graylog Extended Log Format (GELF) is a JSON-based, structured log event format created by Graylog. You can use NXLog Agent’s xm_gelf extension to generate GELF output and send it over UDP or TCP. See our Graylog integration guide for more information on sending telemetry data to Graylog.
This configuration collects syslog messages with the im_file input module.
This module populates the $raw_event field.
It then forwards events to a remote server in GELF over UDP.
<Extension gelf>
Module xm_gelf
</Extension>
<Input system_messages>
Module im_file
File '/var/log/syslog'
</Input>
<Output udp>
Module om_udp
Host 192.168.1.123:12201
OutputType GELF_UDP (1)
</Output>| 1 | Sets the OutputType directive to GELF_UDP, an output writer function provided by xm_gelf. |
The following is a syslog message collected from a Linux host.
<38>Apr 16 15:45:43 SERVER1 sshd[8459]: Failed password for invalid user jdoe from 192.168.1.60 port 38176 ssh2NXLog Agent forwards this event via UDP with the following JSON payload.
{
"version": "1.1",
"EventReceivedTime": "2024-04-16 15:45:56",
"SourceModuleName": "system_messages",
"SourceModuleType": "im_file",
"Hostname": "LOG-SRV",
"ShortMessage": "<38>Apr 16 15:45:43 myhost sshd[8459]: Failed password for inval",
"FullMessage": "<38>Apr 16 15:45:43 myhost sshd[8459]: Failed password for invalid user linda from 192.168.1.60 port 38176 ssh2",
"EventTime": "2024-04-16T15:45:56.556581+02:00",
"SeverityValue": 6,
"MessageSourceAddress": "192.168.1.115"
}NXLog Binary format
The Binary format is a proprietary format that preserves all event fields when sending them to another NXLog Agent instance or NXLog Platform. The NXLog Transport input and output modules use the Binary format by default. You can also use this format with other stream-oriented modules by specifying the InputType and OutputType directives.
See Log records and fields for more information on how NXLog Agent handles data.
This configuration collects syslog messages with the im_file input module and parses them with the xm_syslog module. It then forwards the events to another NXLog Agent instance configured to receive data with im_batchcompress.
<Extension syslog>
Module xm_syslog
</Extension>
<Input system_messages>
Module im_file
File '/var/log/syslog'
Exec parse_syslog(); (1)
</Input>
<Output relay_agent>
Module om_batchcompress
Host 192.168.1.101:2514
</Output>| 1 | Calls the parse_syslog() procedure to parse the event into structured data. |
The following is a syslog message collected from a Linux host.
<38>Apr 16 15:45:43 SERVER1 sshd[8459]: Failed password for invalid user jdoe from 192.168.1.60 port 38176 ssh2When NXLog Agent forwards this message, it will send the following fields to the relay agent. Note that although the timestamp of the syslog message above does not contain the year, the procedure adds it to the $EventTime field to make it a valid timestamp.
| Field | Value |
|---|---|
|
2024-04-16 15:45:56 |
|
2024-04-16 15:45:43 |
|
SERVER1 |
|
Failed password for invalid user jdoe from 192.168.1.60 port 38176 ssh2 |
|
8459 |
|
<38>Apr 16 15:45:43 SERVER1 sshd[8459]: Failed password for invalid user jdoe from 192.168.1.60 port 38176 ssh2 |
|
INFO |
|
2 |
|
system_messages |
|
im_file |
|
sshd |
|
AUTH |
|
4 |
|
INFO |
|
6 |