NXLog Agent log records and fields

NXLog Agent treats each event it collects as a log record. A log record comprises the $raw_event and NXLog Agent core fields in its most basic form. However, NXLog Agent’s power rests on its advanced log parsing capabilities, which can be done automatically by input modules or customized with NXLog language functions. Once NXLog Agent collects and parses a record into structured data, you can use the event fields to filter, normalize, or enrich log records.

What is a log record?

In NXLog Agent, a log record is a single log event and the data related to it. Log events come in many shapes and forms. However, some of the most common log events include:

Text-based log events delimited by a newline. This is NXLog Agent’s default input and output type.
Multiline log events. You can parse these events with NXLog Agent’s multiline parser.
Text-based logs can be in various data formats such as JSON, XML, CSV, or key-value pairs. NXLog Agent has extension modules to parse and output these log formats.
Log events as UDP datagrams. NXLog Agent’s UDP input and output modules can parse and send UDP datagrams.
Logs can be stored in a database table, where each row signifies a log event. The ODBC input and output modules can read and write logs to ODBC-compliant database engines.
The proprietary NXLog Binary format preserves event fields during transit. Use this format when sending logs to NXLog Platform or another NXLog Agent instance. If you need to use another format, carefully consider its characteristics to make sure it will preserve the data you need. For example, JSON has no datetime type, but many software applications and services support it.

Incoming log records

NXLog Agent creates an internal representation of each log event it collects using fields. Each field is typed and represents a log event attribute. These fields pass through the log route and are available in each subsequent module in the chain until the output instance forwards the record to its destination.

The way NXLog Agent parses a log record on collection depends on its type.

Unstructured data

NXLog Agent sets the value of the $raw_event field to the entire text and adds the core fields. Unstructured logs are usually text-based logs that arrive via TCP, UDP, or collected from a file.

The following diagram illustrates the structure of a log record collected by the im_file input.

Figure 1. Example of an unstructured log record

Structured data

NXLog Agent automatically parses structured log records into module-specific fields, such as Windows events, macOS ULS logs, or logs collected from a database. You can find the list of fields in each module’s documentation.

The following diagram illustrates the structure of a log record collected by the im_msvistalog input module.

Figure 2. Example of a structured log record

† Not all input modules that collect structured logs set the $raw_event field.

Outgoing log records

The way NXLog Agent forwards log records depends on the output module.

Stream-oriented modules, such as the TCP, HTTP(s), and File output modules, forward the value of the $raw_event field. Therefore, you must ensure that this field contains all the relevant information. You can use a data conversion procedure, such as to_json() or to_syslog_bsd(), to populate the $raw_event field.
Specialized modules, like the ODBC output module rely on event fields and do not use the $raw_event field. If you’re collecting unstructured data, you must parse log records into structured data with a regular expression or a data parsing procedure such as parse_json() or parse_syslog().

Manipulating log records

You can manipulate log records by adding, deleting, or renaming fields. The NXLog language provides several functions to help you tailor log records according to your needs. For examples, see our how-to on Normalizing logs.