Collect multiline events and traces with NXLog Agent

NXLog Agent expects stream-oriented inputs, such as files and logs sent over TCP or UDP, one record per line by default. Although a newline is the most common way to separate log records, some applications may log multiline events without encapsulation. In this case, you must define how NXLog Agent should process multiline log records with the xm_multiline module.

Look out for one or more of the following multiline event characteristics:

A header or a character sequence that indicates the start of a new log record, such as a timestamp.
A footer or a character sequence marking the end of a log record.
A fixed line count.

Below, we provide examples of using the xm_multiline module to process multiline events.

Detect the event header and footer line

Some events, such as diagnostic or debug logs, are written in a human-readable format and contain events spanning multiple lines. These events often use a character sequence to mark the beginning and end of an event. You can configure the xm_multiline HeaderLine and EndLine directives to define the event header and/or footer.

Example 1. Collecting multiline events with a header and footer

Below is an example of a multiline event from the Siemens SICAM start-stop log file. A header and footer separate each event.

Siemens SICAM start-stop log event

-------------------------------------------------------------------------------
Tue 01/11/2023  9:09:36.01
SSR_BeforeBaseContextStart.bat
Exporting dynamic ASR attributes
ChangelogActivator CHKDYNATTRIB returns ERRORLEVEL 0
ASRTool ExportDynAttr returns ERRORLEVEL 0
-END---------------------------------------------------------------------------

This configuration reads Siemens SICAM start-stop events with the im_file input module. It sets the InputType directive of im_file to the xm_multiline instance name, which, in turn, defines the beginning and end of a new record.

nxlog.conf

define SICAM_PATH   C:\ProgramData\Siemens Energy\SICAM PAS PQS\Temp

<Extension ssr_parser>
    Module        xm_multiline
    HeaderLine    '-------------------------------------------------------------------------------'
    EndLine       '-END---------------------------------------------------------------------------'
</Extension>

<Input sicam_ssr>
    Module        im_file
    File          '%SICAM_PATH%\SSR_BeforeCfeASRManagerStop.log'
    InputType     ssr_parser
</Input>

The HeaderLine and EndLine directives also support regular expressions. Below, we demonstrate how to detect new Apache Tomcat multiline events using a regular expression.

Example 2. Collecting multiline Apache Tomcat events

The first step is to define the regular expression to detect new log lines.

Apache Tomcat log excerpt

Oct 26, 2023 05:28:16 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 1160 ms
Oct 26, 2023 05:28:17 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory manager

Every Apache Tomcat log event starts with a timestamp and can span any number of lines. Therefore, you can detect new events like the above with the following regular expression:

^\w{3} \d{2}, \d{4} \d{2}:\d{2}:\d{2} (?:AM|PM) .*

Once you define your regular expression(s), you can collect the logs with NXLog Agent. This configuration reads Apache Tomcat logs with the im_file input module. It sets the InputType directive of im_file to the xm_multiline instance name, which, in turn, uses a regular expression to detect the beginning of a new log record.

nxlog.conf

<Extension tomcat_parser>
    Module        xm_multiline
    HeaderLine    /^\w{3} \d{2}, \d{4} \d{2}:\d{2}:\d{2} (?:AM|PM) .*/
</Extension>

<Input tomcat>
    Module        im_file
    File          '/opt/tomcat/logs/catalina.*.log'
    InputType     tomcat_parser
</Input>

Because this xm_multiline instance does not specify the EndLine directive, the parser will only know that a log record is complete when it receives a matching HeaderLine. Therefore, the module buffers data until it detects a new log record or the im_file instance’s PollInterval expires. See also the xm_multiline AutoFlush directive.

Process events with a fixed line count

You can use the xm_multiline FixedLineCount directive to collect log events comprising a fixed line count.

Example 3. Collecting events by line count

Below is an example of a multiline audit event containing 16 lines followed by a blank line. We assume that every event in the log file always has the same number of lines.

Input sample

Wed Nov  1 12:06:24 2023 +01:00
LENGTH: "396"
SESSIONID:[7] "1970008"
ENTRYID:[1] "1"
STATEMENT:[1] "1"
USERID:[4] "JSMITH"
USERHOST:[13] "WORKGROUP\PC1"
TERMINAL:[3] "PC1"
ACTION:[3] "100"
RETURNCODE:[1] "0"
COMMENT$TEXT:[126] "Authenticated by: DATABASE;AUTHENTICATED IDENTITY: JSMITH;
Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=10.0.0.102)(PORT=49804))"
OS$USERID:[4] "John"
DBID:[10] "1676771236"
PRIV$USED:[1] "5"
CURRENT_USER:[4] "JSMITH"

This configuration reads the audit log with the im_file input module. It sets the InputType directive of im_file to the xm_multiline instance name, which, in turn, specifies a fixed line count of 16. The xm_multiline instance also uses the Exec directive to discard any empty lines.

nxlog.conf

define DB_PATH        C:\ProgramData\MyDB\logs

<Extension multiline_parser>
    Module            xm_multiline
    FixedLineCount    16
    Exec              if $raw_event =~ /^\s*$/ drop();
</Extension>

<Input db_audit>
    Module            im_file
    File              '%DB_PATH%\audit.log'
    InputType         multiline_parser
</Input>