Collect multi-line logs with NXLog Agent

NXLog Agent expects stream-oriented inputs, such as files and logs sent over TCP or UDP, one record per line by default. Although a newline is the most common way to separate log records, some applications may log multi-line events without encapsulation. In this case, you must define how NXLog Agent should process multi-line log records with the xm_multiline module.

Look out for one or more of the following multi-line event characteristics:

A header or a character sequence that indicates the start of a new log record, such as a timestamp.
A footer or a character sequence marking the end of a log record.
A fixed line count.

Below, we provide examples of using the xm_multiline module to process multi-line logs.

Detect the log header and footer line

Some logs, such as diagnostic or debug logs, are written in a human-readable format and contain events spanning multiple lines. These logs often use a character sequence to mark the beginning and end of an event. You can configure the xm_multiline HeaderLine and EndLine directives to define the event header and/or footer.

Example 1. Collecting multi-line logs with a header and footer

Below is an example of a multi-line event from the Siemens SICAM start-stop log file. A header and footer separate each log event.

Siemens SICAM start-stop log event

-------------------------------------------------------------------------------
Tue 01/11/2023  9:09:36.01
SSR_BeforeBaseContextStart.bat
Exporting dynamic ASR attributes
ChangelogActivator CHKDYNATTRIB returns ERRORLEVEL 0
ASRTool ExportDynAttr returns ERRORLEVEL 0
-END---------------------------------------------------------------------------

This configuration reads Siemens SICAM start-stop logs with the im_file input module. It sets the InputType directive of im_file to the xm_multiline instance name, which, in turn, defines the beginning and end of a new log record.

nxlog.conf

define SICAM_PATH   C:\ProgramData\Siemens Energy\SICAM PAS PQS\Temp

<Extension ssr_parser>
    Module        xm_multiline
    HeaderLine    '-------------------------------------------------------------------------------'
    EndLine       '-END---------------------------------------------------------------------------'
</Extension>

<Input sicam_ssr>
    Module        im_file
    File          '%SICAM_PATH%\SSR_BeforeCfeASRManagerStop.log'
    InputType     ssr_parser
</Input>

The HeaderLine and EndLine directives also support regular expressions. Below, we demonstrate how to detect new Apache Tomcat multi-line events using a regular expression.

Example 2. Collecting multi-line Apache Tomcat logs

The first step is to define the regular expression to detect new log lines.

Apache Tomcat log excerpt

Oct 26, 2023 05:28:16 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 1160 ms
Oct 26, 2023 05:28:17 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory manager

Every Apache Tomcat log event starts with a timestamp and can span any number of lines. Therefore, you can detect new events like the above with the following regular expression:

^\w{3} \d{2}, \d{4} \d{2}:\d{2}:\d{2} (?:AM|PM) .*

Once you define your regular expression(s), you can collect the logs with NXLog Agent. This configuration reads Apache Tomcat logs with the im_file input module. It sets the InputType directive of im_file to the xm_multiline instance name, which, in turn, uses a regular expression to detect the beginning of a new log record.

nxlog.conf

<Extension tomcat_parser>
    Module        xm_multiline
    HeaderLine    /^\w{3} \d{2}, \d{4} \d{2}:\d{2}:\d{2} (?:AM|PM) .*/
</Extension>

<Input tomcat>
    Module        im_file
    File          '/opt/tomcat/logs/catalina.*.log'
    InputType     tomcat_parser
</Input>

Because this xm_multiline instance does not specify the EndLine directive, the parser will only know that a log record is complete when it receives a matching HeaderLine. Therefore, the module buffers data until it detects a new log record or the im_file instance’s PollInterval expires. See also the xm_multiline AutoFlush directive.

Process logs with a fixed line count

You can use the xm_multiline FixedLineCount directive to collect log events comprising a fixed line count.

Example 3. Collecting logs by line count

Below is an example of a multi-line audit event containing 16 lines followed by a blank line. We assume that every event in the log file always has the same number of lines.

Input sample

Wed Nov  1 12:06:24 2023 +01:00
LENGTH: "396"
SESSIONID:[7] "1970008"
ENTRYID:[1] "1"
STATEMENT:[1] "1"
USERID:[4] "JSMITH"
USERHOST:[13] "WORKGROUP\PC1"
TERMINAL:[3] "PC1"
ACTION:[3] "100"
RETURNCODE:[1] "0"
COMMENT$TEXT:[126] "Authenticated by: DATABASE;AUTHENTICATED IDENTITY: JSMITH;
Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=10.0.0.102)(PORT=49804))"
OS$USERID:[4] "John"
DBID:[10] "1676771236"
PRIV$USED:[1] "5"
CURRENT_USER:[4] "JSMITH"

This configuration reads the audit log with the im_file input module. It sets the InputType directive of im_file to the xm_multiline instance name, which, in turn, specifies a fixed line count of 16. The xm_multiline instance also uses the Exec directive to discard any empty lines.

nxlog.conf

define DB_PATH        C:\ProgramData\MyDB\logs

<Extension multiline_parser>
    Module            xm_multiline
    FixedLineCount    16
    Exec              if $raw_event =~ /^\s*$/ drop();
</Extension>

<Input db_audit>
    Module            im_file
    File              '%DB_PATH%\audit.log'
    InputType         multiline_parser
</Input>