Siemens SICAM PAS/PQS

SICAM PAS (Power Automation System) is a modular energy automation solution from Siemens for spatially distributed information systems. Siemens SICAM PQS (Power Quality System) is designed to analyze the fault records and power quality data of networks.

NXLog Agent can be configured to collect and process all types of logs that Siemens SICAM PAS/PQS generates. All examples from this guide were tested on SICAM PAS/PQS version 8.16.

Types of logs

NXLog Agent can collect SICAM PAS/PQS events from the following sources:

Logs in Windows Event Log
File-based logs
Passive network monitoring

Logs in Windows Event Log

NXLog Agent can be configured to process Windows Event Logs produced by SICAM PAS/PQS as shown in the examples below.

Example 1. Processing SICAM PAS/PQS logs based on Event ID

The NXLog Agent configuration example shown here reads and processes Windows Event logs generated by SICAM PAS/PQS. Log filtering is based on the values of the Event ID field.

A sample list of Event IDs is provided in the table below. This is only a small subset of events generated by SICAM PAS/PQS since it is impractical to include all possible event types.

Table 1. Events generated by SICAM PAS/PQS
Event ID	Event Text
0	SARAServer monitoring service stopped
0	Service started successfully.
3	ntservice: The Network Time Protocol Service is stopping.
2000	IpcDaemon started.
0	SICAM PAS PQS Watchdog SICAM PAS/PQS Watchdog has been started.
0	SQLANYs_pasStarting SQL Anywhere.

The following log sample with Event ID 0 was copied from the Windows Event Viewer.

Event sample

Log Name:      Application
Source:        SARAMonitoringService
Date:          1/15/2021 11:47:44 AM
Event ID:      0
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      WIN-5RU7GP5MI4V
Description:
SARAServer monitoring service stopped
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="SARAMonitoringService" />
    <EventID Qualifiers="0">0</EventID>
    <Level>4</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2021-01-15T19:47:44.174351800Z" />
    <EventRecordID>38592</EventRecordID>
    <Channel>Application</Channel>
    <Computer>WIN-5RU7GP5MI4V</Computer>
    <Security />
  </System>
  <EventData>
    <Data>SARAServer monitoring service stopped</Data>
  </EventData>
</Event>

The configuration below uses the im_msvistalog module to collect Windows Event Log data. Data is then converted to JSON using the to_json() procedure of the xm_json module.

nxlog.conf

<Extension json>
    Module    xm_json
</Extension>

<Input from_eventlog>
    Module    im_msvistalog
    <QueryXML>
        <QueryList>
            <Query Id="0" Path="Application">
                <Select Path="Application">
                    *[System[(EventID=0  or EventID=3 or
                    EventID=2000)]]
                </Select>
            </Query>
        </QueryList>
    </QueryXML>
    Exec      to_json();
</Input>

This JSON record is output after NXLog Agent has processed the original Windows Event Log sample shown above.

Output sample in JSON format

{
  "EventTime": "2021-01-15T12:56:02.463413-08:00",
  "Hostname": "WIN-5RU7GP5MI4V",
  "Keywords": "36028797018963968",
  "EventType": "INFO",
  "SeverityValue": 2,
  "Severity": "INFO",
  "EventID": 0,
  "SourceName": "SARAMonitoringService",
  "TaskValue": 0,
  "RecordNumber": 38605,
  "ExecutionProcessID": 0,
  "ExecutionThreadID": 0,
  "Channel": "Application",
  "Message": "SARAServer monitoring service started",
  "Opcode": "Info",
  "Data": "SARAServer monitoring service started",
  "EventReceivedTime": "2021-01-15T12:56:03.416555-08:00",
  "SourceModuleName": "from_eventlog",
  "SourceModuleType": "im_msvistalog"
}

The following table lists each SICAM PAS/PQS service that generates log data in Windows Event Log, along with its display name and the path to its executable.

Table 2. Event sources for Windows Event Log
Service Name	Display Name	Path to Executable
SARAMonitoring	Archive Server monitoring service	C:\Program Files (x86)\Common Files\Siemens\Energy\SARA\SARAService.exe
Usermanagement	Archive Server User Management service	C:\Program Files (x86)\Common Files\Siemens\Energy\Usermanagement\Usermanagement.exe
almservice	Automation License Manager Service	C:\Program Files\Common Files\Siemens\sws\almsrv\almsrv64x.exe
DfAgent	Defragmentation Agent	C:\Program Files (x86)\Common Files\Siemens\Energy\DfAgent\DfAgent.exe
LanguageSwitchService	LanguageSwitchService	C:\Program Files (x86)\Siemens Energy\SICAM\PAS PQS\PlcUI\LanguageSwitchService.exe
NTP	Network Time Protocol	C:\Program Files (x86)\Common Files\Siemens\Energy\NTP\ntpdssl-sag.exe
ApplicationDispatcherSvc	PowerCC Application Dispatcher Service	C:\Siemens\Energy\SySrv\Bin\ApplicationDispatcherSvc.exe
IPCDaemon	PowerCC IPC Daemon	C:\Siemens\Energy\SySrv\Bin\IpcDaemon.exe
SecSNMPAgent	Secure SNMP Agent	C:\Program Files (x86)\Common Files\\Siemens\Energy\SecureSNMPAgent\securesnmpagent.exe
hasplms	Sentinel LDK License Manager	C:\Program Files (x86)\Common Files\Aladdin Shared\HASP\hasplms.exe
SICAMFaultEventManager	SICAM Fault Event Manager	C:\Program Files (x86)\Common Files\Siemens\Energy\FaultEventAgent\FaultEventAgent.exe
SICAMPASPQSGroupManagement	SICAM PAS PQS Group Management	C:\Program Files (x86)\Siemens Energy\SICAM\PAS PQS\binui\SICAMPASPQSGroupManagement.exe
SICAM PAS PQS Watchdog	SICAM PAS PQS Watchdog	C:\Program Files (x86)\Siemens Energy\SICAM\PAS PQS\bin\WatchDog.exe
CollectorCondensing	SICAM PQ Collector Archive Condension	C:\Program Files (x86)\Siemens Energy\SICAM\PQ Analyzer\bin\Siemens.Energy.PowerQuality.CollectorCondension.exe
SQLANYs_pas	SQL Anywhere 17 - Pas	C:\Program Files (x86)\Siemens Energy\SICAM\PAS PQS\bin\SQLAnywhereX.exe
SSR	SSR	C:\Siemens\Energy\SySrv\Bin\PowerCCSSR2.exe

Example 2. Processing SICAM PAS/PQS logs based on event source

This example demonstrates how to configure NXLog Agent to parse and process the SICAM PAS/PQS log sources listed in the Event sources for Windows Event Log table.

The following is a log sample from the SARAMonitoringService event source.

Event sample

Log Name:      Application
Source:        SARAMonitoringService
Date:          1/15/2021 11:47:44 AM
Event ID:      0
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      WIN-5RU7GP5MI4V
Description:
SARAServer monitoring service stopped
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="SARAMonitoringService" />
    <EventID Qualifiers="0">0</EventID>
    <Level>4</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2021-01-15T19:47:44.174351800Z" />
    <EventRecordID>38592</EventRecordID>
    <Channel>Application</Channel>
    <Computer>WIN-5RU7GP5MI4V</Computer>
    <Security />
  </System>
  <EventData>
    <Data>SARAServer monitoring service stopped</Data>
  </EventData>
</Event>

Using the im_msvistalog module, the configuration below instructs NXLog Agent to process only those events generated by the following five SICAM PAS/PQS event sources:

SARAMonitoringService
NTP
IPCDaemon
SICAM PAS PQS Watchdog
SQLANYs_pas

To facilitate more convenient post-processing, the xm_json module enables all messages to be converted to JSON as specified in the Exec directive.

nxlog.conf

<Extension json>
    Module    xm_json
</Extension>

<Input from_eventlog>
    Module    im_msvistalog
    <QueryXML>
        <QueryList>
            <Query Id="0" Path="Application">
                <Select Path="Application">
                    *[System[Provider[@Name='SARAMonitoringService'
                    or @Name='NTP'
                    or @Name='IPCDaemon'
                    or @Name='SICAM PAS PQS Watchdog'
                    or @Name='SQLANYs_pas']]]
                </Select>
            </Query>
        </QueryList>
    </QueryXML>
    Exec      to_json();
</Input>

This JSON record is output after NXLog Agent has processed the SARAMonitoringService event sample.

Output sample in JSON format

{
  "EventTime": "2021-01-15T12:56:02.463413-08:00",
  "Hostname": "WIN-5RU7GP5MI4V",
  "Keywords": "36028797018963968",
  "EventType": "INFO",
  "SeverityValue": 2,
  "Severity": "INFO",
  "EventID": 0,
  "SourceName": "SARAMonitoringService",
  "TaskValue": 0,
  "RecordNumber": 38605,
  "ExecutionProcessID": 0,
  "ExecutionThreadID": 0,
  "Channel": "Application",
  "Message": "SARAServer monitoring service started",
  "Opcode": "Info",
  "Data": "SARAServer monitoring service started",
  "EventReceivedTime": "2021-01-15T12:56:03.416555-08:00",
  "SourceModuleName": "from_eventlog",
  "SourceModuleType": "im_msvistalog"
}

File-based logs

There are several types of file-based SICAM PAS/PQS logs that NXLog Agent can process. The following table provides details about their locations.

Table 3. List of file-based logs
Log type	File Ext.	Location	Description
SQL Anywhere logs	`.log`	C:\ProgramData\Siemens Energy\SICAM PAS PQS\Logs\*	-
Defragmentation agent log	`.log`	C:\ProgramData\Siemens Energy\SICAM PAS PQS\Logs\DfAgent\*	-
Fault Event Agent log	`.log`	C:\ProgramData\Siemens Energy\SICAM PQ Analyzer\Logs\FaultEventAgent\FaultEventAgent.log	Fault messages generated by SICAM PAS/PQS
Watchdog log	`.log`	C:\ProgramData\Siemens Energy\SICAM PAS PQS\Logs\*	WatchDog service logs
Non-default SICAM logs	`.log` `.txt`	C:\ProgramData\Siemens Energy\SICAM PAS PQS\Temp	Database connection error logs SSR_AfterCfeASRManagerStart logs SSR_BeforeCfeASRManagerStop logs
System logs	`.log` `.txt` `.xml`	C:\Siemens\Energy\SySrv\Logs\* C:\Siemens\Energy\SySrv\Logs\ASRManager\RT\CFE\* C:\Siemens\Energy\SySrv\Logs\ASRManager\RT\SMI\* C:\Siemens\Energy\SySrv\Logs\IpcDaemon\* C:\Siemens\Energy\SySrv\Logs\MsgLog\* C:\Siemens\Energy\SySrv\Logs\replicator\* C:\Siemens\Energy\SySrv\Logs\SSR\* C:\Siemens\Energy\SySrv\Logs\Summary\* C:\Siemens\Energy\SySrv\Logs\Changeloggenerator\* C:\Siemens\Energy\SySrv\Logs\Process_Logs	ASRManager logs SSR logs Summary logs PowerCC IPC Daemon logs Replicator logs Config Manager Converter logs
SICAM PQ Analyzer log	`.log`	C:\ProgramData\Siemens Energy\SICAM PQ Analyzer\Logs\DfAgent\* C:\ProgramData\Siemens Energy\SICAM PQ Analyzer\Logs\* C:\ProgramData\Siemens Energy\SICAM PQ Analyzer\Logs\PQAnalyzer\* C:\ProgramData\Siemens Energy\SICAM PQ Analyzer\Logs\SARAServer\* C:\ProgramData\Siemens Energy\SICAM PQ Analyzer\Logs\Setup\*	-
Report log	`.txt`	C:\ProgramData\Siemens\Energy\*	Log entries generated after import and update procedures in SICAM PAS/PQS
Automation License Manager setup log	`.txt`	C:\ProgramData\Siemens\Automation\Logfiles\Setup\*	-
Communication trace logs CFE interfaces logs	`.txt`	C:\ProgramData\Siemens Energy\SICAM PAS PQS\Logs\CFE\*	-
Installation logs	`.log`	C:\ProgramData\Siemens Energy\SICAM PAS PQS\Logs\Setup\*	Events encountered during the installation procedure

The universal configuration can used for processing any of the logs documented in the List of file-based logs table.

SQL Anywhere and Watchdog logs

During installation, SICAM PAS/PQS saves project configuration in an SQL Anywhere database. The SICAM PAS/PQS Watchdog service provides memory management of the SICAM PAS/PQS applications.

SQL Anywhere logs are stored in the C:\ProgramData\Siemens Energy\SICAM PAS PQS\Logs\SQLAnywhereX\SQLAnywhereX.log file. Watchdog logs can be found in the C:\ProgramData\Siemens Energy\SICAM PAS PQS\Logs\WatchDog\WatchDog.log file.

Both SQL Anywhere and Watchdog logs use the same format, which consist of the following fields:

DayOfWeek
EventTime
PartOfDay
EventType
Message

Example 3. Processing SQL Anywhere and Watchdog logs

Event sample

Tue, 08-Dec-2020, 1:26:59 AM  [INF]: SQL Anywhere 17 - Pas has been started.

This configuration uses the im_file module to read SQL Anywhere logs from file. To parse log records, the configuration compares each record to the SW_REGEX regular expression. If a match occurs, new fields are created based on the named capturing groups. The strptime() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime field.

The log record is then converted to JSON using the to_json() procedure from the xm_json module. The drop procedure discards records that do not match the SW_REGEX regular expression.

nxlog.conf

# A regular expression for parsing log data
define SW_REGEX     /(?x)^(?<DayOfWeek>\w+),\s+(\d+.\w+.\d+.\s+)(\d+.\d+.\d+)\
                    \s+(?<PartOfDay>\w+)\s+.(?<EventType>\w+)..\
                    \s+(?<Message>.*)/

# Generic path to the folder with the log file
define SICAM_PATH   C:\ProgramData\Siemens Energy\SICAM PAS PQS\Logs

<Extension json>
    Module          xm_json
</Extension>

<Input from_file>
    Module          im_file
    File            '%SICAM_PATH%\SQLAnywhereX\SQLAnywhereX.log'
    <Exec>

        # Matching events to the regular expression
        if $raw_event =~ %SW_REGEX%
        {

            # Creating a timestamp
            $EventTime = strptime($2 + $3, "%d-%b-%Y, %H:%M:%S");

            # Converting to JSON
            to_json();
        }

        # Discarding messages
        else drop();
    </Exec>
</Input>

Output sample in JSON format

{
  "EventReceivedTime": "2021-01-21T03:57:07.606072-08:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "DayOfWeek": "Tue",
  "EventType": "INF",
  "Message": "SQL Anywhere 17 - Pas has been started.",
  "PartOfDay": "AM",
  "EventTime": "2020-12-08T01:26:59.000000-08:00"
}

Defragmentation Agent and Fault Event Agent logs

The Defragmentation Agent service is used for SICAM PAS/PQS and SICAM PQ Analyzer archive management. The Fault Event Agent service manages fault events in SICAM PAS/PQS or SICAM PQ Analyzer.

Table 4. Paths to log files
Log Type	Path
SICAM PAS/PQS Defragmentation Agent log	C:\ProgramData\Siemens Energy\SICAM PAS PQS\Logs\DfAgent\DfAgent.log
SICAM PQ Analyzer Defragmentation Agent log	C:\ProgramData\Siemens Energy\SICAM PQ Analyzer\Logs\DfAgent\DfAgent.log
SICAM PQ Analyzer Fault Event Agent log	C:\ProgramData\Siemens Energy\SICAM PQ Analyzer\Logs\FaultEventAgent\FaultEventAgent.log

Both types of logs use the same format and consist of the following fields:

EventTime
TID (thread ID)
EventType
Message

Example 4. Processing Defragmentation Agent and Fault Event Agent logs

Event sample

[2020-12-04] [02:54:29:366] [TID-003300] [ERR]  DfAgentService::RegisterForChangesInArchiveversion started

This configuration uses the im_file module to read Defragmentation Agent logs from file. To parse log records, the configuration compares each record to the DFEAG_REGEX regular expression. If a match occurs, new fields are created based on the named capturing groups. The strptime() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime field.

The log record is then converted to JSON using the to_json() procedure from the xm_json module. The drop procedure discards records that do not match the DFEAG_REGEX regular expression.

nxlog.conf

# A regular expression for parsing log data
define DFEAG_REGEX  /(?x)^.(\d+.\d+.\d+).\s+.(\d+.\d+.\d+).\d+.\s+.\w+.\
                    (?<TID>\d+).\s+.(?<EventType>\w+).\s+(?<Message>.*)/

# Generic path to the folder with the log file
define SICAM_PATH   C:\ProgramData\Siemens Energy\SICAM PAS PQS\Logs

<Extension json>
    Module          xm_json
</Extension>

<Input from_file>
    Module          im_file
    File            '%SICAM_PATH%\DfAgent\DfAgent.log'
    <Exec>
        # Matching events to the regular expression
        if $raw_event =~ %DFEAG_REGEX%
        {
            # Creating a timestamp
            $EventTime = strptime($1 + $2, "%Y-%m-%d %T");
            # Converting to JSON
            to_json();
        }
        # Discarding messages
        else drop();
    </Exec>
</Input>

Output sample in JSON format

{
  "EventReceivedTime": "2021-01-25T12:53:57.488243-08:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "EventType": "ERR",
  "Message": "DfAgentService::RegisterForChangesInArchiveversion started",
  "TID": "003300",
  "EventTime": "2020-12-04T02:54:29.000000-08:00"
}

Non-default SICAM logs

Non-default SICAM logs are located in the C:\ProgramData\Siemens Energy\SICAM PAS PQS\Temp directory as shown in the table.

Table 5. SICAM log types
Log type	Path
Database connection errors	C:\ProgramData\Siemens Energy\SICAM PAS PQS\Temp\DBConnectionErrorLog.txt
Start-stop recovery events after CfeASRManager start	C:\ProgramData\Siemens Energy\SICAM PAS PQS\Temp\SSR_AfterCfeASRManagerStart.log
Start-stop recovery events before CfeASRManager stop	C:\ProgramData\Siemens Energy\SICAM PAS PQS\Temp\SSR_BeforeCfeASRManagerStop.log

Database connection errors

Each database connection error message contains the following fields:

Timestamp
Event type description
Event type identifier
Database management system data such as the provider’s name, the connection driver, and the name of the database management system
Event message

Example 5. Processing database connection error logs

Event sample

12/8/2020 6:09:12 AM: ERROR [08001] [SAP][ODBC Driver][SQL Anywhere]Specified database not found

This configuration uses the im_file module to read database connection error logs from a file. To parse log records, the configuration compares each record to the DBCONN_REGEX regular expression. If a match occurs, new fields are created based on the named capturing groups. The parsedate() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime field.

The log record is then converted to JSON using the to_json() procedure from the xm_json module. The drop procedure discards records that do not match the DBCONN_REGEX regular expression.

nxlog.conf

# A regular expression for parsing log data
define DBCONN_REGEX /(?x)^(\d+.\d+.\d+\s+\d+.\d+.\d+.\s+\w+).\s+\
                    (?<EventType>\w+)\s+\[(?<EventTypeID>.*)\]\s+\
                    \[(?<Provider>.*)\]\[(?<Driver>.*)\]\[(?<Service>.*)\]\
                    (?<Message>.*)/

# Generic path to the folder with the log file
define SICAM_PATH    C:\ProgramData\Siemens Energy\SICAM PAS PQS\Temp

<Extension json>
    Module           xm_json
</Extension>

<Input from_file>
    Module           im_file
    File             '%SICAM_PATH%\DBConnectionErrorLog.txt'
    <Exec>
        # Matching events to the regular expression
        if $raw_event =~ %DBCONN_REGEX%
        {
            # Creating a timestamp
            $EventTime = parsedate($1);
            # Converting to JSON
            to_json();
        }
        # Discarding messages
        else drop();
    </Exec>
</Input>

Output sample in JSON format

{
  "EventReceivedTime": "2021-01-26T10:35:53.190174-08:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Driver": "ODBC Driver",
  "EventType": "ERROR",
  "EventTypeID": "08001",
  "Message": "Specified database not found",
  "Provider": "SAP",
  "Service": "SQL Anywhere",
  "EventTime": "2020-12-08T06:09:12.000000-08:00"
}

Start-stop recovery events after CfeASRManager start

This example demonstrates how to process log events from the SSR_AfterCfeASRManagerStart.log file using NXLog Agent.

Example 6. Processing start-stop recovery events generated after CfeASRManager start

Event sample

ASR(Info1    ) 16/1_04:13:01:888 G ReadConfigParams: Registry key SOFTWARE\Siemens\Spectrum\ASRManager for config values not found

This type of message spans several lines and contains two timestamps. Each timestamp can act as a separator to split the message into two parts. Both parts can then be processed as separate messages. The first message of the newly created message pair represents the beginning of an operation. The second message of the pair represents the end of an operation.

First part of the message

Sat 01/16/2021  4:13:03.31
SSR_BeforeBaseContextStart.bat
loading incremental changelog files
Importing dynamic ASR attributes

Second part of the message

Sat 01/16/2021  4:13:03.43
SSR_BeforeBaseContextStart.bat ends

This configuration uses the multiline module for parsing multiline messages. The Headerline directive of this module identifies the beginning of each message.

The im_file module reads events from files using the xm_multiline module as input. The Exec block of im_file compares each log message to the SSRA_REGEX regular expression. If it matches, new fields are created according to the named capturing groups. The strptime() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime field.

The log record is then converted to JSON using the to_json() procedure from the xm_json module. The drop procedure discards records that do not match the SSRA_REGEX regular expression.

nxlog.conf

# A regular expression for parsing log data
define SSRA_REGEX   /(?x)^(?<DayOfWeek>\w+)\s+(\d+.\d+.\d+)\s+(\d+.\d+.\d+).\
                    \d+\s(?<BatchFile>.*\.\w+)\s+(?<Message>.*)/

# Generic path to the folder with the log file
define SICAM_PATH   C:\ProgramData\Siemens Energy\SICAM PAS PQS\Temp

<Extension multiline_ssr_after>
    Module          xm_multiline

    # A regular expression that recognizes the message header
    HeaderLine      /^\w+\s+\d+\/\d+\/\d+\s+\d+\:\d+\:\d+\.\d+/

    # Ignoring lines which contain only dashes
    Exec            if $raw_event =~ /^\-+/ drop();
</Extension>

<Extension json>
    Module          xm_json
</Extension>

<Input from_file>
    Module          im_file
    File            '%SICAM_PATH%\SSR_AfterCfeASRManagerStart.log'

    # Applying multiline parser
    InputType       multiline_ssr_after
    <Exec>

        # Replacing unwanted characters
        $raw_event = replace($raw_event, "\r\n", "");

        # Matching events to the regular expression
        if $raw_event =~ %SSRA_REGEX%
        {
            # Creating a timestamp
            $EventTime = strptime($2 + $3, "%m/%d/%Y %H:%M:%S");

            # Converting to JSON
            to_json();
        }

        # Discarding messages
        else drop();
    </Exec>
</Input>

Output sample showing the first part of the message in JSON format

{
  "EventReceivedTime": "2021-01-27T04:19:10.442655-08:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "BatchFile": "SSR_BeforeBaseContextStart.bat",
  "DayOfWeek": "Sat",
  "Message": "loading incremental changelog files Importing dynamic ASR attributes ",
  "EventTime": "2021-01-16T04:13:03.000000-08:00"
}

Output sample showing the second part of the message in JSON format

{
  "EventReceivedTime": "2021-01-27T04:19:10.442655-08:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "BatchFile": "SSR_BeforeBaseContextStart.bat",
  "DayOfWeek": "Sat",
  "Message": "ends ",
  "EventTime": "2021-01-16T04:13:03.000000-08:00"
}

Start-stop recovery events before CfeASRManager stop

This example explains how to process Start-stop recovery events from the SSR_BeforeCfeASRManagerStop.log file using NXLog Agent.

Example 7. Processing Start-stop recovery events generated before CfeASRManager stop

Event sample

-------------------------------------------------------------------------------
Tue 12/08/2020  9:09:36.01
SSR_BeforeBaseContextStart.bat
Exporting dynamic ASR attributes
ChangelogActivator CHKDYNATTRIB returns ERRORLEVEL 0
ASRTool ExportDynAttr returns ERRORLEVEL 0
-END---------------------------------------------------------------------------

NXLog Agent uses the im_file module to read messages from files. To read multiline messages, the NXLog Agent configuration employs the multiline module. The Headerline directive contains a regular expression to identify the first line of a message.

The im_file module specifies the xm_multiline module instance in the InputType directive to process multiline messages, and the Exec block compares each message to the SSRB_REGEX regular expression.

The strptime() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime field.

The log record is then converted to JSON using the to_json() procedure from the xm_json module. The drop procedure discards records that do not match the SSRB_REGEX regular expression.

nxlog.conf

# A regular expression for parsing log data
define SSRB_REGEX   /(?x)^(?<DayOfWeek>\w+)\s+(\d+.\d+.\d+)\s+(\d+.\d+.\d+).\
                    \d+\s(?<BatchFile>.*\.\w+)(?<Message>.*)\-END.*/

# Generic path to the folder with the log file
define SICAM_PATH   C:\ProgramData\Siemens Energy\SICAM PAS PQS\Temp

<Extension multiline_ssr_before>
    Module          xm_multiline

    # A regular expression that recognizes the message header
    HeaderLine      /^\w+\s+\d+\/\d+\/\d+\s+\d+\:\d+\:\d+\.\d+/
</Extension>

<Extension json>
    Module          xm_json
</Extension>

<Input from_file>
    Module          im_file
    File            '%SICAM_PATH%\SSR_BeforeCfeASRManagerStop.log'
    InputType       multiline_ssr_before
    <Exec>
        # Replacing unwanted characters
        $raw_event = replace($raw_event, "\r\n", "");

        # Matching events to the regular expression
        if $raw_event =~ %SSRB_REGEX%
        {
            # Creating a timestamp
            $EventTime = strptime($2 + $3, "%m/%d/%Y %T");

            # Converting to JSON
            to_json();
        }

        # Discarding messages
        else drop();
    </Exec>
</Input>

Output sample in JSON format

{
  "EventReceivedTime": "2021-01-27T02:18:20.770595-08:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "BatchFile": "SSR_BeforeBaseContextStart.bat",
  "DayOfWeek": "Tue",
  "Message": " Exporting dynamic ASR attributes ChangelogActivator CHKDYNATTRIB returns ERRORLEVEL 0 ASRTool ExportDynAttr returns ERRORLEVEL 0 ",
  "EventTime": "2020-12-08T09:09:36.000000-08:00"
}

System logs

SICAM PAS/PQS generates several types of system-related logs.

Table 6. Types of system logs
Log Type	Path
ASRManager logs	C:\Siemens\Energy\SySrv\Logs\ASRManager\RT\CFE.txt C:\Siemens\Energy\SySrv\Logs\ASRManager\RT\SECURITY\.txt C:\Siemens\Energy\SySrv\Logs\ASRManager\RT\SMI\*.txt
SSR log	C:\Siemens\Energy\SySrv\Logs\SSR\LogXX.txt
Summary log	C:\Siemens\Energy\SySrv\Logs\Summary\LogXX.txt
PowerCC IPC Daemon log	C:\Siemens\Energy\SySrv\Logs\IpcDaemon\*.txt
Replicator log	C:\Siemens\Energy\SySrv\Logs\replicator\*.txt
Config Manager Converter log	C:\Siemens\Energy\SySrv\Logs\ConfigMgrConv_yyyy-mm-dd_hh-mm-ss.log

Application Suite Repository (ASR) Manager and SSR logs

The Start Stop Recovery (SSR) service is responsible for system shutdown and recovery. This service stores its log in the C:\Siemens\Energy\SySrv\Logs\SSR directory. The Application Suite Repository (ASR) Manager log can be found in the C:\Siemens\Energy\SySrv\Logs\ASRManager directory.

Both types of logs use the same format and consist of the following fields:

Timestamp
Event type
Service name
Message

A separate log file is created for each work session. The most recent files end with 00.txt.

Example 8. Processing ASRManager and SSR logs

Event sample

ASR(Info1    ) 16/1_04:13:01:888 G ReadConfigParams: Registry key SOFTWARE\Siemens\Spectrum\ASRManager for config values not found

This NXLog Agent configuration uses the im_file module to read ASRManager logs from file. To parse log records, the configuration compares each record to the ASR_REGEX regular expression. If a match occurs, new fields are created based on the named capturing groups. The strptime() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime field.

The log record is then converted to JSON using the to_json() procedure from the xm_json module. The drop procedure discards records that do not match the ASR_REGEX regular expression.

nxlog.conf

# A regular expression for parsing log data
define ASR_REGEX    /(?x)^(?<ServiceName>\w+)\((?<EventType>\w+).*\)\s+(\d+.\d+).\
                    (\d+.\d+.\d+).\d+\s+(?<EventMessage>.*)\:\
                    (?<EventDescription>.*)/

# Generic path to the folder with the log file
define SYS_PATH     C:\Siemens\Energy\SySrv\Logs

<Extension json>
    Module          xm_json
</Extension>

<Input from_file>
    Module          im_file
    File            '%SYS_PATH%\ASRManager\RT\CFE\Log00.txt'
    <Exec>
        # Matching events to the regular expression
        if $raw_event =~ %ASR_REGEX%
        {
            # Creating a timestamp
            $EventTime = strptime($3 + '/' + year(now()) + $4, "%d/%m/%Y %T");

            # Converting to JSON
            to_json();
        }

        # Discarding messages
        else drop();
    </Exec>
</Input>

Output sample in JSON format

{
  "EventReceivedTime": "2021-01-30T11:55:02.593424-08:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "EventDescription": " Registry key SOFTWARE\\Siemens\\Spectrum\\ASRManager for config values not found",
  "EventMessage": "G ReadConfigParams",
  "EventType": "Info1",
  "ServiceName": "ASR",
  "EventTime": "2021-01-16T04:13:01.000000-08:00"
}

Summary log

The summary log is stored in the C:\Siemens\Energy\SySrv\Logs\Summary directory. Each summary message contains the following fields:

Event type
Service name
Event date
Event time
Event message
Event description

Example 9. Processing summary logs

Event sample

 Warn             SSR 29/1_04:42:21:205 SSR SSRProcess TimeOut: Process RT/WIN-5RU7GP5MI4V PAS Package/PowerAutomation/RemoteServer in state PCS_STOPPING will be killed

This configuration uses the im_file module to read the most recent summary logs from file. To parse log records, the configuration compares each record to the SUM_REGEX regular expression. If a match occurs, new fields are created based on the named capturing groups. The strptime() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime field.

The log record is then converted to JSON using the to_json() procedure from the xm_json module. The drop procedure discards records that do not match the SUM_REGEX regular expression.

nxlog.conf

# A regular expression for parsing log data
define SUM_REGEX    /(?x)^(?<EventType>.*\w+)\s+(?<Service>\w+)\s+(\d+.\d+).\
                    (\d+.\d+.\d+).\d+\s+(?<EventMessage>.*)\:\
                    (?<EventDescription>.*)/

# Generic path to the folder with the log file
define SYS_PATH     C:\Siemens\Energy\SySrv\Logs

<Extension json>
    Module          xm_json
</Extension>

<Input from_file>
    Module          im_file
    File            '%SYS_PATH%\Summary\Log00.txt'
    <Exec>
        # Matching events to the regular expression
        if $raw_event =~ %SUM_REGEX%
        {
            # Creating a timestamp
            $EventTime = strptime($3 + '/' + year(now()) + $4, "%d/%m/%Y %T");
            # Converting to JSON
            to_json();
        }
        # Discarding messages
    else drop();
    </Exec>
</Input>

Output sample in JSON format

{
  "EventReceivedTime": "2021-01-30T10:56:08.499733-08:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "EventDescription": " Process RT/WIN-5RU7GP5MI4V PAS Package/PowerAutomation/RemoteServer in state PCS_STOPPING will be killed",
  "EventMessage": "SSR SSRProcess TimeOut",
  "EventType": " Warn",
  "Service": "SSR",
  "EventTime": "2021-01-29T04:42:21.000000-08:00"
}

PowerCC IPC Daemon log

PowerCC IPC Daemon service is used for inter-process communication of the SICAM PAS/PQS services. All events related to the IPC Daemon service are stored in the C:\Siemens\Energy\SySrv\Logs\IpcDaemon directory.

This log file contains two different log formats. The first format consists of the following fields:

Date
Time
Thread ID
Connection number
Host name
Connection status
Preferred IP address
Connection initialization status

The second format consists of the following fields:

Date
Time
Thread ID
Message

Both formats can be read and processed using the NXLog Agent configuration example below.

Example 10. Processing IPC Daemon logs

Event sample of the first format

01/29/21 10:02:18: 3172:     1       WIN-5RU7GP5MI4V   Yes     0.0.0.0                No

Event sample of the second format

01/29/21 10:02:17: 3172: IpcDaemon started.

This configuration uses the im_file module to read IPC Daemon logs from a file. To parse log records, the configuration compares each record to the IPC_REGEX_1 or the IPC_REGEX_2 regular expressions. If a match occurs, new fields are created based on the named capturing groups. The strptime() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime field.

The log record is then converted to JSON using the to_json() procedure from the xm_json module. The drop procedure discards messages that do not match the IPC_REGEX_1 or the IPC_REGEX_2 regular expressions.

nxlog.conf

# Regular expressions for parsing log data
define IPC_REGEX_1    /(?x)^(\d+.\d+.\d+)\s+(\d+.\d+.\d+).\s+(?<TID>\d+).\s+\
                      (?<Connection>\d+)\s+(?<Name>[\w\d\-.]*)\s+(?<Active>\w+)\
                      \s+(?<PreferredIpAddr>\d+.\d+.\d+.\d+)\s+\
                      (?<InitiateConnection>\w+)/
define IPC_REGEX_2    /(?x)^(\d+.\d+.\d+)\s+(\d+.\d+.\d+).\s+(?<TID>\d+).\s+\
                      (?<Message>(?!.*--).*)/

# Generic path to the folder with the log file
define SYS_PATH       C:\Siemens\Energy\SySrv\Logs

<Extension json>
    Module            xm_json
</Extension>

<Input from_file>
    Module            im_file
    File              '%SYS_PATH%\IpcDaemon\log1.txt'
    <Exec>
        # Matching events to the %IPC_REGEX_1% regular expression
        if $raw_event =~ %IPC_REGEX_1%
        {
            # Creating a timestamp
            $EventTime = strptime($1 + $2, "%m/%d/%y %T");
            # Converting to JSON
            to_json();
        }
        # Matching events to the %IPC_REGEX_2% regular expression
        else if $raw_event =~ %IPC_REGEX_2%
        {
            # Creating a timestamp
            $EventTime = strptime($1 + $2, "%m/%d/%y %T");
            # Converting to JSON
            to_json();
        }
        # Discarding messages
        else drop();
    </Exec>
</Input>

Output sample of the first type in JSON format

{
  "EventReceivedTime": "2021-01-29T12:22:25.269292-08:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Active": "Yes",
  "Connection": "1",
  "InitiateConnection": "No",
  "Name": "WIN-5RU7GP5MI4V",
  "PreferredIpAddr": "0.0.0.0",
  "TID": "3172",
  "EventTime": "2021-01-29T10:02:18.000000-08:00"
}

Output sample of the second type in JSON format

{
  "EventReceivedTime": "2021-01-29T12:22:25.269292-08:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Message": "IpcDaemon started.",
  "TID": "3172",
  "EventTime": "2021-01-29T10:02:17.000000-08:00"
}

Replicator log

Replicator log files are located in the C:\Siemens\Energy\SySrv\Logs\replicator directory. This type of message contains the following fields:

Event time and date
TID
Message text

Example 11. Processing replicator log

Event sample

01/29/21 10:02:24: 2708: Performing periodic background processing

This configuration uses the im_file module to read replicator logs from file. To parse log records, the configuration compares each record to the REP_REGEX regular expression. If a match occurs, new fields are created based on the named capturing groups. The strptime() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime field.

The log record is then converted to JSON using the to_json() procedure from the xm_json module. The drop procedure discards records that do not match the REP_REGEX regular expression.

nxlog.conf

# A regular expression for parsing log data
define REP_REGEX    /(?x)^(\d+.\d+.\d+)\s+(\d+.\d+.\d+).\s+\
                    (?<TID>\d+).\s+(?<Message>.*)/

# Generic path to the folder with the log file
define SYS_PATH     C:\Siemens\Energy\SySrv\Logs

<Extension json>
    Module          xm_json
</Extension>

<Input from_file>
    Module          im_file
    File            '%SYS_PATH%\replicator\log1.txt'
    <Exec>
        # Matching events to the regular expression
        if $raw_event =~ %REP_REGEX%
        {
            # Creating a timestamp
            $EventTime = strptime($1 + $2, "%m/%d/%y %T");
            # Converting to JSON
            to_json();
        }
        # Discarding messages
        else drop();
    </Exec>
</Input>

Output sample in JSON format

{
  "EventReceivedTime": "2021-01-29T12:46:17.065098-08:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Message": "Performing periodic background processing",
  "TID": "2708",
  "EventTime": "2021-01-29T10:02:24.000000-08:00"
}

Config Manager Converter log

The Config Manager Converter compiles XML-formatted configuration files to a binary format and vice versa.

Converter messages are collected and stored in the ConfigMgrConv_yyyy-mm-dd_hh-mm-ss.log file of the C:\Siemens\Energy\SySrv\Logs directory.

Each message of this type contains the following fields:

Event date and time
PID
Event type
Event message

Example 12. Processing Config Manager Converter logs

Event sample

2020-12-4_11-57-44 pid:3292 [Info]: Converting xml to binary per directory. The path is: C:\Siemens\Energy\SySrv\Data\ConfigMgr

This configuration uses the im_file module to read Config Manager Converter logs from file. To parse log records, the configuration compares each record to the CMC_REGEX regular expression. If a match occurs, new fields are created based on the named capturing groups. The strptime() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime field.

The log record is then converted to JSON using the to_json() procedure from the xm_json module. The drop procedure discards records that do not match the CMC_REGEX regular expression.

nxlog.conf

# A regular expression for parsing log data
define CMC_REGEX    /(?x)^(\d+.\d+.\d+).(\d+.\d+.\d+)\s+\w+.(?<PID>\d+)\s+.\
                    (?<EventType>\w+)..\s+(?<Message>.*)/

# Generic path to the folder with the log file
define SYS_PATH     C:\Siemens\Energy\SySrv\Logs

<Extension json>
    Module          xm_json
</Extension>

<Input from_file>
    Module          im_file
    File            '%SYS_PATH%\ConfigMgrConv_2020-12-4_11-57-44.log'
    <Exec> 
        # Matching events to the regular expression
        if $raw_event =~ %CMC_REGEX%
        {
            # Creating a timestamp
            $EventTime = strptime($1 + $2, "%Y-%m-%d %H-%M-%S");

            # Converting to JSON
            to_json();
        }
        # Discarding messages
        else drop();
    </Exec>
</Input>

Output sample in JSON format

{
  "EventReceivedTime": "2021-01-31T07:03:20.361190-08:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "EventType": "Info",
  "Message": "Converting xml to binary per directory. The path is: C:\\Siemens\\Energy\\SySrv\\Data\\ConfigMgr",
  "PID": "3292",
  "EventTime": "2020-12-04T11:57:44.000000-08:00"
}

SICAM PQ Analyzer logs

SICAM PQ Analyzer is a SICAM PAS/PQS component that evaluates archived PQ measuring data, fault records, events, and Post Disturbance Review (PDR) records.

The following table lists both types of SICAM PQ Analyser logs.

Table 7. List of SICAM PQ Analyzer logs
Log type	Path
PQ Analyzer log	C:\ProgramData\Siemens Energy\SICAM PQ Analyzer\Logs\PQAnalyzer\PQAnalyzer.log
Archive Server (SARA Server) trace log	C:\ProgramData\Siemens Energy\SICAM PQ Analyzer\Logs\SARAServer\*.log

PQ Analyzer log

PQ Analyzer logs are stored in the C:\ProgramData\Siemens Energy\SICAM PQ Analyzer\Logs\PQAnalyzer\PQAnalyzer.log file. Each log record contains the following fields:

Event date and time
Event type
Event message

Example 13. Processing PQ Analyzer log

Event sample

04-January-2021 03:03:27.199:   Error: Notification Events handling Thread ID: 16

This configuration uses the im_file module to read PQ Analyzer logs from file. To parse log records, the configuration compares each record to the PQ_REGEX regular expression. If a match occurs, new fields are created based on the named capturing groups. The strptime() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime field.

The log record is then converted to JSON using the to_json() procedure from the xm_json module. The drop procedure discards records that do not match the PQ_REGEX regular expression.

nxlog.conf

# A regular expression for parsing log data
define PQ_REGEX    /(?x)^(\d+.\w+.\d+)\s+(\d+.\d+.\d+).\d+.\s+\
                   (?<EventType>\w+).\s+(?<Message>.*)/

# Generic path to the folder with the log file
define PQ_PATH     C:\ProgramData\Siemens Energy\SICAM PQ Analyzer\Logs

<Extension json>
    Module         xm_json
</Extension>

<Input from_file>
    Module         im_file
    File           '%PQ_PATH%\PQAnalyzer\PQAnalyzer.log'
    <Exec>
        # Matching events to the regular expression
        if $raw_event =~ %PQ_REGEX%
        {
            # Creating a timestamp
            $EventTime = strptime($1 + $2, "%d-%B-%Y%T");
            # Converting to JSON
            to_json();
        }
        # Discarding messages
        else drop();
    </Exec>
</Input>

Output sample in JSON format

{
  "EventReceivedTime": "2021-01-31T10:58:41.329611-08:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "EventType": "Error",
  "Message": "Notification Events handling Thread ID: 16",
  "EventTime": "2021-01-04T03:03:27.000000-08:00"
}

Archive Server (SARA Server) trace log

The Archive Server monitoring service collects its trace log messages in the Runtime.log, Runtime_Error.log, <ArchiveName>.log and <ArchiveName>_Error.log files located in the C:\ProgramData\Siemens Energy\SICAM PQ Analyzer\Logs\SARAServer\ directory. The <ArchiveName>.log and Runtime.log files contain the most recent data. When new log files are created, older log data is moved to the <ArchiveName_X>.log and Runtime_<X>.log files.

Each trace log message begins with a header describing the server name, message text, and event date.

The header is followed by an event message in one of two formats. The first event message format contains the following data:

Event time
Event type
Event message

The second event message format contains only the event time and the event message.

NXLog Agent can read and process SARA Server trace log messages as shown in the following example.

Example 14. Processing Archive Server (SARA Server) trace log

Trace log header

** SARAServer: start trace
** Date: 2021-01-04
**

Sample of event message format 1

12:51:31.128Z|1|0694* Error: The SendMessageToWindowsSysLogger(SendMessageToSysLogServer from SecurityEventLogger function as failed) and return value id = -4.

Sample of event message format 2

14:23:25.268Z|1|1024* 1048576 bytes newly allocated in memory pool

This configuration defines the SARA_HEAD_RE regular expressions to parse message headers. The SARA_REGEX_1 and SARA_REGEX_2 expressions are used for parsing the two types of messages.

The multiline module is specified for parsing multiline header messages. The Headerline and Endline directives of this module identify the beginning and the end of each message header.

The configuration provides separate im_file module instances for parsing headers and single-line events. The Exec blocks of each input module compare event entries to the regular expressions, parse values, and create new fields using named capturing groups.

The log record is then converted to JSON using the to_json() procedure from the xm_json module. The drop procedure discards messages that match neither the SARA_REGEX_1 nor the SARA_REGEX_2 regular expressions.

nxlog.conf

# A regular expression for parsing headers
define SARA_HEAD_RE /(?x)^\**\s+(?<Server>\w+).\s+(?<Message>[\w\s]*)\**\s+\w+.\
                    \s+(?<TraceDate>\d+.\d+.\d+)/

# A regular expression for parsing the first type of messages
define SARA_REGEX_1 /(?x)^(?<EventTime>\d+.\d+.\d+)[\d\w|*.]*\s+\
                    (?<EventType>\w+)\:(?<Message>.*)/

# A regular expression for parsing the second type of messages
define SARA_REGEX_2 /(?x)^(?<EventTime>\d+.\d+.\d+)[\d\w|*.]*\s+(?<Message>.*)/

# Generic path to the folder with log files
define PQ_PATH      C:\ProgramData\Siemens Energy\SICAM PQ Analyzer\Logs

<Extension json>
    Module          xm_json
</Extension>

<Extension multiline_sara>
    Module          xm_multiline

    # A regular expression for identifying message headers
    HeaderLine      /^\*\*\s+\w+\:\s[\w\s]*/

    # A regular expression for identifying message ends
    EndLine         /^\*\*\s+\w+\:\s\d+.\d+.\d+/
</Extension>

<Input from_file_header>
    Module          im_file
    File            '%PQ_PATH%\SARAServer\Demo.Log'
    InputType       multiline_sara
    <Exec>

        # Replacing unwanted characters
        $raw_event = replace($raw_event, "\r\n", "");

        # Matching events to the regular expression and converting to JSON
        if $raw_event =~ %SARA_HEAD_RE% to_json();

        # Discarding messages
        else drop();
    </Exec>
</Input>

<Input from_file>
    Module          im_file
    File            '%PQ_PATH%\SARAServer\Demo.Log'
    <Exec>

        # Matching events to the regular expressions and converting to JSON
        if ($raw_event =~ %SARA_REGEX_1%) or
           ($raw_event =~ %SARA_REGEX_2%) to_json();

        # Discarding messages
        else drop();
    </Exec>
</Input>

Output header sample in JSON format

{
  "EventReceivedTime": "2021-02-01T12:06:00.172465-08:00",
  "SourceModuleName": "from_file_header",
  "SourceModuleType": "im_file",
  "Message": "start trace",
  "Server": "SARAServer",
  "TraceDate": "2021-01-04"
}

Output sample after processing Event message format 1

{
  "EventReceivedTime": "2021-02-01T12:06:00.172465-08:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "EventTime": "12:51:31",
  "EventType": "Error",
  "Message": " The SendMessageToWindowsSysLogger(SendMessageToSysLogServer from SecurityEventLogger function as failed) and return value id = -4."
}

Output sample after processing Event message format 2

{
  "EventReceivedTime": "2021-02-01T12:06:00.172465-08:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "EventTime": "14:23:25",
  "Message": "1048576 bytes newly allocated in memory pool"
}

Report log

Report logs contain information about import and update procedures and are stored in C:\ProgramData\Siemens\Energy\Report.txt. Each report message contains the following data:

Timestamp
Event type description
Message text

Example 15. Processing Report log

Event sample

Report
======

Overview
--------
                        		----------------------------------------------------------------------------------------------------------------------
1/4/2021   7:12:53 AM   Info:	Creating generic report template 'AllCharacteristics_01-27 Delta'...
1/4/2021   7:13:04 AM   Warning:	The creation of the generic report template is completed.- 0 fault(s), 1 warning(s)
                        		----------------------------------------------------------------------------------------------------------------------
1/4/2021   7:15:19 AM   Info:	Starting importing/updating from file 'C:\Users\Public\Documents\Siemens Energy\SICAM PAS PQS\ReportTemplatesScheduled\english\AllCharacteristics_01-27-Delta.TXT'.
1/4/2021   7:15:21 AM   Warning:	   There is no suitable measuring point available for the following diagrams. This results in missing channels or missing diagrams in the report.

                        		   Diagram: 1. Time line Mean Values

                        		    - Mean value/Voltage/A-B/Absolute

                        		    - Mean value/Voltage/B-C/Absolute

                        		    - Mean value/Voltage/C-A/Absolute

This configuration uses the im_file module to collect file-based logs and the multiline module to read multiline log records. To parse log records, the configuration compares each record to the RPT_REGEX regular expression. If a match occurs, new fields are created based on the named capturing groups. The parsedate() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime field.

The log record is then converted to JSON using the to_json() procedure from the xm_json module. The drop procedure discards records that do not match the RPT_REGEX regular expression.

nxlog.conf

# A regular expression for parsing log data
define RPT_REGEX    /(?x)^(\d+.\d+.\d+)\s+(\d+.\d+.\d+.\w+)\s+\
                    (?<EventType>\w+).\s+(?<Message>.*)/

# Generic path to the folder with the log file
define RPT_PATH     C:\ProgramData\Siemens\Energy

<Extension json>
    Module          xm_json
</Extension>

<Extension multiline_report>
    Module          xm_multiline

    # Regular expression for parsing message headers
    HeaderLine      /^\d+.\d+.\d+\s+\d+.\d+.\d+\s+\w+\s+\w+\:\s+.*$/

    # Discarding unwanted lines
    Exec            if $raw_event =~ /^\s+\-{2,}/ drop();
</Extension>

<Input from_file>
    Module          im_file
    File            '%RPT_PATH%\Report.txt'
    InputType       multiline_report
    <Exec>

        # Replacing unwanted characters
        $raw_event = replace($raw_event, "\r\n", " ");
        $raw_event = replace($raw_event, "\t", " ");
        $raw_event =~ s/\s{2,}/ /g;

        # Matching events to the regular expression
        if $raw_event =~ %RPT_REGEX%
        {

            # Creating a timestamp
            $EventTime = parsedate($1 + ' ' + $2);

            # Converting to JSON
            to_json();
        }

        # Discarding messages
        else drop();
    </Exec>
</Input>

Output sample of the single-line event in JSON format

{
  "EventReceivedTime": "2021-09-21T05:25:19.435981-07:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "EventType": "Warning",
  "Message": "The creation of the generic report template is completed.- 0 fault(s), 1 warning(s)",
  "EventTime": "2021-09-21T05:25:04.000000-08:00"
}

Output sample of the multiline event in JSON format

{
  "EventReceivedTime": "2021-02-03T04:39:25.236338-08:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "EventType": "Warning",
  "Message": "There is no suitable measuring point available for the following diagrams. This results in missing channels or missing diagrams in the report. Diagram: 1. Time line Mean Values - Mean value/Voltage/A-B/Absolute - Mean value/Voltage/B-C/Absolute - Mean value/Voltage/C-A/Absolute",
  "EventTime": "2021-01-04T07:15:21.000000-08:00"
}

Universal configuration file

This example contains a universal NXLog Agent configuration that can process any type of file-based SICAM PAS/PQS log.

Example 16. Processing all file-based logs

This configuration is comprised of various sections. In the first section, all regular expressions are defined as constants. The second section defines the absolute paths to the log files that will be read.

The third section loads the extension modules that will be needed for processing the data. The xm_json module is needed for converting the parsed data to JSON. Multiple instances of the multiline module identify multiline messages. The Headerline and Endline directives identify the beginning and the end of each message.

The fourth section is dedicated to defining 17 distinct input instances of the im_file module, unique to each type of log that can be processed. This is where the bulk of the configuration and event processing occurs. Each im_file input instance bears a unique instance name to identify which kind of log or message type it has been configured to read and process:

sql_anywhere
watchdog
df_agent
fe_agent
dbconn
ssr_after
ssr_before
asr
ssr
summary
ipc
replicator
cmc
pq_analyzer
sara_header
sara_server
report

The fifth section loads the om_file module that will take the JSON-formatted data it receives from the various input instances and output it—without any further processing or conversion—to a single file, C:\output.txt.

The sixth and final section, ROUTES, is required since it defines which input instance(s) are routed to which output instance(s). In this case, all input instances are routed to the only output instance that has been defined.

nxlog.conf

# ----------------- REGULAR EXPRESSIONS FOR PARSING DATA -----------------------

define SW_REGEX        /(?x)^(?<DayOfWeek>\w+),\s+(\d+.\w+.\d+.\s+)\
                       (\d+.\d+.\d+)\s+(?<PartOfDay>\w+)\s+.(?<EventType>\w+)\
                       ..\s+(?<Message>.*)/

define DFEAG_REGEX     /(?x)^.(\d+.\d+.\d+).\s+.(\d+.\d+.\d+).\d+.\s+.\w+.\
                       (?<TID>\d+).\s+.(?<EventType>\w+).\s+(?<Message>.*)/

define DBCONN_REGEX    /(?x)^(\d+.\d+.\d+\s+\d+.\d+.\d+.\s+\w+).\s+\
                       (?<EventType>\w+)\s+\[(?<EventTypeID>.*)\]\s+\
                       \[(?<Provider>.*)\]\[(?<Driver>.*)\]\[(?<Service>.*)\]\
                       (?<Message>.*)/

define SSRA_REGEX      /(?x)^(?<DayOfWeek>\w+)\s+(\d+.\d+.\d+)\s+(\d+.\d+.\d+)\
                       .\d+\s(?<BatchFile>.*\.\w+)\s+(?<Message>.*)/

define SSRB_REGEX      /(?x)^(?<DayOfWeek>\w+)\s+(\d+.\d+.\d+)\s+(\d+.\d+.\d+)\
                       .\d+\s(?<BatchFile>.*\.\w+)(?<Message>.*)\-END.*/

define ASR_REGEX       /(?x)^(?<ServiceName>\w+)\((?<EventType>\w+).*\)\
                       \s+(\d+.\d+).(\d+.\d+.\d+).\d+\s+(?<EventMessage>.*)\:\
                       (?<EventDescription>.*)/

define SUM_REGEX       /(?x)^(?<EventType>.*\w+)\s+(?<Service>\w+)\s+(\d+.\d+)\
                       .(\d+.\d+.\d+).\d+\s+(?<EventMessage>.*)\:\
                       (?<EventDescription>.*)/

define IPC_REGEX_1     /(?x)^(\d+.\d+.\d+)\s+(\d+.\d+.\d+).\s+(?<TID>\d+).\s+\
                       (?<Connection>\d+)\s+(?<Name>[\w\d\-.]*)\s+\
                       (?<Active>\w+)\s+(?<PreferredIpAddr>\d+.\d+.\d+.\d+)\s+\
                       (?<InitiateConnection>\w+)/

define IPC_REGEX_2     /(?x)^(\d+.\d+.\d+)\s+(\d+.\d+.\d+).\s+(?<TID>\d+).\s+\
                       (?<Message>(?!.*--).*)/

define REP_REGEX       /(?x)^(\d+.\d+.\d+)\s+(\d+.\d+.\d+).\s+\
                       (?<TID>\d+).\s+(?<Message>.*)/

define CMC_REGEX       /(?x)^(\d+.\d+.\d+).(\d+.\d+.\d+)\s+\w+.(?<PID>\d+)\s+.\
                       (?<EventType>\w+)..\s+(?<Message>.*)/

define PQ_REGEX        /(?x)^(\d+.\w+.\d+)\s+(\d+.\d+.\d+).\d+.\s+\
                       (?<EventType>\w+).\s+(?<Message>.*)/

define SARA_HEAD_RE    /(?x)^\**\s+(?<Server>\w+).\s+(?<Message>[\w\s]*)\**\s+\
                       \w+.\s+(?<TraceDate>\d+.\d+.\d+)/

define SARA_REGEX_1    /(?x)^(?<EventTime>\d+.\d+.\d+)[\d\w|*.]*\s+\
                       (?<EventType>\w+)\:(?<Message>.*)/

define SARA_REGEX_2    /(?x)^(?<EventTime>\d+.\d+.\d+)[\d\w|*.]*\s+\
                       (?<Message>.*)/

define RPT_REGEX       /(?x)^(\d+.\d+.\d+)\s+(\d+.\d+.\d+.\w+)\s+\
                       (?<EventType>\w+).\s+(?<Message>.*)/

# ----------------- PATHS TO LOG FILES -----------------------------------------

define SICAM_PATH      C:\ProgramData\Siemens Energy\SICAM PAS PQS\Logs

define SYS_PATH        C:\Siemens\Energy\SySrv\Logs

define PQ_PATH         C:\ProgramData\Siemens Energy\SICAM PQ Analyzer\Logs

define RPT_PATH        C:\ProgramData\Siemens\Energy

# ----------------- EXTENSION MODULES ------------------------------------------

<Extension json>
    Module             xm_json
</Extension>

<Extension multiline_ssr_after>
    Module             xm_multiline
    HeaderLine         /^\w+\s+\d+\/\d+\/\d+\s+\d+\:\d+\:\d+\.\d+/
    Exec               if $raw_event =~ /^\-+/ drop();
</Extension>

<Extension multiline_ssr_before>
    Module             xm_multiline
    HeaderLine         /^\w+\s+\d+\/\d+\/\d+\s+\d+\:\d+\:\d+\.\d+/
</Extension>

<Extension multiline_sara>
    Module             xm_multiline
    HeaderLine         /^\*\*\s+\w+\:\s[\w\s]*/
    EndLine            /^\*\*\s+\w+\:\s\d+.\d+.\d+/
</Extension>

<Extension multiline_report>
    Module             xm_multiline
    HeaderLine         /^\d+.\d+.\d+\s+\d+.\d+.\d+\s+\w+\s+\w+\:\s+.*$/
    Exec               if $raw_event =~ /^\s+\-{2,}/ drop();
</Extension>

# ----------------- INPUT MODULES ----------------------------------------------

<Input sql_anywhere>
    Module             im_file
    File               '%SICAM_PATH%\SQLAnywhereX\SQLAnywhereX.log'
    <Exec>
        if $raw_event =~ %SW_REGEX%
        {
            $EventTime = strptime($2 + $3, "%d-%b-%Y, %H:%M:%S");
            to_json();
        }
        else drop();
    </Exec>
</Input>

<Input watchdog>
    Module             im_file
    File               '%SICAM_PATH%\WatchDog.log'
    <Exec>
        if $raw_event =~ %SW_REGEX%
        {
            $EventTime = strptime($2 + $3, "%d-%b-%Y, %H:%M:%S");
            to_json();
        }
        else drop();
    </Exec>
</Input>

<Input df_agent>
    Module             im_file
    File               '%SICAM_PATH%\DfAgent\DfAgent.log'
    <Exec>
        if $raw_event =~ %DFEAG_REGEX%
        {
            $EventTime = strptime($1 + $2, "%Y-%m-%d %T");
            to_json();
        }
        else drop();
    </Exec>
</Input>

<Input fe_agent>
    Module             im_file
    File               '%SICAM_PATH%\FaultEventAgent\FaultEventAgent.log'
    <Exec>
        if $raw_event =~ %DFEAG_REGEX%
        {
            $EventTime = strptime($1 + $2, "%Y-%m-%d %T");
            to_json();
        }
        else drop();
    </Exec>
</Input>

<Input dbconn>
    Module             im_file
    File               '%SICAM_PATH%\DBConnectionErrorLog.txt'
    <Exec>
        if $raw_event =~ %DBCONN_REGEX%
        {
            $EventTime = parsedate($1);
            to_json();
        }
        else drop();
    </Exec>
</Input>

<Input ssr_after>
    Module             im_file
    File               '%SICAM_PATH%\SSR_AfterCfeASRManagerStart.log'
    InputType          multiline_ssr_after
    <Exec>
        $raw_event = replace($raw_event, "\r\n", "");
        if $raw_event =~ %SSRA_REGEX%
        {
            $EventTime = strptime($2 + $3, "%m/%d/%Y %H:%M:%S");
            to_json();
        }
        else drop();
    </Exec>
</Input>

<Input ssr_before>
    Module             im_file
    File               '%SICAM_PATH%\SSR_BeforeCfeASRManagerStop.log'
    InputType          multiline_ssr_before
    <Exec>
        $raw_event = replace($raw_event, "\r\n", "");
        if $raw_event =~ %SSRB_REGEX%
        {
            $EventTime = strptime($2 + $3, "%m/%d/%Y %T");
            to_json();
        }
        else drop();
    </Exec>
</Input>

<Input asr>
    Module             im_file
    File               '%SYS_PATH%\ASRManager\RT\CFE\Log00.txt'
    <Exec>
        if $raw_event =~ %ASR_REGEX%
        {
            $EventTime = strptime($3 + '/' + year(now()) + $4, "%d/%m/%Y %T");
            to_json();
        }
        else drop();
    </Exec>
</Input>

<Input ssr>
    Module             im_file
    File               '%SYS_PATH%\SSR\Log00.txt'
    <Exec>
        if $raw_event =~ %ASR_REGEX%
        {
            $EventTime = strptime($3 + '/' + year(now()) + $4, "%d/%m/%Y %T");
            to_json();
        }
        else drop();
    </Exec>
</Input>

<Input summary>
    Module             im_file
    File               '%SYS_PATH%\Summary\Log00.txt'
    <Exec>
        if $raw_event =~ %SUM_REGEX%
        {
            $EventTime = strptime($3 + '/' + year(now()) + $4, "%d/%m/%Y %T");
            to_json();
        }
    else drop();
    </Exec>
</Input>

<Input ipc>
    Module             im_file
    File               '%SYS_PATH%\IpcDaemon\log1.txt'
    <Exec>
        if ($raw_event =~ %IPC_REGEX_1%) or ($raw_event =~ %IPC_REGEX_2%)
        {
            $EventTime = strptime($1 + $2, "%m/%d/%y %T");
            to_json();
        }
        else drop();
    </Exec>
</Input>

<Input replicator>
    Module             im_file
    File               '%SYS_PATH%\replicator\log1.txt'
    <Exec>
        if $raw_event =~ %REP_REGEX%
        {
            $EventTime = strptime($1 + $2, "%m/%d/%y %T");
            to_json();
        }
        else drop();
    </Exec>
</Input>

<Input cmc>
    Module             im_file
    File               '%SYS_PATH%\ConfigMgrConv_2020-12-4_11-57-44.log'
    <Exec>
        if $raw_event =~ %CMC_REGEX%
        {
            $EventTime = strptime($1 + $2, "%Y-%m-%d %H-%M-%S");
            to_json();
        }
        else drop();
    </Exec>
</Input>

<Input pq_analyzer>
    Module             im_file
    File               '%PQ_PATH%\PQAnalyzer\PQAnalyzer.log'
    <Exec>
        if $raw_event =~ %PQ_REGEX%
        {
            $EventTime = strptime($1 + $2, "%d-%B-%Y%T");
            to_json();
        }
        else drop();
    </Exec>
</Input>

<Input sara_header>
    Module             im_file
    File               '%PQ_PATH%\SARAServer\Demo.Log'
    InputType          multiline_sara
    <Exec>
        $raw_event = replace($raw_event, "\r\n", "");
        if $raw_event =~ %SARA_HEAD_RE% to_json();
        else drop();
    </Exec>
</Input>

<Input sara_server>
    Module             im_file
    File               '%PQ_PATH%\SARAServer\Demo.Log'
    <Exec>
        if ($raw_event =~ %SARA_REGEX_1%) or
        ($raw_event =~ %SARA_REGEX_2%) to_json();
        else drop();
    </Exec>
</Input>

<Input report>
    Module             im_file
    File               '%RPT_PATH%\Report.txt'
    InputType          multiline_report
    <Exec>
        $raw_event = replace($raw_event, "\r\n", " ");
        $raw_event = replace($raw_event, "\t", " ");
        $raw_event =~ s/\s{2,}/ /g;
        if $raw_event =~ %RPT_REGEX%
        {
            $EventTime = parsedate($1 + ' ' + $2);
            to_json();
        }
        else drop();
    </Exec>
</Input>

# ----------------- OUTPUT MODULE ----------------------------------------------

<Output to_file>
    Module          om_file
    File            'C:\output.txt'
</Output>

# ----------------- ROUTES -----------------------------------------------------

<Route r1>
    Path               sql_anywhere, df_agent, dbconn, ssr_after => to_file
</Route>

<Route r2>
    Path               ssr_before, asr, summary, ipc, replicator => to_file
</Route>

<Route r3>
    Path               cmc, pq_analyzer, sara_server, report => to_file
</Route>

<Route r4>
    Path               watchdog, fe_agent, ssr, sara_header => to_file
</Route>

Communication trace logs

Communication trace logs contain information about SICAM PAS/PQS communication issues related to various network interfaces. Log files are stored in the C:\ProgramData\Siemens Energy\SICAM PAS PQS\Logs\CFE directory. The log files follow a specific file naming convention:

<name of CFE><name of protocol><name of interface>_trc.txt

Modbus

NXLog Agent can process various protocol-independent and protocol-specific trace logs from the Modbus Master and Modbus Slave interfaces. This section covers the following types of Modbus events:

Byte stream
Interpreted Telegrams
Normalized data
Preprocessed data
Statistics data
Systemaccu events
Miscellaneous events

Byte stream events

This example shows how to configure NXLog Agent to process Modbus data organized as byte blocks. These events contain the following fields:

EventSource
Bytestream
Data

Example 17. Processing byte stream events

Input sample

2021-04-19 15:36:14.652  PM : Byte Stream (Hex)Received(CIpConnectSec::ReadDataResult)IO_READ_ALL RTU=Device(bytes=29 of 29):
                         01 fd 00 00 00 17 01 04 14 80 31 80 44 0f b3 0f
                         b1 0f af 80 40 0f ac 80 3b 0f aa 0f a7

This configuration uses the im_file module to collect file-based logs and the multiline module to read multiline log records. To parse log records, the configuration compares each record to the REGEX_BS regular expression. If a match occurs, new fields are created based on the named capturing groups. The parsedate() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime field.

The log record is then converted to JSON using the to_json() procedure from the xm_json module. The drop procedure discards records that do not match the REGEX_BS regular expression.

nxlog.conf

# A regular expression for parsing log data
define REGEX_BS     /(?x)^(\d+.\d+.\d+.)(\d+.\d+.\d+.\d+)\s*\
                    (?<EventSource>[A-Z]{2,})[\s\:]+\
                    (?<ByteStream>Byte\s*Stream.*)\:\s*(?<Data>[\d\w\s]*)/

# Generic path to the folder with log files
define CFE_PATH     C:\ProgramData\Siemens Energy\SICAM PAS PQS\Logs\CFE

<Extension json>
    Module          xm_json
</Extension>

<Extension multiline>
    Module          xm_multiline
    HeaderLine      /^\d+.\d+.\d+.\d+.\d+.\d+.\d+/
</Extension>

<Input from_file>
    Module          im_file
    File            '%CFE_PATH%\WIN-5RU7GP5MI4VModbus *Interface_trc.txt'
    InputType       multiline
    <Exec>

        # Matching a file header to the regular expression
        if $raw_event =~ %REGEX_BS%
        {

            # Creating a timestamp
            $EventTime = parsedate($1 + $2);

            # Converting to JSON
            to_json();
        }

        # Discarding messages which do not match the regular expression
        else drop();

        # Replacing unwanted characters
        $raw_event =~ s/\\r\\n/ /g;
        $raw_event =~ s/\s{2,}/ /g;
    </Exec>
</Input>

Output sample in JSON format

{
  "EventReceivedTime": "2021-04-19T15:36:14.669993+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "ByteStream": "Byte Stream (Hex)Received(CIpConnectSec::ReadDataResult)IO_READ_ALL RTU=Device(bytes=29 of 29)",
  "Data": "01 fd 00 00 00 17 01 04 14 80 31 80 44 0f b3 0f b1 0f af 80 40 0f ac 80 3b 0f aa 0f a7 ",
  "EventSource": "PM",
  "EventTime": "2021-04-19T15:36:14.652000+03:00"
}

Interpreted Telegrams

Interpreted Telegrams events contain information in a protocol-specific format. This section explains how to parse two types of Interpreted Telegrams.

The first type of Interpreted Telegram contains the following fields:

EventSource
MessageType
RTU
RTU_Address
MsgAddress
Value
Function
Result
Message

Example 18. Processing the first type of Interpreted Telegrams

Input sample

2021-04-19 15:36:14.612  PM : Interpreted Telegram - Received Read Holding Register FC03:  (S=1, R=0)
                         RTU Address:   1 (LinkId=100001)
                         Msg Address: 9
                         Value(INT32): 0x0000h
                         !!! INVALID (HALF incomplete)!!!

This configuration uses the im_file module to collect file-based logs and the multiline module to read multiline log records. To parse log records, the configuration compares each record to the REGEX_ITONE regular expression. If a match occurs, new fields are created based on the named capturing groups. The parsedate() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime field.

The log record is then converted to JSON using the to_json() procedure from the xm_json module. The drop procedure discards records that do not match the REGEX_ITONE regular expression.

nxlog.conf

# A regular expression for parsing log data
define REGEX_ITONE  /(?x)^(\d+.\d+.\d+.)(\d+.\d+.\d+.\d+)\s*\
                    (?<EventSource>[A-Z]{2,})[\s\:]+(?<MessageType>\
                    (?:Interpreted\s*Telegram|Terminated\s*Scan).*)\s*\
                    (?:RTU\s*\:\s*(?<RTU>.*))?\s*(?:RTU\s*Address[\:\s]+\
                    (?<RTU_Address>.*))?\s*(?:Msg\s*Address[\:\s]+\
                    (?<MsgAddress>.*))?\s*(?:Value.*?[\:\s]+(?<Value>.*))?\s*\
                    (?<Function>[\w\s]+FC\d+\:.*)?\s*(?:Result.*?[\:\s]+\
                    (?<Result>.*))?\s*(?<Message>.*)?/

# Generic path to the folder with log files
define CFE_PATH     C:\ProgramData\Siemens Energy\SICAM PAS PQS\Logs\CFE

<Extension json>
    Module          xm_json
</Extension>

<Extension multiline>
    Module          xm_multiline
    HeaderLine      /^\d+.\d+.\d+.\d+.\d+.\d+.\d+/
</Extension>

<Input from_file>
    Module          im_file
    File            '%CFE_PATH%\WIN-5RU7GP5MI4VModbus *Interface_trc.txt'
    InputType       multiline
    <Exec>

        # Matching a file header to the regular expression
        if $raw_event =~ %REGEX_ITONE%
        {

            # Creating a timestamp
            $EventTime = parsedate($1 + $2);

            # Converting to JSON
            to_json();
        }

        # Discarding messages which do not match the regular expression
        else drop();

        # Replacing unwanted characters
        $raw_event =~ s/\\r\\n/ /g;
        $raw_event =~ s/\s{2,}/ /g;
    </Exec>
</Input>

Output sample in JSON format

{
  "EventReceivedTime": "2021-04-19T15:36:14.659229+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "EventSource": "PM",
  "Function": "",
  "Message": "!!! INVALID (HALF incomplete)!!!",
  "MessageType": "Interpreted Telegram - Received Read Holding Register FC03: (S=1, R=0)",
  "MsgAddress": "9",
  "RTU": "",
  "RTU_Address": "1 (LinkId=100001)",
  "Result": "",
  "Value": "0x0000h",
  "EventTime": "2021-04-19T15:36:14.612000+03:00"
}

The second type of Interpreted Telegram contains the following fields:

Loader
SignalType
RTU_Address_LinkId
DP_Address_Index
Value
Info
DSI_Value

Example 19. Processing the second type of Interpreted Telegrams

Input sample

2021-04-19 15:36:14.669  PM : Analog Input(INT):
                         RTU Address/LinkId: Device - 1 / 100001
                         DP - Address/Index: MeU32_7_FC4 - 8
                         Value(M_ME_InputRegister32U)= 262803367
                         Info: CONVERTED OK
                         DSI Value(u)=262803367

This configuration uses the im_file module to collect file-based logs and the multiline module to read multiline log records. To parse log records, the configuration compares each record to the REGEX_ITTWO regular expression. If a match occurs, new fields are created based on the named capturing groups. The parsedate() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime field.

The log record is then converted to JSON using the to_json() procedure from the xm_json module. The drop procedure discards records that do not match the REGEX_ITTWO regular expression.

nxlog.conf

# A regular expression for parsing log data
define REGEX_ITTWO  /(?x)^(\d+.\d+.\d+.)(\d+.\d+.\d+.\d+)\s*\
                    (?<EventSource>[A-Z]{2,})[\s\:]+\
                    (?<SignalType>[\w\s\(\)]+)[\:\s]*\
                    (?:RTU\s*Address\/LinkId[\:\s]+(?<RTU_AddressLinkId>.*))\s*\
                    (?:DP[\-\s]*Address\/Index[\:\s]+(?<DP_AdressIndex>.*))\s*\
                    (?:Value.*?[\=\s]+(?<Value>.*))\s*\
                    (?:Info[\:\s]+(?<Info>.*))\s*\
                    (?:DSI\s*Value.*?[\=\s]+(?<DSI_Value>.*))/

# Generic path to the folder with log files
define CFE_PATH     C:\ProgramData\Siemens Energy\SICAM PAS PQS\Logs\CFE

<Extension json>
    Module          xm_json
</Extension>

<Extension multiline>
    Module          xm_multiline
    HeaderLine      /^\d+.\d+.\d+.\d+.\d+.\d+.\d+/
</Extension>

<Input from_file>
    Module          im_file
    File            '%CFE_PATH%\WIN-5RU7GP5MI4VModbus *Interface_trc.txt'
    InputType       multiline
    <Exec>

        # Matching a file header to the regular expression
        if  $raw_event =~ %REGEX_ITTWO%
        {

            # Creating a timestamp
            $EventTime = parsedate($1 + $2);

            # Converting to JSON
            to_json();
        }

        # Discarding messages which do not match the regular expression
        else drop();

        # Replacing unwanted characters
        $raw_event =~ s/\\r\\n/ /g;
        $raw_event =~ s/\s{2,}/ /g;
    </Exec>
</Input>

Output sample in JSON format

{
  "EventReceivedTime": "2021-04-19T15:36:14.688848+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "DP_AdressIndex": "MeU32_7_FC4 - 8",
  "DSI_Value": "262803367",
  "EventSource": "PM",
  "Info": "CONVERTED OK ",
  "RTU_AddressLinkId": "Device - 1 / 100001",
  "SignalType": "Analog Input(INT)",
  "Value": "262803367",
  "EventTime": "2021-04-19T15:36:14.669000+03:00"
}

Normalized data

This log type contains information about scaled analog values. Each normalized event message contains the following fields:

NormalizedData
NormalizedValue
Source_NormalSource
NormalRange
ChangeOfValueCounter
CoT
AddCause
ToV
InitCat
Ident
Supplementary
Testbit
UserId
AppId
ProtData
DSI_Time
DSI_Time_Utc

Example 20. Processing normalized data

Input sample

2021-04-19 15:36:14.700  Normalized Data : DP_2_FC2 (typeid,type/address: 10003, dtNormInteger / 2) on RTU Device (ThreadId=904)
                         Normalized Integer :3 (signed), Validity : Valid(1)
                         Source/Normal Source : dsiTelemetered(1) / dsiTelemetered(1), Normal Range : dsiNormal(1), Change of Value Counter : 0
                           CoT: Spontaneous, AddCause: Positive, ToV: DoublePointInformation(2)
                           InitCat: IrrelevantInitiatorCategory(0), Ident: IdentificationNumberIrrelevant(0), Supplementary: SupplementaryInformationIrrelevant(0), 0, Testbit: dsiTESTFLAG_Zero(0)
                           UserId: 0, AppId: 0, ProtData: 0x00000000h
                         DSI Time: 2021-04-19 - 15:36:14,700258 (dsiTimeStampValid - dsiTimeQualityUnknown)
                         DSI Time: 2021-04-19 - 12:36:14,700258 (Utc) - dsiTimeStampValid - dsiTimeQualityUnknown

This configuration uses the im_file module to collect file-based logs and the multiline module to read multiline log records. To parse log records, the configuration compares each record to the REGEX_NORMAL regular expression. If a match occurs, new fields are created based on the named capturing groups. The parsedate() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime field.

The log record is then converted to JSON using the to_json() procedure from the xm_json module. The drop procedure discards records that do not match the REGEX_NORMAL regular expression.

nxlog.conf

# A regular expression for parsing log data
define REGEX_NORMAL /(?x)^(\d+.\d+.\d+.)(\d+.\d+.\d+.\d+)\s+\
                    (?:Normalized\s*Data[\s\:]*(?<NormalizedData>.*))[\,\s]*\
                    (?:Normalized\s*(?:Integer|Float)[\s\:]+\
                    (?<NormalizedValue>.*?)\,)\s*(?:Validity[\s\:]+\
                    (?<Validity>.*))\s*(?:Source\/Normal\s*Source[\s\:]+\
                    (?<Source_NormalSource>.*)\,)\s*(?:Normal\s*Range[\s\:]+\
                    (?<NormalRange>.*)\,)\s*\
                    (?:Change\s*of\s*Value\s*Counter[\s\:]+\
                    (?<ChangeOfValueCounter>.*))\s*(?:CoT[\s\:]+(?<CoT>.*)\,)\s*\
                    (?:AddCause[\s\:]+(?<AddCause>.*)\,)\s*\
                    (?:ToV[\s\:]+(?<ToV>.*))\s*(?:InitCat[\s\:]+\
                    (?<InitCat>.*)\,)?\s*(?:Ident[\s\:]+(?<Ident>.*)\,)\s*\
                    (?:Supplementary[\s\:]+(?<Supplementary>.*)\,)\s*\
                    (?:Testbit[\s\:]+(?<Testbit>.*))\s*(?:UserId[\s\:]+\
                    (?<UserId>.*)\,)\s*(?:AppId[\s\:]+(?<AppId>.*)\,)\s*\
                    (?:ProtData[\s\:]+(?<ProtData>.*))\s*\
                    (?:DSI\s*Time[\s\:]+(?<DSI_Time>.*))\s*\
                    (?:DSI\s*Time[\s\:]+(?<DSI_TimeUtc>.*))/

# Generic path to the folder with log files
define CFE_PATH     C:\ProgramData\Siemens Energy\SICAM PAS PQS\Logs\CFE

<Extension json>
    Module          xm_json
</Extension>

<Extension multiline>
    Module          xm_multiline
    HeaderLine      /^\d+.\d+.\d+.\d+.\d+.\d+.\d+/
</Extension>

<Input from_file>
    Module          im_file
    File            '%CFE_PATH%\WIN-5RU7GP5MI4VModbus *Interface_trc.txt'
    InputType       multiline
    <Exec>

        # Matching a file header to the regular expression
        if $raw_event =~ %REGEX_NORMAL%
        {

            # Creating a timestamp
            $EventTime = parsedate($1 + $2);

            # Converting to JSON
            to_json();
        }

        # Discarding messages which do not match the regular expression
        else drop();

        # Replacing unwanted characters
        $raw_event =~ s/\\r\\n/ /g;
        $raw_event =~ s/\s{2,}/ /g;
    </Exec>
</Input>

Output sample in JSON format

{
  "EventReceivedTime": "2021-04-19T15:36:14.735391+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "AddCause": "Positive",
  "AppId": "0",
  "ChangeOfValueCounter": "0",
  "CoT": "Spontaneous",
  "DSI_Time": "2021-04-19 - 15:36:14,700258 (dsiTimeStampValid - dsiTimeQualityUnknown)",
  "DSI_TimeUtc": "2021-04-19 - 12:36:14,700258 (Utc) - dsiTimeStampValid - dsiTimeQualityUnknown",
  "Ident": "IdentificationNumberIrrelevant(0)",
  "InitCat": "IrrelevantInitiatorCategory(0)",
  "NormalRange": "dsiNormal(1)",
  "NormalizedData": "DP_2_FC2 (typeid,type/address: 10003, dtNormInteger / 2) on RTU Device (ThreadId=904)",
  "NormalizedValue": "3 (signed)",
  "ProtData": "0x00000000h",
  "Source_NormalSource": "dsiTelemetered(1) / dsiTelemetered(1)",
  "Supplementary": "SupplementaryInformationIrrelevant(0), 0",
  "Testbit": "dsiTESTFLAG_Zero(0)",
  "ToV": "DoublePointInformation(2)",
  "UserId": "0",
  "Validity": "Valid(1)",
  "EventTime": "2021-04-19T15:36:14.700000+03:00"
}

Preprocessed data

NXLog Agent can read and manipulate two types of preprocessed logs:

Events before preprocessing
Preprocessed events

The first example shows how to process event messages that haven’t been preprocessed. Each message of this type contains the following fields:

DataBeforePreprocessing
DataEvent
Qualifier(s)
CFE Data
CoT
AddCause
ToV
InitCat
Ident
Supplementary
Testbit
UserId
AppId
ProtData
CFE Time

Example 21. Processing events before preprocessing

Input sample

2021-04-19 23:33:31.038  Data before preprocessing: 'Control center' / 'MeU32_7_FC3' (typeid/address: 10026/12) value forwarded to RCC 'Interface'
                         AnalogDataEvent INT: -1439418443 , Qualifier(s) :
                         CFE Data: cfeValid,  cfeTelemetered,  cfeInitVal,  cfeNormal
                         CFE AddInfo: CoT: cfeCOT_Spontaneous, AddCause: cfeAddC_Positive, ToV: cfeTOV_MeasuredValueStandardized(29)
                           InitCat: cfeIC_IrrelevantInitiatorCategory(0), Ident: cfeIdNo_IdentificationNumberIrrelevant(0), 0, Supplementary: cfeSI_SupplementaryInformationIrrelevant, 0, Testbit: cfeTESTFLAG_Zero(0)
                           UserId: 0, AppId: 0, ProtData: 0x00000000h
                         CFE Time: 2021-04-19 - 20:33:31,034460 (Utc) - dsiTimeStampValid - dsiTimeQualityUnknown

This configuration uses the im_file module to collect file-based logs and the multiline module to read multiline log records. To parse log records, the configuration compares each record to the REGEX_PBEF regular expression. If a match occurs, new fields are created based on the named capturing groups. The parsedate() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime field.

The log record is then converted to JSON using the to_json() procedure from the xm_json module. The drop procedure discards records that do not match the REGEX_PBEF regular expression.

nxlog.conf

# A regular expression for parsing log data
define REGEX_PBEF   /(?x)^(\d+.\d+.\d+.)(\d+.\d+.\d+.\d+)\s*\
                    (?:Data\sbefore\spreprocessing\:\s*\
                    (?<DataBeforePreprocessing>.*))\s*\
                    (?<DataEvent>\w*DataEvent.*)\,\s*\
                    (?:Qualifier\(s\)\s*\:(?<Qualifier_s>.*?))\s*\
                    (?:CFE\s*Data\:\s*(?<CFE_Data>.*))\s*(?:CFE\s*AddInfo\:\s*)\
                    (?:CoT\:\s*(?<CoT>.*)\,)\s*\
                    (?:AddCause\:\s*(?<AddCause>.*)\,)\s*(?:ToV\:\s*(?<ToV>.*))\s*\
                    (?:InitCat\:\s*(?<InitCat>.*)\,)\s*\
                    (?:Ident\:\s*(?<Ident>.*)\,)\s*\
                    (?:Supplementary\:\s*(?<Supplementary>.*)\,)\s*\
                    (?:Testbit\:\s*(?<Testbit>.*))\s*\
                    (?:UserId\:\s*(?<UserId>.*)\,)\s*\
                    (?:AppId\:\s*(?<AppId>.*)\,)\s*\
                    (?:ProtData\:\s*(?<ProtData>.*))\s*\
                    (?:CFE\s*Time\:\s*(?<CFE_Time>.*))/

# Generic path to the folder with log files
define CFE_PATH     C:\ProgramData\Siemens Energy\SICAM PAS PQS\Logs\CFE

<Extension json>
    Module          xm_json
</Extension>

<Extension multiline>
    Module          xm_multiline
    HeaderLine      /^\d+.\d+.\d+.\d+.\d+.\d+.\d+/
</Extension>

<Input from_file>
    Module          im_file
    File            '%CFE_PATH%\WIN-5RU7GP5MI4VModbus *Interface_trc.txt'
    InputType       multiline
    <Exec>

        # Matching a file header to the regular expression
        if  $raw_event =~ %REGEX_PBEF%
        {

            # Creating a timestamp
            $EventTime = parsedate($1 + $2);

            # Converting to JSON
            to_json();
        }

        # Discarding messages which do not match the regular expression
        else drop();

        # Replacing unwanted characters
        $raw_event =~ s/\\r\\n/ /g;
        $raw_event =~ s/\s{2,}/ /g;
    </Exec>
</Input>

Output sample in JSON format

{
  "EventReceivedTime": "2021-04-19T23:37:39.883101+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "AddCause": "cfeAddC_Positive",
  "AppId": "0",
  "CFE_Data": "cfeValid, cfeTelemetered, cfeInitVal, cfeNormal",
  "CFE_Time": "2021-04-19 - 20:33:31,034460 (Utc) - dsiTimeStampValid - dsiTimeQualityUnknown",
  "CoT": "cfeCOT_Spontaneous",
  "DataBeforePreprocessing": "'Control center' / 'MeU32_7_FC3' (typeid/address: 10026/12) value forwarded to RCC 'Interface'",
  "DataEvent": "AnalogDataEvent INT: -1439418443 ",
  "Ident": "cfeIdNo_IdentificationNumberIrrelevant(0), 0",
  "InitCat": "cfeIC_IrrelevantInitiatorCategory(0)",
  "ProtData": "0x00000000h",
  "Qualifier_s": "",
  "Supplementary": "cfeSI_SupplementaryInformationIrrelevant, 0",
  "Testbit": "cfeTESTFLAG_Zero(0)",
  "ToV": "cfeTOV_MeasuredValueStandardized(29)",
  "UserId": "0",
  "EventTime": "2021-04-19T23:33:31.038000+03:00"
}

The next example shows how to process preprocessed events. Each message of this type contains the following fields:

PreprocessedData
DataEvent
DSI Quality
CFE Quality
DSI Time
CFE Time
CoT
AddCause
ToV
InitCat
Ident
Supplementary
Testbit
UserId
AppId
ProtData

Example 22. Processing preprocessed events

Input sample

2021-04-19 15:36:14.669  Preprocessed Data to [RTI]: --  MeU32_7_FC4 -- on RTU Device
                         AnalogDataEvent INT: 262803367 (unsigned),
                         DSI Quality: Valid,  dsiTelemetered,  dsiTelemetered,  dsiNormal;
                         CFE Quality: cfeValid,  cfeTelemetered,  cfeTelemetered,  cfeNormal;
                         DSI Time: 2021-04-19 - 12:36:14,669993 (Utc) - dsiTimeStampValid - dsiTimeQualityUnknown
                         CFE Time: 2021-04-19 - 12:36:14,669993 (Utc) - dsiTimeStampValid - dsiTimeQualityUnknown
                         CFE AddInfo:   CoT: cfeCOT_Spontaneous, AddCause: cfeAddC_Positive, ToV: cfeTOV_MeasuredValueStandardized(29)
                           InitCat: cfeIC_IrrelevantInitiatorCategory(0), Ident: cfeIdNo_IdentificationNumberIrrelevant(0), Supplementary: cfeSI_SupplementaryInformationIrrelevant, 0, Testbit: cfeTESTFLAG_Zero(0)
                           UserId: 0, AppId: 0, ProtData: 0x00000000h

This configuration uses the im_file module to collect file-based logs and the multiline module to read multiline log records. To parse log records, the configuration compares each record to the REGEX_PREP regular expression. If a match occurs, new fields are created based on the named capturing groups. The parsedate() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime field.

The log record is then converted to JSON using the to_json() procedure from the xm_json module. The drop procedure discards records that do not match the REGEX_PREP regular expression.

nxlog.conf

# A regular expression for parsing log data
define REGEX_PREP   /(?x)^(\d+.\d+.\d+.)(\d+.\d+.\d+.\d+)\s*\
                    (?:Preprocessed\s*Data.*?\:\s*(?<PreprocessedData>.*))?\s*\
                    (?<DataEvent>\w+DataEvent.*)\,\s*\
                    (?:DSI\s*Quality[\s\:]+(?<DSI_Quality>.*)\;)\s*\
                    (?:CFE\s*Quality[\s\:]+(?<CFE_Quality>.*)\;)\s*\
                    (?:DSI\s*Time[\s\:]+(?<DSI_Time>.*))\s*\
                    (?:CFE\s*Time[\s\:]+(?<CFE_Time>.*))\s*\
                    (?:CFE\s*AddInfo[\s\:]+)(?:CoT[\s\:]+(?<CoT>.*)\,)\s*\
                    (?:AddCause[\s\:]+(?<AddCause>.*)\,)\s*\
                    (?:ToV[\s\:]+(?<ToV>.*))\s*\
                    (?:InitCat[\s\:]+(?<InitCat>.*)\,)\s*\
                    (?:Ident[\s\:]+(?<Ident>.*)\,)\s*\
                    (?:Supplementary[\s\:]+(?<Supplementary>.*)\,)\s*\
                    (?:Testbit[\s\:]+(?<Testbit>.*))\s*\
                    (?:UserId[\s\:]+(?<UserId>.*)\,)\s*\
                    (?:AppId[\s\:]+(?<AppId>.*)\,)\s*\
                    (?:ProtData[\s\:]+(?<ProtData>.*))/

# Generic path to the folder with log files
define CFE_PATH     C:\ProgramData\Siemens Energy\SICAM PAS PQS\Logs\CFE

<Extension json>
    Module          xm_json
</Extension>

<Extension multiline>
    Module          xm_multiline
    HeaderLine      /^\d+.\d+.\d+.\d+.\d+.\d+.\d+/
</Extension>

<Input from_file>
    Module          im_file
    File            '%CFE_PATH%\WIN-5RU7GP5MI4VModbus *Interface_trc.txt'
    InputType       multiline
    <Exec>

        # Matching a file header to the regular expression
        if  $raw_event =~ %REGEX_PREP%
        {

            # Creating a timestamp
            $EventTime = parsedate($1 + $2);

            # Converting to JSON
            to_json();
        }

        # Discarding messages which do not match the regular expression
        else drop();

        # Replacing unwanted characters
        $raw_event =~ s/\\r\\n/ /g;
        $raw_event =~ s/\s{2,}/ /g;
    </Exec>
</Input>

Output sample in JSON format

{
  "EventReceivedTime": "2021-04-19T15:36:14.693415+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "AddCause": "cfeAddC_Positive",
  "AppId": "0",
  "CFE_Quality": "cfeValid, cfeTelemetered, cfeTelemetered, cfeNormal",
  "CFE_Time": "2021-04-19 - 12:36:14,669993 (Utc) - dsiTimeStampValid - dsiTimeQualityUnknown",
  "CoT": "cfeCOT_Spontaneous",
  "DSI_Quality": "Valid, dsiTelemetered, dsiTelemetered, dsiNormal",
  "DSI_Time": "2021-04-19 - 12:36:14,669993 (Utc) - dsiTimeStampValid - dsiTimeQualityUnknown",
  "DataEvent": "AnalogDataEvent INT: 262803367 (unsigned)",
  "Ident": "cfeIdNo_IdentificationNumberIrrelevant(0)",
  "InitCat": "cfeIC_IrrelevantInitiatorCategory(0)",
  "PreprocessedData": "-- MeU32_7_FC4 -- on RTU Device",
  "ProtData": "0x00000000h",
  "Supplementary": "cfeSI_SupplementaryInformationIrrelevant, 0",
  "Testbit": "cfeTESTFLAG_Zero(0)",
  "ToV": "cfeTOV_MeasuredValueStandardized(29)",
  "UserId": "0",
  "EventTime": "2021-04-19T15:36:14.669000+03:00"
}

Statistics events

Statistics events contain the following fields:

Statistics
Accumulators
Analog
Digitals
Fleetings
Positions
Bitpattern
Total

Example 23. Processing statistics events

Input sample

2021-04-19 23:25:30.985  Statistics: Received Info Objects since 2021-04-19 23:19:10,961
                           Accumulators :            0
                           Analog       :          414
                           Digitals     :          690
                           Fleetings    :            0
                           Positions    :            0
                           Bitpattern   :            0
                           Total        :         1104

This configuration uses the im_file module to collect file-based logs and the multiline module to read multiline log records. To parse log records, the configuration compares each record to the REGEX_STAT regular expression. If a match occurs, new fields are created based on the named capturing groups. The parsedate() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime field.

The log record is then converted to JSON using the to_json() procedure from the xm_json module. The drop procedure discards records that do not match the REGEX_STAT regular expression.

nxlog.conf

# A regular expression for parsing log data
define REGEX_STAT   /(?x)^(\d+.\d+.\d+.)(\d+.\d+.\d+.\d+)\s*\
                    (?:Statistics[\s\:]+(?<Statistics>.*))\s*\
                    (?:Accumulators[\s\:]+(?<Accumulators>.*))\s*\
                    (?:Analog[\s\:]+(?<Analog>.*))\s*\
                    (?:Digitals[\s\:]+(?<Digitals>.*))\s*\
                    (?:Fleetings[\s\:]+(?<Fleetings>.*))\s*\
                    (?:Positions[\s\:]+(?<Positions>.*))\s*\
                    (?:Bitpattern[\s\:]+(?<Bitpattern>.*))\s*\
                    (?:Total[\s\:]+(?<Total>.*))/

# Generic path to the folder with log files
define CFE_PATH     C:\ProgramData\Siemens Energy\SICAM PAS PQS\Logs\CFE

<Extension json>
    Module          xm_json
</Extension>

<Extension multiline>
    Module          xm_multiline
    HeaderLine      /^\d+.\d+.\d+.\d+.\d+.\d+.\d+/
</Extension>

<Input from_file>
    Module          im_file
    File            '%CFE_PATH%\WIN-5RU7GP5MI4VModbus *Interface_trc.txt'
    InputType       multiline
    PollInterval    0.01
    ReadFromLast    TRUE
    <Exec>

        # Matching a file header to the regular expression
        if  $raw_event =~ %REGEX_STAT%
        {

            # Creating a timestamp
            $EventTime = parsedate($1 + $2);

            # Converting to JSON
            to_json();
        }

        # Discarding messages which do not match the regular expression
        else drop();

        # Replacing unwanted characters
        $raw_event =~ s/\\r\\n/ /g;
        $raw_event =~ s/\s{2,}/ /g;
    </Exec>
</Input>

Output sample in JSON format

{
  "EventReceivedTime": "2021-04-19T23:25:32.005917+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Accumulators": "0",
  "Analog": "414",
  "Bitpattern": "0",
  "Digitals": "690",
  "Fleetings": "0",
  "Positions": "0",
  "Statistics": "Received Info Objects since 2021-04-19 23:19:10,961",
  "Total": "1104",
  "EventTime": "2021-04-19T23:25:30.985000+03:00"
}

Systemaccu events

Example 24. Processing Systemaccu events

Input sample

2021-04-19 22:13:17.094  PM : Systemaccu Interface=TESTInterface(502):
                                   Last StopError Code=0(1)
                                   Last Error Code=0(1)
                                    =>Error Uknown (Grp= 0)
                                   Sent Messages=215(0)
                                   Received Messages=215(0)
                                   Receive Errors=0(0)
                                   Timeouts=0(0)
                                   Scan OverRuns=0(1)
                                   Scan Requests=215(1)
                                   Command Requests=0(1)
                                   Received Repeated Messages=0(3)
                                   UnexpectedMessages=0(3)
                                   AuthorizationFailures=0(3)
                                   AuthenticationFailures=0(3)
                                   ReplyTimeouts=0(3)
                                   RekeysDueToAuthenticationFailure=0(3)
                                   TotalMessagesSent=0(3)
                                   TotalMessagesReceived=0(3)
                                   CriticalMessagesSent=0(3)
                                   CriticalMessagesReceived=0(3)
                                   DiscardedMessages=0(3)
                                   ErrorMessagesSent=0(3)
                                   ErrorMessagesRxed=0(3)
                                   SuccessfulAuthentications=0(3)
                                   SessionKeyChanges=0(3)
                                   FailedSessionKeyChanges=0(3)
                                   UpdateKeyChanges=0(3)
                                   FailedUpdateKeyChanges=0(3)
                                   RekeysDueToRestarts=0(3)
                                   Error Data Length=0(1)
                                   Failure Interface=0(1)
                                   Failure Device=0(3)
                                   Invalid Message=0(1)
                                   Error Msg Checksum=0(1)
                                   Error Clock=0(3)
                                   Error Block Sync=0(3)
                                   Error CTS=0(3)
                                   Error Framing=0(3)
                                   Error Overrun=0(3)
                                   Error InputBuf Overflow=0(3)
                                   Error Parity=0(3)
                                   Error OutputBuf Overflow=0(3)
                                   Error DSR=0(3)
                                   Error DCD=0(3)
                                   Error Control=0(1)
                                   Error SW Logic=0(1)
                                   Error Timeout=0(1)
                                   Error Not Supported=0(1)
                                   Error Configuration=0(1)

This configuration uses the im_file module to collect file-based logs and the multiline module to read multiline log records. To parse log records, the configuration compares each record to the REGEX_SA regular expression. If a match occurs, new fields are created based on the named capturing groups. The parsedate() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime field.

The log record is then converted to JSON using the to_json() procedure from the xm_json module. The drop procedure discards records that do not match the REGEX_SA regular expression.

nxlog.conf

# A regular expression for parsing log data
define REGEX_SA     /(?x)^(\d+.\d+.\d+.)(\d+.\d+.\d+.\d+)\s*\
                    (?<EventSource>[A-Z]{2,})[\s\:]+Systemaccu\s*\
                    (?:RTU\=(?<RTU>.*))?[\:\s]*\
                    (?:Interface\=(?<Interface>.*))?[\:\s]*\
                    (?:Last\s*StopError\s*Code\=(?<LastStopErrorCode>.*))\s*\
                    (?:Last\s*Error\s*Code\=(?<LastErrorCode>.*))\s*\
                    (?:.*?Error\s*Uknown\s*(?<ErrorUknown>.*))\s*\
                    (?:Sent\s*Messages\=(?<SentMessages>.*))\s*\
                    (?:Received\s*Messages\=(?<ReceivedMessages>.*))\s*\
                    (?:Receive\s*Errors\=(?<ReceiveErrors>.*))\s*\
                    (?:Timeouts\=(?<Timeouts>.*))\s*\
                    (?:Scan\s*OverRuns\=(?<ScanOverRuns>.*))\s*\
                    (?:Scan\s*Requests\=(?<ScanRequests>.*))\s*\
                    (?:Command\s*Requests\=(?<CommandRequests>.*))\s*\
                    (?:Received\s*Repeated\s*Messages\=\
                    (?<ReceivedRepeatedMessages>.*))\s*\
                    (?:UnexpectedMessages\=(?<UnexpectedMessages>.*))\s*\
                    (?:AuthorizationFailures\=(?<AuthorizationFailures>.*))\s*\
                    (?:AuthenticationFailures\=(?<AuthenticationFailures>.*))\s*\
                    (?:ReplyTimeouts\=(?<ReplyTimeouts>.*))\s*\
                    (?:RekeysDueToAuthenticationFailure\=\
                    (?<RekeysDueToAuthenticationFailure>.*))\s*\
                    (?:TotalMessagesSent\=(?<TotalMessagesSent>.*))\s*\
                    (?:TotalMessagesReceived\=(?<TotalMessagesReceived>.*))\s*\
                    (?:CriticalMessagesSent\=(?<CriticalMessagesSent>.*))\s*\
                    (?:CriticalMessagesReceived\=\
                    (?<CriticalMessagesReceived>.*))?\s*\
                    (?:DiscardedMessages\=(?<DiscardedMessages>.*))?\s*\
                    (?:ErrorMessagesSent\=(?<ErrorMessagesSent>.*))\s*\
                    (?:ErrorMessagesRxed\=(?<ErrorMessagesRxed>.*))\s*\
                    (?:SuccessfulAuthentications\=\
                    (?<SuccessfulAuthentications>.*))\s*\
                    (?:SessionKeyChanges\=(?<SessionKeyChanges>.*))\s*\
                    (?:FailedSessionKeyChanges\=(?<FailedSessionKeyChanges>.*))\s*\
                    (?:UpdateKeyChanges\=(?<UpdateKeyChanges>.*))?\s*\
                    (?:FailedUpdateKeyChanges\=(?<FailedUpdateKeyChanges>.*))\s*\
                    (?:RekeysDueToRestarts\=(?<RekeysDueToRestarts>.*))\s*\
                    (?:Error\s*Data\s*Length\=(?<ErrorDataLength>.*))\s*\
                    (?:Failure\s*Interface\=(?<FailureInterface>.*))\s*\
                    (?:Failure\s*Device\=(?<FailureDevice>.*))\s*\
                    (?:Invalid\s*Message\=(?<InvalidMessage>.*))\s*\
                    (?:Error\s*Msg\s*Checksum\=(?<ErrorMsgChecksum>.*))\s*\
                    (?:Error\s*Clock\=(?<ErrorClock>.*))\s*\
                    (?:Error\s*Block\s*Sync\=(?<ErrorBlockSync>.*))\s*\
                    (?:Error\s*CTS\=(?<ErrorCTS>.*))\s*\
                    (?:Error\s*Framing\=(?<ErrorFraming>.*))\s*\
                    (?:Error\s*Overrun\=(?<ErrorOverrun>.*))\s*\
                    (?:Error\s*InputBuf\s*Overflow\=\
                    (?<ErrorInputBufOverflow>.*))\s*\
                    (?:Error\s*Parity\=(?<ErrorParity>.*))\s*\
                    (?:Error\s*OutputBuf\s*Overflow\=\
                    (?<ErrorOutputBufOverflow>.*))\s*\
                    (?:Error\s*DSR\=(?<ErrorDSR>.*))\s*\
                    (?:Error\s*DCD\=(?<ErrorDCD>.*))\s*\
                    (?:Error\s*Control\=(?<ErrorControl>.*))\s*\
                    (?:Error\s*SW\s*Logic\=(?<ErrorSWLogic>.*))\s*\
                    (?:Error\s*Timeout\=(?<ErrorTimeout>.*))\s*\
                    (?:Error\s*Not\s*Supported\=(?<ErrorNotSupported>.*))\s*\
                    (?:Error\s*Configuration\=(?<ErrorConfiguration>.*))/

# Generic path to the folder with log files
define CFE_PATH     C:\ProgramData\Siemens Energy\SICAM PAS PQS\Logs\CFE

<Extension json>
    Module          xm_json
</Extension>

<Extension multiline>
    Module          xm_multiline
    HeaderLine      /^\d+.\d+.\d+.\d+.\d+.\d+.\d+/
</Extension>

<Input from_file>
    Module          im_file
    File            '%CFE_PATH%\WIN-5RU7GP5MI4VModbus *Interface_trc.txt'
    InputType       multiline
    <Exec>

        # Matching a file header to the regular expression
        if $raw_event =~ %REGEX_SA%
        {

            # Creating a timestamp
            $EventTime = parsedate($1 + $2);

            # Converting to JSON
            to_json();
        }

        # Discarding messages which do not match the regular expression
        else drop();

        # Replacing unwanted characters
        $raw_event =~ s/\\r\\n/ /g;
        $raw_event =~ s/\s{2,}/ /g;
    </Exec>
</Input>

Output sample in JSON format

{
  "EventReceivedTime": "2021-04-19T22:13:17.521540+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "AuthenticationFailures": "0(3)",
  "AuthorizationFailures": "0(3)",
  "CommandRequests": "0(1)",
  "CriticalMessagesReceived": "0(3)",
  "CriticalMessagesSent": "0(3)",
  "DiscardedMessages": "0(3)",
  "ErrorBlockSync": "0(3)",
  "ErrorCTS": "0(3)",
  "ErrorClock": "0(3)",
  "ErrorConfiguration": "0(1)",
  "ErrorControl": "0(1)",
  "ErrorDCD": "0(3)",
  "ErrorDSR": "0(3)",
  "ErrorDataLength": "0(1)",
  "ErrorFraming": "0(3)",
  "ErrorInputBufOverflow": "0(3)",
  "ErrorMessagesRxed": "0(3)",
  "ErrorMessagesSent": "0(3)",
  "ErrorMsgChecksum": "0(1)",
  "ErrorNotSupported": "0(1)",
  "ErrorOutputBufOverflow": "0(3)",
  "ErrorOverrun": "0(3)",
  "ErrorParity": "0(3)",
  "ErrorSWLogic": "0(1)",
  "ErrorTimeout": "0(1)",
  "ErrorUknown": "(Grp= 0)",
  "EventSource": "PM",
  "FailedSessionKeyChanges": "0(3)",
  "FailedUpdateKeyChanges": "0(3)",
  "FailureDevice": "0(3)",
  "FailureInterface": "0(1)",
  "Interface": "TESTInterface(502):",
  "InvalidMessage": "0(1)",
  "LastErrorCode": "0(1)",
  "LastStopErrorCode": "0(1)",
  "RTU": "",
  "ReceiveErrors": "0(0)",
  "ReceivedMessages": "215(0)",
  "ReceivedRepeatedMessages": "0(3)",
  "RekeysDueToAuthenticationFailure": "0(3)",
  "RekeysDueToRestarts": "0(3)",
  "ReplyTimeouts": "0(3)",
  "ScanOverRuns": "0(1)",
  "ScanRequests": "215(1)",
  "SentMessages": "215(0)",
  "SessionKeyChanges": "0(3)",
  "SuccessfulAuthentications": "0(3)",
  "Timeouts": "0(0)",
  "TotalMessagesReceived": "0(3)",
  "TotalMessagesSent": "0(3)",
  "UnexpectedMessages": "0(3)",
  "UpdateKeyChanges": "0(3)",
  "EventTime": "2021-04-19T22:13:17.094000+03:00"
}

Miscellaneous events

Miscellaneous event logs contain the following fields:

Tid
Severity
EventSource
Process
Properties

Some log records may not contain the first four fields. In such cases, only the Properties field will be available.

Example 25. Processing miscellaneous events

Input sample

2021-04-19 22:37:44.238  (Tid: 5088)[INF]TAD: CCfeTadConnectionTcp::StartSend WSASend with OvP(TcpSend, BufLen[0]=58/[1]=30, StartPos=0) HdrCpy: TaDHeader: (Len=82, Seq=98924, IF) GUID:3e995751-af75-415d-abd9-5d991e2b48be, AddrPL:-:-, Info, TrCtrl=c0, t:2021-4-19 22:37:38,845,
                          Msg: TadMsg(Ref=2) TaDHeader: (Len=82, Seq=98924, IF) GUID:3e995751-af75-415d-abd9-5d991e2b48be, AddrPL:-:-, Info, TrCtrl=c0, t:2021-4-19 22:37:38,845,
                          Txt(s=30): GPT: GPT Setting Last Error 0

This configuration uses the im_file module to collect file-based logs and the multiline module to read multiline log records. To parse log records, the configuration compares each record to the REGEX_MISC regular expression. If a match occurs, new fields are created based on the named capturing groups. The parsedate() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime field.

The log record is then converted to JSON using the to_json() procedure from the xm_json module. The drop procedure discards records that do not match the REGEX_MISC regular expression.

nxlog.conf

# A regular expression for parsing log data
define REGEX_MISC   /(?x)^(?(?=.*\s+(?:Interpreted\sTelegram|Analog\sInput\
                    \(INT\)\:|Terminated\sScan|Normalized\sData|Preprocessed\sData\
                    |Data\sbefore\spreprocessing\:|Byte\sStream|\w+\:{2}\w+\
                    \son\sdevice|Statistics\:|Systemaccu)\s+)\
                    |(\d+.\d+.\d+.)(\d+.\d+.\d+.\d+)[\s\:\-\<\>]*\
                    (?:\(Tid[\:\s]+(?<Tid>.*?)\))?\s*(?:\[(?<Severity>.*?)\])?\s*\
                    (?<EventSource>[A-Z]+(?=[\s\:]+)|data|error|void)?[\s\:\-\>\<]*\
                    (?<Process>\w+(?:\(.*?\))?(?:\[.*?\])?\:{2}\w+\
                    (?:\(.*?\))?(?:\[.*?\])?)?[\s\:\-]*\
                    (?<Properties>[\s\S]*)?)$/

# Generic path to the folder with log files
define CFE_PATH     C:\ProgramData\Siemens Energy\SICAM PAS PQS\Logs\CFE

<Extension json>
    Module          xm_json
</Extension>

<Extension multiline>
    Module          xm_multiline
    HeaderLine      /^\d+.\d+.\d+.\d+.\d+.\d+.\d+/
</Extension>

<Input from_file>
    Module          im_file
    File            '%CFE_PATH%\WIN-5RU7GP5MI4VModbus *Interface_trc.txt'
    InputType       multiline
    <Exec>

        # Matching a file header to the regular expression
        if  $raw_event =~ %REGEX_MISC%
        {

            # Creating a timestamp
            $EventTime = parsedate($1 + $2);

            # Converting to JSON
            to_json();
        }

        # Discarding messages which do not match the regular expression
        else drop();

        # Replacing unwanted characters
        $raw_event =~ s/\\r\\n/ /g;
        $raw_event =~ s/\s{2,}/ /g;
    </Exec>
</Input>

Output sample in JSON format

{
  "EventReceivedTime": "2021-04-19T22:37:46.010802+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "EventSource": "TAD",
  "Process": "CCfeTadConnectionTcp::StartSend",
  "Properties": "WSASend with OvP(TcpSend, BufLen[0]=58/[1]=30, StartPos=0) HdrCpy: TaDHeader: (Len=82, Seq=98924, IF) GUID:3e995751-af75-415d-abd9-5d991e2b48be, AddrPL:-:-, Info, TrCtrl=c0, t:2021-4-19 22:37:38,845, Msg: TadMsg(Ref=2) TaDHeader: (Len=82, Seq=98924, IF) GUID:3e995751-af75-415d-abd9-5d991e2b48be, AddrPL:-:-, Info, TrCtrl=c0, t:2021-4-19 22:37:38,845, Txt(s=30): GPT: GPT Setting Last Error 0 ",
  "Severity": "INF",
  "Tid": "5088",
  "EventTime": "2021-04-19T22:37:44.238000+03:00"
}

IEC 60870-5-104

This section explains how to configure NXLog Agent to read and process events related to the IEC 60870-5-104 protocol that are stored in the SICAM PAS/PQS communication trace log.

This section describes how to process byte stream and Interpreted Telegrams events.

Byte stream events

Byte stream messages related to the IEC 60870-5-104 protocol contain the following fields:

Process
DeviceName
Data

NXLog Agent can read and process these events as shown in the example below.

Example 26. Processing IEC 60870-5-104 byte stream events

Input sample

2021-04-21 09:34:05.911  clTE::read on device Device IEC104:
                         68 12 72 00 04 00 0F 01 03 00 05 0D 58 1B 00 BE
                         02 00 00 18

This configuration uses the im_file module to collect file-based logs and the multiline module to read multiline log records. To parse log records, the configuration compares each record to the REGEX_IECBS regular expression. If a match occurs, new fields are created based on the named capturing groups. The parsedate() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime field.

The log record is then converted to JSON using the to_json() procedure from the xm_json module. The drop procedure discards records that do not match the REGEX_IECBS regular expression.

nxlog.conf

# A regular expression for parsing log data
define REGEX_IECBS  /(?x)^(\d+.\d+.\d+.)(\d+.\d+.\d+.\d+)\s*\
                    (?<Process>\w+\:{2}\w+\s*on\s*device)\s*\
                    (?<DeviceName>.*?)\:\s*(?<Data>[\d\w\s]*)/

# Generic path to the folder with log files
define CFE_PATH     C:\ProgramData\Siemens Energy\SICAM PAS PQS\Logs\CFE

<Extension json>
    Module          xm_json
</Extension>

<Extension multiline>
    Module          xm_multiline
    HeaderLine      /^\d+.\d+.\d+.\d+.\d+.\d+.\d+/
</Extension>

<Input from_file>
    Module          im_file
    File            '%CFE_PATH%\WIN-5RU7GP5MI4VModbus *Interface_trc.txt'
    InputType       multiline
    <Exec>

        # Matching a file header to the regular expression
        if  $raw_event =~ %REGEX_IECBS%
        {
            # Creating a timestamp
            $EventTime = parsedate($1 + $2);

            # Converting to JSON
            to_json();
        }

        # Discarding messages which do not match the regular expression
        else drop();

        # Replacing unwanted characters
        $raw_event =~ s/\\r\\n/ /g;
        $raw_event =~ s/\s{2,}/ /g;
    </Exec>
</Input>

Output sample in JSON format

{
  "EventReceivedTime": "2021-04-21T09:34:05.936732+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Data": "68 12 72 00 04 00 0F 01 03 00 05 0D 58 1B 00 BE 02 00 00 18 ",
  "DeviceName": "Device IEC104",
  "Process": "clTE::read on device",
  "EventTime": "2021-04-21T09:34:05.911000+03:00"
}

Interpreted Telegrams

Each Interpreted Telegram contains the following fields:

MessageType
CommonAddress
LinkAddress
ObjectAddress
Type
CauseOfTransmission
StructuredQualifier
Value
SequenceNo
Qualifiers
Error

NXLog Agent can parse and process Interpreted Telegrams using the following configuration.

Example 27. Processing IEC 60870-5-104 Interpreted Telegrams events

Input sample

2021-04-21 09:34:05.911  Interpreted Telegram Received info object:
                         Common Address: 3333
                         Link Address: 0
                         Object Address: 7000
                         Type: Integrated totals (15)
                         Cause of Transmission: Spontaneously(3)
                         Structured Qualifier: false
                         Value : 702, Sequence No.: 24 Qualifiers:
                         Error : The received info object doesn't match with a configured object address, info type or common address.

This configuration uses the im_file module to collect file-based logs and the multiline module to read multiline log records. To parse log records, the configuration compares each record to the REGEX_IECIT regular expression. If a match occurs, new fields are created based on the named capturing groups. The parsedate() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime field.

The log record is then converted to JSON using the to_json() procedure from the xm_json module. The drop procedure discards records that do not match the REGEX_IECIT regular expression.

nxlog.conf

# A regular expression for parsing log data
define REGEX_IECIT  /(?x)^(\d+.\d+.\d+.)(\d+.\d+.\d+.\d+)\s*\
                    (?<MessageType>Interpreted\s*Telegram.*)?\:\s*\
                    (?:Common\s*Address[\:\s]+(?<CommonAddress>.*))\s*\
                    (?:Link\s*Address[\:\s]+(?<LinkAddress>.*))\s*\
                    (?:Object\s*Address[\:\s]+(?<ObjectAddress>.*))\s*\
                    (?:Type[\:\s]+(?<Type>.*))\s*\
                    (?:Cause\s*of\s*Transmission[\:\s]+\
                    (?<CauseOfTransmission>.*))\s*(?:Structured\s*Qualifier[\:\s]+\
                    (?<StructuredQualifier>.*))?\s*(?:Value[\:\s]+\
                    (?<Value>.*?))\,\s*(?:Sequence\s*No\.[\:\s]+\
                    (?<SequenceNo>.*?))\s*(?:Qualifiers\:\s(?<Qualifiers>.*))\s*\
                    (?:Error[\:\s]+(?<Error>.*))?/

# Generic path to the folder with log files
define CFE_PATH     C:\ProgramData\Siemens Energy\SICAM PAS PQS\Logs\CFE

<Extension json>
    Module          xm_json
</Extension>

<Extension multiline>
    Module          xm_multiline
    HeaderLine      /^\d+.\d+.\d+.\d+.\d+.\d+.\d+/
</Extension>

<Input from_file>
    Module          im_file
    File            '%CFE_PATH%\WIN-5RU7GP5MI4VModbus *Interface_trc.txt'
    InputType       multiline
    <Exec>

        # Matching a file header to the regular expression
        if  $raw_event =~ %REGEX_IECIT%
        {

            # Creating a timestamp
            $EventTime = parsedate($1 + $2);

            # Converting to JSON
            to_json();
        }

        # Discarding messages which do not match the regular expression
        else drop();

        # Replacing unwanted characters
        $raw_event =~ s/\\r\\n/ /g;
        $raw_event =~ s/\s{2,}/ /g;
    </Exec>
</Input>

Output sample in JSON format

{
  "EventReceivedTime": "2021-04-21T09:34:06.924008+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "CauseOfTransmission": "Spontaneously(3) ",
  "CommonAddress": "3333",
  "Error": "The received info object doesn't match with a configured object address, info type or common address.",
  "MessageType": "Interpreted Telegram Received info object",
  "ObjectAddress": "7000",
  "Qualifiers": "",
  "SequenceNo": "24",
  "StructuredQualifier": "false",
  "Type": "Integrated totals (15)",
  "Value": "702",
  "LinkAddress": "0",
  "EventTime": "2021-04-21T09:34:05.911000+03:00"
}

DNP3

DNP3 logs produced by SICAM PAS/PQS contain the following fields:

DP_AddressIndex
EventSource
Interpreted_Telegram
LinkAddress
Status
Time
Value

The following example explains how to process protocol-specific Interpreted Telegrams related to DNP3 using NXLog Agent.

Example 28. Processing protocol-specific events of DNP3

Input sample

2021-04-21 14:39:57.455  PM : Interpreted Telegram - Received Event Analog Input:
                         Link Address: 4 (LinkId=100004)
                         DP - Address/Index: 1
                         Value(sf): 14481.8
                         Status: Online
                         Time(1970): 1619015996sec, 322ms, present, valid

This configuration uses the im_file module to collect file-based logs and the multiline module to read multiline log records. To parse log records, the configuration compares each record to the REGEX_DPN3 regular expression. If a match occurs, new fields are created based on the named capturing groups. The parsedate() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime field.

The log record is then converted to JSON using the to_json() procedure from the xm_json module. The drop procedure discards records that do not match the REGEX_DPN3 regular expression.

nxlog.conf

# A regular expression for parsing log data
define REGEX_DPN3   /(?x)^(\d+.\d+.\d+.)(\d+.\d+.\d+.\d+)\s*\
                    (?<EventSource>[A-Z]{2,})[\s\:]+\
                    (?:Interpreted\s*Telegram[\s\-]+\
                    (?<Interpreted_Telegram>.*))?\s*\
                    (?:Link\s*Address[\s\:]+(?<LinkAddress>.*))?\s*\
                    (?:DP[\s\-]+Address\/Index[\s\:]+(?<DP_AddressIndex>.*))?\s*\
                    (?:Value.*?[\s\:]+(?<Value>.*))?\s*\
                    (?:Status[\s\:]+(?<Status>.*))?\s*\
                    (?:Time.*?[\s\:]+(?<Time>.*))?\s*$/

# Generic path to the folder with log files
define CFE_PATH     C:\ProgramData\Siemens Energy\SICAM PAS PQS\Logs\CFE

<Extension json>
    Module          xm_json
</Extension>

<Extension multiline>
    Module          xm_multiline
    HeaderLine      /^\d+.\d+.\d+.\d+.\d+.\d+.\d+/
</Extension>

<Input from_file>
    Module          im_file
    File            '%CFE_PATH%\WIN-5RU7GP5MI4VModbus *Interface_trc.txt'
    InputType       multiline
    <Exec>

        # Matching a file header to the regular expression
        if  $raw_event =~ %REGEX_DPN3%
        {

            # Creating a timestamp
            $EventTime = parsedate($1 + $2);

            # Converting to JSON
            to_json();
        }

        # Discarding messages which do not match the regular expression
        else drop();

        # Replacing unwanted characters
        $raw_event =~ s/\\r\\n/ /g;
        $raw_event =~ s/\s{2,}/ /g;
    </Exec>
</Input>

Output sample in JSON format

{
  "EventReceivedTime": "2021-09-07T07:52:04.999288-07:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "DP_AddressIndex": "1",
  "EventSource": "PM",
  "Interpreted_Telegram": "Received Event Analog Input:",
  "LinkAddress": "4 (LinkId=100004)",
  "Status": "Online",
  "Time": "1619015996sec, 322ms, present, valid",
  "Value": "14481.8",
  "EventTime": "2021-04-21T14:39:57.455000-07:00"
}

Universal configuration file

This NXLog Agent configuration can process any type of communication log that SICAM PAS/PQS generates.

Example 29. Processing all communication trace logs

The fourth section configures an instance of the im_file module that uses the 12 regular expression constants for filtering and parsing the fields for each type of communication trace log.

The fifth section loads the om_file module that will take the JSON-formatted data it receives from the input instance and output it—without any further processing or conversion—to a single file, C:\output.txt.

The sixth and final section, ROUTES, is used for defining which input instance(s) are routed to which output instance(s). In this case, with only one input instance and one output instance, only one route can be defined.

nxlog.conf

# ----------------- REGULAR EXPRESSIONS FOR PARSING DATA -----------------------

define REGEX_BS     /(?x)^(\d+.\d+.\d+.)(\d+.\d+.\d+.\d+)\s*\
                    (?<EventSource>[A-Z]{2,})[\s\:]+\
                    (?<ByteStream>Byte\s*Stream.*)\:\s*(?<Data>[\d\w\s]*)/

define REGEX_ITONE  /(?x)^(\d+.\d+.\d+.)(\d+.\d+.\d+.\d+)\s*\
                    (?<EventSource>[A-Z]{2,})[\s\:]+(?<MessageType>\
                    (?:Interpreted\s*Telegram|Terminated\s*Scan).*)\s*\
                    (?:RTU\s*\:\s*(?<RTU>.*))?\s*(?:RTU\s*Address[\:\s]+\
                    (?<RTU_Address>.*))?\s*(?:Msg\s*Address[\:\s]+\
                    (?<MsgAddress>.*))?\s*(?:Value.*?[\:\s]+(?<Value>.*))?\s*\
                    (?<Function>[\w\s]+FC\d+\:.*)?\s*(?:Result.*?[\:\s]+\
                    (?<Result>.*))?\s*(?<Message>.*)?/

define REGEX_ITTWO  /(?x)^(\d+.\d+.\d+.)(\d+.\d+.\d+.\d+)\s*\
                    (?<EventSource>[A-Z]{2,})[\s\:]+\
                    (?<SignalType>[\w\s\(\)]+)[\:\s]*\
                    (?:RTU\s*Address\/LinkId[\:\s]+(?<RTU_AddressLinkId>.*))\s*\
                    (?:DP[\-\s]*Address\/Index[\:\s]+(?<DP_AdressIndex>.*))\s*\
                    (?:Value.*?[\=\s]+(?<Value>.*))\s*\
                    (?:Info[\:\s]+(?<Info>.*))\s*\
                    (?:DSI\s*Value.*?[\=\s]+(?<DSI_Value>.*))/

define REGEX_NORMAL /(?x)^(\d+.\d+.\d+.)(\d+.\d+.\d+.\d+)\s+\
                    (?:Normalized\s*Data[\s\:]*(?<NormalizedData>.*))[\,\s]*\
                    (?:Normalized\s*(?:Integer|Float)[\s\:]+\
                    (?<NormalizedValue>.*?)\,)\s*(?:Validity[\s\:]+\
                    (?<Validity>.*))\s*(?:Source\/Normal\s*Source[\s\:]+\
                    (?<Source_NormalSource>.*)\,)\s*(?:Normal\s*Range[\s\:]+\
                    (?<NormalRange>.*)\,)\s*\
                    (?:Change\s*of\s*Value\s*Counter[\s\:]+\
                    (?<ChangeOfValueCounter>.*))\s*(?:CoT[\s\:]+(?<CoT>.*)\,)\s*\
                    (?:AddCause[\s\:]+(?<AddCause>.*)\,)\s*\
                    (?:ToV[\s\:]+(?<ToV>.*))\s*(?:InitCat[\s\:]+\
                    (?<InitCat>.*)\,)?\s*(?:Ident[\s\:]+(?<Ident>.*)\,)\s*\
                    (?:Supplementary[\s\:]+(?<Supplementary>.*)\,)\s*\
                    (?:Testbit[\s\:]+(?<Testbit>.*))\s*(?:UserId[\s\:]+\
                    (?<UserId>.*)\,)\s*(?:AppId[\s\:]+(?<AppId>.*)\,)\s*\
                    (?:ProtData[\s\:]+(?<ProtData>.*))\s*\
                    (?:DSI\s*Time[\s\:]+(?<DSI_Time>.*))\s*\
                    (?:DSI\s*Time[\s\:]+(?<DSI_TimeUtc>.*))/

define REGEX_PBEF   /(?x)^(\d+.\d+.\d+.)(\d+.\d+.\d+.\d+)\s*\
                    (?:Data\sbefore\spreprocessing\:\s*\
                    (?<DataBeforePreprocessing>.*))\s*\
                    (?<DataEvent>\w*DataEvent.*)\,\s*\
                    (?:Qualifier\(s\)\s*\:(?<Qualifier_s>.*?))\s*\
                    (?:CFE\s*Data\:\s*(?<CFE_Data>.*))\s*(?:CFE\s*AddInfo\:\s*)\
                    (?:CoT\:\s*(?<CoT>.*)\,)\s*\
                    (?:AddCause\:\s*(?<AddCause>.*)\,)\s*(?:ToV\:\s*(?<ToV>.*))\s*\
                    (?:InitCat\:\s*(?<InitCat>.*)\,)\s*\
                    (?:Ident\:\s*(?<Ident>.*)\,)\s*\
                    (?:Supplementary\:\s*(?<Supplementary>.*)\,)\s*\
                    (?:Testbit\:\s*(?<Testbit>.*))\s*\
                    (?:UserId\:\s*(?<UserId>.*)\,)\s*\
                    (?:AppId\:\s*(?<AppId>.*)\,)\s*\
                    (?:ProtData\:\s*(?<ProtData>.*))\s*\
                    (?:CFE\s*Time\:\s*(?<CFE_Time>.*))/

define REGEX_PREP   /(?x)^(\d+.\d+.\d+.)(\d+.\d+.\d+.\d+)\s*\
                    (?:Preprocessed\s*Data.*?\:\s*(?<PreprocessedData>.*))?\s*\
                    (?<DataEvent>\w+DataEvent.*)\,\s*\
                    (?:DSI\s*Quality[\s\:]+(?<DSI_Quality>.*)\;)\s*\
                    (?:CFE\s*Quality[\s\:]+(?<CFE_Quality>.*)\;)\s*\
                    (?:DSI\s*Time[\s\:]+(?<DSI_Time>.*))\s*\
                    (?:CFE\s*Time[\s\:]+(?<CFE_Time>.*))\s*\
                    (?:CFE\s*AddInfo[\s\:]+)(?:CoT[\s\:]+(?<CoT>.*)\,)\s*\
                    (?:AddCause[\s\:]+(?<AddCause>.*)\,)\s*\
                    (?:ToV[\s\:]+(?<ToV>.*))\s*\
                    (?:InitCat[\s\:]+(?<InitCat>.*)\,)\s*\
                    (?:Ident[\s\:]+(?<Ident>.*)\,)\s*\
                    (?:Supplementary[\s\:]+(?<Supplementary>.*)\,)\s*\
                    (?:Testbit[\s\:]+(?<Testbit>.*))\s*\
                    (?:UserId[\s\:]+(?<UserId>.*)\,)\s*\
                    (?:AppId[\s\:]+(?<AppId>.*)\,)\s*\
                    (?:ProtData[\s\:]+(?<ProtData>.*))/

define REGEX_STAT   /(?x)^(\d+.\d+.\d+.)(\d+.\d+.\d+.\d+)\s*\
                    (?:Statistics[\s\:]+(?<Statistics>.*))\s*\
                    (?:Accumulators[\s\:]+(?<Accumulators>.*))\s*\
                    (?:Analog[\s\:]+(?<Analog>.*))\s*\
                    (?:Digitals[\s\:]+(?<Digitals>.*))\s*\
                    (?:Fleetings[\s\:]+(?<Fleetings>.*))\s*\
                    (?:Positions[\s\:]+(?<Positions>.*))\s*\
                    (?:Bitpattern[\s\:]+(?<Bitpattern>.*))\s*\
                    (?:Total[\s\:]+(?<Total>.*))/

define REGEX_SA     /(?x)^(\d+.\d+.\d+.)(\d+.\d+.\d+.\d+)\s*\
                    (?<EventSource>[A-Z]{2,})[\s\:]+Systemaccu\s*\
                    (?:RTU\=(?<RTU>.*))?[\:\s]*\
                    (?:Interface\=(?<Interface>.*))?[\:\s]*\
                    (?:Last\s*StopError\s*Code\=(?<LastStopErrorCode>.*))\s*\
                    (?:Last\s*Error\s*Code\=(?<LastErrorCode>.*))\s*\
                    (?:.*?Error\s*Uknown\s*(?<ErrorUknown>.*))\s*\
                    (?:Sent\s*Messages\=(?<SentMessages>.*))\s*\
                    (?:Received\s*Messages\=(?<ReceivedMessages>.*))\s*\
                    (?:Receive\s*Errors\=(?<ReceiveErrors>.*))\s*\
                    (?:Timeouts\=(?<Timeouts>.*))\s*\
                    (?:Scan\s*OverRuns\=(?<ScanOverRuns>.*))\s*\
                    (?:Scan\s*Requests\=(?<ScanRequests>.*))\s*\
                    (?:Command\s*Requests\=(?<CommandRequests>.*))\s*\
                    (?:Received\s*Repeated\s*Messages\=\
                    (?<ReceivedRepeatedMessages>.*))\s*\
                    (?:UnexpectedMessages\=(?<UnexpectedMessages>.*))\s*\
                    (?:AuthorizationFailures\=(?<AuthorizationFailures>.*))\s*\
                    (?:AuthenticationFailures\=(?<AuthenticationFailures>.*))\s*\
                    (?:ReplyTimeouts\=(?<ReplyTimeouts>.*))\s*\
                    (?:RekeysDueToAuthenticationFailure\=\
                    (?<RekeysDueToAuthenticationFailure>.*))\s*\
                    (?:TotalMessagesSent\=(?<TotalMessagesSent>.*))\s*\
                    (?:TotalMessagesReceived\=(?<TotalMessagesReceived>.*))\s*\
                    (?:CriticalMessagesSent\=(?<CriticalMessagesSent>.*))\s*\
                    (?:CriticalMessagesReceived\=\
                    (?<CriticalMessagesReceived>.*))?\s*\
                    (?:DiscardedMessages\=(?<DiscardedMessages>.*))?\s*\
                    (?:ErrorMessagesSent\=(?<ErrorMessagesSent>.*))\s*\
                    (?:ErrorMessagesRxed\=(?<ErrorMessagesRxed>.*))\s*\
                    (?:SuccessfulAuthentications\=\
                    (?<SuccessfulAuthentications>.*))\s*\
                    (?:SessionKeyChanges\=(?<SessionKeyChanges>.*))\s*\
                    (?:FailedSessionKeyChanges\=(?<FailedSessionKeyChanges>.*))\s*\
                    (?:UpdateKeyChanges\=(?<UpdateKeyChanges>.*))?\s*\
                    (?:FailedUpdateKeyChanges\=(?<FailedUpdateKeyChanges>.*))\s*\
                    (?:RekeysDueToRestarts\=(?<RekeysDueToRestarts>.*))\s*\
                    (?:Error\s*Data\s*Length\=(?<ErrorDataLength>.*))\s*\
                    (?:Failure\s*Interface\=(?<FailureInterface>.*))\s*\
                    (?:Failure\s*Device\=(?<FailureDevice>.*))\s*\
                    (?:Invalid\s*Message\=(?<InvalidMessage>.*))\s*\
                    (?:Error\s*Msg\s*Checksum\=(?<ErrorMsgChecksum>.*))\s*\
                    (?:Error\s*Clock\=(?<ErrorClock>.*))\s*\
                    (?:Error\s*Block\s*Sync\=(?<ErrorBlockSync>.*))\s*\
                    (?:Error\s*CTS\=(?<ErrorCTS>.*))\s*\
                    (?:Error\s*Framing\=(?<ErrorFraming>.*))\s*\
                    (?:Error\s*Overrun\=(?<ErrorOverrun>.*))\s*\
                    (?:Error\s*InputBuf\s*Overflow\=\
                    (?<ErrorInputBufOverflow>.*))\s*\
                    (?:Error\s*Parity\=(?<ErrorParity>.*))\s*\
                    (?:Error\s*OutputBuf\s*Overflow\=\
                    (?<ErrorOutputBufOverflow>.*))\s*\
                    (?:Error\s*DSR\=(?<ErrorDSR>.*))\s*\
                    (?:Error\s*DCD\=(?<ErrorDCD>.*))\s*\
                    (?:Error\s*Control\=(?<ErrorControl>.*))\s*\
                    (?:Error\s*SW\s*Logic\=(?<ErrorSWLogic>.*))\s*\
                    (?:Error\s*Timeout\=(?<ErrorTimeout>.*))\s*\
                    (?:Error\s*Not\s*Supported\=(?<ErrorNotSupported>.*))\s*\
                    (?:Error\s*Configuration\=(?<ErrorConfiguration>.*))/

define REGEX_MISC   /(?x)^(?(?=.*\s+(?:Interpreted\sTelegram|Analog\sInput\
                    \(INT\)\:|Terminated\sScan|Normalized\sData|Preprocessed\sData\
                    |Data\sbefore\spreprocessing\:|Byte\sStream|\w+\:{2}\w+\
                    \son\sdevice|Statistics\:|Systemaccu)\s+)\
                    |(\d+.\d+.\d+.)(\d+.\d+.\d+.\d+)[\s\:\-\<\>]*\
                    (?:\(Tid[\:\s]+(?<Tid>.*?)\))?\s*(?:\[(?<Severity>.*?)\])?\s*\
                    (?<EventSource>[A-Z]+(?=[\s\:]+)|data|error|void)?[\s\:\-\>\<]*\
                    (?<Process>\w+(?:\(.*?\))?(?:\[.*?\])?\:{2}\w+\
                    (?:\(.*?\))?(?:\[.*?\])?)?[\s\:\-]*\
                    (?<Properties>[\s\S]*)?)$/

define REGEX_IECBS  /(?x)^(\d+.\d+.\d+.)(\d+.\d+.\d+.\d+)\s*\
                    (?<Process>\w+\:{2}\w+\s*on\s*device)\s*\
                    (?<DeviceName>.*?)\:\s*(?<Data>[\d\w\s]*)/

define REGEX_IECIT  /(?x)^(\d+.\d+.\d+.)(\d+.\d+.\d+.\d+)\s*\
                    (?<MessageType>Interpreted\s*Telegram.*)?\:\s*\
                    (?:Common\s*Address[\:\s]+(?<CommonAddress>.*))\s*\
                    (?:Link\s*Address[\:\s]+(?<LinkAddress>.*))\s*\
                    (?:Object\s*Address[\:\s]+(?<ObjectAddress>.*))\s*\
                    (?:Type[\:\s]+(?<Type>.*))\s*\
                    (?:Cause\s*of\s*Transmission[\:\s]+\
                    (?<CauseOfTransmission>.*))\s*(?:Structured\s*Qualifier[\:\s]+\
                    (?<StructuredQualifier>.*))?\s*(?:Value[\:\s]+\
                    (?<Value>.*?))\,\s*(?:Sequence\s*No\.[\:\s]+\
                    (?<SequenceNo>.*?))\s*(?:Qualifiers\:\s(?<Qualifiers>.*))\s*\
                    (?:Error[\:\s]+(?<Error>.*))?/

define REGEX_DPN3   /(?x)^(\d+.\d+.\d+.)(\d+.\d+.\d+.\d+)\s*\
                    (?<EventSource>[A-Z]{2,})[\s\:]+\
                    (?:Interpreted\s*Telegram[\s\-]+\
                    (?<Interpreted_Telegram>.*))?\s*\
                    (?:Link\s*Address[\s\:]+(?<LinkAddress>.*))?\s*\
                    (?:DP[\s\-]+Address\/Index[\s\:]+(?<DP_AddressIndex>.*))?\s*\
                    (?:Value.*?[\s\:]+(?<Value>.*))?\s*\
                    (?:Status[\s\:]+(?<Status>.*))?\s*\
                    (?:Time.*?[\s\:]+(?<Time>.*))?\s*$/

# ----------------- PATH TO LOG FILES ------------------------------------------

define CFE_PATH     C:\ProgramData\Siemens Energy\SICAM PAS PQS\Logs\CFE

# ----------------- EXTENSION MODULES ------------------------------------------

<Extension json>
    Module          xm_json
</Extension>

<Extension multiline>
    Module          xm_multiline
    HeaderLine      /^\d+.\d+.\d+.\d+.\d+.\d+.\d+/
</Extension>

# ----------------- INPUT MODULE -----------------------------------------------

<Input from_file>
    Module          im_file
    File            '%CFE_PATH%\WIN-5RU7GP5MI4VModbus *Interface_trc.txt'
    InputType       multiline
    <Exec>
        if ($raw_event =~ %REGEX_BS%)    or ($raw_event =~ %REGEX_ITONE%)  or
           ($raw_event =~ %REGEX_ITTWO%) or ($raw_event =~ %REGEX_NORMAL%) or
           ($raw_event =~ %REGEX_PBEF%)  or ($raw_event =~ %REGEX_PREP%)   or
           ($raw_event =~ %REGEX_STAT%)  or ($raw_event =~ %REGEX_SA%)     or
           ($raw_event =~ %REGEX_MISC%)  or ($raw_event =~ %REGEX_IECBS%)  or
           ($raw_event =~ %REGEX_IECIT%) or ($raw_event =~ %REGEX_DPN3%)
        {
            $EventTime = parsedate($1 + $2);
            to_json();
        }
        else drop();
        $raw_event =~ s/\\r\\n/ /g;
        $raw_event =~ s/\s{2,}/ /g;
    </Exec>
</Input>

# ----------------- OUTPUT MODULE ----------------------------------------------

<Output to_file>
    Module          om_file
    File            'C:\output.txt'
</Output>

# ----------------- ROUTE ------------------------------------------------------

<Route r1>
    Path            from_file => to_file
</Route>

Passive network monitoring

SICAM PAS/PQS can be configured to use a set of communication protocols to communicate with control centers, bay devices, substations, and components for data exchange with industrial automation systems. To passively monitor SICAM PAS/PQS network traffic, NXLog Agent provides the im_pcap module.

This section describes how to configure NXLog Agent to monitor the following industrial protocols that SICAM PAS/PQS uses:

Modbus TCP
BACnet
DNP3
IEC 60870-5-104

Modbus TCP

Modbus is an application protocol providing client/server communication between industrial devices connected to different networks. SICAM PAS/PQS supports the Modbus Slave interface to communicate with high-level control centers, and the Modbus Master interface to connect to bay devices and substations. Each Modbus transaction consists of a client request followed by a server response.

Example 30. Capturing SCADA Modbus packets

This configuration uses the im_pcap module to capture network packets. The Dev directive specifies the network interface. The Protocol group directive denotes the modbus protocol. The Exec directive converts the captured messages to JSON using the to_json() procedure.

nxlog.conf

<Extension _json>
    Module    xm_json
</Extension>

<Input pcap_protocol>
    Module    im_pcap
    # Name of a network device or interface
    Dev       \Device\NPF_{159289BE-CE80-47DB-A659-2F8BF277C9C6}
    # Protocol type
    <Protocol>
        Type    modbus
    </Protocol>
    # Converting to JSON
    Exec      to_json();
</Input>

Modbus query sample in JSON format

{
  "modbus.function_code": "Read Holding Registers (03)",
  "modbus.length": "6",
  "modbus.prot_id": "0",
  "modbus.query.read_holding_regs.qty_of_regs": "3",
  "modbus.query.read_holding_regs.starting_address": "0",
  "modbus.trans_id": "585",
  "modbus.unit_id": "1",
  "EventTime": "2021-08-16T11:27:50.598481+03:00",
  "EventReceivedTime": "2021-08-16T11:27:51.524063+03:00",
  "SourceModuleName": "pcap_protocol",
  "SourceModuleType": "im_pcap"
}

Modbus response sample in JSON format

{
  "modbus.function_code": "Read Holding Registers (03)",
  "modbus.length": "9",
  "modbus.prot_id": "0",
  "modbus.response.read_holding_regs.byte_count": "6",
  "modbus.response.read_holding_regs.registers": "355, 1573, 460",
  "modbus.trans_id": "585",
  "modbus.unit_id": "1",
  "EventTime": "2021-08-16T11:27:50.609556+03:00",
  "EventReceivedTime": "2021-08-16T11:27:51.524063+03:00",
  "SourceModuleName": "pcap_protocol",
  "SourceModuleType": "im_pcap"
}

BACnet

BACnet (Building Automation and Management Networks) is a communication protocol standard that connects control products from various vendors. BACnet is used in HVAC, lighting control, and fire and life systems.

This example explains how to capture BACnet packets using NXLog Agent.

Example 31. Capturing SCADA BACnet packets

This configuration uses the Dev and Protocol directives from the im_pcap module module to specify the network interface and protocol for capturing network packets. Once captured, data is converted to JSON using the to_json() procedure of the xm_json module.

nxlog.conf

<Extension _json>
    Module    xm_json
</Extension>

<Input pcap_protocol>
    Module    im_pcap
    # Name of a network device/interface
    Dev       \Device\NPF_{159289BE-CE80-47DB-A659-2F8BF277C9C6}
    # Protocol type
    <Protocol>
        Type    bacnet
    </Protocol>
    # Conversion to JSON
    Exec      to_json();
</Input>

BACnet query sample in JSON format

{
  "bacnet.apdu.bacnet_confirmed_request.invoke_id": "225",
  "bacnet.apdu.bacnet_confirmed_request.max_resp": "1476",
  "bacnet.apdu.bacnet_confirmed_request.max_segs": "Unspecified",
  "bacnet.apdu.bacnet_confirmed_request.more_segments_follow": "false",
  "bacnet.apdu.bacnet_confirmed_request.segmented": "false",
  "bacnet.apdu.bacnet_confirmed_request.segmented_accepted": "true",
  "bacnet.apdu.bacnet_confirmed_request.service_choice": "Read Property Multiple (14)",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.0.object_identifier.instance_number": "0",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.0.object_identifier.type": "analog-input (0)",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.1": "Opening Tag (1)",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.2.0.property_identifier": "description (28)",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.3": "Closing Tag (1)",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.4.object_identifier.instance_number": "0",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.4.object_identifier.type": "analog-input (0)",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.5": "Opening Tag (1)",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.6.0.property_identifier": "present-value (85)",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.7": "Closing Tag (1)",
  "bacnet.apdu.pdu_type": "BACnet-Confirmed-Request-PDU (0x00)",
  "bacnet.bvlc.function": "Original-Unicast-NPDU (0x0A)",
  "bacnet.bvlc.length": "38",
  "bacnet.bvlc.type": "BACnet/IP (Annex J) (0x81)",
  "bacnet.npdu.control": "0x0024",
  "bacnet.npdu.control.contains": "BACnet APDU message (0)",
  "bacnet.npdu.control.dst_spec": "DNET, DLEN, Hop Count present (1)",
  "bacnet.npdu.control.prio": "Normal message",
  "bacnet.npdu.control.reply_expected": "Yes (1)",
  "bacnet.npdu.control.src_spec": "SNET, SLEN, SADR absent (0)",
  "bacnet.npdu.version": "0x0001",
  "EventTime": "2021-08-17T08:02:07.507714+03:00",
  "EventReceivedTime": "2021-08-17T08:02:08.076729+03:00",
  "SourceModuleName": "pcap_protocol",
  "SourceModuleType": "im_pcap"
}

BACnet response sample in JSON format

{
  "bacnet.apdu.bacnet_complexack.more_segments_follow": "false",
  "bacnet.apdu.bacnet_complexack.original_invoke_id": "225",
  "bacnet.apdu.bacnet_complexack.segmented": "false",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.0.0.object_identifier.instance_number": "0",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.0.0.object_identifier.type": "analog-input (0)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.0.1": "Opening Tag (1)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.0.2.property_identifier": "description (28)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.0.3.records.0": "Opening Tag (4)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.0.3.records.1": "NXLog EE (ANSI X3.4/UTF-8 (0))",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.0.3.records.2": "Closing Tag (4)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.0.4": "Closing Tag (1)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.1.0.object_identifier.instance_number": "0",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.1.0.object_identifier.type": "analog-input (0)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.1.1": "Opening Tag (1)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.1.2.property_identifier": "present-value (85)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.1.3.records.0": "Opening Tag (4)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.1.3.records.1": "-624000.000000",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.1.3.records.2": "Closing Tag (4)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.1.4": "Closing Tag (1)",
  "bacnet.apdu.bacnet_complexack.service_choice": "Read Property Multiple (14)",
  "bacnet.apdu.pdu_type": "BACnet-Complex-ACK-PDU (0x03)",
  "bacnet.bvlc.function": "Original-Unicast-NPDU (0x0A)",
  "bacnet.bvlc.length": "56",
  "bacnet.bvlc.type": "BACnet/IP (Annex J) (0x81)",
  "bacnet.npdu.control": "0x0008",
  "bacnet.npdu.control.contains": "BACnet APDU message (0)",
  "bacnet.npdu.control.dst_spec": "DNET, DLEN, DADR, Hop Count absent (0)",
  "bacnet.npdu.control.prio": "Normal message",
  "bacnet.npdu.control.reply_expected": "No (0)",
  "bacnet.npdu.control.src_spec": "SNET, SLEN, SADR present (1)",
  "bacnet.npdu.version": "0x0001",
  "EventTime": "2021-08-17T08:02:07.536180+03:00",
  "EventReceivedTime": "2021-08-17T08:02:08.077730+03:00",
  "SourceModuleName": "pcap_protocol",
  "SourceModuleType": "im_pcap"
}

DNP3

DNP3 is an open, multi-layered communication protocol used in the oil, gas, water, and transportation industries that supports authentication, encryption, metadata, and timestamping.

The following example shows how to capture DNP3 packets using NXLog Agent.

Example 32. Capturing DNP3 packets

The following configuration uses the Dev and Protocol directives from the im_pcap module to define the network interface and protocols that will be used for capturing network packets. Once captured, data is converted to JSON using the to_json() procedure of the xm_json module.

nxlog.conf

<Extension _json>
    Module    xm_json
</Extension>

<Input pcap_protocol>
    Module    im_pcap
    # Name of a network device/interface
    Dev       \Device\NPF_{159289BE-CE80-47DB-A659-2F8BF277C9C6}
    # Protocol type
    <Protocol>
        Type    dnp3
    </Protocol>
    # Conversion to JSON
    Exec      to_json();
</Input>

DNP3 request sample in JSON format

{
  "dnp3.application_layer.control.con": "0",
  "dnp3.application_layer.control.fin": "1",
  "dnp3.application_layer.control.fir": "1",
  "dnp3.application_layer.control.sequence": "15",
  "dnp3.application_layer.control.uns": "0",
  "dnp3.application_layer.function_code": "Read",
  "dnp3.application_layer.object0.count": "0",
  "dnp3.application_layer.object0.group": "60",
  "dnp3.application_layer.object0.name": "Class objects - Class 3 data",
  "dnp3.application_layer.object0.variation": "4",
  "dnp3.application_layer.object1.count": "0",
  "dnp3.application_layer.object1.group": "60",
  "dnp3.application_layer.object1.name": "Class objects - Class 2 data",
  "dnp3.application_layer.object1.variation": "3",
  "dnp3.application_layer.object2.count": "0",
  "dnp3.application_layer.object2.group": "60",
  "dnp3.application_layer.object2.name": "Class objects - Class 1 data",
  "dnp3.application_layer.object2.variation": "2",
  "dnp3.application_layer.object3.count": "0",
  "dnp3.application_layer.object3.group": "60",
  "dnp3.application_layer.object3.name": "Class objects - Class 0 data",
  "dnp3.application_layer.object3.variation": "1",
  "dnp3.data_layer.control": "0xC4",
  "dnp3.data_layer.control.dir": "1",
  "dnp3.data_layer.control.fcb": "0",
  "dnp3.data_layer.control.fcv": "0",
  "dnp3.data_layer.control.function_code": "Unconfirmed User Data",
  "dnp3.data_layer.control.prm": "1",
  "dnp3.data_layer.destination": "1",
  "dnp3.data_layer.length": "20",
  "dnp3.data_layer.source": "2",
  "dnp3.data_layer.start_bytes": "0x0564",
  "dnp3.transport.fin": "1",
  "dnp3.transport.fir": "1",
  "dnp3.transport.sequence": "49",
  "EventTime": "2021-08-16T12:53:28.998131+03:00",
  "EventReceivedTime": "2021-08-16T12:53:29.376352+03:00",
  "SourceModuleName": "pcap_protocol",
  "SourceModuleType": "im_pcap"
}

DNP3 response sample in JSON format

{
  "dnp3.application_layer.control.con": "0",
  "dnp3.application_layer.control.fin": "1",
  "dnp3.application_layer.control.fir": "1",
  "dnp3.application_layer.control.sequence": "15",
  "dnp3.application_layer.control.uns": "0",
  "dnp3.application_layer.function_code": "Response",
  "dnp3.application_layer.internal_indications.already_executing": "0",
  "dnp3.application_layer.internal_indications.broadcast": "0",
  "dnp3.application_layer.internal_indications.class1_events": "0",
  "dnp3.application_layer.internal_indications.class2_events": "0",
  "dnp3.application_layer.internal_indications.class3_events": "0",
  "dnp3.application_layer.internal_indications.config_corrupt": "0",
  "dnp3.application_layer.internal_indications.device_restart": "0",
  "dnp3.application_layer.internal_indications.device_trouble": "0",
  "dnp3.application_layer.internal_indications.events_buffer_overflow": "0",
  "dnp3.application_layer.internal_indications.local_control": "0",
  "dnp3.application_layer.internal_indications.need_time": "0",
  "dnp3.application_layer.internal_indications.no_func_code_support": "0",
  "dnp3.application_layer.internal_indications.object_unknown": "0",
  "dnp3.application_layer.internal_indications.parameter_error": "0",
  "dnp3.application_layer.internal_indications.reserved": "0 (expected 0)",
  "dnp3.application_layer.object0.count": "1",
  "dnp3.application_layer.object0.group": "30",
  "dnp3.application_layer.object0.name": "Analog input - single-precision, floating-point with flag",
  "dnp3.application_layer.object0.point0.flags": "[ONLINE]",
  "dnp3.application_layer.object0.point0.value": "314.980011",
  "dnp3.application_layer.object0.variation": "5",
  "dnp3.data_layer.control": "0x44",
  "dnp3.data_layer.control.dir": "0",
  "dnp3.data_layer.control.fcb": "0",
  "dnp3.data_layer.control.fcv": "0",
  "dnp3.data_layer.control.function_code": "Unconfirmed User Data",
  "dnp3.data_layer.control.prm": "1",
  "dnp3.data_layer.destination": "2",
  "dnp3.data_layer.length": "20",
  "dnp3.data_layer.source": "1",
  "dnp3.data_layer.start_bytes": "0x0564",
  "dnp3.transport.fin": "1",
  "dnp3.transport.fir": "1",
  "dnp3.transport.sequence": "53",
  "EventTime": "2021-08-16T12:53:29.049416+03:00",
  "EventReceivedTime": "2021-08-16T12:53:29.378329+03:00",
  "SourceModuleName": "pcap_protocol",
  "SourceModuleType": "im_pcap"
}

IEC 60870-5-104

IEC 60870-5-104 is an application-oriented protocol used in the electrical engineering and power system automation sectors to enable communication between control stations and substations over TCP.

This example demonstrates how NXLog Agent can capture network packets of the IEC 60870-5-104 protocol.

Example 33. Capturing IEC 60870-5-104 packets

This configuration uses the Dev and Protocol directives of the im_pcap module to define the network interface and protocols that will be used for capturing network packets. Once captured, data is converted to JSON using the to_json() procedure from the xm_json module.

nxlog.conf

<Extension _json>
    Module    xm_json
</Extension>

<Input pcap_protocol>
    Module    im_pcap
    # Name of a network device/interface
    Dev           \Device\NPF_{159289BE-CE80-47DB-A659-2F8BF277C9C6}
    # Protocol type
    <Protocol>
        Type    iec104apci
    </Protocol>
    <Protocol>
        Type    iec104asdu
    </Protocol>
    # Conversion to JSON
    Exec      to_json();
</Input>

IEC 60870-5-104 sample in JSON format

{
  "iec104.apci.receive_sequence_number": "4",
  "iec104.apci.send_sequence_number": "2619",
  "iec104.apci.type": "Information (I)",
  "iec104.asdu.data": {
    "io": [
      {
        "ioa": 1000,
        "ie": [
          {
            "type": "NVA",
            "value": "0.438507 (14369)"
          },
          {
            "type": "QDS",
            "invalid": false,
            "not-topical": false,
            "substituted": false,
            "blocked": false,
            "overflow": false
          },
          {
            "type": "CP56Time2A",
            "milliseconds": 54639,
            "minutes": 23,
            "hours": 3,
            "day-of-week": 0,
            "day-of-month": 16,
            "month": 8,
            "year": 21
          }
        ],
        "ies": 3
      }
    ],
    "ios": 1
  },
  "iec104.asdu.dui.cause_of_transmission": "Spontaneous (3)",
  "iec104.asdu.dui.coa": "1",
  "iec104.asdu.dui.num_records": "1",
  "iec104.asdu.dui.org": "0",
  "iec104.asdu.dui.pn": "0",
  "iec104.asdu.dui.sq": "FALSE",
  "iec104.asdu.dui.test_bit": "0",
  "iec104.asdu.dui.type": "M_ME_TD_1",
  "EventTime": "2021-08-16T13:23:54.518715+03:00",
  "EventReceivedTime": "2021-08-16T13:23:55.530255+03:00",
  "SourceModuleName": "pcap_protocol",
  "SourceModuleType": "im_pcap"
}

Disclaimer

While we endeavor to keep the information in our guides up to date and correct, NXLog makes no representations or warranties of any kind, express or implied about the completeness, accuracy, reliability, suitability, or availability of the content represented here. We update our screenshots and instructions on a best-effort basis.

The accurateness of the content was tested and proved to be working in our lab environment at the time of the last revision with the following software versions:

Siemens Sicam PAS/PQS V8.16 (08.16.05.995)
NXLog Agent version 5.3.7166

Last revision: 7 January 2022