Siemens SIMATIC PCS 7

SIMATIC PCS 7 is a Supervisory Control and Data Acquisition (SCADA) solution from Siemens.

NXLog Agent can be configured to read and process all kinds of logs produced by Siemens SIMATIC PCS 7.

Types of logs

NXLog Agent can collect events from SIMATIC PCS 7 from the following sources:

Logs in Windows Event Log

NXLog Agent can be configured to process Windows Event Logs produced by SIMATIC WinCC, as shown in the examples below.

Example 1. Processing WinCC logs based on the Event ID values

This example contains NXLog Agent configuration for reading and processing Windows Event logs generated by WinCC. Log filtering is based on the values of the Event ID field.

A sample list of values for Event ID is provided in the table below. This is only a small subset of events generated by WinCC since it is impractical to include all possible events.

Table 1. Events generated by WinCC
Event ID Event Text

256

S7 PCAdapter Plugin successfully started.

1005

The "SIMATIC NET Configuration Service" service was started.

1026

The SIMATIC PC Station is ready for communication.

1027

The SIMATIC PC Station is not ready for communication.

4096

The service SIMATIC NET Station Manager was started.

4132

!! Service started!!

The following log sample with Event ID 4096 was copied from the Windows Event Viewer.

Event sample
Log Name:      Application
Source:        SIMATIC NET Station Manager
Date:          16.06.2020 14:09:51
Event ID:      4096
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      WINSRV99
Description:
The service SIMATIC NET Station Manager was started
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="SIMATIC NET Station Manager" />
    <EventID Qualifiers="16386">4096</EventID>
    <Level>4</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2020-06-16T11:09:51.480603800Z" />
    <EventRecordID>20030</EventRecordID>
    <Channel>Application</Channel>
    <Computer>WINSRV99</Computer>
    <Security />
  </System>
  <EventData>
    <Data>SIMATIC NET Station Manager</Data>
  </EventData>
</Event>

Using the im_msvistalog module, the configuration below instructs NXLog Agent to read data from the log sources listed in the table above. To facilitate more convenient post-processing, the xm_json module enables all messages to be converted to JSON as specified in the Exec directive.

nxlog.conf
<Extension json>
    Module    xm_json
</Extension>

<Input from_eventlog>
    Module    im_msvistalog
    <QueryXML>
        <QueryList>
            <Query Id="0">
                <Select Path="Application">
                    *[System[(EventID=256  or EventID=1005 or 
                              EventID=1026 or EventID=1027 or
                              EventID=4096 or EventID=4132)]]
                </Select>
            </Query>
        </QueryList>
    </QueryXML>
    Exec    to_json();
</Input>

Below is the message with Event ID 4096 after it has been processed by NXLog Agent and converted to JSON.

Output
{
  "EventTime": "2020-06-16T14:09:51.480603+03:00",
  "Hostname": "WINSRV99",
  "Keywords": "36028797018963968",
  "EventType": "INFO",
  "SeverityValue": 2,
  "Severity": "INFO",
  "EventID": 4096,
  "SourceName": "SIMATIC NET Station Manager",
  "TaskValue": 0,
  "RecordNumber": 20030,
  "ExecutionProcessID": 0,
  "ExecutionThreadID": 0,
  "Channel": "Application",
  "Message": "The service SIMATIC NET Station Manager was started",
  "EventData": "<Data>SIMATIC NET Station Manager</Data>",
  "EventReceivedTime": "2020-06-16T14:09:51.666231+03:00",
  "SourceModuleName": "from_eventlog",
  "SourceModuleType": "im_msvistalog"
}

The following table lists the WinCC services that generate log data in the Windows Event Log, their display names, and the path to their executables.

Table 2. Event sources in Windows Event Log
Service Name Display Name Path to Executable

CCAgent

CCAgent

C:\Program Files (x86)\Common Files\Siemens\ACE\bin\CCAgent.EXE

CCArchiveConnMon

CCArchiveConnMon

C:\Program Files (x86)\Common Files\Siemens\bin\CCArchiveConnMon.exe

CCArchiveManagerService

CCArchiveManagerService

C:\Program Files (x86)\Common Files\Siemens\CommonArchiving\CCArchiveManager.EXE

CCCAPHServer

CCCAPHServer

C:\Program Files (x86)\Common Files\Siemens\CommonArchiving\CCArchiveManager.EXE

CCDBUtils

CCDBUtils

C:\Program Files (x86)\Common Files\Siemens\CommonArchiving\CCDBUtils.EXE

CCEClient

CCEClient

C:\Program Files (x86)\Common Files\Siemens\ACE\bin\CCEClient_x64.exe

CCEServer

CCEServer

C:\Program Files (x86)\Common Files\Siemens\ACE\bin\CCEServer_x64.exe

CCOPC.XMLWrapper

CCOPC.XMLWrapper

C:\Program Files (x86)\SIEMENS\WINCC\opc\XMLDataAccess\bin\DA2XML.exe

CCOpcUaImporter

CCOpcUaImporter

C:\Program Files (x86)\SIEMENS\WINCC\OPC\UAClient\UaConfigServer\CCOpcUaImporter.exe

CCPerfMon

CCPerfMon

C:\Program Files (x86)\Common Files\Siemens\bin\CCPerfMon.exe

CCRedundancyAgent-Service

CCRedundancyAgent-Service

C:\Program Files (x86)\Common Files\Siemens\CommonArchiving\CCRedundancyAgent.exe

CCRemoteService

CCRemoteService

C:\Program Files (x86)\Common Files\Siemens\bin\CCRemoteService.exe

OpcUaServerWinCC

OpcUaServerWinCC

C:\Program Files (x86)\SIEMENS\WINCC\OPC\UAServer\OpcUaServerWinCC.exe

RC_Service

RC_Service

C:\Program Files (x86)\Common Files\Siemens\bin\rc_servicex.exe

RedundancyControl

RedundancyControl

C:\Program Files (x86)\Common Files\Siemens\ace\bin\RedundancyControl.exe

RedundancyState

RedundancyState

C:\Program Files (x86)\Common Files\Siemens\ace\bin\RedundancyState.exe

s7hspsvx

S7 HSP Service

C:\Program Files (x86)\Common Files\Siemens\bin\s7hspsvx.exe

s7oiehsx64

S7DOS Help Service

C:\Program Files\Common Files\Siemens\Automation\Simatic OAM\bin\s7oiehsx64.exe

S7DOS SCP Remote

S7DOS SCP Remote

C:\Program Files\Common Files\Siemens\Automation\Simatic OAM\bin\S7O.TunnelServiceHost.exe

SCS Distribution Service

SCS Distribution Service

C:\Program Files (x86)\Common Files\Siemens\ACE\bin\SCSDistServiceX.exe

SCSFsX

SCSFsX

C:\Program Files (x86)\Common Files\Siemens\ACE\bin\SCSFsX.exe

SCSMonitor

SCSMonitor

C:\Program Files (x86)\Common Files\Siemens\ace\bin\SCSMX.exe

SIMATIC BATCH Database server

SIMATIC BATCH Database server

C:\Program Files (x86)\SIEMENS\BATCH\FastObjectsServer64.exe" -config "C:\ProgramData\Siemens\Automation\Batch\sbdata\global\ptserver.cfg" -serviceName "SIMATIC BATCH Database server

bfedsx

SIMATIC BATCH Equipment Service

C:\Program Files (x86)\SIEMENS\WINCC\bin\sbwccopts\sbeds32ux.exe

bfprojectsrvx

SIMATIC BATCH Project Service

C:\Program Files (x86)\SIEMENS\BATCH\sbprojectsrv32ux.exe

bfpublisherx

SIMATIC BATCH Publisher Service

C:\Program Files (x86)\SIEMENS\BATCH\sbpublisher32ux.exe

bflaunchcoordinatorservicex

SIMATIC BATCH Server Control Service

C:\Program Files (x86)\SIEMENS\BATCH\sblaunchcoordinatorservice32ux.exe

SIMATIC Management Agent

SIMATIC Management Agent

C:\Program Files (x86)\SIEMENS\SIMATIC Management Console\SMAgent_32.exe

rc_serverhostx

SIMATIC Route Control Server

C:\Program Files (x86)\SIEMENS\RCS\bin\rc_serverhostx.exe

SIMATIC PnDiscovery Service

SIMATIC PnDiscovery Service

C:\Program Files\Common Files\Siemens\Automation\Simatic OAM\bin\s7oPNDiscoveryx64.exe

SIMATIC PDM Asset Service

SIMATIC PDM Asset Service

C:\Program Files (x86)\SIEMENS\SIMATIC_PDM\bin\PdmAssetService_x.exe

S7TraceServiceX

SIMATIC Trace Service

C:\Program Files\Common Files\Siemens\Automation\TraceEngine\bin\S7TraceService64X.exe

SIMATIC NET Configuration Service

SIMATIC NET Configuration Service

C:\Program Files (x86)\SIEMENS\SIMATIC.NET\opc2\bincfg\SServCFG.exe

SIMATIC NET Configuration Server

SIMATIC NET Configuration Server

C:\Program Files (x86)\SIEMENS\SIMATIC.NET\opc2\bincfg\scorecfg.exe

SIMATIC NET Core Server DP

SIMATIC NET Core Server DP

C:\Program Files (x86)\SIEMENS\SIMATIC.NET\opc2\bindp\scoredp.exe

SIMATIC NET Core Server DP2

SIMATIC NET Core Server DP2

C:\Program Files (x86)\SIEMENS\SIMATIC.NET\opc2\bindp2\scoredp2.exe

SIMATIC NET Core Server FDL

SIMATIC NET Core Server FDL

C:\Program Files (x86)\SIEMENS\SIMATIC.NET\opc2\binfdl\scorefdl.exe

SIMATIC NET Core Server PD

SIMATIC NET Core Server PD

C:\Program Files (x86)\SIEMENS\SIMATIC.NET\opc2\binpd\scorepd.exe

SIMATIC NET Core Server PROFINET IO

SIMATIC NET Core Server PROFINET IO

C:\Program Files (x86)\SIEMENS\SIMATIC.NET\opc2\binpnio\scorepnio.exe

SIMATIC NET Core Server S7

SIMATIC NET Core Server S7

C:\Program Files (x86)\SIEMENS\SIMATIC.NET\opc2\binS7\SCoreS7.exe

SIMATIC NET Core Server S7OPT

SIMATIC NET Core Server S7OPT

C:\Program Files (x86)\SIEMENS\SIMATIC.NET\opc2\binS7opt\SCoreS7opt.exe

SIMATIC NET Core Server SNMP

SIMATIC NET Core Server SNMP

C:\Program Files (x86)\SIEMENS\SIMATIC.NET\opc2\binSNMP\scoresnmp.exe

SIMATIC NET Core Server SR

SIMATIC NET Core Server SR

C:\Program Files (x86)\SIEMENS\SIMATIC.NET\opc2\binsr\scoresr.exe

SIMATIC NET PnP Manager

SIMATIC NET PnP Manager

C:\Program Files (x86)\SIEMENS\SIMATIC.NET\SimNetCom\simnetpnpman.exe

SIMATIC NET RouteManager

SIMATIC NET RouteManager

C:\Program Files\Common Files\Siemens\S7wnrmsx\s7wnrmsx.exe

SIMATIC NET SOFTNET-IE RNA

SIMATIC NET SOFTNET-IE RNA

C:\Program Files\SIEMENS\SOFTNET-IE RNA\SIMATIC NET SOFTNET-IE RNA x64.exe

StatMgr

SIMATIC NET Station Manager

C:\Program Files\Common Files\Siemens\S7wnsmsx\s7wnsmsx.exe

SIMATIC NET Synchronization Service

SIMATIC NET Synchronization Service

C:\Program Files\Common Files\Siemens\Automation\Simatic OAM\bin\sim9sync.exe

CCAlgIAlarmDataCollector

SIMATIC WinCC CCAlgIAlarmDataCollector

C:\Program Files (x86)\SIEMENS\WINCC\bin\CCAlgIAlarmDataCollector.exe

CCAlgRtServer

SIMATIC WinCC CCAlgRtServer

C:\Program Files (x86)\SIEMENS\WINCC\bin\CcAlgRtServer.exe

CCCSigRTServer

SIMATIC WinCC CCCSigRTServer

C:\Program Files (x86)\SIEMENS\WINCC\bin\CCCSigRTServer.exe

CCDeltaLoader

SIMATIC WinCC CCDeltaLoader

C:\Program Files (x86)\SIEMENS\WINCC\bin\CCDeltaLoader.exe

CCLBMRTServer

SIMATIC WinCC CCLBMRTServer

C:\Program Files (x86)\SIEMENS\WINCC\bin\CCLBMRTServer.exe

CCNSInfo2Provider

SIMATIC WinCC CCNSInfo2Provider

C:\Program Files (x86)\SIEMENS\WINCC\bin\CCNSInfo2Provider.exe

CCPackageMgr

SIMATIC WinCC CCPackageMgr

C:\Program Files (x86)\SIEMENS\WINCC\bin\CCPackageMgr.exe

CCProfileServer

SIMATIC WinCC CCProfileServer

C:\Program Files (x86)\SIEMENS\WINCC\bin\CCProfileServer.exe

CCProjectMgr

SIMATIC WinCC CCProjectMgr

C:\Program Files (x86)\SIEMENS\WINCC\bin\CCProjectMgr.exe

CCPtmRTServer

SIMATIC WinCC CCPtmRTServer

C:\Program Files (x86)\SIEMENS\WINCC\bin\CCPtmRTServer.exe

CCSsmRTServer

SIMATIC WinCC CCSsmRTServer

C:\Program Files (x86)\SIEMENS\WINCC\bin\CCSsmRTServer.exe

CCSystemDiagnosticsHost

SIMATIC WinCC CCSystemDiagnosticsHost

C:\Program Files (x86)\SIEMENS\WINCC\bin\CCSystemDiagnosticsHost.exe

CCTextServer

SIMATIC WinCC CCTextServer

C:\Program Files (x86)\SIEMENS\WINCC\bin\CCTextServer.exe

CCTlgServer

SIMATIC WinCC CCTlgServer

C:\Program Files (x86)\SIEMENS\WINCC\bin\CCTlgServer.exe

CCTMTimeSyncServer

SIMATIC WinCC CCTMTimeSyncServer

C:\Program Files (x86)\SIEMENS\WINCC\bin\CCTMTimeSyncServer.exe

CCTTRTServer

SIMATIC WinCC CCTTRTServer

C:\Program Files (x86)\SIEMENS\WINCC\bin\CCTTRTServer.exe

CCUsrAcv

SIMATIC WinCC CCUsrAcv

C:\Program Files (x86)\SIEMENS\WINCC\bin\CCUsrAcv.exe

CCRtsLoader

SIMATIC WinCC Data Manager

C:\Program Files (x86)\SIEMENS\WINCC\bin\CCRtsLoader.exe

CCLicenseService

SIMATIC WinCC License Service

C:\Program Files (x86)\Common Files\Siemens\bin\CCLicenseService.exe

ReportScheduler

SIMATIC WinCC ReportScheduler

C:\Program Files (x86)\SIEMENS\WinCC\WebNavigator\DataMonitor\bin\ReportScheduler.exe

MSSQL$WINCC

SQL Server (WINCC)

C:\Program Files (x86)\Microsoft SQL Server\MSSQL12.WINCC\MSSQL\Binn\sqlservr.exe" -sWINCC

SQLAgent$WINCC

SQL Server Agent (WINCC)

C:\Program Files (x86)\Microsoft SQL Server\MSSQL12.WINCC\MSSQL\Binn\SQLAGENT.EXE" -i WINCC

Example 2. Processing WinCC logs based on event source names

This example demonstrates how to apply NXLog Agent to parse and process WinCC log sources listed in the table above.

The following log sample of the SIMATIC Management Agent event source was copied from the Windows Event Viewer.

Event sample
Log Name:      Application
Source:        SIMATIC Management Agent
Date:          19.06.2020 17:15:53
Event ID:      0
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      WINSRV99
Description:
Service started/resumed
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="SIMATIC Management Agent" />
    <EventID Qualifiers="0">0</EventID>
    <Level>4</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2020-06-19T14:15:53.966166700Z" />
    <EventRecordID>20534</EventRecordID>
    <Channel>Application</Channel>
    <Computer>WINSRV99</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Service started/resumed</Data>
  </EventData>
</Event>

Using im_msvistalog module, the configuration below instructs NXLog Agent to process only those events generated by the following four SIMATIC WinCC event sources:

  • S7 PC Adapter USB A2 Service Plugin

  • SIMATIC Management Agent

  • SIMATIC NET PnP Manager

  • SIMATIC NET RouteManager.

To facilitate more convenient post-processing, the xm_json module enables all messages to be converted to JSON as specified in the Exec directive.

nxlog.conf
<Extension json>
    Module    xm_json
</Extension>

<Input from_eventlog>
    Module    im_msvistalog
    <QueryXML>
        <QueryList>
            <Query Id="0" Path="Application">
                <Select Path="Application">*
                [System[Provider[@Name='S7 PC Adapter USB A2 Service Plugin' 
                or @Name='SIMATIC Management Agent' 
                or @Name='SIMATIC NET PnP Manager' 
                or @Name='SIMATIC NET RouteManager']]]
                </Select>
            </Query>
        </QueryList>
    </QueryXML>
    Exec    to_json();
</Input>

Below is the message from the SIMATIC Management Agent event source after it has been processed by NXLog Agent and converted to JSON.

Output
{
  "EventTime": "2020-06-19T17:15:53.966166+03:00",
  "Hostname": "WINSRV99",
  "Keywords": "36028797018963968",
  "EventType": "INFO",
  "SeverityValue": 2,
  "Severity": "INFO",
  "EventID": 0,
  "SourceName": "SIMATIC Management Agent",
  "TaskValue": 0,
  "RecordNumber": 20534,
  "ExecutionProcessID": 0,
  "ExecutionThreadID": 0,
  "Channel": "Application",
  "Message": "Service started/resumed",
  "Category": "%1",
  "Opcode": "Info",
  "EventData": "<Data>Service started/resumed</Data>",
  "EventReceivedTime": "2020-06-19T17:15:54.078372+03:00",
  "SourceModuleName": "from_eventlog",
  "SourceModuleType": "im_msvistalog"
}

File-based logs

The Siemens SCADA system produces various kinds of logs stored in files.

The Automation, Engineering, and Operator stations create file-based logs and, in most cases, do not follow a unified formatting scheme. To tackle the inconsistent formatting of these logs, NXLog Agent can be configured to use regular expressions that is specific to each messaging structure.

File-based logs of PCS7 in this section are grouped into the following categories:

This section will be filled with new configuration examples.

WinCC system diagnostics logs

The table below provides information about WinCC system diagnostics logs.

Table 3. WinCC system diagnostics logs
Name File
Ext.
Location Description

Alarm logging server logs

.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\CCAlgRtCollectAlarm.log

Channel diagnostics logs

.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\SIMATIC_S7_Protocol_Suite_xx.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\SimotionTA_xx.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\SIMATIC_S7-1200_S7-1500_Channel_xx.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\System_Info.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\SIMATIC_S5_PROFIBUS_FDL_xx.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\OPC.log

In case of communication errors, the corresponding log files provide the detailed descriptions of the events

Data Manager runtime logs

.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\CCDmRtServer.log

WinCC data Manager runtime logs

Delta loader logs

.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\CCDeltaLoader.log

Delta loading within integrated projects

OPC UA server logs

.log

C:\ProgramData\Siemens\Automation\Logfiles\Sinec2.log

Operator messages logs

.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\WinCC_Op_xx.log

Operator messages which are generated during WinCC runtime

Project manager logs

.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\PMActivate.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\PMAutostart.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\PMDeactivate.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\PMDiagnose.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\PMProjectFile.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\PMService.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\PMStart.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\PMStop.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\PMTrace.log

Project manager events (open, close, start, stop, trace)

Script logs

.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\Script.log

Access violation in C or VB scripts generates the "Script.log" file. It contains, for example, image names, object names, properties/events, script names, and line number of the error in the script

Tag archiving logs

.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\CCTlgCommonArchiving.log

Tag logging and alarm server logs

.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\CCAlgRtCommonArchiving.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\CCTlgRtCommonArchiving.log

Text server logs

.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\CCTextServer.log

Trace files and diagnostics logs of the option packages

.log

C:\ProgramData\Siemens\Logs*

C:\Users\All Users\Siemens\Logs\*

User context surrogate logs

.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\CUCSurrogate.log

WinCC user context surrogate logs

WinCC license information

.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\License_Info.log

Information about used licenses

WinCC runtime events logs

.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\WinCC_Sys_xx.log

Events that occur during WinCC runtime

WinCC runtime logs

.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\PDLRT_GUI.log

WinCC startup logs

.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\WinCC_SStart_xx.log

Events that occur during WinCC startup in the runtime

Data Manager, Tag Archiving, and Servers logs

The following types of logs can be processed with a single configuration of NXLog Agent:

Table 4. List of logs
Log Type Path

Alarm logging server logs

C:\Program Files(x86)\Siemens\WinCC\diagnose\CCAlgRtCollectAlarm.log

Data Manager runtime logs

C:\Program Files(x86)\Siemens\WinCC\diagnose\CCDmRtServer.log

Tag archiving logs

C:\Program Files(x86)\Siemens\WinCC\diagnose\CCTlgCommonArchiving.log

Tag logging and alarm server logs

C:\Program Files(x86)\Siemens\WinCC\diagnose\CCAlgRtCommonArchiving.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\CCTlgRtCommonArchiving.log

Text server logs

C:\Program Files(x86)\Siemens\WinCC\diagnose\CCTextServer.log

The example below demonstrates how to read and process Data Manager runtime logs from the C:\Program Files (x86)\SIEMENS\WinCC\diagnose\CCDmRtServer.log file. To apply the configuration for other types of logs from the table above, simply replace the file name in the configuration.

Example 3. Processing Data Manager, Tag Archiving, and Servers logs

In this example, the input sample of a Data Manager runtime event shows which fields can be parsed: the timestamp of the event, the PID, TID, a short string that represents a parameter, another string terminated by a colon (:) conveying the loader type and a verbose message.

Input sample
12.05.2020 13:40:35.357 PID=6128 TID=15760 C  *CCDmRtServer: CChannel::StopCyclicReadRequest() on channel SIMATIC S7 Protocol Suite failed with error 0x800401fd in line SharedMemory.cpp (@387)

This NXLog Agent configuration needs to load the xm_charconv extension module because this type of log file is UCS-2LE encoded. In the _charconv extension instance, the LineReader directive sets the encoding type to be used. In the from_file input instance, the InputType directive is used for referencing the xm_charconv extension instance by name, _charconv, so that the log file data can be read using the correct encoding and then converted to UTF-8.

Reading UCS-2LE encoded logs
<Extension _charconv>
    Module        xm_charconv
    # Conversion from UCS-2LE to UTF-8
    LineReader    UCS-2LE
</Extension>

In the configuration file, the regular expression used for capturing these six fields is defined as the constant DATA_MANAGER_REGEX. The constant DIAGNOSE_PATH defines the absolute path to the log file. These constants will make the from_file input instance more readable and easier to maintain.

The logic for parsing and filtering is defined within the Exec block of the from_file instance of the im_file input module. All parsed fields that do not require additional processing are captured using the named capturing groups defined in DATA_MANAGER_REGEX. Each field name is defined within angle brackets (< >) that will determine which event field the captured value will be assigned to. The event record will be enriched with the following new fields: $PID, $TID, $ParamFive, $Loader, and $Message.

The timestamp string is captured by the regular expression as $1 and $2. However, parsedate() cannot readily parse them due to the non-standard string format. The strptime() function allows a custom format to be defined as the second argument, which it then uses for parsing and converting the string from the first argument to the datetime data type. This value is then assigned to the explicitly defined $EventTime field.

The drop procedure discards records that do not match the DATA_MANAGER_REGEX regular expression.

Then, by calling the to_json() procedure of the JSON (xm_json) extension module, the newly parsed fields along with the core fields are added to the event record and formatted as JSON before being routed to any output instances.

The following NXLog Agent configuration combines all the steps described above.

nxlog.conf
# Regular expression to read the message contents with named groups
define DATA_MANAGER_REGEX /(?x)^(\d+.\d+.\d+)\s+(\d+:\d+:\d+).\d+\
                          \s+PID=(?<PID>\d+)\s+TID=(?<TID>\d+)\s+\
                          (?<ParamFive>\w+)\s+\S(?<Loader>\w+):\s+\
                          (?<Message>.*)/
# Path to the folder with log files
define DIAGNOSE_PATH      C:\Program Files (x86)\SIEMENS\WinCC\diagnose

<Extension json>
    Module        xm_json
</Extension>

<Extension _charconv>
    Module        xm_charconv
    # Conversion from UCS-2LE to UTF-8
    LineReader    UCS-2LE
</Extension>

<Input from_file>
    Module        im_file
    File          '%DIAGNOSE_PATH%\CCDmRtServer.log'
    InputType     _charconv
    <Exec>
        if $raw_event =~ %DATA_MANAGER_REGEX%
        {
            # Creating a timestamp from the data and time values
            $EventTime = strptime($1 + $2,"%d.%m.%Y %T");
            to_json();    # Conversion to JSON format
        }
        else drop();      # Discarding messages not matching DATA_MANAGER_REGEX
    </Exec>
</Input>

The sample below depicts the processed event in JSON format.

Output sample
{
  "EventReceivedTime": "2020-10-21T21:36:30.324551+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Loader": "CCDmRtServer",
  "Message": "CChannel::StopCyclicReadRequest() on channel SIMATIC S7 Protocol Suite failed with error 0x800401fd in line SharedMemory.cpp (@387)",
  "PID": "6128",
  "ParamFive": "C",
  "TID": "15760",
  "EventTime": "2020-05-12T13:40:35.000000+03:00"
}

Channel Diagnostics logs

Channel diagnostics logs provide an overview of the status of each active connection at runtime. These log files are stored in the diagnose subdirectory contained within the WinCC directory, as indicated in the following table.

Table 5. List of files
Path Notice

C:\Program Files(x86)\Siemens\WinCC\diagnose\SIMATIC_S7_Protocol_Suite_*.log

Rotated file

C:\Program Files(x86)\Siemens\WinCC\diagnose\SimotionTA_*.log

Rotated file

C:\Program Files(x86)\Siemens\WinCC\diagnose\SIMATIC_S7-1200_S7-1500_Channel_*.log

Rotated file

C:\Program Files(x86)\Siemens\WinCC\diagnose\SIMATIC_S5_PROFIBUS_FDL_*.log

Rotated file

C:\Program Files(x86)\Siemens\WinCC\diagnose\System_Info.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\OPC.log

Filenames containing a wildcard character (*) indicate that they are rotated. For instance: SIMATIC_S7_Protocol_Suite_01.log, SIMATIC_S7_Protocol_Suite_02.log, etc.

NXLog Agent can collect the unstructured channel diagnostics data, parse it, and then convert it to structured data, which can then be stored as files or forwarded over the network. The example below explains how to perform all these steps.

Example 4. Processing Channel Diagnostics logs

All channel diagnostics log messages exhibit the following data patterns:

  • Date in the yyyy-mm-dd format.

  • Time in the hh:mm:ss,fff format.

  • Message type or severity level, for example, INFO.

  • Message text containing arbitrary textual information.

The following message sample shows how the data is arranged.

Input sample
2020-05-04 14:29:05,502 INFO     | LogFileName   : C:\Program Files (x86)\SIEMENS\WINCC\Diagnose\SIMATIC_S7_Protocol_Suite_01.LOG

Logs can be read via the im_file module. To track multiple files simultaneously. See the section on wildcards in the File (im_file) documentation.

Because these logs exhibit consistent data patterns, specific fields can be parsed using the following regular expression defined as CHANNEL_REGEX:

Defining CHANNEL_REGEX
define CHANNEL_REGEX /(?x)^(\d+-\d+-\d+)\s(\d+:\d+:\d+),\
                     \d+\s+(?<Type>\w+)\s+(?<Message>.*)/

The logic for parsing and filtering is defined within the Exec block of the from_file instance of the im_file input module. All parsed fields that do not require additional processing are captured using the named capturing groups defined in CHANNEL_REGEX. Each field name is defined within angle brackets (< >) that will determine which event field the captured value will be assigned to. The event record will be enriched with the $Message field in this example.

The timestamp string is captured by the regular expression as $1 and $2. However, parsedate() cannot readily parse them due to the non-standard string format. The strptime() function allows a custom format to be defined as the second argument, which it then uses for parsing and converting the string from the first argument to the datetime data type. This value is then assigned to the explicitly defined $EventTime field.

Then, by calling the to_json() procedure of the JSON (xm_json) extension module, the newly parsed fields along with the core fields are added to the event record and formatted as JSON before being routed to any output instances.

The following NXLog Agent configuration combines all the steps described above.

nxlog.conf
# Regular expression to read the message contents
define CHANNEL_REGEX /(?x)^(\d+-\d+-\d+)\s(\d+:\d+:\d+),\
                     \d+\s+(?<Type>\w+)\s+(?<Message>.*)/
# Path to the folder with log files
define DIAGNOSE_PATH C:\Program Files (x86)\SIEMENS\WinCC\diagnose

<Extension json>
    Module        xm_json
</Extension>

<Input from_file>
    Module        im_file
    # Using the asterisk in the file name enables reading of rotated files
    File          '%DIAGNOSE_PATH%\SIMATIC_S7_Protocol_Suite_*.log'
    <Exec>
        if $raw_event =~ %CHANNEL_REGEX%
        {
            # Creating a timestamp from the data and time values
            $EventTime = strptime($1 + $2,"%Y-%m-%d %T");
            to_json();    # Conversion to JSON format
        }
        else drop();      # Discarding messages not matching CHANNEL_REGEX
    </Exec>
</Input>

The sample below depicts the processed event in JSON format.

Output sample
{
  "EventReceivedTime": "2020-10-08T23:11:36.670055+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Message": "| LogFileName   : C:\\Program Files (x86)\\SIEMENS\\WINCC\\Diagnose\\SIMATIC_S7_Protocol_Suite_01.LOG",
  "Type": "INFO",
  "EventTime": "2020-05-04T14:29:05.000000+03:00"
}
Backslash characters (\) are escaped by default during processing as prescribed by the official JSON definition of a string ("Any codepoint except " or \ or control characters"). See the Introducing JSON section on the JSON website for more details.

Delta Loader logs

With the WinCC/Redundancy option, delta loader support eliminates the need to download the entire project when a project is modified. It downloads only changes that have been made to the project. Additionally, the changes can be applied to an activated project without deactivating it.

Delta loader logs are collected and stored in the CCDeltaLoader.log in the C:\Program Files (x86)\Siemens\WinCC\diagnose directory.

The example below demonstrates how to read, process, and save processed messages using NXLog Agent.

Example 5. Processing Delta Loader logs

This delta loader log sample exhibits three types of information that can be parsed: the delta loader name, a timestamp of the event formatted as dd.mm.yyyy hh:mm:ss, and a message that can vary depending on the nature of the event. In this case, it contains a state change (logging started) along with the absolute path and name the file that will include the events logged.

Input sample
CCDeltaLoader.exe : 24.04.2020 23:14:31 : Log C:\Program Files (x86)\SIEMENS\WINCC\Diagnose\CCDeltaLoader.Log started.

This NXLog Agent configuration needs to load the xm_charconv extension module because this type of log file is UCS-2LE encoded. In the _charconv extension instance, the LineReader directive sets the encoding type to be used. In the from_file input instance, the InputType directive is used for referencing the xm_charconv extension instance by name, _charconv, so that the log file data can be read using the correct encoding and then converted to UTF-8.

Reading UCS-2LE encoded logs
<Extension _charconv>
    Module        xm_charconv
    # Conversion from UCS-2LE to UTF-8
    LineReader    UCS-2LE
</Extension>

Because these logs exhibit consistent data patterns, specific fields can be parsed using the following regular expression defined as DELTA_REGEX:

Defining DELTA_REGEX
define DELTA_REGEX   /(?x)^(?<Loader>[a-zA-Z.]*)\s+:\s+(\d+.\d+.\d+)\
                     \s(\d+:\d+:\d+)\s:\s(?<Message>[\w():.\\\s]*)/

The logic for parsing and filtering is defined within the Exec block of the from_file instance of the im_file input module. All parsed fields that do not require additional processing are captured using the named capturing groups defined in DELTA_REGEX. Each field name is defined within angle brackets (< >) that will determine which event field the captured value will be assigned to. $Loader and $Message.

The timestamp string is captured by the regular expression as $2 and $3. However, parsedate() cannot readily parse them due to the non-standard string format. The strptime() function allows a custom format to be defined as the second argument, which it then uses for parsing and converting the string from the first argument to the datetime data type. This value is then assigned to the explicitly defined $EventTime field.

The drop procedure discards records that do not match the DELTA_REGEX regular expression.

Then, by calling the to_json() procedure of the JSON (xm_json) extension module, the newly parsed fields along with the core fields are added to the event record and formatted as JSON before being routed to any output instances.

The following NXLog Agent configuration combines all the steps described above.

nxlog.conf
# Regular expression with named groups for reading messages
define DELTA_REGEX   /(?x)^(?<Loader>[a-zA-Z.]*)\s+:\s+(\d+.\d+.\d+)\
                     \s(\d+:\d+:\d+)\s:\s(?<Message>[\w():.\\\s]*)/
# Path to the folder with logs
define DIAGNOSE_PATH C:\Program Files (x86)\siemens\wincc\diagnose

<Extension json>
    Module        xm_json
</Extension>

<Extension _charconv>
    Module        xm_charconv
    # Conversion from UCS-2LE to UTF-8
    LineReader    UCS-2LE
</Extension>

<Input from_file>
    Module        im_file
    File          '%DIAGNOSE_PATH%\CCDeltaLoader.Log'
    InputType     _charconv
    <Exec>
        if $raw_event =~ %DELTA_REGEX%
        {
            # Creating a timestamp from the date and time values
            $EventTime = strptime($2 + $3,"%d.%m.%Y %T");
            to_json();    # Conversion to JSON
        }
        else drop();      # Discarding if the DELTA_REGEX pattern is not matched
    </Exec>
</Input>

The sample below depicts the processed event in JSON format.

Output sample
{
  "EventReceivedTime": "2020-10-08T23:31:09.157668+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Loader": "CCDeltaLoader.exe",
  "Message": "Log C:\\Program Files (x86)\\SIEMENS\\WINCC\\Diagnose\\CCDeltaLoader.Log started.",
  "EventTime": "2020-04-24T23:14:31.000000+03:00"
}
Backslash characters (\) are escaped by default during processing as prescribed by the official JSON definition of a string ("Any codepoint except " or \ or control characters"). See the Introducing JSON section on the JSON website for more details.

OPC UA Server logs

Open Platform Communications (OPC) is a consortium responsible for creating and maintaining standards and specification groups for industrial automation. These standards are used to normalize how data from sensors and other industrial devices process and communicate data between each other.

OPC Unified Architecture (OPC UA) is the new machine-to-machine communications protocol that is used extensively in industrial automation and the IoT. It has incorporated the functionality of all previous OPC specifications, is no longer tied to a single operating system, and includes modern technologies such as secure authentication and client-server architecture.

The OPC UA Server logs are stored in the following file:

C:\ProgramData\Siemens\Automation\Logfiles\sinec2.log

NXLog Agent can be configured to process this kind of log. See the example below.

Example 6. Processing OPC UA Server logs

This OPC UA Server log sample taken from the sinec2.log file contains multiple lines for a single event. In such a case, NXLog Agent needs to be configured for multiline processing. The first line of the message will be treated as a message header containing the date, time, event type, and event ID data.

Input sample
05/05/2020 	 17:49:51 	 Event Type: E 	 Event ID: 2134
The computer name no longer matches the computer name in the OPC UA configuration.

Update the OPC UA configuration with the 'SIMATIC NET Configuration --> OPC Settings --> OPC UA Certificates --> Recreate OPC UA configuration' function of the 'Communication Settings' program.

The NXLog Agent configuration below defines a constants called OPC_UA_PATH defines the absolute path to the log file. Another constant, OPC_UA_LOG_HEADER, defines the regular expression for finding message headers.

To correctly process multiline event logs, a pattern needs to be defined as a regular expression that describes the header line of an event. In the following xm_multiline extension instance, the HeaderLine directive specifies the regular expression to be used for finding the header line of each event.

Enabling multiline event processing
<Extension _multiline>
    Module        xm_multiline
    # Parsing the header line
    HeaderLine    OPC_UA_LOG_HEADER
</Extension>

In the from_file input instance, the InputType directive is used for referencing the xm_multiline extension instance by name, _multiline, which will enable the instance to establish the beginning and end of each event.

To improve the readability of the parsed messages, the Exec block invokes the replace() function to replace all occurrences of \r\n (Windows line endings) and \t (tabs) with spaces.

Then, by calling the to_json() procedure of the JSON (xm_json) extension module, the newly parsed fields along with the core fields are added to the event record and formatted as JSON before being routed to any output instances.

The following NXLog Agent configuration combines all the steps described above.

nxlog.conf
# Regular expression for recognizing the message header
define OPC_UA_LOG_HEADER /(?x)^\d+\/\d+\/\d+\s+\d+:\d+:\d+\s+[\d\w\s:]*\d+/
# Path to the folder with log files
define OPC_UA_PATH       C:\ProgramData\Siemens\Automation\Logfiles

<Extension _multiline>
    Module        xm_multiline
    # Parsing the header line
    HeaderLine    %OPC_UA_LOG_HEADER%
</Extension>

<Extension json>
    Module        xm_json
</Extension>

<Input from_file>
    Module        im_file
    File          '%OPC_UA_PATH%\sinec2.log'
    InputType     _multiline
    <Exec>
        $Message = $raw_event;
        # Replacing the `\r\n` and `\t` combinations with white spaces
        $Message = replace($Message, "\r\n" , " ");
        $Message = replace($Message, "\t" , " ");
        to_json();        # Conversion of the read data to JSON
    </Exec>
</Input>

The sample below depicts the processed event in JSON format.

Output sample
{
  "EventReceivedTime": "2020-09-30T19:07:46.225581+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Message": "05/05/2020   17:49:51   Event Type: E   Event ID: 2134  The computer name no longer matches the computer name in the OPC UA configuration.  Update the OPC UA configuration with the 'SIMATIC NET Configuration --> OPC Settings --> OPC UA Certificates --> Recreate OPC UA configuration' function of the 'Communication Settings' program."
}

Project Manager logs

There are multiple types of Project Manager logs which are all stored in the C:\Program Files(x86)\Siemens\WinCC\diagnose directory as shown in the table below.

Table 6. List of files with Project Manager logs

C:\Program Files(x86)\Siemens\WinCC\diagnose\PMActivate.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\PMAutostart.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\PMDeactivate.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\PMDiagnose.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\PMProjectFile.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\PMService.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\PMStart.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\PMStop.log

C:\Program Files(x86)\Siemens\WinCC\diagnose\PMTrace.log

Since NXLog Agent supports wildcards in filenames and these log files all start with PM and have a .log filename extension, only a single input instance is needed to read them. The following example demonstrates how NXLog Agent can be configured to process Project Manager log events and output them as structured data which can then be saved to a file or forwarded over the network.

Example 7. Processing Project Manager logs

Based on the Project Manager input sample below, the following fields can be parsed:

  • Date in the yyyy-mm-dd format

  • Time in the hh:mm:ss format

  • Type, in this case, this is the PM combination

  • PID or process ID information

  • TID or thread ID information

  • Arbitrary message text

Input sample
2020-10-08 21:44:17.969 PM( 1228-02768) CServiceModule::Run Service is started

This NXLog Agent configuration defines two constants: PM_REGEX and DIAGNOSE_PATH which defines the absolute path to the log files. In the from_file input instance, the File directive uses DIAGNOSE_PATH and the wildcarded filename PM*.log to read all the logs.

Because these logs exhibit consistent data patterns, specific fields can be parsed using the following regular expression defined as PM_REGEX:

Defining PM_REGEX
define PM_REGEX       /(?x)^(\d+-\d+-\d+)\s+(\d+:\d+:\d+).\d+\s+\
                      (?<Type>\w+)\(\s*(?<PID>\d*)\-(?<TID>\d*)\)\
                      \s*(?<Message>.*)/

The logic for parsing and filtering is defined within the Exec block of the from_file instance of the im_file input module. All parsed fields that do not require additional processing are captured using the named capturing groups defined in PM_REGEX. Each field name is defined within angle brackets (< >) that will determine which event field the captured value will be assigned to. In this example, the event record will be enriched with the following fields: $Type, $PID, $TID, and $Message.

The timestamp string is captured by the regular expression as $1 and $2. However, parsedate() cannot readily parse them due to the non-standard string format. The strptime() function allows a custom format to be defined as the second argument, which it then uses for parsing and converting the string from the first argument to the datetime data type. This value is then assigned to the explicitly defined $EventTime field.

The drop procedure discards records that do not match the PM_REGEX regular expression.

Then, by calling the to_json() procedure of the JSON (xm_json) extension module, the newly parsed fields along with the core fields are added to the event record and formatted as JSON before being routed to any output instances.

The following NXLog Agent configuration combines all the steps described above.

nxlog.conf
# Regular expression for parsing log entries with named groups
define PM_REGEX       /(?x)^(\d+-\d+-\d+)\s+(\d+:\d+:\d+).\d+\s+\
                      (?<Type>\w+)\(\s*(?<PID>\d*)\-(?<TID>\d*)\)\
                      \s*(?<Message>.*)/
# Path to the folder with log files
define DIAGNOSE_PATH  C:\Program Files (x86)\SIEMENS\WinCC\diagnose

<Extension json>
    Module        xm_json
</Extension>

<Input from_file>
    Module        im_file
    File          '%DIAGNOSE_PATH%\PM*.log'
    <Exec>
        if $raw_event =~ %PM_REGEX%
        {
            # Transforming the date and time values to the single timestamp
            $EventTime = strptime($1 + $2,"%Y-%m-%d %T");
            to_json();    # Converting to JSON
        }
        else drop();      # Discarding messages if not matching the PM_REGEX
    </Exec>
</Input>

The sample below depicts the processed event in JSON format.

Output sample
{
  "EventReceivedTime": "2020-10-21T18:53:08.336289+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Message": "CServiceModule::Run Service is started",
  "PID": "1228",
  "TID": "02768",
  "Type": "PM",
  "EventTime": "2020-10-08T21:44:17.000000+03:00"
}

Trace Files and Diagnostics logs

Trace files and diagnostic logs are stored as files in the following directories:

  • C:\ProgramData\Siemens\Logs\

  • C:\Users\All Users\Siemens\Logs\

The example below explains how to process trace files and diagnostics logs using NXLog Agent. Processing here means that the data will be parsed, processed, and formatted as structured data, which is usually either saved to a file or forwarded over the network.

Example 8. Processing Trace Files and Diagnostics logs

This log sample contains various fields that can be parsed, such as the entry number, timestamp, process ID (PID), thread ID (TID), address, process abbreviation, ParamSeven, and two message fields.

Input sample
      3; 2020-04-20T22:08:48.327Z;  3356;  2844; [0x5A098658]; BCS\ECT; E; CBCSEqmControlConfig::LoadConfigFromIniFile; "GetCurrentDbIdent failed with hr=0x80040500"

This NXLog Agent configuration defines two constants: TRACE_REGEX and TRACE_PATH which defines the absolute path to the trace files. In the from_file input instance, the File directive uses TRACE_PATH and a wildcard (*) to read all files in this directory.

Because these logs exhibit consistent data patterns, specific fields can be parsed using the following regular expression defined as TRACE_REGEX:

Defining TRACE_REGEX
define TRACE_REGEX  /(?x)^[^\w\d]*(\d*);\s*(\d*-\d*\
                    -\d*\w\d*:\d*:\d*.\d*\w);\s*(?<PID>\d*);\
                    \s*(?<TID>\d*);\s*\[(?<Address>\w*\d*)\];\
                    \s(?<Process>[\w\\]*);\s*(?<ParamSeven>\w*)\
                    ;(?<MessageOne>.*);\s*\"(?<MessageTwo>.*)\"/

The logic for parsing and filtering is defined within the Exec block of the from_file instance of the im_file input module. All parsed fields that do not require additional processing are captured using the named capturing groups defined in TRACE_REGEX. Each field name is defined within angle brackets (< >) that will determine which event field the captured value will be assigned to. In this example, the event record will be enriched with the following fields: $PID, $TID, $Address, $Process, $ParamSeven, $MessageOne, and $MessageTwo.

The first field to be captured as $1 using TRACE_REGEX is the $EntryNumber which is then converted to an integer using the integer() function. The second field to be captured as $2 is the timestamp of the event which is in a standardized string format that parsedate() can convert to the datetime data type.

The drop procedure discards records that do not match the TRACE_REGEX regular expression.

Then, by calling the to_json() procedure of the JSON (xm_json) extension module, the newly parsed fields along with the core fields are added to the event record and formatted as JSON before being routed to any output instances.

The following NXLog Agent configuration combines all the steps described above.

nxlog.conf
# Regular expression for parsing logs
define TRACE_REGEX  /(?x)^[^\w\d]*(\d*);\s*(\d*-\d*\
                    -\d*\w\d*:\d*:\d*.\d*\w);\s*(?<PID>\d*);\
                    \s*(?<TID>\d*);\s*\[(?<Address>\w*\d*)\];\
                    \s(?<Process>[\w\\]*);\s*(?<ParamSeven>\w*)\
                    ;(?<MessageOne>.*);\s*\"(?<MessageTwo>.*)\"/
# Path to the folder with the trace files
define TRACE_PATH   C:\ProgramData\Siemens\Logs\REGSVR32\trace

<Extension json>
    Module        xm_json
</Extension>

<Input from_file>
    Module        im_file
    File          '%TRACE_PATH%\*'
    <Exec>
        if $raw_event =~ %TRACE_REGEX% 
        {
            # Converting to integer
            $EntryNumber = integer($1);
            # Parsing the timestamp
            $Timestamp = parsedate($2);
            to_json();    # Converting to JSON
        }
        else drop();      # Discarding messages if not matching TRACE_REGEX
    </Exec>
</Input>

The sample below depicts the processed event in JSON format.

Output sample
{
  "EventReceivedTime": "2020-10-21T22:22:58.971760+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Address": "0x5A098658",
  "MessageOne": " CBCSEqmControlConfig::LoadConfigFromIniFile",
  "MessageTwo": "GetCurrentDbIdent failed with hr=0x80040500",
  "PID": "3356",
  "ParamSeven": "E",
  "Process": "BCS\\ECT",
  "TID": "2844",
  "EntryNumber": 3,
  "Timestamp": "2020-04-21T01:08:48.327000+03:00"
}

License, Runtime, and User Context Surrogate logs

WinCC license information, runtime logs, and user context surrogate logs have the same structure of their log messages and are stored under the following paths:

Table 7. Log types
Log Type Path

User Context Surrogate Logs

C:\Program Files(x86)\Siemens\WinCC\diagnose\CUCSurrogate.log

WinCC License Information

C:\Program Files(x86)\Siemens\WinCC\diagnose\License_Info.log

WinCC Runtime Logs

C:\Program Files(x86)\Siemens\WinCC\diagnose\PDLRT_GUI.log

The example below explains how these types of logs can be processed with NXLog Agent. In this example, user context surrogate logs are selected for processing.

Example 9. Processing User Context Surrogate logs

The following event sample was taken from a CUCSurrogate.log file. To process WinCC license information or runtime logs, substitute the filename CUCSurrogate.log assigned to the File directive of the from_file input instance with either License_Info.log or PDLRT_GUI.log.

Input sample
08.10.2020 21:44:27   OnChangeState [0]

Based on the sample event, only the timestamp and a message field can be parsed.

Because these logs exhibit consistent data patterns, specific fields can be parsed using the following regular expression defined as EVENT_REGEX:

Defining EVENT_REGEX
define EVENT_REGEX   /(?x)^(\d+.\d+.\d+)\s+(\d+:\d+:\d+)\s+(?<Message>.*)/

The logic for parsing and filtering is defined within the Exec block of the from_file instance of the im_file input module. All parsed fields that do not require additional processing are captured using the named capturing groups defined in EVENT_REGEX. Each field name is defined within angle brackets (< >) that will determine which event field the captured value will be assigned to. In this example, the event record will be enriched with the $Message field.

The timestamp string is captured by the regular expression as $1 and $2. However, parsedate() cannot readily parse them due to the non-standard string format. The strptime() function allows a custom format to be defined as the second argument, which it then uses for parsing and converting the string from the first argument to the datetime data type. This value is then assigned to the explicitly defined $EventTime field.

The drop procedure discards records that do not match the EVENT_REGEX regular expression.

Then, by calling the to_json() procedure of the JSON (xm_json) extension module, the newly parsed fields along with the core fields are added to the event record and formatted as JSON before being routed to any output instances.

The following NXLog Agent configuration combines all the steps described above.

nxlog.conf
# Regular expression for parsing user context surrogate logs
define EVENT_REGEX   /(?x)^(\d+.\d+.\d+)\s+(\d+:\d+:\d+)\s+(?<Message>.*)/
# Path to the folder with logs
define DIAGNOSE_PATH C:\Program Files (x86)\SIEMENS\WinCC\diagnose

<Extension json>
    Module        xm_json
</Extension>

<Input from_file>
    Module        im_file
    File          '%DIAGNOSE_PATH%\CUCSurrogate.log'
    <Exec>
        if $raw_event =~ %EVENT_REGEX%
        {
            # Converting the date and time values to the single timestamp
            $EventTime = strptime($1 + $2,"%d.%m.%Y %T");
            to_json();    # Converting the data to JSON format
        }
        else drop();      # Discarding messages if not matching EVENT_REGEX
    </Exec>
</Input>

The sample below depicts the processed event in JSON format.

Output sample
{
  "EventReceivedTime": "2020-10-09T13:35:47.981193+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Message": "OnChangeState [0]",
  "EventTime": "2020-10-08T21:44:27.000000+03:00"
}

Startup, Runtime, and Operator logs

WinCC runtime event logs, startup logs, and operator logs exhibit similar data patterns and can be processed using a single NXLog Agent configuration.

The table below lists the path and filename pattern associated with each type of WinCC log.

Table 8. Log types and paths
Log Type Path

WinCC runtime event logs

C:\Program Files (x86)\Siemens\WinCC\diagnose\WinCC_Sys_*.log

WinCC startup logs

C:\Program Files (x86)\Siemens\WinCC\diagnose\WinCC_SStart_*.log

Operator logs

C:\Program Files(x86)\Siemens\WinCC\diagnose\WinCC_Op_*.log

Since the im_file module supports wildcards, a filename pattern can be used to process all the files listed in the table above, including multiple files of the same log type, e.g., WinCC_Sys_01.log, WinCC_Sys_02.log, etc. See the section on wildcards in the File (im_file) documentation for details.

Example 10. Processing Startup, Runtime, and Operator logs

This a sample event which is available under the following path:

C:\Program Files (x86)\Siemens\WinCC\diagnose\WinCC_SStart_*.log

Input sample
2303,30.04.2022,23:49:41:770,1000204,4,,WINSRV99,DataManager Runtime,S7$Program(1)	Connection 'S7$Program(1)' disconnected,MSG_STATE_COME

Based on the sample above, the following fields can be parsed:

  • ID (numeric)

  • Timestamp (string format: dd.mm.yyyy,hh:mm:ss:fff)

  • Message Number

  • Message Class (numeric)

  • Station

  • Application

  • ErrorMessage

  • State

This NXLog Agent configuration needs to load the xm_charconv extension module because this type of log file is UCS-2LE encoded. In the _charconv extension instance, the LineReader directive sets the encoding type to be used. In the from_file input instance, the InputType directive is used for referencing the xm_charconv extension instance by name, _charconv, so that the log file data can be read using the correct encoding and then converted to UTF-8.

Reading UCS-2LE encoded logs
<Extension _charconv>
    Module        xm_charconv
    # Conversion from UCS-2LE to UTF-8
    LineReader    UCS-2LE
</Extension>

Because these logs exhibit consistent data patterns, specific fields can be parsed using the following regular expressions defined by STARTUP_REGEX and OP_REGEX:

Defining regular expressions
# Regular expression for WinCC startup and runtime logs
define STARTUP_REGEX /(?x)^(?<ID>\d+)\,(\d+.\d+.\d+)\,(\d+\:\d+\:\d+)\:\d+\,\
                     (?<MessageNumber>\d+)\,(?<MessageClass>\d+)\,\
                     (?<UserName>.*?)\,(?<Station>.*?)\,(?<Application>.*?)\,\
                     (?<ErrorMessage>.*?)\,(?<State>.*[^,])$/

# Regular expression for operator log messages
define OP_REGEX      /(?x)^(?<ID>\d+)\,(\d+.\d+.\d+)\,(\d+:\d+:\d+):\d+\,\
                    (?<MessageNumber>\d+)\,(?<MessageClass>\d+)\,\
                    (?<UserName>.*?)\,(?<Station>.*?)\,(?<Application>.*?)\,\
                    (?<ErrorMessage>.*?)\,(?<VarName>.*?)\,(?<OldValue>.*?)\,\
                    (?<NewValue>.*?)\,(?<State>.*?)\,(?<Reason>.*?)\,+/

The logic for parsing and filtering is defined within the Exec block of the from_file instance of the im_file input module. All parsed fields that do not require additional processing are captured using the named capturing groups defined in STARTUP_REGEX or OP_REGEX. Each field name is defined within angle brackets (< >) that will determine which event field the captured value will be assigned to. In this example, depending on which of the two regular expressions an event matches, the event record will be enriched with a subset of the following fields: MessageNumber, MessageClass, UserName, $Station, $Application, $ErrorMessage, $State, VarName, OldValue, NewValue, Reason.

The timestamp string is captured by the regular expression as $2 and $3. However, parsedate() cannot readily parse them due to the non-standard string format. The strptime() function allows a custom format to be defined as the second argument, which it then uses for parsing and converting the string from the first argument to the datetime data type. This value is then assigned to the explicitly defined $EventTime field.

The drop procedure discards records that match neither the STARTUP_REGEX nor the OP_REGEX regular expressions.

Then, by calling the to_json() procedure of the JSON (xm_json) extension module, the newly parsed fields along with the core fields are added to the event record and formatted as JSON before being routed to any output instances.

The following NXLog Agent configuration combines all the steps described above.

nxlog.conf
# Regular expression for WinCC startup and runtime logs
define STARTUP_REGEX /(?x)^(?<ID>\d+)\,(\d+.\d+.\d+)\,(\d+\:\d+\:\d+)\:\d+\,\
                     (?<MessageNumber>\d+)\,(?<MessageClass>\d+)\,\
                     (?<UserName>.*?)\,(?<Station>.*?)\,(?<Application>.*?)\,\
                     (?<ErrorMessage>.*?)\,(?<State>.*[^,])$/

# Regular expression for operator log messages
define OP_REGEX      /(?x)^(?<ID>\d+)\,(\d+.\d+.\d+)\,(\d+:\d+:\d+):\d+\,\
                     (?<MessageNumber>\d+)\,(?<MessageClass>\d+)\,\
                     (?<UserName>.*?)\,(?<Station>.*?)\,(?<Application>.*?)\,\
                     (?<ErrorMessage>.*?)\,(?<VarName>.*?)\,(?<OldValue>.*?)\,\
                     (?<NewValue>.*?)\,(?<State>.*?)\,(?<Reason>.*?)\,+/

# Path to the folder with log files
define DIAGNOSE_PATH C:\Program Files (x86)\Siemens\WinCC\diagnose

<Extension json>
    Module        xm_json
</Extension>

<Extension _charconv>
    Module        xm_charconv
    # Converting data from UCS-2LE to UTF-8
    LineReader    UCS-2LE
</Extension>

<Input from_file>
    Module        im_file
    File          '%DIAGNOSE_PATH%\WinCC_[OS]*.log'
    InputType     _charconv
    <Exec>
        # Replaces unwanted characters
        $raw_event =~ s/\t/ /g;
        # Matches the events with a regular expressions
        if $raw_event =~ %STARTUP_REGEX% or $raw_event =~ %OP_REGEX%
        {
            # Processing timestamp
            $EventTime = strptime($2 + $3,"%d.%m.%Y%T");
            # Converting to JSON
            to_json();
        }
        # Discarding messages if they do not match
        else drop();
    </Exec>
</Input>

The sample below depicts the processed event in JSON format.

Output sample
{
  "EventReceivedTime": "2022-04-30T23:49:42.785431+02:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Application": "DataManager Runtime",
  "ErrorMessage": "S7$Program(1) Connection 'S7$Program(1)' disconnected",
  "ID": "2303",
  "MessageClass": "4",
  "MessageNumber": "1000204",
  "State": "MSG_STATE_COME",
  "Station": "WINSRV99",
  "UserName": "",
  "EventTime": "2022-04-30T23:49:41.000000+02:00"
}

WinCC SQL Server logs

Microsoft SQL Server logs are rotated as shown in the following table.

Table 9. WinCC SQL Server logs
Name File
Ext.
Location Description

Error Logs

.1

.2

…​

C:\Program Files (x86)\Microsoft SQL Server\MSSQL12.WINCC\MSSQL\Log\ERRORLOG.*

Rotated files

SQL Agent Logs

.1

.2

…​

C:\Program Files (x86)\Microsoft SQL Server\MSSQL12.WINCC\MSSQL\Log\SQLAGENT.*

Rotated files

NXLog Agent can be configured to process SQL Server logs.

SQL Agent logs

SQL Agent logs are located in C:\Program Files (x86)\Microsoft SQL Server\MSSQL12.WINCC\MSSQL\Log\ where they are rotated in the same way as SQLAGENT.* log files.

The following example explains how to process SQL Agent logs using NXLog Agent.

Example 11. Processing SQL Agent logs

This sample event from an SQL Agent log contains a timestamp and a message.

Input sample
2020-10-08 21:40:01 - ? [495] The SQL Server Agent startup service account is WORKGROUP\WINSRV99$.

This NXLog Agent configuration needs to load the xm_charconv extension module because this type of log file is UCS-2LE encoded. In the _charconv extension instance, the LineReader directive sets the encoding type to be used. In the from_file input instance, the InputType directive is used for referencing the xm_charconv extension instance by name, _charconv, so that the log file data can be read using the correct encoding and then converted to UTF-8.

Reading UCS-2LE encoded logs
<Extension _charconv>
    Module        xm_charconv
    # Conversion from UCS-2LE to UTF-8
    LineReader    UCS-2LE
</Extension>

Because these logs exhibit consistent data patterns, specific fields can be parsed using the following regular expression defined as EVENT_SQLAGENT:

Defining EVENT_SQLAGENT
define EVENT_SQLAGENT  /(?x)^(\d+-\d+-\d+)\s+(\d+:\d+:\d+)\
                       \s+-\s+\W\s+(?<Message>.*)/

The logic for parsing and filtering is defined within the Exec block of the from_file instance of the im_file input module. All parsed fields that do not require additional processing are captured using the named capturing groups defined in EVENT_SQLAGENT. Each field name is defined within angle brackets (< >) that will determine which event field the captured value will be assigned to. In this example, the event record will be enriched with the $Message field.

The timestamp string is captured by the regular expression as $1 and $2. However, parsedate() cannot readily parse them due to the non-standard string format. The strptime() function allows a custom format to be defined as the second argument, which it then uses for parsing and converting the string from the first argument to the datetime data type. This value is then assigned to the explicitly defined $EventTime field.

The drop procedure discards records that do not match the EVENT_SQLAGENT regular expression.

Then, by calling the to_json() procedure of the JSON (xm_json) extension module, the newly parsed fields along with the core fields are added to the event record and formatted as JSON before being routed to any output instances.

The following NXLog Agent configuration combines all the steps described above.

nxlog.conf
# Regular expression for reading logs
define EVENT_SQLAGENT  /(?x)^(\d+-\d+-\d+)\s+(\d+:\d+:\d+)\
                       \s+-\s+\W\s+(?<Message>.*)/
# Path to the folder with SQL log files
define SQL_SERVER_PATH C:\Program Files (x86)\Microsoft SQL Server

<Extension _charconv>
    Module        xm_charconv
    # Converting from UCS-2LE to UTF-8
    LineReader    UCS-2LE
</Extension>

<Extension json>
    Module        xm_json
</Extension>

<Input from_file>
    Module        im_file
    File          '%SQL_SERVER_PATH%\MSSQL12.WINCC\MSSQL\Log\SQLAGENT.*'
    InputType     _charconv
    <Exec>
        if $raw_event =~ %EVENT_SQLAGENT%
        {
            # Converting of the data and time values to the single timestamp
            $EventTime = strptime($1 + $2, "%Y-%m-%d %T");
            to_json();    # Converting to JSON
        }
        else drop();      # Discarding messages if not matching EVENT_SQLAGENT
    </Exec>
</Input>

The sample below depicts the processed event in JSON format.

Output sample
{
  "EventReceivedTime": "2020-10-18T23:13:17.546939+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Message": "[495] The SQL Server Agent startup service account is WORKGROUP\\WINSRV99$.",
  "EventTime": "2020-10-08T21:40:01.000000+03:00"
}

OS Project logs

Project logs on an Operator Station are stored in various files and formats, as listed in the table below.

Table 10. OS Project logs
Name File
Ext.
Location Description

Alarm logs

.mdf

.ldf

[MultiprojectName]\[OS_projectName]\wincproj\[OS_serverName]\ArchiveManager\AlarmLogging\*

Alarm and message archiving

Create/update block icons logs

.log

[MultiprojectName][OS_projectName]\wincproj[OS_serverName]\THPOfile.log

[MultiprojectName][OS_projectName]\AMOBJS\THPOfile.log

Shows which block icons are created and which tags are associated with them

Fast tag logging

.mdf

.ldf

[MultiprojectName]\[OS_projectName]\wincproj\[OS_serverName]\ArchiveManager\TagloggingFast\*

Process data archiving with the cycle shorter than 1 min

OS compilation logs

.zip

.log

[MultiprojectName][OS_projectName]\protocols\OS_Compiling\yyyymmdd_hhmmss_TransferLog.zip\TRANSFER.log

[MultiprojectName][OS_projectName]\TRANSFER.log

[MultiprojectName][OS_projectName]\wincproj[OS_serverName]\TRANSFER.log

OS compilation log, type of changes

.log

[MultiprojectName][OS_projectName]\protocols[yyyymmdd_hhmmss_USER].LOG

Slow tag logging

.mdf

.ldf

[MultiprojectName]\[OS_projectName]\wincproj[OS_serverName]\ArchiveManager\TagloggingSlow\*

Process data archiving with the cycle longer than 1 min

User archives

.txt

[MultiprojectName][OS_projectName]\wincproj[OS_serverName]\UA\UALogFile.txt

OS Event Log associated with the user archive

User Archives

The WinCC User Archives is a user-configurable database system that enables users to store custom data fields.

The logs generated by User Archives are stored in UALogFile.txt in the path listed above, where they can readily be processed by NXLog Agent.

Example 12. Processing User Archives

This sample event was taken from UALogFile.txt.

Input sample
2020-02-10 08:48:12.803 [p=11912, t=0x3370] Sync: Redundancy is not activated

Based on the sample above, the following fields can be parsed:

  • Timestamp (string format: yyyy-mm-dd hh:mm:ss)

  • PID (numeric process ID)

  • TID (numeric thread ID)

  • Message

Because these logs exhibit consistent data patterns, specific fields can be parsed using the following regular expression defined as ARCHIVE_REGEX:

Defining ARCHIVE_REGEX
define ARCHIVE_REGEX /(?x)^(\d+-\d+-\d+)\s+(\d+:\d+:\d+).\
                     \d+\s+\[p=(?<PID>\d+),\s+t=(?<TID>.*)\
                     \]\s+(?<Message>.*)/

The logic for parsing and filtering is defined within the Exec block of the from_file instance of the im_file input module. All parsed fields that do not require additional processing are captured using the named capturing groups defined in ARCHIVE_REGEX. Each field name is defined within angle brackets (< >) that will determine which event field the captured value will be assigned to. In this example, the event record will be enriched with the following fields: $PID, $TID, and $Message.

The timestamp string is captured by the regular expression as $1 and $2. However, parsedate() cannot readily parse them due to the non-standard string format. The strptime() function allows a custom format to be defined as the second argument, which it then uses for parsing and converting the string from the first argument to the datetime data type. This value is then assigned to the explicitly defined $EventTime field.

The drop procedure discards records that do not match the ARCHIVE_REGEX regular expression.

Then, by calling the to_json() procedure of the JSON (xm_json) extension module, the newly parsed fields along with the core fields are added to the event record and formatted as JSON before being routed to any output instances.

The following NXLog Agent configuration combines all the steps described above.

nxlog.conf
# Regular expression for parsing user archives
define ARCHIVE_REGEX /(?x)^(\d+-\d+-\d+)\s+(\d+:\d+:\d+).\
                     \d+\s+\[p=(?<PID>\d+),\s+t=(?<TID>.*)\
                     \]\s+(?<Message>.*)/
# Path to the folder with logs
define PROJECT_PATH  C:\Users\Engineer\Documents\Project\BEV_MP

<Extension json>
    Module        xm_json
</Extension>

<Input from_file>
    Module        im_file
    File          '%PROJECT_PATH%\Hmi\wincproj\HMI\UA\UALogFile.txt'
    <Exec>
        if $raw_event =~ %ARCHIVE_REGEX%
        {
            # Converting the date and time values to the single timestamp
            $EventTime = strptime($1 + $2,"%Y-%m-%d %T");
            to_json();    # Converting parsed data to JSON
        }
        else drop();      # Discarding messages if not matching ARCHIVE_REGEX
    </Exec>
</Input>

The sample below depicts the processed event in JSON format.

Output sample
{
  "EventReceivedTime": "2020-10-21T22:58:52.266852+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Message": "Sync: Redundancy is not activated",
  "PID": "11912",
  "TID": "0x3370",
  "EventTime": "2020-02-10T08:48:12.000000+02:00"
}

AS Project logs

The Automation Station logs are collected in several files, which are listed in the table below.

Table 11. AS Project logs
Name File
Ext.
Location Description

AS compilation logs

.log

[MultiprojectName][AS_projectName]\protocols\*

- AS compilation logs
- Downloaded logs of the Automation Station
- Block types logs
- Logs from module drivers
- Read back logs
- Textual interconnections logs

Protocol of loading

.log

[MultiprojectName]\[AS_projectName]\ES_LOC\1\GEN\LOADPROT.log

Load reference

.log

[MultiprojectName]\[AS_projectName]\ES_LOC\1\GEN\LOADREFS.log

Blocks that have been loaded to the PLC

Protocol of Loading

Log entries generated by the loading protocol are collected in the LOADPROT.log file is located in the Automation Station project directory. The following example explains how to configure NXLog Agent for processing AS protocol of loading logs.

Example 13. Processing the Protocol of Loading logs

The following three input samples are related to the protocol of loading.

Each protocol of loading event starts with a header containing the protocol type, path, and timestamp.

Protocol header
 ***** Protocol Of Loading <BEV_Prj\SIMATIC 400(1)\CPU 417-5 H PN/DP\S7 Program(1)\Charts> on 03/13/20 11:53:04

Based on this header event sample, these fields can be parsed:

  • Type

  • Path

  • Date (string format: mm/dd/yy)

  • Time (string format: hh:mm:ss)

Two types of events are generated after the header event:

The first type of logs
[11:56:00.828 AM] EM_CIP_PROC.EM_CIP_PROC [DB133] (OVERLOAD)

Based on this first type of event sample, these fields can be parsed:

  • Time (string format: hh:mm:ss.sss)

  • Daypart (AM/PM)

  • Process

  • ParamFour

  • Status

The second type of logs
[11:59:33.618 AM] POOL_DB_DBE (LOADED)

Based on this second type of event sample, these fields can be parsed:

  • Time (string format: hh:mm:ss.sss)

  • Daypart (AM/PM)

  • Process

  • Status

Because these logs exhibit consistent data patterns, specific fields can be parsed using the following regular expressions defined by REGEX_LOADPROT_HEADER, REGEX_LOADPROT_ONE, and REGEX_LOADPROT_TWO:

Defining regular expressions
# Regular expression to recognize the message header
define REGEX_LOADPROT_HEADER /(?x)^\s?\**(?<Type>[\w\s]*)\<(?<Path>\
                             [-_ \\\/\(\)\d*\w*]*)\>\s*on\s*(?<Date>\
                             \d+\/\d+\/\d+)\s*(?<Time>\d+:\d+:\d+)/
# Regular expression for processing the first type of logs
define REGEX_LOADPROT_ONE    /(?x)^\[?(?<Time>\d+:\d+:\d+.\d+)\s*\
                             (?<Daypart>AM|PM)\]?\s*(?<Process>\
                             [._ \(\)~@\-\w]*)\[(?<ParamFour>[\w\d]*)\]\
                             \s*\((?<Status>[\w]*)\)/
# Regular expression for processing the second type of logs
define REGEX_LOADPROT_TWO    /(?x)^\[?(?<Time>\d+:\d+:\d+.\d+)\s*\
                             (?<Daypart>AM|PM)\]?\s*(?<Process>.*)\
                             \s*\((?<Status>.*)\)/

The logic for parsing and filtering is defined within the Exec block of the from_file instance of the im_file input module. All parsed fields that do not require additional processing are captured using the named capturing groups contained defined by REGEX_LOADPROT_HEADER, REGEX_LOADPROT_ONE or REGEX_LOADPROT_TWO. Each field name is defined within angle brackets (< >) that will determine which event field the captured value will be assigned to. In this example, depending on which of the three regular expressions an event matches, the event record will be enriched with a subset of the following fields: $Type, $Path, $Date, $Time, $Daypart, $Process, $ParamFour, and/or $Status.

The drop procedure discards records that match neither the REGEX_LOADPROT_HEADER nor the REGEX_LOADPROT_ONE nor the REGEX_LOADPROT_TWO regular expressions.

Then, by calling the to_json() procedure of the JSON (xm_json) extension module, the newly parsed fields along with the core fields are added to the event record and formatted as JSON before being routed to any output instances.

The following NXLog Agent configuration combines all the steps described above.

nxlog.conf
# Regular expression to recognize the message header
define REGEX_LOADPROT_HEADER /(?x)^\s?\**(?<Type>[\w\s]*)\<(?<Path>\
                             [-_ \\\/\(\)\d*\w*]*)\>\s*on\s*(?<Date>\
                             \d+\/\d+\/\d+)\s*(?<Time>\d+:\d+:\d+)/
# Regular expression for processing the first type of logs
define REGEX_LOADPROT_ONE    /(?x)^\[?(?<Time>\d+:\d+:\d+.\d+)\s*\
                             (?<Daypart>AM|PM)\]?\s*(?<Process>\
                             [._ \(\)~@\-\w]*)\[(?<ParamFour>[\w\d]*)\]\
                             \s*\((?<Status>[\w]*)\)/
# Regular expression for processing the second type of logs
define REGEX_LOADPROT_TWO    /(?x)^\[?(?<Time>\d+:\d+:\d+.\d+)\s*\
                             (?<Daypart>AM|PM)\]?\s*(?<Process>.*)\
                             \s*\((?<Status>.*)\)/
# Path to the Automation Station project folder 
define PROJECT_PATH          C:\Users\Engineer\Documents\Project\BEV_MP

<Extension json>
    Module        xm_json
</Extension>

<Input from_file>
    Module        im_file
    File          '%PROJECT_PATH%\BEV_Prj\ES_LOC\1\GEN\loadprot.log'
    <Exec>
        # Converting parsed data to JSON
        if $raw_event =~ %REGEX_LOADPROT_HEADER% to_json();
        else if $raw_event =~ %REGEX_LOADPROT_ONE% to_json();
        else if $raw_event =~ %REGEX_LOADPROT_TWO% to_json();
        else drop();      # Discard event if it doesn't match a/the regular expression
    </Exec>
</Input>

The samples below depict the processed events in JSON format.

Output sample of the Protocol Header
{
  "EventReceivedTime": "2020-10-09T22:15:10.553269+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Date": "03/13/20",
  "Path": "BEV_Prj\\SIMATIC 400(1)\\CPU 417-5 H PN/DP\\S7 Program(1)\\Charts",
  "Time": "11:53:04",
  "Type": " Protocol Of Loading "
}
Output sample of the first type of logs
{
  "EventReceivedTime": "2020-10-09T22:15:10.553269+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Daypart": "AM",
  "ParamFour": "DB133",
  "Process": "EM_CIP_PROC.EM_CIP_PROC ",
  "Status": "OVERLOAD",
  "Time": "11:56:00.828"
}
Output sample of the second type of logs
{
  "EventReceivedTime": "2020-10-09T22:48:45.054997+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Daypart": "AM",
  "Process": "POOL_DB_DBE ",
  "Status": "LOADED",
  "Time": "11:59:33.618"
}

Load reference

Load reference logs are stored in the Automation Station project directory as LOADREFS.log.

NXLog Agent can be configured to process load reference logs, as demonstrated in the following example.

Example 14. Processing load reference logs

Load reference logs are comprised of two types of events: header events and regular log events.

Input header sample
 ***** LoadRefs of <BEV_Prj\SIMATIC 400(1)\CPU 417-5 H PN/DP\S7 Program(1)\Charts> on 05/07/20 16:49:04

Based on this header event sample, these fields can be parsed:

  • Type

  • Path

  • Date (string format: mm/dd/yy)

  • Time (string format: hh:mm:ss)

Input log sample
OB55 [FC30]          (LD ) -> (LD ) -> @(1).SIMATIC_400(1)_1 [DB76] (LD )

Based on this regular log event sample, these fields can be parsed:

  • ParamOne

  • ParamTwo

  • ParamThree

  • ParamFour

  • ParamFive

  • ParamSix

  • ParamSeven

Because these logs exhibit consistent data patterns, specific fields can be parsed using the following regular expressions defined by REGEX_LOADREF_HEADER and REGEX_LOADREF_ONE:

Defining regular expressions
# Regular expression for parsing headers
define REGEX_LOADREF_HEADER /(?x)^\s?\**(?<Type>[\w ]*)\<(?<Path>\
                            [-_ \\\/\(\)\d*\w*]*)\>\s*on\s*(?<Date>\
                            \d+\/\d+\/\d+)\s*(?<Time>\d+:\d+:\d+)/
# Regular expression for parsing log entries
define REGEX_LOADREF_ONE    /(?x)^(?<ParamOne>[\(\)@\d\w_]*)\s*\
                            \[(?<ParamTwo>[\w\d]*)\]\s*\((?<ParamThree>[\w]\
                            *\s*)\s*?\)\s*->\s*\((?<ParamFour>[\w]*\s*)\)\s*\
                            ->\s(?<ParamFive>[\(\)@\w_.-]*)\s*(?:\[(?<ParamSix>\
                            [\w\d]*)\])?\s*\((?<ParamSeven>[\w]*)\s*\)/

The logic for parsing and filtering is defined within the Exec block of the from_file instance of the im_file input module. All parsed fields that do not require additional processing are captured using the named capturing groups defined in REGEX_LOADREF_HEADER or REGEX_LOADREF_ONE. Each field name is defined within angle brackets (< >) that will determine which event field the captured value will be assigned to. In this example, depending on which of the two regular expressions an event matches, the event record will be enriched with a subset of the following fields: $Type, $Path, $Date, $Time, $ParamOne, $ParamTwo, $ParamThree, $ParamFour, $ParamFive, $ParamSix, $ParamSeven.

The drop procedure discards records that match neither the REGEX_LOADREF_HEADER nor the REGEX_LOADREF_ONE regular expressions.

Then, by calling the to_json() procedure of the JSON (xm_json) extension module, the newly parsed fields along with the core fields are added to the event record and formatted as JSON before being routed to any output instances.

The following NXLog Agent configuration combines all the steps described above.

nxlog.conf
# Regular expression for parsing headers
define REGEX_LOADREF_HEADER /(?x)^\s?\**(?<Type>[\w ]*)\<(?<Path>\
                            [-_ \\\/\(\)\d*\w*]*)\>\s*on\s*(?<Date>\
                            \d+\/\d+\/\d+)\s*(?<Time>\d+:\d+:\d+)/
# Regular expression for parsing log entries
define REGEX_LOADREF_ONE    /(?x)^(?<ParamOne>[\(\)@\d\w_]*)\s*\
                            \[(?<ParamTwo>[\w\d]*)\]\s*\((?<ParamThree>[\w]\
                            *\s*)\s*?\)\s*->\s*\((?<ParamFour>[\w]*\s*)\)\s*\
                            ->\s(?<ParamFive>[\(\)@\w_.-]*)\s*(?:\[(?<ParamSix>\
                            [\w\d]*)\])?\s*\((?<ParamSeven>[\w]*)\s*\)/
# Path to the AS project folder
define PROJECT_PATH         C:\Users\Engineer\Documents\Project\BEV_MP

<Extension json>
    Module        xm_json
</Extension>

<Input from_file>
    Module        im_file
    File          '%PROJECT_PATH%\VR_Lib\ES_LOC\1\GEN\loadrefs.log'
    <Exec>
        if $raw_event =~ %REGEX_LOADREF_HEADER% 
        {
            $Type = replace($Type, " LoadRefs of ", "LoadRefs");
            to_json();    # Converting to JSON format
        }
        else if $raw_event =~ %REGEX_LOADREF_ONE% to_json();
        else drop();      # Discard event if it doesn't match a/the regular expression
    </Exec>
</Input>

The samples below depict the processed events in JSON format.

Output header sample
{
  "EventReceivedTime": "2020-10-12T12:33:10.992481+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Date": "05/07/20",
  "Path": "BEV_Prj\\SIMATIC 400(1)\\CPU 417-5 H PN/DP\\S7 Program(1)\\Charts",
  "Time": "16:49:04",
  "Type": "LoadRefs"
}
Backslash characters (\) are escaped by default during processing as prescribed by the official JSON definition of a string ("Any codepoint except " or \ or control characters"). See the Introducing JSON section on the JSON website for more details.
Output log sample
{
  "EventReceivedTime": "2020-10-11T17:29:12.059800+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "ParamFive": "@(1).SIMATIC_400(1)_1",
  "ParamFour": "LD ",
  "ParamOne": "OB55",
  "ParamSeven": "LD",
  "ParamSix": "DB76",
  "ParamThree": "LD ",
  "ParamTwo": "FC30"
}

Multiproject logs

The following table lists the various types of Multi-project logs.

Table 12. Multiproject logs
Name File
Ext.
Location Description

Full log of the shared declarations synchronization

.log

[MultiprojectName]\Global\GDAlignError.log

Import and export assistant logs

.log

[MultiprojectName]\[MasterLibraryName]\Global\[ProcessTagType_name].log

[MultiprojectName]\[MasterLibraryName]\Global\Adjust.log

[MultiprojectName]\[MasterLibraryName]\Global\Modify.log

Update block types logs

.txt

[MultiprojectName]\[MasterLibraryName]@CentralBstActualize.TXT

[MultiprojectName]\[MasterLibraryName]@CentralBstActualize_tmp.TXT

Updating blocks (currently used in the project) from the master library after the adaptation

Full log of the Shared Declarations Synchronization

Logs of the shared declarations synchronization are stored in the GDAlignError.log file of the multi-project directory.

NXLog Agent can be configured to process shared declarations synchronization logs, as demonstrated in the following example.

Example 15. Processing the Shared Declarations Synchronization logs

This sample event shows how this log type spans multiple lines. The entirety of each multiline event will be flattened to a single $Message field.

Input sample
Synchronization of the shared declarations on 07.05.2020 14:17:59
Selected object: Shared Declarations BEV_Lib
Actions were performed during the synchronization of the shared declarations in the following projects/libraries  and a log was created:

Project BEV_Prj

Project HMI

To correctly process multiline event logs, a pattern needs to be defined as a regular expression that describes the header line of an event. In the following xm_multiline extension instance, the HeaderLine directive specifies the regular expression to be used for finding the header line of each event.

Enabling multiline event processing
<Extension _multiline>
    Module        xm_multiline
    # Parsing the header line
    HeaderLine    /^Synchronization.*on\s+\d+.\d+.\d+\s+\d+:\d+:\d+$/
</Extension>

In the from_file input instance, the InputType directive is used for referencing the xm_multiline extension instance by name, _multiline, which will enable the instance to establish the beginning and end of each event.

To improve the readability of the parsed messages, the Exec block invokes the replace() function to replace all occurrences of \r\n (Windows line endings).

Then, by calling the to_json() procedure of the JSON (xm_json) extension module, the newly parsed fields along with the core fields are added to the event record and formatted as JSON before being routed to any output instances.

The following NXLog Agent configuration combines all the steps described above.

nxlog.conf
# Path to the multiproject folder
define PROJECT_PATH C:\Users\Engineer\Documents\Project\BEV_MP

<Extension _multiline>
    Module        xm_multiline
    # Regular expression to recognize the message header
    HeaderLine    /^Synchronization.*on\s+\d+.\d+.\d+\s+\d+:\d+:\d+$/
</Extension>

<Extension json>
    Module        xm_json
</Extension>

<Input from_file>
    Module        im_file
    File          '%PROJECT_PATH%\BEV_MP\Global\GDAlignError.log'
    InputType     _multiline
    <Exec>
        $Message = $raw_event;
        # Replacing the `\r\n` combinations with white spaces
        $Message = replace($Message, "\r\n" , " ");
        to_json();        # Converting the parsed data to JSON format
    </Exec>
</Input>

The sample below depicts the processed event in JSON format.

Output sample
{
  "EventReceivedTime": "2020-09-30T13:56:54.967315+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Message": "Synchronization of the shared declarations on 07.05.2020 14:17:59 Selected object: Shared Declarations BEV_Lib Actions were performed during the synchronization of the shared declarations in the following projects/libraries  and a log was created:  Project BEV_Prj  Project HMI "
}

Import and Export Assistant logs

The following table lists the various types of import and export assistant logs.

Table 13. List of files

[MultiprojectName]\[MasterLibraryName]\Global\[ProcessTagType_name].log

[MultiprojectName]\[MasterLibraryName]\Global\Adjust.log

[MultiprojectName]\[MasterLibraryName]\Global\Modify.log

NXLog Agent can be configured to process import and export assistant logs, as demonstrated in the following example.

Example 16. Processing the Import and Export Assistant logs

Assistant logs are comprised of two types of events: header events and regular log events.

To process other files from the list above, simply replace the file name against the File directive of the im_file module.

Input header sample
Import/Export Assistant log file from:  Wednesday, February 26, 2020 11:25:51

Based on this header event sample, these fields can be parsed:

  • DayOfWeek

  • Message

  • Month (name)

  • Time (string format: hh:mm:ss)

  • Date (day of the month, string format: MM)

  • Year (string format: YYYY)

Input log sample
BEV_Prj\PCELL\CIP\RETURN\CM\\LSHRM3\LH_RM3.EN	OK	Process tag parameter flag reset.

Based on this regular event sample, these fields can be parsed:

  • Path

  • Status

  • Message

Because these logs exhibit consistent data patterns, specific fields can be parsed using the following regular expressions defined by ASSIST_REGEX_HEADER and ASSIST_REGEX_LOG:

Defining the regular expressions
# Regular expression for parsing headers
define ASSIST_REGEX_HEADER}  /(?x)^(?<Message>.*)from:\s*(?<DayOfWeek>\w*),\
                            \s*(?<Month>\w*)\s*(\d*),\s*(\d*)\
                            \s*(?<Time>\d*:\d*:\d*)/
# Regular expression for parsing log entries
define ASSIST_REGEX_LOG     /(?x)^(?<Path>.*)\s+(?<Status>OK|ERROR|Error)\
                            \s+(?<Message>.*)/

The logic for parsing and filtering is defined within the Exec block of the from_file instance of the im_file input module. All parsed fields that do not require additional processing are captured using the named capturing groups defined in ASSIST_REGEX_HEADER or ASSIST_REGEX_LOG. Each field name is defined within angle brackets (< >) that will determine which event field the captured value will be assigned to. In this example, depending on which of the two regular expressions an event matches, the event record will be enriched with a subset of the following fields: $Message, $DayOfWeek, $Month, $Time, $Path, $Status, and/or $Message.

For events that match ASSIST_REGEX_HEADER, the fourth and fifth unnamed capture groups ($4 and $5) are used to create the fields $Date (the day of the month) and $Year are converted to integers using the integer() function.

The drop procedure discards records that match neither the ASSIST_REGEX_HEADER nor the ASSIST_REGEX_LOG regular expressions.

Then, by calling the to_json() procedure of the JSON (xm_json) extension module, the newly parsed fields along with the core fields are added to the event record and formatted as JSON before being routed to any output instances.

The following NXLog Agent configuration combines all the steps described above.

nxlog.conf
# Regular expression for parsing headers
define ASSIST_REGEX_HEADER  /(?x)^(?<Message>.*)from:\s*(?<DayOfWeek>\w*),\
                            \s*(?<Month>\w*)\s*(\d*),\s*(\d*)\
                            \s*(?<Time>\d*:\d*:\d*)/
# Regular expression for parsing log entries
define ASSIST_REGEX_LOG     /(?x)^(?<Path>.*)\s+(?<Status>OK|ERROR|Error)\
                            \s+(?<Message>.*)/
# Path to the multiproject folder
define PROJECT_PATH         C:\Users\Engineer\Documents\Project\BEV_MP

<Extension json>
    Module        xm_json
</Extension>

<Input from_file>
    Module        im_file
    File          '%PROJECT_PATH%\BEV_Lib\Global\modify.log'
    <Exec>
        # Converting messages to JSON format
        if $raw_event =~ %ASSIST_REGEX_HEADER%
        {
            # Converting the date and year values to integers
            $Date = integer($4);
            $Year = integer($5);
            to_json();    # Converting to JSON
        }
        else if $raw_event =~ %ASSIST_REGEX_LOG% to_json(); # Converting to JSON
        else drop();      # Discard event if it doesn't match a/the regular expression
    </Exec>
</Input>

The samples below depict the processed events in JSON format.

Output header sample
{
  "EventReceivedTime": "2020-10-22T13:11:42.590475+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "DayOfWeek": "Wednesday",
  "Message": "Import/Export Assistant log file ",
  "Month": "February",
  "Time": "11:25:51",
  "Date": 26,
  "Year": 2020
}
Output log sample
{
  "EventReceivedTime": "2020-10-12T15:55:18.931486+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Message": "Process tag parameter flag reset.",
  "Path": "BEV_Prj\\PCELL\\CIP\\RETURN\\CM\\\\LSHRM3\\LH_RM3.EN",
  "Status": "OK"
}
Backslash characters (\) are escaped by default during processing as prescribed by the official JSON definition of a string ("Any codepoint except " or \ or control characters"). See the Introducing JSON section on the JSON website for more details.

Update Block Types logs

The update block types logs are stored in two files as listed below.

Table 14. List of files

[MultiprojectName]\[MasterLibraryName]@CentralBstActualize.TXT

[MultiprojectName]\[MasterLibraryName]@CentralBstActualize_tmp.TXT

NXLog Agent can be configured to process update block types logs as demonstrated in the following example.

Example 17. Processing Update Block Types logs

The following event sample spans multiple lines.

Input sample
The following S7 programs were selected for checking:
-> BEV_Lib\S7 Program(1)
-> BEV_Prj\SIMATIC 400(1)\CPU 417-5 H PN/DP\S7 Program(1)
-> VR_Lib\S7 Program(1)

No different blocks found. The central type import will be closed.

End of central type update 05.05.2020 19:47:54

To correctly process multiline event logs, a pattern needs to be defined as a regular expression that describes the header line of an event. In the following xm_multiline extension instance, the HeaderLine directive specifies the regular expression to be used for finding the header line of each event.

Enabling multiline event processing
<Extension _multiline>
    Module        xm_multiline
    # Parsing the header line
    HeaderLine    /^.*:$/
</Extension>

In the from_file input instance, the InputType directive is used for referencing the xm_multiline extension instance by name, _multiline, which will enable the instance to establish the beginning and end of each event.

The entire event will be assigned to the $Message field. Spaces are substituted for -> and \r\n (Windows line endings) to make the message text more readable.

Then, by calling the to_json() procedure of the JSON (xm_json) extension module, the newly parsed fields along with the core fields are added to the event record and formatted as JSON before being routed to any output instances.

The following NXLog Agent configuration combines all the steps described above.

nxlog.conf
# The path to the multiproject folder
define PROJECT_PATH C:\Users\Engineer\Documents\Project\BEV_MP

<Extension _multiline>
    Module        xm_multiline
    HeaderLine    /^.*:$/
</Extension>

<Extension json>
    Module        xm_json
</Extension>

<Input from_file>
    Module        im_file
    File          '%PROJECT_PATH%\BEV_Lib\@CentralBstActualize*.TXT'
    InputType     _multiline
    <Exec>
        $Message = $raw_event;
        # Replacing the `-> ` and `\r\n` combinations with white spaces
        $Message = replace($Message, "-> ", " ");
        $Message = replace($Message, "\r\n", " ");
        to_json();        # Converting the result to JSON
    </Exec>
</Input>

The sample below depicts the processed event in JSON format.

Output sample
{
  "EventReceivedTime": "2020-09-30T15:01:37.686454+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Message": "The following S7 programs were selected for checking:  BEV_Lib\\S7 Program(1)  BEV_Prj\\SIMATIC 400(1)\\CPU 417-5 H PN/DP\\S7 Program(1)  VR_Lib\\S7 Program(1)  No different blocks found. The central type import will be closed.  End of central type update 05.05.2020 19:47:54"
}
Backslash characters (\) are escaped by default during processing as prescribed by the official JSON definition of a string ("Any codepoint except " or \ or control characters"). See the Introducing JSON section on the JSON website for more details.

Batch logs

Table 15. Batch logs
Name File
Ext.
Location Description

Batch logs

.txt

.xml

[MultiprojectName]\BatchPrj\[Batch_ID]*

[MultiprojectName]\[AS_projectName]\BatchPrj\[Batch_ID]\*

C:\ProgramData\Siemens\Automation\Logfiles\Batch*

C:\Users\All Users\Siemens\Automation\Logfiles\Batch\*

- Generated batch type logs
- Batch instances compilation logs
- Batch instances merge logs
- Batch process cell validation report
- Report on cell transfer messages to OS, which is the part of the batch process
- Batch process cell download logs
- Additional log files
- Batch system logs

Universal configuration file

NXLog Agent can be configured to process multiple types of PCS7 log files using a single configuration. The example below combines the processing of all the files listed in this guide.

Example 18. Processing all Siemens SIMATIC PCS 7 log files with a single configuration

This example is structured with extra headers for easier readability. Each main section is separated by comments.

nxlog.conf
# ---------------- REGULAR EXPRESSIONS FOR PARSING DATA ------------------------

define DATA_MANAGER_REGEX /(?x)^(\d+.\d+.\d+)\s+(\d+:\d+:\d+).\d+\
                          \s+PID=(?<PID>\d+)\s+TID=(?<TID>\d+)\s+\
                          (?<ParamFive>\w+)\s+\S(?<Loader>\w+):\s+\
                          (?<Message>.*)/

define CHANNEL_REGEX      /(?x)^(\d+-\d+-\d+)\s(\d+:\d+:\d+),\
                          \d+\s+(?<Type>\w+)\s+(?<Message>.*)/

define DELTA_REGEX        /(?x)^(?<Loader>[a-zA-Z.]*)\s+:\s+(\d+.\d+.\d+)\
                          \s(\d+:\d+:\d+)\s:\s(?<Message>[\w():.\\\s]*)/

define OPC_UA_LOG_HEADER  /(?x)^\d+\/\d+\/\d+\s+\d+:\d+:\d+\s+[\d\w\s:]*\d+/

define PM_REGEX           /(?x)^(\d+-\d+-\d+)\s+(\d+:\d+:\d+).\d+\s+\
                          (?<Type>\w+)\(\s*(?<PID>\d*)\-(?<TID>\d*)\)\
                          \s*(?<Message>.*)/

define TRACE_REGEX        /(?x)^[^\w\d]*(\d*);\s*(\d*-\d*\
                          -\d*\w\d*:\d*:\d*.\d*\w);\s*(?<PID>\d*);\
                          \s*(?<TID>\d*);\s*\[(?<Address>\w*\d*)\];\
                          \s(?<Process>[\w\\]*);\s*(?<ParamSeven>\w*)\
                          ;(?<MessageOne>.*);\s*\"(?<MessageTwo>.*)\"/

define EVENT_REGEX        /(?x)^(\d+.\d+.\d+)\s+(\d+:\d+:\d+)\s+(?<Message>.*)/

define STARTUP_REGEX      /(?x)^(\d+),(\d+.\d+.\d+),(\d+:\d+:\d+):\
                          \d+,(\d+),(\d+),,(?<Station>.*),\
                          (?<Runtime>.*),(?<Message>.*),(?<State>.*)/

define EVENT_SQLAGENT     /(?x)^(\d+-\d+-\d+)\s+(\d+:\d+:\d+)\
                          \s+-\s+\W\s+(?<Message>.*)/

define ARCHIVE_REGEX      /(?x)^(\d+-\d+-\d+)\s+(\d+:\d+:\d+).\
                          \d+\s+\[p=(?<PID>\d+),\s+t=(?<TID>.*)\
                          \]\s+(?<Message>.*)/

define REGEX_LP_HEADER    /(?x)^\s?\**(?<Type>[\w ]*)\<(?<Path>\
                          [-_ \\\/\(\)\d*\w*]*)\>\s*on\s*(?<Date>\
                          \d+\/\d+\/\d+)\s*(?<Time>\d+:\d+:\d+)/

define REGEX_LP_ONE       /(?x)^\[?(?<Time>\d+:\d+:\d+.\d+)\s*\
                          (?<Daypart>AM|PM)\]?\s*(?<Process>\
                          [._ \(\)~@\-\w]*)\[(?<ParamFour>[\w\d]*)\]\
                          \s*\((?<Status>[\w]*)\)/

define REGEX_LP_TWO       /(?x)^\[?(?<Time>\d+:\d+:\d+.\d+)\s*\
                          (?<Daypart>AM|PM)\]?\s*(?<Process>.*)\
                          \s*\((?<Status>.*)\)/

define REGEX_LR_HEADER    /(?x)^\s?\**(?<Type>[\w ]*)\<(?<Path>\
                          [-_ \\\/\(\)\d*\w*]*)\>\s*on\s*(?<Date>\
                          \d+\/\d+\/\d+)\s*(?<Time>\d+:\d+:\d+)/

define REGEX_LR_ONE       /(?x)^(?<ParamOne>[\(\)@\d\w_]*)\s*\
                          \[(?<ParamTwo>[\w\d]*)\]\s*\((?<ParamThree>[\w]\
                          *\s*)\s*?\)\s*->\s*\((?<ParamFour>[\w]*\s*)\)\s*\
                          ->\s(?<ParamFive>[\(\)@\w_.-]*)\s*(?:\[(?<ParamSix>\
                          [\w\d]*)\])?\s*\((?<ParamSeven>[\w]*)\s*\)/

define ASSIST_REGEX_HEAD  /(?x)^(?<Message>.*)from:\s*(?<DayOfWeek>\w*),\
                          \s*(?<Month>\w*)\s*(\d*),\s*(\d*)\
                          \s*(?<Time>\d*:\d*:\d*)/

define ASSIST_REGEX_LOG   /(?x)^(?<Path>.*)\s+(?<Status>OK|ERROR|Error)\
                          \s+(?<Message>.*)/

# ---------------- PATHS TO LOG FILES ------------------------------------------

define DIAGNOSE_PATH      C:\Program Files (x86)\SIEMENS\WinCC\diagnose

define OPC_UA_PATH        C:\ProgramData\Siemens\Automation\Logfiles

define TRACE_PATH         C:\ProgramData\Siemens\Logs\REGSVR32\trace

define SQL_SERVER_PATH    C:\Program Files (x86)\Microsoft SQL Server

define PROJECT_PATH       C:\Users\Engineer\Documents\Project\BEV_MP

# ---------------- EXTENSION MODULES -------------------------------------------

<Extension json>
    Module        xm_json
</Extension>

<Extension _charconv>
    Module        xm_charconv
    LineReader    UCS-2LE
</Extension>

<Extension _multi_opc_ua>
    Module        xm_multiline
    HeaderLine    %OPC_UA_LOG_HEADER%
</Extension>

<Extension _multi_shared>
    Module        xm_multiline
    HeaderLine    /^Synchronization.*on\s+\d+.\d+.\d+\s+\d+:\d+:\d+$/
</Extension>

<Extension _multi_update_block>
    Module        xm_multiline
    HeaderLine    /^.*:$/
</Extension>

# ---------------- INPUT MODULES -----------------------------------------------

<Input data_manager>
    Module        im_file
    File          '%DIAGNOSE_PATH%\CCDmRtServer.log'
    InputType     _charconv
    <Exec>
        if $raw_event =~ %DATA_MANAGER_REGEX%
        {
            $EventTime = strptime($1 + $2,"%d.%m.%Y %T");
            to_json();
        }
        else drop();
    </Exec>
</Input>

<Input channel>
    Module        im_file
    File          '%DIAGNOSE_PATH%\SIMATIC_S7_Protocol_Suite_*.log'
    <Exec>
        if $raw_event =~ %CHANNEL_REGEX%
        {
            $EventTime = strptime($1 + $2,"%Y-%m-%d %T");
            to_json();
        }
        else drop();
    </Exec>
</Input>

<Input delta>
    Module        im_file
    File          '%DIAGNOSE_PATH%\CCDeltaLoader.Log'
    InputType     _charconv
    <Exec>
        if $raw_event =~ %DELTA_REGEX%
        {
            $EventTime = strptime($2 + $3,"%d.%m.%Y %T");
            to_json();
        }
        else drop();
    </Exec>
</Input>

<Input opc_ua>
    Module        im_file
    File          '%OPC_UA_PATH%\sinec2.log'
    InputType     _multi_opc_ua
    <Exec>
        $Message = $raw_event;
        $Message = replace($Message, "\r\n" , " ");
        $Message = replace($Message, "\t" , " ");
        to_json();
    </Exec>
</Input>

<Input pm>
    Module        im_file
    File          '%DIAGNOSE_PATH%\PM*.log'
    <Exec>
        if $raw_event =~ %PM_REGEX%
        {
            $EventTime = strptime($1 + $2,"%Y-%m-%d %T");
            to_json();
        }
        else drop();
    </Exec>
</Input>

<Input trace>
    Module        im_file
    File          '%TRACE_PATH%\*'
    <Exec>
        if $raw_event =~ %TRACE_REGEX%
        {
            $EntryNumber = integer($1);
            $Timestamp = parsedate($2);
            to_json();
        }
		else drop();
    </Exec>
</Input>

<Input surrogate>
    Module        im_file
    File          '%DIAGNOSE_PATH%\CUCSurrogate.log'
    <Exec>
        if $raw_event =~ %EVENT_REGEX%
        {
            $EventTime = strptime($1 + $2,"%d.%m.%Y %T");
            to_json();
        }
        else drop();
    </Exec>
</Input>

<Input startup>
    Module        im_file
    File          '%DIAGNOSE_PATH%\WinCC_SStart_*.log'
    InputType     _charconv
    <Exec>
        if $raw_event =~ %STARTUP_REGEX%
        {
            $ID = integer($1);
            $EventTime = strptime($2 + $3,"%d.%m.%Y%T");
            $MessageNumber = integer($4);
            $MessageClass = integer($5);
            to_json();
        }
        else drop();
    </Exec>
</Input>

<Input sqlagent>
    Module        im_file
    File          '%SQL_SERVER_PATH%\MSSQL12.WINCC\MSSQL\Log\SQLAGENT.*'
    InputType     _charconv
    <Exec>
        if $raw_event =~ %EVENT_SQLAGENT%
        {
            $EventTime = strptime($1 + $2, "%Y-%m-%d %T");
            to_json();
        }
        else drop();
    </Exec>
</Input>

<Input archive>
    Module        im_file
    File          '%PROJECT_PATH%\Hmi\wincproj\HMI\UA\UALogFile.txt'
    <Exec>
        if $raw_event =~ %ARCHIVE_REGEX%
        {
            $EventTime = strptime($1 + $2,"%Y-%m-%d %T");
            to_json();
        }
        else drop();
    </Exec>
</Input>

<Input loadprot>
    Module        im_file
    File          '%PROJECT_PATH%\BEV_Prj\ES_LOC\1\GEN\loadprot.log'
    <Exec>
        if $raw_event =~ %REGEX_LP_HEADER% to_json();
        else if $raw_event =~ %REGEX_LP_ONE% to_json();
        else if $raw_event =~ %REGEX_LP_TWO% to_json();
        else drop();
    </Exec>
</Input>

<Input loadref>
    Module        im_file
    File          '%PROJECT_PATH%\VR_Lib\ES_LOC\1\GEN\loadrefs.log'
    <Exec>
        if $raw_event =~ %REGEX_LR_HEADER% to_json();
        else if $raw_event =~ %REGEX_LR_ONE% to_json();
        else drop();
    </Exec>
</Input>

<Input shared_declarations>
    Module        im_file
    File          '%PROJECT_PATH%\BEV_MP\Global\GDAlignError.log'
    InputType     _multi_shared
    <Exec>
        $Message = $raw_event;
        $Message = replace($Message, "\r\n" , " ");
        to_json();
    </Exec>
</Input>

<Input import_export>
    Module        im_file
    File          '%PROJECT_PATH%\BEV_Lib\Global\modify.log'
    <Exec>
        if $raw_event =~ %ASSIST_REGEX_HEAD%
        {
            $Date = integer($4);
            $Year = integer($5);
            to_json();
        }
        else if $raw_event =~ %ASSIST_REGEX_LOG% to_json();
        else drop();
    </Exec>
</Input>

<Input update_block>
    Module        im_file
    File          '%PROJECT_PATH%\BEV_Lib\@CentralBstActualize*.TXT'
    InputType     _multi_update_block
    <Exec>
        $Message = $raw_event;
        $Message = replace($Message, "-> ", " ");
        $Message = replace($Message, "\r\n", " ");
        to_json();
    </Exec>
</Input>

<Output to_file>
    Module        om_file
    File          'C:\output.txt'
</Output>

# ---------------- ROUTES ------------------------------------------------------

<Route r1>
    Path data_manager, channel, delta, opc_ua, pm, trace => to_file
</Route>

<Route r2>
    Path surrogate, startup, sqlagent, archive, loadprot => to_file
</Route>

<Route r3>
    Path loadref, shared_declarations, import_export => to_file
</Route>

<Route r4>
    Path update_block => to_file
</Route>

Network monitoring

NXLog Agent’s im_pcap module provides support to passively monitor network traffic by generating logs for various protocols. This capability of NXLog Agent is another source for collecting events based on the network communication to and from SIMATIC PCS7 devices and controller computers.

SIMATIC WinCC supports a number of communication drivers (channels) for connecting Automation Stations (AS) with bus systems. These drivers (channels) are software components that help establish communication between WinCC and Automation Station (AS) as well as enable the supply of tags with process values.

The following SIMATIC WinCC communication drivers were used to capture traffic samples for this chapter:

  • Modbus TCP/IP

  • SIMATIC S7-1200, S7-1500 Channel

  • OPC Channel

This section describes how to monitor network traffic of the following industrial protocols:

Modbus TCP

Modbus is an application protocol used in industrial automation systems. Modbus TCP describes how to apply the Modbus protocol using the Ethernet interface along with the TCP/IP transport and network layers. Each Modbus transaction consists of a client request followed by a server response.

Example 19. Capturing Modbus packets

Using the Dev directive in the im_pcap module, this NXLog Agent configuration specifies the network interface it will use for capturing packets. The Protocol group directive denotes the modbus protocol for monitoring. The Exec block of im_pcap calls the to_json() procedure of the xm_json module to convert messages to JSON.

nxlog.conf
<Extension _json>
    Module        xm_json
</Extension>

<Input pcap_protocol>
    Module        im_pcap
    # Name of a network device/interface
    Dev           \Device\NPF_{475C04FC-859D-42F5-8BF1-765D0D6C6518}
    # Protocol type
    <Protocol>
        Type      modbus
    </Protocol>
    # Conversion to JSON
    Exec          to_json();
</Input>
Modbus query sample
{
  "modbus.function_code": "Read Holding Registers (03)",
  "modbus.length": "6",
  "modbus.prot_id": "0",
  "modbus.query.read_holding_regs.qty_of_regs": "3",
  "modbus.query.read_holding_regs.starting_address": "0",
  "modbus.trans_id": "636",
  "modbus.unit_id": "1",
  "EventTime": "2021-08-05T16:48:08.064553+03:00",
  "EventReceivedTime": "2021-08-05T16:48:08.369641+03:00",
  "SourceModuleName": "pcap_protocol",
  "SourceModuleType": "im_pcap"
}
Modbus response sample
{
  "modbus.function_code": "Read Holding Registers (03)",
  "modbus.length": "9",
  "modbus.prot_id": "0",
  "modbus.response.read_holding_regs.byte_count": "6",
  "modbus.response.read_holding_regs.registers": "251, 247, 241",
  "modbus.trans_id": "636",
  "modbus.unit_id": "1",
  "EventTime": "2021-08-05T16:48:08.075087+03:00",
  "EventReceivedTime": "2021-08-05T16:48:08.369641+03:00",
  "SourceModuleName": "pcap_protocol",
  "SourceModuleType": "im_pcap"
}

ISO-on-TCP (RFC1006)

ISO-on-TCP, also known as RFC1006, is an attempt to combine two different Ethernet data transfer methods, package-oriented data formatting (ISO) and easy address routing (TCP/IP), thus combining the advantages of both.

Example 20. Capturing ISO-on-TCP packets

In this configuration, NXLog Agent uses the im_pcap module to passively monitor network traffic. The Dev directive of this module specifies the network interface for capturing packets. The Protocol group directive is defined twice so that the tpkt and cotp protocols can be monitored simultaneously. The Exec block of this module calls the to_json() procedure of the xm_json module to convert messages to JSON.

nxlog.conf
<Extension _json>
    Module        xm_json
</Extension>

<Input pcap_protocol>
    Module        im_pcap
    # Name of a network device/interface
    Dev           \Device\NPF_{475C04FC-859D-42F5-8BF1-765D0D6C6518}
    # Protocol type
    <Protocol>
        Type      tpkt
    </Protocol>
    <Protocol>
        Type      cotp
    </Protocol>
    # Conversion to JSON
    Exec          to_json();
</Input>
ISO-on-TCP sample
{
  "cotp.tpdu_eot": "true",
  "cotp.tpdu_nr": "0",
  "cotp.tpdu_type": "Data Transfer",
  "tpkt.length": "131",
  "tpkt.rsvd": "0",
  "tpkt.vrsn": "3",
  "EventTime": "2021-08-09T18:52:31.802672+03:00",
  "EventReceivedTime": "2021-08-09T18:52:32.046606+03:00",
  "SourceModuleName": "pcap_protocol",
  "SourceModuleType": "im_pcap"
}

BACnet

Building Automation and Control Network (BACnet) is a communication protocol designed for building automation and control systems. BACnet/IP describes how BACnet can use UDP to send data to connected devices across multiple subnets. NXLog Agent can be configured to capture BACnet packets.

Example 21. Capturing BACnet packets

To provide passive network monitoring, NXLog Agent uses the im_pcap module. In this case, the Dev directive of this module specifies the network interface it will use for capturing packets. The Protocol group directive of im_pcap specifies the bacnet protocol for capturing. The Exec block of im_pcap converts each message to JSON.

nxlog.conf
<Extension _json>
    Module        xm_json
</Extension>

<Input pcap_protocol>
    Module        im_pcap
    # Name of a network device/interface
    Dev           \Device\NPF_{475C04FC-859D-42F5-8BF1-765D0D6C6518}
    # Protocol type
    <Protocol>
        Type      bacnet
    </Protocol>
    # Conversion to JSON
    Exec          to_json();
</Input>
BACnet query sample
{
  "bacnet.apdu.bacnet_confirmed_request.invoke_id": "67",
  "bacnet.apdu.bacnet_confirmed_request.max_resp": "1476",
  "bacnet.apdu.bacnet_confirmed_request.max_segs": "Unspecified",
  "bacnet.apdu.bacnet_confirmed_request.more_segments_follow": "false",
  "bacnet.apdu.bacnet_confirmed_request.segmented": "false",
  "bacnet.apdu.bacnet_confirmed_request.segmented_accepted": "true",
  "bacnet.apdu.bacnet_confirmed_request.service_choice": "Read Property Multiple (14)",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.0.object_identifier.instance_number": "0",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.0.object_identifier.type": "binary-input (3)",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.1": "Opening Tag (1)",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.10.0.property_identifier": "change-of-state-time (16)",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.11": "Closing Tag (1)",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.12.object_identifier.instance_number": "0",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.12.object_identifier.type": "binary-input (3)",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.13": "Opening Tag (1)",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.14.0.property_identifier": "change-of-state-count (15)",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.15": "Closing Tag (1)",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.2.0.property_identifier": "elapsed-active-time (33)",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.3": "Closing Tag (1)",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.4.object_identifier.instance_number": "0",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.4.object_identifier.type": "binary-input (3)",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.5": "Opening Tag (1)",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.6.0.property_identifier": "description (28)",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.7": "Closing Tag (1)",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.8.object_identifier.instance_number": "0",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.8.object_identifier.type": "binary-input (3)",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.9": "Opening Tag (1)",
  "bacnet.apdu.pdu_type": "BACnet-Confirmed-Request-PDU (0x00)",
  "bacnet.bvlc.function": "Original-Unicast-NPDU (0x0A)",
  "bacnet.bvlc.length": "56",
  "bacnet.bvlc.type": "BACnet/IP (Annex J) (0x81)",
  "bacnet.npdu.control": "0x0024",
  "bacnet.npdu.control.contains": "BACnet APDU message (0)",
  "bacnet.npdu.control.dst_spec": "DNET, DLEN, Hop Count present (1)",
  "bacnet.npdu.control.prio": "Normal message",
  "bacnet.npdu.control.reply_expected": "Yes (1)",
  "bacnet.npdu.control.src_spec": "SNET, SLEN, SADR absent (0)",
  "bacnet.npdu.version": "0x0001",
  "EventTime": "2021-08-10T09:58:20.492180+03:00",
  "EventReceivedTime": "2021-08-10T09:58:20.684789+03:00",
  "SourceModuleName": "pcap_protocol",
  "SourceModuleType": "im_pcap"
}
BACnet response sample
{
  "bacnet.apdu.bacnet_complexack.more_segments_follow": "false",
  "bacnet.apdu.bacnet_complexack.original_invoke_id": "67",
  "bacnet.apdu.bacnet_complexack.segmented": "false",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.0.0.object_identifier.instance_number": "0",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.0.0.object_identifier.type": "binary-input (3)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.0.1": "Opening Tag (1)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.0.2.property_identifier": "elapsed-active-time (33)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.0.3.records.0": "Opening Tag (4)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.0.3.records.1": "826",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.0.3.records.2": "Closing Tag (4)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.0.4": "Closing Tag (1)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.1.0.object_identifier.instance_number": "0",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.1.0.object_identifier.type": "binary-input (3)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.1.1": "Opening Tag (1)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.1.2.property_identifier": "description (28)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.1.3.records.0": "Opening Tag (4)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.1.3.records.1": "NXLog BACnet Test Tag (ANSI X3.4/UTF-8 (0))",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.1.3.records.2": "Closing Tag (4)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.1.4": "Closing Tag (1)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.2.0.object_identifier.instance_number": "0",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.2.0.object_identifier.type": "binary-input (3)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.2.1": "Opening Tag (1)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.2.2.property_identifier": "change-of-state-time (16)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.2.3.records.0": "Opening Tag (4)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.2.3.records.1": "2021-8-9 (Day of week: 1)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.2.3.records.2": "23:58:19.19",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.2.3.records.3": "Closing Tag (4)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.2.4": "Closing Tag (1)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.3.0.object_identifier.instance_number": "0",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.3.0.object_identifier.type": "binary-input (3)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.3.1": "Opening Tag (1)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.3.2.property_identifier": "change-of-state-count (15)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.3.3.records.0": "Opening Tag (4)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.3.3.records.1": "331",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.3.3.records.2": "Closing Tag (4)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.3.4": "Closing Tag (1)",
  "bacnet.apdu.bacnet_complexack.service_choice": "Read Property Multiple (14)",
  "bacnet.apdu.pdu_type": "BACnet-Complex-ACK-PDU (0x03)",
  "bacnet.bvlc.function": "Original-Unicast-NPDU (0x0A)",
  "bacnet.bvlc.length": "102",
  "bacnet.bvlc.type": "BACnet/IP (Annex J) (0x81)",
  "bacnet.npdu.control": "0x0008",
  "bacnet.npdu.control.contains": "BACnet APDU message (0)",
  "bacnet.npdu.control.dst_spec": "DNET, DLEN, DADR, Hop Count absent (0)",
  "bacnet.npdu.control.prio": "Normal message",
  "bacnet.npdu.control.reply_expected": "No (0)",
  "bacnet.npdu.control.src_spec": "SNET, SLEN, SADR present (1)",
  "bacnet.npdu.version": "0x0001",
  "EventTime": "2021-08-10T09:58:20.535150+03:00",
  "EventReceivedTime": "2021-08-10T09:58:20.684789+03:00",
  "SourceModuleName": "pcap_protocol",
  "SourceModuleType": "im_pcap"
}

DNP3

DNP3 is a protocol for communication between process automation components used by public utilities for supplying electricity and water. This protocol is based on the Enhanced Performance Architecture (EPA) model, a simplified model of ISO/OSI to specify the data link layer, the application layer, and the transport pseudo-layer. NXLog Agent can be configured to capture packets transmitted over DNP3.

Example 22. Capturing DNP3 packets

This configuration uses the im_pcap module for passive network monitoring. The Dev directive of this module specifies the network interface it will use for capturing packets. The Protocol group directive denotes the dnp3 protocol for capturing. The Exec block of im_pcap calls the to_json() procedure of the xm_json module to convert messages to JSON.

nxlog.conf
<Extension _json>
    Module        xm_json
</Extension>

<Input pcap_protocol>
    Module        im_pcap
    # Name of a network device/interface
    Dev           \Device\NPF_{475C04FC-859D-42F5-8BF1-765D0D6C6518}
    # Protocol type
    <Protocol>
        Type      dnp3
    </Protocol>
    # Conversion to JSON
    Exec          to_json();
</Input>
DNP3 request sample
{
  "dnp3.application_layer.control.con": "0",
  "dnp3.application_layer.control.fin": "1",
  "dnp3.application_layer.control.fir": "1",
  "dnp3.application_layer.control.sequence": "6",
  "dnp3.application_layer.control.uns": "0",
  "dnp3.application_layer.function_code": "Read",
  "dnp3.application_layer.object0.count": "0",
  "dnp3.application_layer.object0.group": "60",
  "dnp3.application_layer.object0.name": "Class objects - Class 1 data",
  "dnp3.application_layer.object0.variation": "2",
  "dnp3.application_layer.object1.count": "0",
  "dnp3.application_layer.object1.group": "60",
  "dnp3.application_layer.object1.name": "Class objects - Class 2 data",
  "dnp3.application_layer.object1.variation": "3",
  "dnp3.application_layer.object2.count": "0",
  "dnp3.application_layer.object2.group": "60",
  "dnp3.application_layer.object2.name": "Class objects - Class 3 data",
  "dnp3.application_layer.object2.variation": "4",
  "dnp3.application_layer.object3.count": "0",
  "dnp3.application_layer.object3.group": "60",
  "dnp3.application_layer.object3.name": "Class objects - Class 0 data",
  "dnp3.application_layer.object3.variation": "1",
  "dnp3.data_layer.control": "0xC4",
  "dnp3.data_layer.control.dir": "1",
  "dnp3.data_layer.control.fcb": "0",
  "dnp3.data_layer.control.fcv": "0",
  "dnp3.data_layer.control.function_code": "Unconfirmed User Data",
  "dnp3.data_layer.control.prm": "1",
  "dnp3.data_layer.destination": "1",
  "dnp3.data_layer.length": "20",
  "dnp3.data_layer.source": "2",
  "dnp3.data_layer.start_bytes": "0x0564",
  "dnp3.transport.fin": "1",
  "dnp3.transport.fir": "1",
  "dnp3.transport.sequence": "55",
  "EventTime": "2021-08-10T18:57:43.423933+03:00",
  "EventReceivedTime": "2021-08-10T18:57:43.596147+03:00",
  "SourceModuleName": "pcap_protocol",
  "SourceModuleType": "im_pcap"
}
DNP3 response sample
{
  "dnp3.application_layer.control.con": "0",
  "dnp3.application_layer.control.fin": "1",
  "dnp3.application_layer.control.fir": "1",
  "dnp3.application_layer.control.sequence": "6",
  "dnp3.application_layer.control.uns": "0",
  "dnp3.application_layer.function_code": "Response",
  "dnp3.application_layer.internal_indications.already_executing": "0",
  "dnp3.application_layer.internal_indications.broadcast": "0",
  "dnp3.application_layer.internal_indications.class1_events": "0",
  "dnp3.application_layer.internal_indications.class2_events": "0",
  "dnp3.application_layer.internal_indications.class3_events": "0",
  "dnp3.application_layer.internal_indications.config_corrupt": "0",
  "dnp3.application_layer.internal_indications.device_restart": "0",
  "dnp3.application_layer.internal_indications.device_trouble": "0",
  "dnp3.application_layer.internal_indications.events_buffer_overflow": "0",
  "dnp3.application_layer.internal_indications.local_control": "0",
  "dnp3.application_layer.internal_indications.need_time": "0",
  "dnp3.application_layer.internal_indications.no_func_code_support": "0",
  "dnp3.application_layer.internal_indications.object_unknown": "0",
  "dnp3.application_layer.internal_indications.parameter_error": "0",
  "dnp3.application_layer.internal_indications.reserved": "0 (expected 0)",
  "dnp3.application_layer.object0.count": "1",
  "dnp3.application_layer.object0.group": "30",
  "dnp3.application_layer.object0.name": "Analog input - single-precision, floating-point with flag",
  "dnp3.application_layer.object0.point0.flags": "[ONLINE]",
  "dnp3.application_layer.object0.point0.value": "314.980011",
  "dnp3.application_layer.object0.variation": "5",
  "dnp3.data_layer.control": "0x44",
  "dnp3.data_layer.control.dir": "0",
  "dnp3.data_layer.control.fcb": "0",
  "dnp3.data_layer.control.fcv": "0",
  "dnp3.data_layer.control.function_code": "Unconfirmed User Data",
  "dnp3.data_layer.control.prm": "1",
  "dnp3.data_layer.destination": "2",
  "dnp3.data_layer.length": "20",
  "dnp3.data_layer.source": "1",
  "dnp3.data_layer.start_bytes": "0x0564",
  "dnp3.transport.fin": "1",
  "dnp3.transport.fir": "1",
  "dnp3.transport.sequence": "55",
  "EventTime": "2021-08-10T18:57:43.480934+03:00",
  "EventReceivedTime": "2021-08-10T18:57:43.598076+03:00",
  "SourceModuleName": "pcap_protocol",
  "SourceModuleType": "im_pcap"
}

IEC 60870-5-104

IEC 60870–5‑104 is an extension of the IEC 60870-5-101 protocol and combines transport, network, link, and physical layers to enable communication between control stations and substations using TCP/IP. NXLog Agent can be configured to monitor network traffic that uses this protocol.

Example 23. Capturing IEC 60870-5-104 packets

To passively monitor network traffic, NXLog Agent uses the im_pcap module. The Dev directive of this module specifies the network interface it will use for capturing packets. The Protocol group directive is defined twice so that the iec104apci and iec104asdu protocols can be monitored simultaneously. The iec104apci and iec104asdu protocols are being monitored. The Exec block of this module calls the to_json() procedure of the xm_json module to convert messages to JSON.

nxlog.conf
<Extension _json>
    Module        xm_json
</Extension>

<Input pcap_protocol>
    Module        im_pcap
    # Name of a network device/interface
    Dev           \Device\NPF_{475C04FC-859D-42F5-8BF1-765D0D6C6518}
    # Protocol type
    <Protocol>
        Type    iec104apci
    </Protocol>
    <Protocol>
        Type    iec104asdu
    </Protocol>	
    # Conversion to JSON
    Exec          to_json();
</Input>
IEC 60870-5-104 sample
{
  "iec104.apci.receive_sequence_number": "10",
  "iec104.apci.send_sequence_number": "309",
  "iec104.apci.type": "Information (I)",
  "iec104.asdu.data": {
    "io": [
      {
        "ioa": 1000,
        "ie": [
          {
            "type": "NVA",
            "value": "0.968719 (31743)"
          },
          {
            "type": "QDS",
            "invalid": false,
            "not-topical": false,
            "substituted": false,
            "blocked": false,
            "overflow": false
          },
          {
            "type": "CP56Time2A",
            "milliseconds": 2690,
            "minutes": 35,
            "hours": 13,
            "day-of-week": 0,
            "day-of-month": 10,
            "month": 8,
            "year": 21
          }
        ],
        "ies": 3
      }
    ],
    "ios": 1
  },
  "iec104.asdu.dui.cause_of_transmission": "Spontaneous (3)",
  "iec104.asdu.dui.coa": "1",
  "iec104.asdu.dui.num_records": "1",
  "iec104.asdu.dui.org": "0",
  "iec104.asdu.dui.pn": "0",
  "iec104.asdu.dui.sq": "FALSE",
  "iec104.asdu.dui.test_bit": "0",
  "iec104.asdu.dui.type": "M_ME_TD_1",
  "EventTime": "2021-08-10T23:35:02.821209+03:00",
  "EventReceivedTime": "2021-08-10T23:35:03.394000+03:00",
  "SourceModuleName": "pcap_protocol",
  "SourceModuleType": "im_pcap"
}
Disclaimer

While we endeavor to keep the information in our guides up to date and correct, NXLog makes no representations or warranties of any kind, express or implied about the completeness, accuracy, reliability, suitability, or availability of the content represented here. We update our screenshots and instructions on a best-effort basis.

The accurateness of the content was tested and proved to be working in our lab environment at the time of the last revision with the following software versions:

NXLog Agent version 5.5.7535
Microsoft Windows 10
Ubuntu 20.04.4 LTS
RHEL 7

Last revision: 23 May 2022