NXLog Legacy Documentation

General Electric CIMPLICITY

CIMPLICITY HMI is a client-server HMI (human-machine interface) and SCADA solution from General Electric. For convenience, this technology will be referred to as CIMPLICITY in this guide.

CIMPLICITY SCADA servers are responsible for collecting and distributing data within a production environment. CIMPLICITY clients connect to servers to view and manage the data collected.

NXLog can be configured to collect and process all types of logs produced by General Electric CIMPLICITY.

Types of logs

NXLog can collect events from the following sources:

Logs from Windows Event Log
File-based logs
Passive network monitoring

Logs from Windows Event Log

This table contains the CIMPLICITY services which generate Windows Event Log data along with their display names and executables.

Table 1. List of CIMPLICITY services
Service Name	Display Name	Path to Executable
BSV	BSV BACnet Value Service	C:\Program Files (x86)\Proficy\Proficy Drivers\Win32\bsv.exe
DNPDrv	Catapult DNP3	C:\PROGRAM FILES (X86)\PROFICY\PROFICY CIMPLICITY\EXE\DNPDRV.EXE /Service
CIMPLICITY Advanced Viewer	CIMPLICITY Advanced Viewer	C:\Program Files (x86)\Proficy\Proficy CIMPLICITY\exe\ptopc.exe
CIMBroadcastService	CIMPLICITY Broadcast Service	C:\Windows\SysWoW64\CIMBroadcastService.exe
CIMConfigService	CIMPLICITY Configuration Microservice	"C:\Program Files (x86)\Proficy\Proficy CIMPLICITY\exe\cim_config_service.exe" serve -config "C:\Program Files (x86)\Proficy\Proficy CIMPLICITY\data\cim_config_service.json"
CIMPLICITY	CIMPLICITY HMI Service	C:\Windows\SysWoW64\cimplicity.exe
CIMPLICITYNGINX	CIMPLICITY NGINX Server	C:\Program Files (x86)\Proficy\Proficy CIMPLICITY\Web\exe\nssm.exe
CimProxy	CIMPLICITY Proxy Service	C:\Program Files (x86)\Proficy\Proficy CIMPLICITY\exe\CimProxy.exe
CimplicityViewConnectionService	CimplicityViewConnectionService	C:\Program Files (x86)\Proficy\Proficy CIMPLICITY\exe\CimplicityViewConnectionService.exe
FirstPAGEAlarmServer	FirstPAGE Alarm Manager Server	C:\Program Files (x86)\Proficy\Proficy CIMPLICITY\ALARMCAST\EXE\fpamserver.exe
FirstPAGEServer	FirstPAGE Server	C:\Program Files (x86)\Proficy\Proficy CIMPLICITY\ALARMCAST\EXE\fpserver.exe
CCFLIC4	Helper Service for Proficy Licensing	C:\Program Files (x86)\Proficy\Proficy Common\Proficy Common Licensing\CCFLIC4.exe
OpcUaBrowseService	OPC UA Browser Microservice	"C:\Program Files (x86)\Proficy\Proficy CIMPLICITY\exe\opcua_browse_service.exe" serve -config "C:\Program Files (x86)\Proficy\Proficy CIMPLICITY\data\opcua-browse-config.json"
ProficyDrivers	Proficy Drivers	C:\Program Files (x86)\Proficy\Proficy Drivers\Win32\ProficyDrivers.exe
CCFLIC0	Proficy Licensing	C:\Program Files (x86)\Proficy\Proficy Common\Proficy Common Licensing\CCFLIC0.exe
TrkCollector	Tracker Collector Service	C:\Program Files (x86)\Proficy\Proficy CIMPLICITY\exe\TrkCollector.exe
TrkEventServer	Tracker Event Service	C:\Program Files (x86)\Proficy\Proficy CIMPLICITY\exe\TrkEventServer.exe

Log data can be collected using the Event IDs of log entries or the event source names.

This table lists the Event IDs related to CIMPLICITY.

Table 2. Event IDs and their description
Event ID	Event text
10	Service startup
13	Service shutdown
1040	Service `CIMPLICITYNGINX` received `STOP` control, which will be handled
1008	Started `C:\Program Files (x86)\Proficy\Proficy CIMPLICITY\Web\exe\nginx.exe` for the `CIMPLICITYNGINX` service in `C:\Program Files (x86)\Proficy\Proficy CIMPLICITY\Web\exe`
1011	Killing process `C:\Program Files (x86)\Proficy\Proficy CIMPLICITY\Web\exe\nginx.exe` because service `CIMPLICITYNGINX` is stopping
1023	Killing process tree of process `2232` for service `CIMPLICITYNGINX` with exit code `0`
1027	Killing `PID 1396` in process tree of `PID 2972` because service `CIMPLICITYNGINX` is stopping

NXLog can be configured to process events from the table above.

Example 1. Processing CIMPLICITY logs based on the Event ID values

CIMPLICITY produced this sample event with Event ID 1040 from Windows Event Log.

Event sample

Log Name:      Application
Source:        nssm
Date:          3/4/2021 8:07:17 AM
Event ID:      1040
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      WIN-5RU7GP5MI4V
Description:
Service CIMPLICITYNGINX received STOP control, which will be handled.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="nssm" />
    <EventID Qualifiers="16384">1040</EventID>
    <Level>4</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2021-03-04T16:07:17.471075800Z" />
    <EventRecordID>12552</EventRecordID>
    <Channel>Application</Channel>
    <Computer>WIN-5RU7GP5MI4V</Computer>
    <Security />
  </System>
  <EventData>
    <Data>CIMPLICITYNGINX</Data>
    <Data>STOP</Data>
  </EventData>
</Event>

This NXLog configuration uses the im_msvistalog module to read data from the Application channel of Windows Event Log. It also lists all Event IDs related to CIMPLICITY, including the one from the sample above. The Exec directive uses the to_json() procedure to format the output as JSON.

nxlog.conf

<Extension json>
    Module    xm_json
</Extension>

<Input eventlog>
    Module    im_msvistalog
    #XML query to read Windows Event Logs based on EventID
    <QueryXML>
        <QueryList>
            <Query Id="0" Path="Application">
                <Select Path="Application">
                    *[System[(EventID=10 or EventID=13 or
                    EventID=1008 or EventID=1011 or
                    EventID=1023 or EventID=1027 or
                    EventID=1040)]]
                </Select>
            </Query>
        </QueryList>
    </QueryXML>
    # Conversion to JSON
    Exec      to_json();
</Input>

Output sample in JSON

{
  "EventTime": "2021-03-04T08:07:17.471075-08:00",
  "Hostname": "WIN-5RU7GP5MI4V",
  "Keywords": "36028797018963968",
  "EventType": "INFO",
  "SeverityValue": 2,
  "Severity": "INFO",
  "EventID": 1040,
  "SourceName": "nssm",
  "TaskValue": 0,
  "RecordNumber": 12552,
  "ExecutionProcessID": 0,
  "ExecutionThreadID": 0,
  "Channel": "Application",
  "Message": "Service CIMPLICITYNGINX received STOP control, which will be handled.",
  "Data": "CIMPLICITYNGINX",
  "Data_1": "STOP",
  "EventReceivedTime": "2021-03-04T08:07:18.439411-08:00",
  "SourceModuleName": "eventlog",
  "SourceModuleType": "im_msvistalog"
}

As previously mentioned, NXLog can filter Windows Event log entries by their event source names.

Example 2. Processing CIMPLICITY logs based on the event source names

CIMPLICITY collected this sample event from the OpcUaBrowseService log source.

Event sample

Log Name:      Application
Source:        OpcUaBrowseService
Date:          3/4/2021 12:53:44 PM
Event ID:      13
Task Category: Service
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      WIN-5RU7GP5MI4V
Description:
Service shutdown, see details for more information.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="OpcUaBrowseService" />
    <EventID Qualifiers="16384">13</EventID>
    <Level>4</Level>
    <Task>1</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2021-03-04T20:53:44.180193300Z" />
    <EventRecordID>12583</EventRecordID>
    <Channel>Application</Channel>
    <Computer>WIN-5RU7GP5MI4V</Computer>
    <Security />
  </System>
  <EventData>
    <Data>UABrowse service is stopping</Data>
  </EventData>
</Event>

This NXLog configuration uses the im_msvistalog module to read Windows Event Log entries. The QueryXML directive of this module specifies the Application channel and the following sources to filter data by:

nssm
CIMConfigService
OpcUaBrowseService
CimProxy
CIMPLICITY Advanced Viewer

nxlog.conf

<Extension json>
    Module    xm_json
</Extension>

<Input eventlog>
    Module    im_msvistalog
    # XML query to read Windows Event Log data
    <QueryXML>
        <QueryList>
            <Query Id="0" Path="Application">
                <Select Path="Application">*
                    [System[Provider[@Name='nssm' or
                        @Name='CIMConfigService' or
                        @Name='OpcUaBrowseService' or
                        @Name='CimProxy' or
                        @Name='CIMPLICITY Advanced Viewer']]]
                </Select>
            </Query>
        </QueryList>
    </QueryXML>
    # Conversion to JSON
    Exec    to_json();
</Input>

The following output sample demonstrates the OpcUaBrowseService event source message after processing by NXLog. As before, formatted as JSON.

Output sample in JSON

{
  "EventTime": "2021-03-04T12:53:44.180193-08:00",
  "Hostname": "WIN-5RU7GP5MI4V",
  "Keywords": "36028797018963968",
  "EventType": "INFO",
  "SeverityValue": 2,
  "Severity": "INFO",
  "EventID": 13,
  "SourceName": "OpcUaBrowseService",
  "TaskValue": 1,
  "RecordNumber": 12583,
  "ExecutionProcessID": 0,
  "ExecutionThreadID": 0,
  "Channel": "Application",
  "Message": "Service shutdown, see details for more information.",
  "Category": "Service",
  "Data": "UABrowse service is stopping",
  "EventReceivedTime": "2021-03-04T12:53:45.010271-08:00",
  "SourceModuleName": "eventlog",
  "SourceModuleType": "im_msvistalog"
}

File-based log collection

This table lists the various types of file-based logs that General Electric CIMPLICITY generates.

Table 3. List of file-based logs
Log type	File ext.	Location	Description
Project status and system status logs	`.csv`, `.txt`, `.LOG`	`…\<project directory>\log\COR_RECSTAT.CSV` `…\<project directory>\log\COR_RECSTAT.txt` `…\<project directory>\log\COR_STATUS.LOG` `…\<CIMPLICITY Installation>\log\COR_STATUS.LOG`
Protocol stack trace logs	`.err`	`…\<project directory>\log\MTCPSI_RP.err`	Communication messages that the Modbus TCP Slave Interface generates
Web configuration services logs	`.log`	`…\<CIMPLICITY Installation>\log\cim_config_service.log`	Interaction logs between the web server and CIMPLICITY
Counters log files	`.CSV`	`…\<CIMPLICITY Installation>\log\PERFDATA_*_DATA.CSV`	Entries about the CIMPLICITY counters acquisition that occurs while running the `w32rtr.exe` router process
Optional OPC client debug tracing	`.LOG`	`…\<project directory>\log\MASTER_OPC_*_<DeviceID>.LOG` …\<project directory>\log\`MASTER_OPC_x.LOG	Messages about communication problems between the OPC client and the OPC server
Catapult DNP3 driver logs	`.log`	`C:\Program Files\Proficy\Proficy Cimplicity\exe\DNP_Logs\dnp_YYYYMMDD_HHMM.log` `C:\Program Files\Proficy\Proficy Cimplicity\exe\DNP_Logs\DNP_SOE<xx>.log`

In this section, we cover how to process each type of log individually. However, there is a universal configuration file at the end that provides a single NXLog configuration for processing all of the file-based logs.

Project status and system status logs

Project and system status log files contain binary data that cannot be viewed directly. However, you can use the CIMPLICITY Log Viewer to view such events and save them in any format listed in this table.

Table 4. List of formats for project and/or system status logs
Format	Location
CSV	`…\<project directory>\log\COR_RECSTAT.CSV`
TXT	`…\<project directory>\log\COR_RECSTAT.txt`
ASCII	`…\<project directory>\log\COR_STATUS.LOG` `…\<CIMPLICITY Installation>\log\COR_STATUS.LOG`

Each status log message from this table contains the following fields that can be parsed and processed by NXLog:

Date/Time - contains the date and time values when the message was logged.
Status - specifies the message type. The available values are Failure, Warning, and Success.
Process - stores the name of the process that generated the log message.
Procedure - contains the name of the procedure that generated the log message.
Source - specifies the name of the error class.
Code - is the primary value used by software for expressing the error type.
Reference - contains the number that can be used to determine the location of the condition that caused the error.
Message - explains the condition that caused the log message to be created.
PID - contains the process identification number.
SourceCode - is contained in the ASCII-formatted logs.

Example 3. Processing the CSV version of status logs

This event sample was taken from the COR_RECSTAT.CSV file.

Event sample

3/4/2021|6:04:10 AM|Success|6544|MRTUSI_RP|Main|0|Program shutdown completed successfully.

In this NXLog configuration, regular expressions are used for parsing the fields. The regular expression for message parsing is defined by the COR_RECSTAT_CSV_REGEX constant. The PRJ_PATH constant specifies the absolute path to the log file.

The xm_multiline module is used to parse events that span multiple lines. Here, the HeaderLine directive defines the regular expression for parsing event headers.

The Exec block removes any carriage return or line feed (CRLF) characters in the event record. The core $EventTime field is created by taking the parsed date and time components and passing them to the strptime() function. The processed event record is formatted to JSON by calling the to_json() procedure of the xm_json module.

nxlog.conf

# A regular expression defined as a constant to read the content of the logs
define COR_RECSTAT_CSV_REGEX    /(?x)^(\d+.\d+.\d+)\|(\d+.\d+.\d+.\w+)\|\
                                (?<Status>\w+)\|(?<PID>\d+)\|(?<Process>.*?)\|\
                                (?<Procedure>.*?)\|(?<Reference>\d+)\|\
                                (?<Message>.*?)\|(?<Source>.*?)\|\
                                (?<Code>\d+)/
# Part of the log path defined as a constant
define PRJ_PATH                 C:\Users\Administrator\Documents\DummyPrj\log

<Extension json>
    Module        xm_json
</Extension>

<Extension multiline_status>
    Module        xm_multiline
    # Regular expression to look for the header of the message
    HeaderLine    /^\d+.\d+.\d+\|\d+.\d+.\d+.\w+/
</Extension>

<Input from_file>
    Module        im_file
    File          '%PRJ_PATH%\COR_RECSTAT.CSV'
    InputType     multiline_status
    <Exec>
        # Replaces unwanted characters
        $raw_event = replace($raw_event, "\r", "");
        $raw_event = replace($raw_event, "\n", "");

        # Matches the events with a regular expression
        if $raw_event =~ %COR_RECSTAT_CSV_REGEX%
        {
            # Creates the timestamp
            $EventTime = strptime($1 + $2, "%m/%d/%Y %T %p");
            # Formats the result as JSON
            to_json();
        }
        # Discard event if it doesn't match a/the regular expression
        else drop();
    </Exec>
</Input>

The following sample represents a processed event in JSON format.

Output sample in JSON

{
  "EventReceivedTime": "2021-03-06T05:17:38.586649-08:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Code": "6",
  "Message": "Program shutdown completed successfully.",
  "PID": "6544",
  "Procedure": "Main",
  "Process": "MRTUSI_RP",
  "Reference": "0",
  "Source": "COR_MRTUSI_ERR",
  "Status": "Success",
  "EventTime": "2021-03-04T06:04:10.000000-08:00"
}

Project status and system status logs can be stored as unstructured event data in plain text files (.txt). The following example shows how to process such logs.

Example 4. Processing the text version of status logs

The following event sample was taken from the COR_RECSTAT.txt file.

Event sample

3/8/2021   4:04:18 AM    Success 3228       PM_MCP                           ProjectStop                       0
Project DUMMYPRJ is stopped.
Error of type: COR_PM_ERR,  Code: 0

Because the event spans multiple lines, the xm_multiline module is used here to process it as a single-line message. The HeaderLine directive of this module specifies the regular expression to identify the first line of each event.

The Exec block contains instructions to replace unwanted characters and parse the event using the COR_RECSTAT_TXT_REGEX expression. Each captured field is added to the event record as per named capturing groups defined in angle brackets (< >) of the regular expression. The strptime() function returns the timestamp captured as fields $1 and $2 with the datetime data type. Finally, the to_json() procedure of the xm_json module formats all parsed fields to JSON.

nxlog.conf

# A regular expression defined as a constant to read the content of the logs
define COR_RECSTAT_TXT_REGEX  /(?x)^(\d+.\d+.\d+)\s+(\d+.\d+.\d+.\w+)\
                              \s+(?<Status>\w+)\s+(?<PID>\d+)\s+(?<Process>\w+)\
                              \s+(?<Procedure>.*?)\s+(?<Reference>\d+)\s+\
                              (?<Message>.*?)\s+(?:Error\s+of\s+type\:\s+)\
                              (?<Source>.*?)\,\s+(?:Code\:\s+)(?<Code>\d+)/
# Part of the log path defined as a constant
define PRJ_PATH               C:\Users\Administrator\Documents\DummyPrj\log

<Extension json>
    Module        xm_json
</Extension>

<Extension multiline_status_text>
    Module        xm_multiline
    # Regular expression to look for the header of the message
    HeaderLine    /^\d+.\d+.\d+\s+\d+.\d+.\d+.\w+/
</Extension>

<Input from_file>
    Module        im_file
    File          '%PRJ_PATH%\COR_RECSTAT.txt'
    InputType     multiline_status_text
    <Exec>
        # Replaces unwanted characters
        $raw_event = replace($raw_event, "\r", " ");
        $raw_event = replace($raw_event, "\n", " ");

        # Matches the events with a regular expression
        if $raw_event =~ %COR_RECSTAT_TXT_REGEX%
        {
            # Creates the timestamp
            $EventTime = strptime($1 + $2, "%m/%d/%Y %T %p");
            # Formats the result as JSON
            to_json();
        }
        # Discard event if it doesn't match a/the regular expression
        else drop();
    </Exec>
</Input>

This is the output sample of the message after it has been processed by NXLog.

Output sample in JSON

{
  "EventReceivedTime": "2021-03-08T05:34:02.040898-08:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Code": "0",
  "Message": "Project DUMMYPRJ is stopped.",
  "PID": "3228",
  "Procedure": "ProjectStop",
  "Process": "PM_MCP",
  "Reference": "0",
  "Source": "COR_PM_ERR",
  "Status": "Success",
  "EventTime": "2021-03-08T04:04:18.000000-08:00"
}

CIMPLICITY can be configured to save and automatically update status log entries as ASCII text. In this case, CIMPLICITY creates the COR_STATUS.LOG file in the …\<project directory>\log folder and/or system folder …\<CIMPLICITY Installation>\log\ on startup.

CIMPLICITY log files with the .err extension other than user processes, are also stored in the project directory. Some of these logs have the same structure as the COR_STATUS.LOG file.

Example 5. Processing the ASCII-formatted version of status logs

The following event sample was taken from the COR_STATUS.LOG file.

Event sample

Mon Mar  8 20:31:26 2021 failure       1892       CfgCab bGetTADBConnect     83
item does not exist
 dbms_def $TADB
Error of type COR_SC_ERR:
source 130   code 111

The NXLog configuration contains the COR_STATUS_REGEX and PRJ_PATH constants. The former specifies the regular expressions to parse the event contents, while the latter defines the absolute path to the folder with the log file.

Since events span multiple lines, they should be processed by the xm_multiline module. Here, the HeaderLine directive specifies the regular expression to search for the first line and thus identify the boundary of each event.

The Exec directive block of the input module replaces the \r and \n characters with the whitespace character and parses the event using the COR_STATUS_REGEX regular expression. The original event time is processed using the strptime() function to create a timestamp. After processing, messages are converted to JSON using the to_json() procedure of the xm_json module.

nxlog.conf

# A regular expression defined as a constant to read the content of the logs
define COR_STATUS_REGEX     /(?x)^(\w+\s+\w+\s+\d+\s+\d+.\d+.\d+\s+\d+)\s+\
                            (?<Status>\w+)\s+(?<PID>\d+)\s+\
                            (?<Process>\<.*?\>|.*?)\s+(?<Procedure>.*?)\s+\
                            (?<Reference>\d+)\s+(?<Message>.*?)\s+\
                            (?:Error\s+of\s+type\s+)(?<Source>.*?)\:\s+\
                            (?:source\s+)(?<SourceCode>\d+)\s+(?:code\s+)\
                            (?<Code>\d+)/
# Part of the log path defined as a constant
define PRJ_PATH             C:\Users\Administrator\Documents\DummyPrj\log

<Extension json>
    Module        xm_json
</Extension>

<Extension multiline_process>
    Module        xm_multiline
    # A regular expression that matches the header (first line) of an event
    HeaderLine    /^\w+\s+\w+\s+\d+\s+\d+.\d+.\d+\s+\d+/
</Extension>

<Input from_file>
    Module        im_file
    File          '%PRJ_PATH%\COR_STATUS.LOG'
    InputType     multiline_process
    <Exec>
        # Replaces unwanted characters
        $raw_event = replace($raw_event, "\r", " ");
        $raw_event = replace($raw_event, "\n", " ");
        $raw_event =~ s/\s{2,}/ /g;

        # Matches the events with a regular expression
        if $raw_event =~ %COR_STATUS_REGEX%
        {
            # Creates the timestamp
            $EventTime = strptime($1, "%a %b %e %T %Y");
            # Formats the result as JSON
            to_json();
        }
        # Discard event if it doesn't match a/the regular expression
        else drop();
    </Exec>
</Input>

Below is the output sample after it has been converted to JSON.

Output sample in JSON

{
  "EventReceivedTime": "2021-03-08T23:26:46.097086-08:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Code": "111",
  "Message": "item does not exist dbms_def $TADB",
  "PID": "1892",
  "Procedure": "bGetTADBConnect",
  "Process": "CfgCab",
  "Reference": "83",
  "Source": "COR_SC_ERR",
  "SourceCode": "130",
  "Status": "failure",
  "EventTime": "2021-03-08T20:31:26.000000-08:00"
}

Protocol stack trace logs

The protocol stack trace logs contain communication messages from the Modbus TCP Slave Interface.

Each event message contains the timestamp, direction (expressed as either -> or <-), connection ID, and a message body.

By default, protocol stack trace logging is disabled. If enabled, the Modbus TCP slave interface stores trace messages in the MTCPSI_RP.err file of the …\<project directory>\log directory.

Each log entry is comprised of three types of parts: header, event message, and footer. Both header and footer messages contain event timestamps and verbose message texts.

The structure of the header and the footer is exactly the same. The same regular expression can be used to parse them.

Example 6. Processing protocol stack trace logs

These samples represent a typical header and event message.

Header sample

2021-03-11 04:50:14.415 Started Modbus/TCP Slave Interface (MTCPSI_RP) - using IP address 127.0.0.1

Event message sample

2021-03-11 04:50:17.370 -> [id = 1216] 13F90000000F 03030C04D211D722C5000000000000

To parse the message and its header, the NXLog configuration defines the MTCPSI_REGEX and MTCPSI_HEADER_REGEX constants with regular expressions. The PRJ_PATH constant defines the full path to the project directory.

The Exec block of the im_file module compares each line of the input file to the two regular expressions. In case a line matches either of them, the parsedate() function processes the timestamp data and saves the result to the $EventTime field. Other information is parsed and saved to fields according to the names of the named capturing groups. The result is converted to JSON using the to_json() procedure of the xm_json module. In case of non-match, entries are discarded using the drop() procedure.

nxlog.conf

# Regular expressions defined to read the contents of log entries
define MTCPSI_HEADER_REGEX  /(?x)^(\d+.\d+.\d+.\d+.\d+.\d+.\d+)\s+\
                            (?<Message>.*)/
define MTCPSI_REGEX         /(?x)^(\d+.\d+.\d+.\d+.\d+.\d+.\d+)\s+\
                            (?<Direction>[\<\>\-]*)\s+(?:\[id\s+\=)\s+\
                            (?<ConnectionID>\d+)\]\s+(?<Data>[\d\w\s]*)\s+/
# Part of the log path defined as a constant
define PRJ_PATH             C:\Users\Administrator\Documents\DummyPrj\log

<Extension json>
    Module        xm_json
</Extension>

<Input from_file>
    Module        im_file
    File          '%PRJ_PATH%\MTCPSI_RP.err'
    <Exec>
        # Matches the events with a regular expression
        if ($raw_event =~ %MTCPSI_REGEX%) or
           ($raw_event =~ %MTCPSI_HEADER_REGEX%)
        {
            # Creates the timestamp
            $EventTime = parsedate($1);
            # Converts to JSON
            to_json();
        }
        # Discard event if it doesn't match a/the regular expression
        else drop();
    </Exec>
</Input>

Here we see the JSON-formatted header and message samples after NXLog has processed them.

Output header sample

{
  "EventReceivedTime": "2021-03-11T04:50:14.609195-08:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Message": "Started Modbus/TCP Slave Interface (MTCPSI_RP) - using IP address 127.0.0.1",
  "EventTime": "2021-03-11T04:50:14.415000-08:00"
}

Output message sample

{
  "EventReceivedTime": "2021-03-11T04:50:17.621151-08:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "ConnectionID": "1216",
  "Data": "13F90000000F 03030C04D211D722C5000000000000",
  "Direction": "->",
  "EventTime": "2021-03-11T04:50:17.370000-08:00"
}

Web configuration services logs

The OPC UA browse service and cim-config service web configuration services provide interaction of the web server with CIMPLICITY. Log files of these services are located in the …\<CIMPLICITY Installation>\log\ directory.

Each log entry of this type consists of the following fields:

EventTime
Service
Status
Loader
Message

The example below demonstrates how NXLog can process the file-based logs of the web configuration services.

Example 7. Processing log messages of web configuration services

CIMPLICITY web configuration services produced this log message sample.

Sample message

[2021-03-09 19:15:38.568] [CIMConfigService] [info] [CimConfigService.cpp:165] CIMPLICITY Config REST API is available at https://localhost:4855/cim-config/v1

To read messages like this sample, this NXLog configuration uses the WEBCONF_REGEX constant with the regular expression. The PROFICY_PATH constant defines the path to the folder with files.

The Exec block of the im_file module compares each message to WEBCONF_REGEX. If it matches, the parsedate() function is called to convert the first captured string to a datetime value and assigns it to the $EventTime field. All other fields are created and values assigned to them according to the named capturing groups of WEBCONF_REGEX. All fields are converted to JSON using the to_json() procedure of the xm_json module. If not matched, a message is discarded using the drop() procedure.

nxlog.conf

# A regular expression for reading the contents of the logs
define WEBCONF_REGEX    /(?x)^\[(\d+.\d+.\d+.\d+.\d+.\d+.\d+)\]\s+\
                        \[(?<Service>\w+)\]\s+\[(?<Status>\w+)\]\s+\
                        \[(?<Loader>.*?)\]\s(?<Message>.*)/
# Part of the log path defined as a constant
define PROFICY_PATH     C:\Program Files (x86)\Proficy\Proficy CIMPLICITY\log

<Extension json>
    Module        xm_json
</Extension>

<Input from_file>
    Module        im_file
    File          '%PROFICY_PATH%\cim_config_service.log'
    <Exec>
        # Matches the events with a regular expression
        if $raw_event =~ %WEBCONF_REGEX%
        {
            # Creates the timestamp
            $EventTime = parsedate($1);
            # Formats the result as JSON
            to_json();
        }
        # Discard event if it doesn't match a/the regular expression
        else drop();
    </Exec>
</Input>

The following sample represents the processed event in JSON format.

Output sample in JSON

{
  "EventReceivedTime": "2021-03-09T21:15:49.428591-08:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Loader": "CimConfigService.cpp:165",
  "Message": "CIMPLICITY Config REST API is available at https://localhost:4855/cim-config/v1",
  "Service": "CIMConfigService",
  "Status": "info",
  "EventTime": "2021-03-09T19:15:38.568000-08:00"
}

Counters log files

The counters logs contain information about the CIMPLICITY counters acquisition that occurs while running the w32rtr.exe router process.

Once the pre-configured size limit of the log file has been reached, all log files are rotated.

Counters log data are contained in the PERFDATA_*_DATA.CSV rotated files of the …\<CIMPLICITY Installation>\log\ folder. Each counters log message contains the date and time when the event was generated, along with the Process_PrivateBytes_Total and Objects_Threads fields.

NXLog can be configured to process CIMPLICITY counters logs.

Example 8. Processing counters logs

Sample message

"03/10/2021 00:48:20.361","1200619520","1249"

To parse the log entries like this sample, the NXLog configuration defines the PERFDATA_REGEX constant with a regular expression. The first capturing group is used to to capture the value for the EventTime field. The other two named capturing groups parse the values for the Process_PrivateBytes_Total and Objects_Threads fields. The PROFICY_PATH constant defines the path to log files.

The wildcard character (* in PERFDATA_*_DATA.CSV) is used to read rotated files like PERFDATA_1_DATA.CSV and PERFDATA_2_DATA.CSV. The Exec block of im_file matches messages against PERFDATA_REGEX. In case of match, the strptime() function is called to create the timestamp and write the result to the $EventTime field. The other fields are created using named capturing groups. The result is converted to JSON using the to_json() procedure of xm_json module. The drop() procedure discards messages that do not match PERFDATA_REGEX.

nxlog.conf

# A regular expression defined as a constant to read the content of the logs
define PERFDATA_REGEX   /(?x)^\"(\d+.\d+.\d+.\d+.\d+.\d+).\d+\"\,\
                        \"(?<Process_PrivateBytes_Total>\d+)\"\,\
                        \"(?<Objects_Threads>\d+)\"/
# Part of the log path defined as a constant
define PROFICY_PATH     C:\Program Files (x86)\Proficy\Proficy CIMPLICITY\log

<Extension json>
    Module        xm_json
</Extension>

<Input from_file>
    Module        im_file
    File          '%PROFICY_PATH%\PERFDATA_*_DATA.CSV'
    <Exec>
        # Matches the events with a regular expression
        if $raw_event =~ %PERFDATA_REGEX%
        {
            # Creates the timestamp
            $EventTime = strptime($1, "%m/%d/%Y %T");
            # Formats the result as JSON
            to_json();
        }
        # Discard event if it doesn't match a/the regular expression
        else drop();
    </Exec>
</Input>

Below is the output sample containing the JSON-formatted message.

Output sample in JSON

{
  "EventReceivedTime": "2021-03-10T03:06:50.213183-08:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Objects_Threads": "1249",
  "Process_PrivateBytes_Total": "1200619520",
  "EventTime": "2021-03-10T00:48:20.000000-08:00"
}

Optional OPC client debug tracing

To diagnose communication problems between the OPC client and the OPC server, CIMPLICITY has the ability to enable tracing. The trace file is located in the project log directory …\<project directory>\log.

Depending on the specific device parameters, the trace data can be stored either in the CIMPLICITY port trace file like MASTER_OPC_x.LOG or in its own rotated file named as per the device ID like MASTER_OPC_*_<DeviceID>.LOG, where the asterisk (*) is the wildcard character substituting numerals.

Each trace record contains several fields of data separated by the vertical bar character (|):

Timestamp - 20210314.0725577939
Device ID - OPC_KEP
Thread ID - TOOLKIT
Message - INI parameter [TraceLevel] Value [2]
Result (error) Code - 0x00000000(The operation completed successfully. )
Group ID - contains no data
Point ID - contains no data
Point Address - contains no data
Point Value - contains no data
Point Quality - 0x0000
Trace Message ID - 0x00000000
Trace Message Group ID - 04097

NXLog can be configured to process OPC client debug tracing of CIMPLICITY.

Example 9. Processing optional OPC client debug trace logs

The following event sample was taken from the MASTER_OPC_*_<DeviceID>.LOG rotated file.

Event sample

20210314.0725577939|OPC_KEP|TOOLKIT|[DEVICE] INI parameter [TraceLevel] Value [2]|0x00000000(The operation completed successfully. )|||||0x0000|0x00000000|04097

In this NXLog configuration two constants are defined: OPCTRC_REGEX is assigned the regular expression for parsing fields from each message and and PRJ_PATH is the absolute path to the trace logs. The Exec block of the im_file module compares each message to OPCTRC_REGEX. If it matches, the strptime() function is called to convert the captured string to a datetime value and assigns it to the $EventTime field. All other fields are created using the named capturing groups. The parsed fields are converted to JSON using the to_json() procedure of xm_json. The non-matching messages are discarded using the drop() procedure.

nxlog.conf

# A regular expression defined as a constant to read the content of the logs
define OPCTRC_REGEX     /(?x)^(\d+)\.(\d{6})\d+\|(?<DeviceID>.*?)\|\
                        (?<ThreadID>.*?)\|(?<Message>.*?)\|(?<ResultCode>.*?)\|\
                        (?<GroupID>.*?)\|(?<PointID>.*?)\|(?<PointAddress>.*?)\|\
                        (?<PointValue>.*?)\|(?<PointQuality>.*?)\|\
                        (?<TraceMessageID>.*?)\|(?<TraceMessageGroupID>.*)/
# Part of the log path defined as a constant
define PRJ_PATH         C:\Users\Administrator\Documents\DummyPrj\log

<Extension json>
    Module        xm_json
</Extension>

<Input from_file>
    Module        im_file
    File          '%PRJ_PATH%\MASTER_OPC_*_OPC_KEP.LOG'
    <Exec>
        # Matches the events with a regular expression
        if $raw_event =~ %OPCTRC_REGEX%
        {
            # Creates the timestamp
            $EventTime = strptime($1 + $2, "%Y%m%d%H%M%S");
            # Formats the result as JSON
            to_json();
        }
        # Discard event if it doesn't match a/the regular expression
        else drop();
    </Exec>
</Input>

This output sample below represents a processed event in JSON format.

Output sample in JSON

{
  "EventReceivedTime": "2021-03-14T07:25:59.316562-07:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "DeviceID": "OPC_KEP",
  "GroupID": "",
  "Message": "[DEVICE] INI parameter [TraceLevel] Value [2]",
  "PointAddress": "",
  "PointID": "",
  "PointQuality": "0x0000",
  "PointValue": "",
  "ResultCode": "0x00000000(The operation completed successfully. )",
  "ThreadID": "TOOLKIT",
  "TraceMessageGroupID": "04097",
  "TraceMessageID": "0x00000000",
  "EventTime": "2021-03-14T07:25:57.000000-07:00"
}

Catapult DNP3 driver logs

Catapult DNP3 is a fully compliant OPC server that natively supports CIMPLICITY HMI SCADA systems.

This type of log stores the following types of event messages:

Channel messages
Sequence Of Events messages
Driver debug messages
Device messages
Error messages
Changes to the configuration

By default, log data of this type is located in rotated files as per the table below.

Table 5. List of Catapult DNP driver logs
Log type	Location
Catapult DNP3 driver main log	`C:\Program Files\Proficy\Proficy Cimplicity\exe\DNP_Logs\dnp_YYYYMMDD_HHMM.log`
SOE (Sequence of Event) log	`C:\Program Files\Proficy\Proficy Cimplicity\exe\DNP_Logs\DNP_SOE<xx>.log`

Each Catapult DNP3 driver main log message from the main log file consists of the following fields:

Date - 3/15/2021
Time - 04:21:34
Source - Channel CH_1
EventType - Error
Message - RefreshChannelStatus: iResult = 0 [Socket Error (RefreshChannelStatus) - Socket Still Connecting (Error Code = 0)]

Example 10. Processing Catapult DNP3 driver main log

Below is the sample of Catapult DNP3 driver main log.

Message sample

3/15/2021	"04:21:34"	:277	Channel	CH_1	Error		RefreshChannelStatus: iResult = 0 [Socket Error (RefreshChannelStatus) - Socket Still Connecting (Error Code = 0)]

In this NXLog configuration two constants are defined: DNP_REGEX is assigned the regular expression for parsing fields from each message and and DNP_PATH is the absolute path to the log files.

The Exec block of the im_file module compares each message to DNP_REGEX. If it matches, the strptime() function is called to convert the captured string to a datetime value and assigns it to the $EventTime field. All other fields are created according to the named capturing groups of DNP_REGEX. The result is converted to JSON using the to_json() procedure of xm_json. If not matched, messages are discarded using the drop() procedure.

nxlog.conf

# A regular expression defined as a constant to read the content of the logs
define DNP_REGEX    /(?x)^(\d+.\d+.\d+)\s+\"(\d+.\d+.\d+)\"\s+\:\d+\s+\
                    (?<Source>.*?(?=\s+\w+\s{2}))\s+(?<EventType>\w+)\s+\
                    (?<Message>.*)/
# Part of the log path defined as a constant
define DNP_PATH     C:\Program Files (x86)\Proficy\Proficy CIMPLICITY\exe\DNP_Logs

<Extension json>
    Module        xm_json
</Extension>

<Input from_file>
    Module        im_file
    File          '%DNP_PATH%\dnp_20210315_0018.log'
    <Exec>
        # Replaces unwanted characters
        $raw_event = replace($raw_event, "\t", " ");
        # Matches the events with a regular expression
        if $raw_event =~ %DNP_REGEX%
        {
            # Creates the timestamp
            $EventTime = strptime($1 + $2, "%m/%d/%Y %T");
            # Formats the result as JSON
            to_json();
        }
        # Discard event if it doesn't match a/the regular expression
        else drop();
    </Exec>
</Input>

Below is the output sample of the message in JSON format.

Output sample in JSON

{
  "EventReceivedTime": "2021-03-15T04:30:36.589913-07:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "EventType": "Error",
  "Message": "RefreshChannelStatus: iResult = 0 [Socket Error (RefreshChannelStatus) - Socket Still Connecting (Error Code = 0)]",
  "Source": "Channel CH_1",
  "EventTime": "2021-03-15T04:21:34.000000-07:00"
}

Sequence of Events (SOE) log files record the changes in the values of important points in the SCADA system generated and timestamped by the DNP3 slave.

The SOE log message sample contains the SOE log entry from the DNP_SOE**.log rotated file. The asterisk (*) symbol in the filename is the wildcard symbol which is substituted with any numeral like DNP_SOE01.log, DNP_SOE02.log, etc when NXLog is running.

Each SOE event contains the following data fields that NXLog can parse and process:

EventTime contains the date and time when the event was received by the DNP3 Driver. In the sample, this is 2020 08 20 04:29:34.198
MessageType contains the SOE value in the sample
Address contains the address of the DNP3 point that generate SOE messages in the DEVICENAME.OBJ.VAR.IDX format. In the sample above, this is DNP_SIM.2.3.0
RawValue is 81 in the sample
SOE_Timestamp describes the actual time of the event captured by the DNP3 slave instance. In the sample above, this is 2020 08 20 14:29:33.000

Example 11. Processing Catapult DNP3 driver SOE logs

Message sample

2020 08 20 04:29:34.198     SOE DNP_SIM.2.3.0,81,2020 08 20 14:29:33.000

The NXLog configuration below defines the DNPSOE_REGEX and DNP_PATH constants. The first constant contains the regular expression for parsing SOE messages. The second constant defines the absolute path to the log files.

DNPSOE_REGEX contains several capturing groups. If the first two groups are matched, the captured values are concatenated and passed as a single string to the strptime() function which returns a datetime value that it assigns to the $EventTime field. All other fields are created and the values assigned to them according to the named capturing groups. The parsed fields are converted to JSON using the to_json() procedure of the xm_json module. If not matched, messages are discarded using the drop() procedure.

nxlog.conf

# A regular expression defined as a constant to read the content of the logs
define DNPSOE_REGEX /(?x)^(\d+.\d+.\d+.\d+.\d+.\d+)\.\d+\s+(?<MessageType>\w+)\
                    \s+(?<Address>.*?)\,(?<RawValue>.*?)\,(.*?)\.\d+/
# Part of the log path defined as a constant
define DNP_PATH     C:\Program Files (x86)\Proficy\Proficy CIMPLICITY\exe\DNP_Logs

<Extension json>
    Module        xm_json
</Extension>

<Input from_file>
    Module        im_file
    File          '%DNP_PATH%\dnp_soe**.log'
    <Exec>
        # Matches the events with a regular expression
        if $raw_event =~ %DNPSOE_REGEX%
        {
            # Creates the timestamps
            $EventTime = strptime($1, "%Y %m %d %T");
            $SOE_TimeStamp = strptime($5, "%Y %m %d %T");
            # Formats the result as JSON
            to_json();
        }
        # Discard event if it doesn't match a/the regular expression
        else drop();
    </Exec>
</Input>

Output sample in JSON

{
  "EventReceivedTime": "2021-03-15T21:10:16.231804-07:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Address": "DNP_SIM.2.3.0",
  "MessageType": "SOE",
  "RawValue": "81",
  "EventTime": "2020-08-20T04:29:34.000000-07:00",
  "SOE_TimeStamp": "2020-08-20T14:29:33.000000-07:00"
}

Universal configuration for file-based logs

In this example, the various configuration components needed for parsing and processing each type of file-based log source are combined into a single configuration for the sake of convenience.

Example 12. Processing all file-based logs

This configuration is divided into six parts. In the first part, all regular expressions are defined as constants. The second part of the configuration defines the paths to the folders with log files.

The third part loads the extension modules. The xm_json module is needed for converting the parsed data to JSON. Multiple instances of the xm_multiline module are needed for defining the HeaderLine which differs from one multiline log source to another.

The fourth part handles input—the bulk of the configuration—by defining separate im_file module instances, one for each of the nine log sources.

The fifth part of the configuration uses the om_file module to save the JSON-formatted data to file.

The sixth and final part, the ROUTES section, routes the streams of JSON-formatted data from all the input instances to the output module.

nxlog.conf

# --------------- REGULAR EXPRESSIONS FOR PARSING DATA -------------------------

define COR_RECSTAT_CSV_REGEX    /(?x)^(\d+.\d+.\d+)\|(\d+.\d+.\d+.\w+)\|\
                                (?<Status>\w+)\|(?<PID>\d+)\|(?<Process>.*?)\|\
                                (?<Procedure>.*?)\|(?<Reference>\d+)\|\
                                (?<ErrorMessage>.*?)\|(?<Source>.*?)\|\
                                (?<Code>\d+)/

define COR_RECSTAT_TXT_REGEX    /(?x)^(\d+.\d+.\d+)\s+(\d+.\d+.\d+.\w+)\
                                \s+(?<Status>\w+)\s+(?<PID>\d+)\s+\
                                (?<Process>\w+)\s+(?<Procedure>.*?)\s+\
                                (?<Reference>\d+)\s+(?<Message>.*?)\s+\
                                (?:Error\s+of\s+type\:\s+)(?<Source>.*?)\,\
                                \s+(?:Code\:\s+)(?<Code>\d+)/

define COR_STATUS_REGEX         /(?x)^(\w+\s+\w+\s+\d+\s+\d+.\d+.\d+\s+\d+)\s+\
                                (?<Status>\w+)\s+(?<PID>\d+)\s+\
                                (?<Process>\<.*?\>|.*?)\s+(?<Procedure>.*?)\s+\
                                (?<Reference>\d+)\s+(?<Message>.*?)\s+\
                                (?:Error\s+of\s+type\s+)(?<Source>.*?)\:\s+\
                                (?:source\s+)(?<SourceCode>\d+)\s+(?:code\s+)\
                                (?<Code>\d+)/

define MTCPSI_HEADER_REGEX      /(?x)^(\d+.\d+.\d+.\d+.\d+.\d+.\d+)\s+\
                                (?<Message>.*)/

define MTCPSI_REGEX             /(?x)^(\d+.\d+.\d+.\d+.\d+.\d+.\d+)\s+\
                                (?<Direction>[\<\>\-]*)\s+(?:\[id\s+\=)\s+\
                                (?<ConnectionID>\d+)\]\s+(?<Data>[\d\w\s]*)\s+/

define WEBCONF_REGEX            /(?x)^\[(\d+.\d+.\d+.\d+.\d+.\d+.\d+)\]\s+\
                                \[(?<Service>\w+)\]\s+\[(?<Status>\w+)\]\s+\
                                \[(?<Loader>.*?)\]\s(?<Message>.*)/

define PERFDATA_REGEX           /(?x)^\"(\d+.\d+.\d+.\d+.\d+.\d+).\d+\"\,\
                                \"(?<Process_PrivateBytes_Total>\d+)\"\,\
                                \"(?<Objects_Threads>\d+)\"/

define OPCTRC_REGEX             /(?x)^(\d+)\.(\d{6})\d+\|(?<DeviceID>.*?)\|\
                                (?<ThreadID>.*?)\|(?<Message>.*?)\|\
                                (?<ResultCode>.*?)\|(?<GroupID>.*?)\|\
                                (?<PointID>.*?)\|(?<PointAddress>.*?)\|\
                                (?<PointValue>.*?)\|(?<PointQuality>.*?)\|\
                                (?<TraceMessageID>.*?)\|\
                                (?<TraceMessageGroupID>.*)/

define DNP_REGEX                /(?x)^(\d+.\d+.\d+)\s+\"(\d+.\d+.\d+)\"\s+\:\
                                \d+\s+(?<Source>.*?(?=\s+\w+\s{2}))\s+\
                                (?<EventType>\w+)\s+(?<Message>.*)/

define DNPSOE_REGEX	            /(?x)^(\d+.\d+.\d+.\d+.\d+.\d+)\.\d+\s+\
                                (?<MessageType>\w+)\s+(?<Address>.*?)\,\
                                (?<RawValue>.*?)\,(.*?)\.\d+/

# -------------------------- PATHS TO LOG FILES --------------------------------

define PRJ_PATH   C:\Users\Administrator\Documents\DummyPrj\log

define PROF_PATH  C:\Program Files (x86)\Proficy\Proficy CIMPLICITY\log

define DNP_PATH   C:\Program Files (x86)\Proficy\Proficy CIMPLICITY\exe\DNP_Logs

# -------------------------- EXTENSION MODULES ---------------------------------

<Extension json>
    Module        xm_json
</Extension>

<Extension multiline_recstat_csv>
    Module        xm_multiline
    HeaderLine    /^\d+.\d+.\d+\|\d+.\d+.\d+.\w+/
</Extension>

<Extension multiline_recstat_text>
    Module        xm_multiline
    HeaderLine    /^\d+.\d+.\d+\s+\d+.\d+.\d+.\w+/
</Extension>

<Extension multiline_cor_status>
    Module        xm_multiline
    HeaderLine    /^\w+\s+\w+\s+\d+\s+\d+.\d+.\d+\s+\d+/
</Extension>

# -------------------------- INPUT MODULES -------------------------------------

<Input recstat_csv>
    Module        im_file
    File          '%PRJ_PATH%\COR_RECSTAT.CSV'
    InputType     multiline_recstat_csv
    <Exec>
        $raw_event = replace($raw_event, "\r", "");
        $raw_event = replace($raw_event, "\n", "");
        if $raw_event =~ %COR_RECSTAT_CSV_REGEX%
        {
            $EventTime = strptime($1 + $2, "%m/%d/%Y %T %p");
            to_json();
        }
        else drop();
    </Exec>
</Input>

<Input recstat_text>
    Module        im_file
    File          '%PRJ_PATH%\COR_RECSTAT.txt'
    InputType     multiline_recstat_text
    <Exec>
        $raw_event = replace($raw_event, "\r", " ");
        $raw_event = replace($raw_event, "\n", " ");
        if $raw_event =~ %COR_RECSTAT_TXT_REGEX%
        {
            $EventTime = strptime($1 + $2, "%m/%d/%Y %T %p");
            to_json();
        }
        else drop();
    </Exec>
</Input>

<Input cor_status>
    Module        im_file
    File          '%PRJ_PATH%\COR_STATUS.LOG'
    InputType     multiline_cor_status
    <Exec>
        $raw_event = replace($raw_event, "\r", " ");
        $raw_event = replace($raw_event, "\n", " ");
        $raw_event =~ s/\s{2,}/ /g;
        if $raw_event =~ %COR_STATUS_REGEX%
        {
            $EventTime = strptime($1, "%a %b %e %T %Y");
            to_json();
        }
        else drop();
    </Exec>
</Input>

<Input protocol_stack_trace>
    Module        im_file
    File          '%PRJ_PATH%\MTCPSI_RP.err'
    <Exec>
        if ($raw_event =~ %MTCPSI_REGEX%) or
           ($raw_event =~ %MTCPSI_HEADER_REGEX%)
        {
            $EventTime = parsedate($1);
            to_json();
        }
        else drop();
    </Exec>
</Input>

<Input web_conf>
    Module        im_file
    File          '%PROF_PATH%\cim_config_service.log'
    <Exec>
        if $raw_event =~ %WEBCONF_REGEX%
        {
            $EventTime = parsedate($1);
            to_json();
        }
        else drop();
    </Exec>
</Input>

<Input perfdata>
    Module        im_file
    File          '%PROF_PATH%\PERFDATA_*_DATA.CSV'
    <Exec>
        if $raw_event =~ %PERFDATA_REGEX%
        {
            $EventTime = strptime($1, "%m/%d/%Y %T");
            to_json();
        }
        else drop();
    </Exec>
</Input>

<Input opcclient>
    Module        im_file
    File          '%PRJ_PATH%\MASTER_OPC_*_OPC_KEP.LOG'
    <Exec>
        if $raw_event =~ %OPCTRC_REGEX%
        {
            $EventTime = strptime($1 + $2, "%Y%m%d%H%M%S");
            to_json();
        }
        else drop();
    </Exec>
</Input>

<Input dnp>
    Module        im_file
    File          '%DNP_PATH%\dnp_20210315_0018.log'
    <Exec>
        $raw_event = replace($raw_event, "\t", " ");
        if $raw_event =~ %DNP_REGEX%
        {
            $EventTime = strptime($1 + $2, "%m/%d/%Y %T");
            to_json();
        }
        else drop();
    </Exec>
</Input>

<Input dnp_soe>
    Module        im_file
    File          '%DNP_PATH%\dnp_soe**.log'
    <Exec>
        if $raw_event =~ %DNPSOE_REGEX%
        {
            $EventTime = strptime($1, "%Y %m %d %T");
            $SOE_TimeStamp = strptime($5, "%Y %m %d %T");
            to_json();
        }
        else drop();
    </Exec>
</Input>

# -------------------------- OUTPUT MODULE -------------------------------------

<Output to_file>
    Module        om_file
    File          'C:\output.txt'
</Output>

# ------------------------------ ROUTES ----------------------------------------

<Route r1>
    Path    recstat_csv, recstat_text, cor_status => to_file
</Route>

<Route r2>
    Path    protocol_stack_trace, web_conf, perfdata, opcclient => to_file
</Route>

<Route r3>
    Path    dnp, dnp_soe => to_file
</Route>

Network monitoring

NXLog Enterprise Edition exclusive feature

CIMPLICITY can be configured to use a wide range of communication protocols and provide a diversity of OPC functions. To passively monitor CIMPLICITY’s network traffic, NXLog uses the im_pcap module.

This section describes how to monitor network traffic of the following industrial protocols:

Modbus TCP
BACnet
DNP3
IEC 60870-5-104

Modbus TCP

Modbus is an application protocol used in industrial automation systems. Modbus TCP describes how to apply the Modbus protocol using the Ethernet interface along with the TCP/IP transport and network layers. Each Modbus transaction consists of a client request followed by a server response.

Example 13. Capturing SCADA Modbus packets

Using the Dev directive in the im_pcap module, this NXLog configuration specifies the network interface it will use for capturing packets. The Protocol group directive denotes the modbus protocol for monitoring. The Exec block of im_pcap calls the to_json() procedure of the xm_json module to convert messages to JSON.

nxlog.conf

<Extension _json>
    Module        xm_json
</Extension>

<Input pcap_modbus>
    Module        im_pcap
    # Name of a network device/interface
    Dev           \Device\NPF_{E8451A5D-5676-4BEF-8ED5-68911C97D953}
    <Protocol>
        # Protocol type
        Type      modbus
    </Protocol>
    # Conversion to JSON
    Exec          to_json();
</Input>

Modbus TCP query sample

{
  "modbus.function_code": "Read Holding Registers (03)",
  "modbus.length": "6",
  "modbus.prot_id": "0",
  "modbus.query.read_holding_regs.qty_of_regs": "3",
  "modbus.query.read_holding_regs.starting_address": "0",
  "modbus.trans_id": "844",
  "modbus.unit_id": "1",
  "EventTime": "2021-07-16T02:34:08.606328-07:00",
  "EventReceivedTime": "2021-07-16T02:34:09.241839-07:00",
  "SourceModuleName": "pcap",
  "SourceModuleType": "im_pcap"
}

Modbus TCP response sample

{
  "modbus.function_code": "Read Holding Registers (03)",
  "modbus.length": "9",
  "modbus.prot_id": "0",
  "modbus.response.read_holding_regs.byte_count": "6",
  "modbus.response.read_holding_regs.registers": "3260, 2937, 3152",
  "modbus.trans_id": "844",
  "modbus.unit_id": "1",
  "EventTime": "2021-07-16T02:34:08.623539-07:00",
  "EventReceivedTime": "2021-07-16T02:34:09.241839-07:00",
  "SourceModuleName": "pcap",
  "SourceModuleType": "im_pcap"
}

BACnet

Building Automation and Control Network (BACnet) is a communication protocol designed for building automation and control systems. BACnet/IP describes how BACnet can use UDP to send data to connected devices across multiple subnets. NXLog can be configured to capture BACnet packets.

Example 14. Capturing SCADA BACnet packets

To provide passive network monitoring, NXLog uses the im_pcap module. In this case, the Dev directive of this module specifies the network interface it will use for capturing packets. The Protocol group directive of im_pcap specifies the bacnet protocol for capturing. The Exec block of im_pcap converts each message to JSON.

nxlog.conf

<Extension _json>
    Module        xm_json
</Extension>

<Input pcap_bacnet>
    Module        im_pcap
    # Name of a network device/interface
    Dev           \Device\NPF_{E8451A5D-5676-4BEF-8ED5-68911C97D953}
    <Protocol>
        # Protocol type
        Type      bacnet
    </Protocol>
    Exec          to_json();
</Input>

BACnet request

{
  "bacnet.apdu.bacnet_confirmed_request.invoke_id": "159",
  "bacnet.apdu.bacnet_confirmed_request.max_resp": "1476",
  "bacnet.apdu.bacnet_confirmed_request.max_segs": "16",
  "bacnet.apdu.bacnet_confirmed_request.more_segments_follow": "false",
  "bacnet.apdu.bacnet_confirmed_request.segmented": "false",
  "bacnet.apdu.bacnet_confirmed_request.segmented_accepted": "true",
  "bacnet.apdu.bacnet_confirmed_request.service_choice": "Read Property (12)",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.0.object_identifier.instance_number": "0",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.0.object_identifier.type": "analog-input (0)",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.1.property_identifier": "present-value (85)",
  "bacnet.apdu.pdu_type": "BACnet-Confirmed-Request-PDU (0x00)",
  "bacnet.bvlc.function": "Original-Unicast-NPDU (0x0A)",
  "bacnet.bvlc.length": "27",
  "bacnet.bvlc.type": "BACnet/IP (Annex J) (0x81)",
  "bacnet.npdu.control": "0x0024",
  "bacnet.npdu.control.contains": "BACnet APDU message (0)",
  "bacnet.npdu.control.dst_spec": "DNET, DLEN, Hop Count present (1)",
  "bacnet.npdu.control.prio": "Normal message",
  "bacnet.npdu.control.reply_expected": "Yes (1)",
  "bacnet.npdu.control.src_spec": "SNET, SLEN, SADR absent (0)",
  "bacnet.npdu.version": "0x0001",
  "EventTime": "2021-08-02T13:39:36.318342-07:00",
  "EventReceivedTime": "2021-08-02T13:39:38.714328-07:00",
  "SourceModuleName": "pcap",
  "SourceModuleType": "im_pcap"
}

BACnet acknowledgment

{
  "bacnet.apdu.bacnet_complexack.more_segments_follow": "false",
  "bacnet.apdu.bacnet_complexack.original_invoke_id": "159",
  "bacnet.apdu.bacnet_complexack.segmented": "false",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.object_identifier.instance_number": "0",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.object_identifier.type": "analog-input (0)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.1.property_identifier": "present-value (85)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.2.records.0": "Opening Tag (3)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.2.records.1": "963000.125000",
  "bacnet.apdu.bacnet_complexack.service_ack.records.2.records.2": "Closing Tag (3)",
  "bacnet.apdu.bacnet_complexack.service_choice": "Read Property (12)",
  "bacnet.apdu.pdu_type": "BACnet-Complex-ACK-PDU (0x03)",
  "bacnet.bvlc.function": "Original-Unicast-NPDU (0x0A)",
  "bacnet.bvlc.length": "32",
  "bacnet.bvlc.type": "BACnet/IP (Annex J) (0x81)",
  "bacnet.npdu.control": "0x0008",
  "bacnet.npdu.control.contains": "BACnet APDU message (0)",
  "bacnet.npdu.control.dst_spec": "DNET, DLEN, DADR, Hop Count absent (0)",
  "bacnet.npdu.control.prio": "Normal message",
  "bacnet.npdu.control.reply_expected": "No (0)",
  "bacnet.npdu.control.src_spec": "SNET, SLEN, SADR present (1)",
  "bacnet.npdu.version": "0x0001",
  "EventTime": "2021-08-02T13:39:36.647469-07:00",
  "EventReceivedTime": "2021-08-02T13:39:38.777825-07:00",
  "SourceModuleName": "pcap",
  "SourceModuleType": "im_pcap"
}

DNP3

DNP3 is a protocol for communication between process automation components used by public utilities for supplying electricity and water. This protocol is based on the Enhanced Performance Architecture (EPA) model, simplified model of ISO/OSI to specify the data link layer, the application layer, and the transport pseudo-layer. NXLog can be configured to capture packets transmitted over DNP3.

Example 15. Capturing SCADA DNP3 packets

This configuration uses the im_pcap module for passive network monitoring. The Dev directive of this module specifies the network interface it will use for capturing packets. The Protocol group directive denotes the dnp3 protocol for capturing. The Exec block of im_pcap calls the to_json() procedure of the xm_json module to convert messages to JSON.

nxlog.conf

<Extension _json>
    Module        xm_json
</Extension>

<Input pcap_dnp>
    Module        im_pcap
    # Name of a network device/interface
    Dev           \Device\NPF_{E8451A5D-5676-4BEF-8ED5-68911C97D953}
    <Protocol>
        # Protocol type
        Type      dnp3
    </Protocol>
    # Conversion to JSON
    Exec          to_json();
</Input>

DNP3 read data objects request

{
  "dnp3.application_layer.control.con": "0",
  "dnp3.application_layer.control.fin": "1",
  "dnp3.application_layer.control.fir": "1",
  "dnp3.application_layer.control.sequence": "7",
  "dnp3.application_layer.control.uns": "0",
  "dnp3.application_layer.function_code": "Read",
  "dnp3.application_layer.object0.count": "0",
  "dnp3.application_layer.object0.group": "60",
  "dnp3.application_layer.object0.name": "Class objects - Class 0 data",
  "dnp3.application_layer.object0.variation": "1",
  "dnp3.application_layer.object1.count": "0",
  "dnp3.application_layer.object1.group": "60",
  "dnp3.application_layer.object1.name": "Class objects - Class 1 data",
  "dnp3.application_layer.object1.variation": "2",
  "dnp3.application_layer.object2.count": "0",
  "dnp3.application_layer.object2.group": "60",
  "dnp3.application_layer.object2.name": "Class objects - Class 2 data",
  "dnp3.application_layer.object2.variation": "3",
  "dnp3.application_layer.object3.count": "0",
  "dnp3.application_layer.object3.group": "60",
  "dnp3.application_layer.object3.name": "Class objects - Class 3 data",
  "dnp3.application_layer.object3.variation": "4",
  "dnp3.data_layer.control": "0xC4",
  "dnp3.data_layer.control.dir": "1",
  "dnp3.data_layer.control.fcb": "0",
  "dnp3.data_layer.control.fcv": "0",
  "dnp3.data_layer.control.function_code": "Unconfirmed User Data",
  "dnp3.data_layer.control.prm": "1",
  "dnp3.data_layer.destination": "1",
  "dnp3.data_layer.length": "20",
  "dnp3.data_layer.source": "2",
  "dnp3.data_layer.start_bytes": "0x0564",
  "dnp3.transport.fin": "1",
  "dnp3.transport.fir": "1",
  "dnp3.transport.sequence": "31",
  "EventTime": "2021-08-03T12:42:00.129992-07:00",
  "EventReceivedTime": "2021-08-03T12:42:00.860759-07:00",
  "SourceModuleName": "pcap",
  "SourceModuleType": "im_pcap"
}

DNP3 data objects response

{
  "dnp3.application_layer.control.con": "0",
  "dnp3.application_layer.control.fin": "1",
  "dnp3.application_layer.control.fir": "1",
  "dnp3.application_layer.control.sequence": "7",
  "dnp3.application_layer.control.uns": "0",
  "dnp3.application_layer.function_code": "Response",
  "dnp3.application_layer.internal_indications.already_executing": "0",
  "dnp3.application_layer.internal_indications.broadcast": "0",
  "dnp3.application_layer.internal_indications.class1_events": "0",
  "dnp3.application_layer.internal_indications.class2_events": "0",
  "dnp3.application_layer.internal_indications.class3_events": "0",
  "dnp3.application_layer.internal_indications.config_corrupt": "0",
  "dnp3.application_layer.internal_indications.device_restart": "0",
  "dnp3.application_layer.internal_indications.device_trouble": "0",
  "dnp3.application_layer.internal_indications.events_buffer_overflow": "0",
  "dnp3.application_layer.internal_indications.local_control": "0",
  "dnp3.application_layer.internal_indications.need_time": "1",
  "dnp3.application_layer.internal_indications.no_func_code_support": "0",
  "dnp3.application_layer.internal_indications.object_unknown": "0",
  "dnp3.application_layer.internal_indications.parameter_error": "0",
  "dnp3.application_layer.internal_indications.reserved": "0 (expected 0)",
  "dnp3.application_layer.object0.count": "1",
  "dnp3.application_layer.object0.group": "30",
  "dnp3.application_layer.object0.name": "Analog input - single-precision, floating-point with flag",
  "dnp3.application_layer.object0.point0.flags": "[ONLINE]",
  "dnp3.application_layer.object0.point0.value": "2.540000",
  "dnp3.application_layer.object0.variation": "5",
  "dnp3.data_layer.control": "0x44",
  "dnp3.data_layer.control.dir": "0",
  "dnp3.data_layer.control.fcb": "0",
  "dnp3.data_layer.control.fcv": "0",
  "dnp3.data_layer.control.function_code": "Unconfirmed User Data",
  "dnp3.data_layer.control.prm": "1",
  "dnp3.data_layer.destination": "2",
  "dnp3.data_layer.length": "20",
  "dnp3.data_layer.source": "1",
  "dnp3.data_layer.start_bytes": "0x0564",
  "dnp3.transport.fin": "1",
  "dnp3.transport.fir": "1",
  "dnp3.transport.sequence": "40",
  "EventTime": "2021-08-03T12:42:00.133436-07:00",
  "EventReceivedTime": "2021-08-03T12:42:00.861419-07:00",
  "SourceModuleName": "pcap",
  "SourceModuleType": "im_pcap"
}

IEC 60870-5-104

IEC 60870–5‑104 is an extension of the IEC 60870-5-101 protocol and combines transport, network, link, and physical layers to enable communication between control stations and substations using TCP/IP. NXLog can be configured to monitor network traffic that uses this protocol.

Example 16. Capturing SCADA IEC 60870-5-104 packets

To passively monitor network traffic, NXLog uses the im_pcap module. The Dev directive of this module specifies the network interface it will use for capturing packets. The Protocol group directive is defined twice so that the iec104apci and iec104asdu protocols can be monitored simultaneously. The Exec block of this module calls the to_json() procedure of the xm_json module to convert messages to JSON.

nxlog.conf

<Extension _json>
    Module        xm_json
</Extension>

<Input pcap_iec>
    Module        im_pcap
    # Name of a network device/interface
    Dev           \Device\NPF_{E8451A5D-5676-4BEF-8ED5-68911C97D953}
    <Protocol>
        # Protocol types
        Type      iec104apci
    </Protocol>
    <Protocol>
        Type      iec104asdu
    </Protocol>
    # Converting to JSON
    Exec          to_json();
</Input>

IEC 60870-5-104 request sample

{
  "iec104.apci.receive_sequence_number": "24589",
  "iec104.apci.send_sequence_number": "12552",
  "iec104.apci.type": "Information (I)",
  "iec104.asdu.data": {
    "io": [
      {
        "ioa": 1020
      }
    ],
    "ios": 1
  },
  "iec104.asdu.dui.cause_of_transmission": "Request or requested (5)",
  "iec104.asdu.dui.coa": "1",
  "iec104.asdu.dui.num_records": "1",
  "iec104.asdu.dui.org": "0",
  "iec104.asdu.dui.pn": "0",
  "iec104.asdu.dui.sq": "FALSE",
  "iec104.asdu.dui.test_bit": "0",
  "iec104.asdu.dui.type": "C_RD_NA_1",
  "EventTime": "2021-07-19T13:07:54.073324-07:00",
  "EventReceivedTime": "2021-07-19T13:07:54.886786-07:00",
  "SourceModuleName": "pcap",
  "SourceModuleType": "im_pcap"
}

IEC 60870-5-104 response sample

{
  "iec104.apci.receive_sequence_number": "12553",
  "iec104.apci.send_sequence_number": "24589",
  "iec104.apci.type": "Information (I)",
  "iec104.asdu.data": {
    "io": [
      {
        "ioa": 1020,
        "ie": [
          {
            "type": "R32",
            "value": 10243
          },
          {
            "type": "QDS",
            "invalid": false,
            "not-topical": false,
            "substituted": false,
            "blocked": false,
            "overflow": false
          },
          {
            "type": "CP56Time2A",
            "milliseconds": 54006,
            "minutes": 7,
            "hours": 13,
            "day-of-week": 0,
            "day-of-month": 19,
            "month": 7,
            "year": 21
          }
        ],
        "ies": 3
      }
    ],
    "ios": 1
  },
  "iec104.asdu.dui.cause_of_transmission": "Request or requested (5)",
  "iec104.asdu.dui.coa": "1",
  "iec104.asdu.dui.num_records": "1",
  "iec104.asdu.dui.org": "0",
  "iec104.asdu.dui.pn": "0",
  "iec104.asdu.dui.sq": "FALSE",
  "iec104.asdu.dui.test_bit": "0",
  "iec104.asdu.dui.type": "M_ME_TF_1",
  "EventTime": "2021-07-19T13:07:54.078755-07:00",
  "EventReceivedTime": "2021-07-19T13:07:54.887761-07:00",
  "SourceModuleName": "pcap",
  "SourceModuleType": "im_pcap"
}

Disclaimer

While we endeavor to keep the information in this topic up to date and correct, NXLog makes no representations or warranties of any kind, express or implied about the completeness, accuracy, reliability, suitability, or availability of the content represented here. We update our screenshots and instructions on a best-effort basis.

The accurateness of the content was tested and proved to be working in our lab environment at the time of the last revision with the following software versions:

General Electric CIMPLICITY 11.0
NXLog 5.3.7166 running on Microsoft Windows Server 2016 Standard

Last revision: 21 July 2021