ABB MicroSCADA Pro SYS600

MicroSCADA Pro SYS 600 is a modular, programmable and scalable supervisory control and data acquisition (SCADA) system designed mainly for network control applications and substation automation.

It is mainly used in the electric power industry but can supervise and control heat and water/wastewater applications, industrial processes, traffic, etc.

Types of logs

NXLog Agent can collect events from the following sources:

Logs from Windows Event Log
File-based logs
Passive network monitoring

Logs from Windows Event Log

This table includes the MicroSCADA Pro SYS600 services that generate Windows Event Log entries, their display names, source names, and executables.

Table 1. MicroSCADA Pro SYS600 services
Service Name/Display Name	Source	Path to Executable	Description
ABB Authentication Service	ABB Ldap Service	`C:\Program Files (x86)\ABB\ABB Authentication Service 3.0.6291.0\OpenLDAP\OpenLDAP_SetupService.exe`	OpenLDAP_SetupService
ABB Authentication Service	Slapd	`C:\Program Files (x86)\ABB\ABB Authentication Service 3.0.6291.0\OpenLDAP\slapd.exe`	OpenLDAP-2.4.39
MicroSCADA	ProcessStarter/ MicroSCADA	`C:\Program Files (x86)\ABB\MicroSCADA Pro\Services\ProcessStarterService.exe`	ProcessStarterService

Log data can be collected using the Event IDs of log entries or the event source names.

The following table lists several Event IDs related to MicroSCADA Pro SYS600.

Table 2. Event IDs and their description
Event ID	Source	Message
`0`	ABB Ldap Service	Service started successfully Service has successfully shut down
`0`	Slapd	Slapd started Slapd Process stopped Slapd Process is not running
`1`	MicroSCADA	The MicroSCADA service has started
`12`	MicroSCADA	The MicroSCADA service has stopped
`61`	MicroSCADA	The MicroSCADA Service is performing a forced shut down
`12000`	MicroSCADA	MicroSCADA AMQP Broker has stopped working (dd/mm/yyyy hh:mm:ss AM Windows error code: 0)
`12001`	MicroSCADA	MicroSCADA logging service has stopped working (dd/mm/yyyy hh:mm:ss AM Windows error code: 0)
`12009`	MigrationTool	MicroSCADA Migration Tool has stopped working (dd/mm/yyyy hh:mm:ss AM Windows error code: 0)

Example 1. Processing MicroSCADA Pro SYS600 logs based on Event ID

Event sample

Log Name:      Application
Source:        MicroSCADA
Date:          8/23/2021 7:47:51 AM
Event ID:      1
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      WIN-5RU7GP5MI4V
Description:
The MicroSCADA service has been started
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="MicroSCADA" />
    <EventID Qualifiers="0">1</EventID>
    <Level>4</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2021-08-23T04:47:51.510311900Z" />
    <EventRecordID>6844</EventRecordID>
    <Channel>Application</Channel>
    <Computer>WIN-5RU7GP5MI4V</Computer>
    <Security />
  </System>
  <EventData>
    <Data>The MicroSCADA service has been started</Data>
  </EventData>
</Event>

This NXLog Agent configuration uses the im_msvistalog module to read data from the Windows Event Log Application channel. It also lists all Event IDs related to MicroSCADA Pro SYS600.

The Exec directive uses the to_json() procedure to format the output as JSON.

nxlog.conf

<Extension json>
    Module    xm_json
</Extension>

<Input eventlog>
    Module    im_msvistalog
    #XML query to read Windows Event Logs based on EventID
    <QueryXML>
        <QueryList>
            <Query Id="0" Path="Application">
                <Select Path="Application">
                    *[System[(EventID=0 or EventID=1)]]
                </Select>
            </Query>
        </QueryList>
    </QueryXML>
    # Conversion to JSON
    Exec      to_json();
</Input>

Output sample in JSON

{
  "EventTime": "2021-08-23T07:47:51.510311+03:00",
  "Hostname": "WIN-5RU7GP5MI4V",
  "Keywords": "36028797018963968",
  "EventType": "INFO",
  "SeverityValue": 2,
  "Severity": "INFO",
  "EventID": 1,
  "SourceName": "MicroSCADA",
  "TaskValue": 0,
  "RecordNumber": 6844,
  "ExecutionProcessID": 0,
  "ExecutionThreadID": 0,
  "Channel": "Application",
  "Message": "The MicroSCADA service has been started",
  "Opcode": "Info",
  "Data": "The MicroSCADA service has been started",
  "EventReceivedTime": "2021-08-23T07:47:52.572990+03:00",
  "SourceModuleName": "eventlog",
  "SourceModuleType": "im_msvistalog"
}

As previously mentioned, NXLog Agent can filter Windows Event Log entries by source name.

Example 2. Processing MicroSCADA Pro SYS600 logs based on source name

MicroSCADA Pro SYS600 collected this sample event from the ABB Ldap Service log source.

Event sample

Log Name:      Application
Source:        ABB Ldap Service
Date:          8/23/2021 7:48:19 AM
Event ID:      0
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      WIN-5RU7GP5MI4V
Description:
Service stopped successfully.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="ABB Ldap Service" />
    <EventID Qualifiers="0">0</EventID>
    <Level>4</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2021-08-23T04:48:19.697675000Z" />
    <EventRecordID>6851</EventRecordID>
    <Channel>Application</Channel>
    <Computer>WIN-5RU7GP5MI4V</Computer>
    <Security />
  </System>
  <EventData>
    <Data>Service stopped successfully.</Data>
  </EventData>
</Event>

This NXLog Agent configuration uses the im_msvistalog module to read Windows Event Log entries. The QueryXML directive of this module specifies the Application channel and the following sources to filter data by:

MicroSCADA
ProcessStarter
ABB Ldap Service
Slapd

nxlog.conf

<Extension json>
    Module    xm_json
</Extension>

<Input eventlog>
    Module    im_msvistalog
    # XML query to read Windows Event Log data
    <QueryXML>
        <QueryList>
            <Query Id="0" Path="Application">
                <Select Path="Application">*
                    [System[Provider[@Name='MicroSCADA' or
                        @Name='ProcessStarter' or
                        @Name='ABB Ldap Service' or
                        @Name='Slapd']]]
                </Select>
            </Query>
        </QueryList>
    </QueryXML>
    # Conversion to JSON
    Exec      to_json();
</Input>

The following output sample shows the ABB Ldap Service event record after NXLog Agent processed and converted it to JSON.

Output sample in JSON

{
  "EventTime": "2021-08-23T08:22:13.463495+03:00",
  "Hostname": "WIN-5RU7GP5MI4V",
  "Keywords": "36028797018963968",
  "EventType": "INFO",
  "SeverityValue": 2,
  "Severity": "INFO",
  "EventID": 0,
  "SourceName": "ABB Ldap Service",
  "TaskValue": 0,
  "RecordNumber": 6866,
  "ExecutionProcessID": 0,
  "ExecutionThreadID": 0,
  "Channel": "Application",
  "Message": "Service started successfully.",
  "Opcode": "Info",
  "Data": "Service started successfully.",
  "EventReceivedTime": "2021-08-23T08:22:14.245403+03:00",
  "SourceModuleName": "eventlog",
  "SourceModuleType": "im_msvistalog"
}

File-based logs

MicroSCADA Pro SYS600 produces several file-based logs that NXLog Agent can collect and process. This table briefly describes these logs and their locations.

Table 3. File-based logs
Log type	Location	Description
System log	`…\sc\sys\active\sys_\SYS_LOG.CSV`	SYS600 base system event log
System message log	`…\sc\sys\active\sys_\sys_msglog.txt`	Messages that occur during the SYS600 startup and operation
Migration Tool output log	`…\sc\sys\active\sys_\MigrationToolOutput.log`	A summary of error and warning messages of process displays and symbols migration from Monitor Pro to Monitor Pro+
Browser (CEF) log	`…\sc\sys\active\sys_\cef_log.log`
User Activity log	`…\sc\apl\main\apl_\APL_UAL2020_UAL.txt …\sc\apl\main\apl_\APL_UAL2020_UAL.csv`	User activity related events
CET (Communication Engineering Tool) for IEC 61850 OPC Server log	`…\PCMDatabases\LogFiles\Common_MS-CET_IEC61850_Log_yyyymmdd.txt`	CET for IEC 61850 OPC Server activity events

System log

Whenever a problem occurs, the SYS 600 base system displays the appropriate message in the Notification Window and writes the related events to the system log file SYS_LOG.CSV located in the …\sc\sys\active\sys_\ directory.

The system log file SYS_LOG.CSV contains system events such as:

Configuration errors in the standard base system configuration file SYS_BASCON.COM
External programs failures
Report and printer spool processes memory pool problems
Any error messages related to the start-up or configuration of the PC-NET process communication unit

System log events can be either single-line or multiline.

The following event samples are from the SYS_LOG.CSV file.

Multiline input sample

2,"2020-07-10T12:09:52.519Z","Base System",,"SYS ","INFO","SYS_600  9.4 FP2  PRODUCTION  2016-06-09 19:30:00
Customer: MICROSCADA PRO ENGINEERING CENTER
Site: ONLY FOR DEMO USE
System ID: 3011407, Key: 1-BGHN-DAOG-OANK-EDOF
Base System:  9.4 FP2 HF2  PRODUCTION  2017-07-17","","","","","","","",""

Single-line input sample

4,"2020-07-10T12:09:53.297Z","Display Migration",,"MIGR","INFO","Started display migration (System symbols)...","","MicroSCADA","","","WIN-5RU7GP5MI4V","","",""

In this NXLog Agent configuration, constant SYS_REGEX stores the regular expression for parsing specific fields. The constant LOG_PATH defines the log file’s absolute path to the directory.

Multiple lines should be processed by the xm_multiline module. The HeaderLine directive of this module defines the regular expression used for detecting the start of each new event entry.

The first Exec directive block of the input module replaces the \r (CR) and \n (LF) characters with a single space to transform multiline events records to single-line events. The last string replacement removes extraneous whitespace.

The second Exec directive block of the input module takes the second string parsed using the SYS_REGEX regular expression, a timestamp, and uses the parsedate() function to convert it to a Datetime data type for the $EventTime field.

Each event record is then converted to JSON format using the to_json() procedure of the xm_json module.

nxlog.conf

# A regular expression defined as a constant to read the content of the logs
define SYS_REGEX  /(?x)^(?<Number>\d+)\,\
                  \"(\d+.\d+.\d+\w\d+.\d+.\d+.\d+\w+)\"\,\
                  \"(?<Source>.*?)\"\,(?<Application>.*?)\,\
                  \"(?<Category>.*?)\"\,\"(?<Severity>.*?)\"\,\
                  \"(?<Message>.*?)\"\,\"(?<Username>.*?)\"\,\
                  \"(?<OS_Username>.*?)\"\,\"(?<Client_IP>.*?)\"\,\
                  \"(?<Service_IP>.*?)\"\,\"(?<Computer_Name>.*?)\"\,\
                  \"(?<DMS_Device>.*?)\"\,\"(?<DMS_Region>.*?)\"\,\
                  \"(?<DMS_Area>.*?)\"/

# Part of the log path defined as a constant
define LOG_PATH   C:\sc\sys\active\sys_

<Extension json>
    Module        xm_json
</Extension>

<Extension multiline_sys>
    Module        xm_multiline
    # Regular expression to look for the header of the message
    HeaderLine    /\d+\,\"\d+\-\d+\-\d+\w\d+\:\d+\:\d+.\d+\w\"\,/
</Extension>

<Input from_file>
    Module        im_file
    File          '%LOG_PATH%\SYS_LOG.CSV'
    InputType     multiline_sys
    <Exec>
        # Replaces unwanted characters
        $raw_event =~ s/\\n/ /g;
        $raw_event =~ s/\\r/ /g;
        $raw_event =~ s/\s{2,}/ /g;
    </Exec>

    <Exec>
        # Matches the events with a regular expression
        if $raw_event =~ %SYS_REGEX%
        {
            # Creates the timestamp
            $EventTime = parsedate($2);
            # Formats the result as JSON
            to_json();
        }
        # Discard event if it doesn't match a/the regular expression
        else drop();
    </Exec>
</Input>

Below is the processed record in JSON format.

Output sample in JSON

{
  "EventReceivedTime": "2021-08-21T22:57:41.813030+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Application": "",
  "Category": "SYS ",
  "Client_IP": "",
  "Computer_Name": "",
  "DMS_Area": "",
  "DMS_Device": "",
  "DMS_Region": "",
  "Message": "SYS_600 9.4 FP2 PRODUCTION 2016-06-09 19:30:00 Customer: MICROSCADA PRO ENGINEERING CENTER Site: ONLY FOR DEMO USE System ID: 3011407, Key: 1-BGHN-DAOG-OANK-EDOF Base System: 9.4 FP2 HF2 PRODUCTION 2017-07-17",
  "Number": "2",
  "OS_Username": "",
  "Service_IP": "",
  "Severity": "INFO",
  "Source": "Base System",
  "Username": "",
  "EventTime": "2020-07-10T15:09:52.519000+03:00"
}

System message log

The sys_msglog.txt file contains MicroSCADA Pro SYS600 start-up and operation messages. The path to this log file contains …\sc\sys\active\sys_\.

The sample below represents a typical event from the sys_msglog.txt file. Three fields can be parsed from each log record:

Timestamp
Severity
Message

Input sample

20-06-28 12:40:32 INFO Fonts converted succesfully for VS_LOCAL_1280X960_MAIN_WIN-5RU7GP5MI4V <LOCAL>

This NXLog Agent configuration uses the SYS_MSG_REGEX constant, which defines the regular expression used for parsing specific fields from each event record. The LOG_PATH constant stores the absolute path to the log file.

The Exec block of the im_file input module compares each message to SYS_MSG_REGEX. For events that match this regular expression, parsed fields are created according to the named capturing groups defined in SYS_MSG_REGEX. Each event record is then converted to JSON using the to_json() procedure of the xm_json module. Events that do not match SYS_MSG_REGEX are discarded using the drop() procedure.

nxlog.conf

# A regular expression defined as a constant to read the content of the logs
define SYS_MSG_REGEX  /(?x)^(\d+.\d+.\d+.\d+.\d+.\d+)?\s*\
                      (?<Severity>[A-Z]+)?\s+(?<Message>.*)/

# Part of the log path defined as a constant
define LOG_PATH       C:\sc\sys\active\sys_

<Extension json>
    Module    xm_json
</Extension>

<Input from_file>
    Module    im_file
    File      '%LOG_PATH%\sys_msglog.txt'
    <Exec>
        # Matches the events with a regular expression
        if $raw_event =~ %SYS_MSG_REGEX%
        {
            # Creates the timestamp
            $EventTime = strptime($1, "%y-%m-%d %T");
            # Formats the result as JSON
            to_json();
        }
        # Discard event if it doesn't match a/the regular expression
        else drop();
    </Exec>
</Input>

Below is the processed record in JSON format.

Output sample in JSON

{
  "EventReceivedTime": "2021-08-23T17:08:27.935577+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Message": "Fonts converted succesfully for VS_LOCAL_1280X960_MAIN_WIN-5RU7GP5MI4V <LOCAL>",
  "Severity": "INFO",
  "EventTime": "2020-06-28T12:40:32.000000+03:00"
}

Migration Tool output log

To use displays and symbols available in Monitor Pro, they must be converted to Monitor Pro+ using a migration tool.

A list of possible errors and warnings from the application and symbol conversion status are logged to the MigrationToolOutput.log file located in the …\sc\sys\active\sys_\ directory.

Below is an input sample of the Migration Tool output log.

Input sample

6/28/2020 11:57:45 AM: [Info] Running process: MigrationTool.exe -log console -s -v Info

This NXLog Agent configuration defines two constants. The MOT_REGEX constant stores the regular expression used for parsing specific fields from each event record. The LOG_PATH constant defines the absolute path to the log file.

Each event entry is matched to the MOT_REGEX in the Exec block of the im_file module. All matched fields are created along with their values according to the named capturing groups feature. The original timestamp is created using the parsedate() function and assigned to the EventTime field.

Messages are discarded using the drop() procedure if the match fails. The parsed fields are subsequently converted to JSON using the to_json() procedure of the xm_json module.

nxlog.conf

# A regular expression defined as a constant to read the content of the logs
define MTO_REGEX  /(?x)^(\d+.\d+.\d+.\d+.\d+.\d+.\w+)\:\s*\
                  \[(?<Severity>.*?)\]\s*(?<Message>.*)/

# Part of the log path defined as a constant
define LOG_PATH   C:\sc\sys\active\sys_

<Extension json>
    Module    xm_json
</Extension>

<Input from_file>
    Module    im_file
    File      '%LOG_PATH%\MigrationToolOutput.log'
    <Exec>
        # Matches the events with a regular expression
        if $raw_event =~ %MTO_REGEX%
        {
            # Creates the timestamp
            $EventTime = parsedate($1);
            # Formats the result as JSON
            to_json();
        }
        # Discard event if it doesn't match a/the regular expression
        else drop();
    </Exec>
</Input>

Below is the processed record in JSON format.

Output sample in JSON

{
  "EventReceivedTime": "2021-08-23T17:23:18.107198+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Message": "Running process: MigrationTool.exe -log console -s -v Info",
  "Severity": "Info",
  "EventTime": "2020-06-28T11:57:45.000000+03:00"
}

Browser (CEF) log

The Browser (CEF) logging feature is a part of the User Activity Log used for debugging purposes. Messages are added to the cef_log.log whenever the Monitor Pro+ user interface starts.

The following event sample is from cef_log.log.

Input sample

[0412/230908:INFO:CONSOLE(730)] "Frame: doLogout", source: http://localhost:8080/js/framewindow.js (730)

This NXLog Agent configuration uses the regular expression defined as constant CEF_REGEX used for parsing specific fields from each event record. The constant LOG_PATH stores the absolute path to the log file.

The im_file module reads each event and uses CEF_REGEX to test for a match. If an event matches, the parsedate() function converts values of the first two captured groups to a Datetime value, which is then assigned to the $EventTime field. All other fields are created along with their values according to the named capturing groups of CEF_REGEX.

The parsed fields are converted to JSON format using the to_json() procedure of the xm_json module. Messages that do not match CEF_REGEX are discarded using the drop() procedure.

nxlog.conf

# A regular expression defined as a constant to read the content of the logs
define CEF_REGEX  /(?x)^\[(\d+)\/(\d+)\:(?<Severity>.*?)\:(?<Source>.*?)\]\
                  \s*(?<Message>.*)/

# Part of the log path defined as a constant
define LOG_PATH   C:\sc\sys\active\sys_

<Extension json>
    Module    xm_json
</Extension>

<Input from_file>
    Module    im_file
    File      '%LOG_PATH%\cef_log.log'
    <Exec>
        # Matches the events with a regular expression
        if $raw_event =~ %CEF_REGEX%
        {
            # Creates the timestamp
            $EventTime = strptime($1 + year(now()) + $2, "%m%d%Y%H%M%S");
            # Formats the result as JSON
            to_json();
        }
        # Discard event if it doesn't match a/the regular expression
        else drop();
    </Exec>
</Input>

Below is the processed record in JSON format.

Output sample in JSON

{
  "EventReceivedTime": "2020-04-12T23:09:09.340130+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Message": "\"Frame: doLogout\", source: http://localhost:8080/js/framewindow.js (730)",
  "Severity": "INFO",
  "Source": "CONSOLE(730)",
  "EventTime": "2020-04-12T23:09:08.000000+03:00"
}

User Activity log

MicroSCADA Pro SYS600 logs user activity-related events to a local file named APL_UALyyyy.UAL, where yyyy stands for the year. This file can be viewed using the User Activity Log list included in SYS600 Monitor Pro.

User Activity Log can be exported to a .csv or a .txt file:

APL_UAL2020_UAL.csv
APL_UAL2020_UAL.txt

User Activity log generates events comprised of the following fields:

SOE (sequence number of event)
Event ID
Time
Event Text
User
Severity
Source
Sender IP
Extra Info

The sample below shows the event entry taken from APL_UAL2020_UAL.txt.

Input sample

26,1110,2020-07-10 15:13:42.223,Log-in successful,Administrator,0,MAIN,,Administrator,WIN-5RU7GP5MI4V(127.0.0.1)

The original log file is UCS-2LE encoded and can’t be processed correctly without additional conversion. The xm_charconv extension module facilitates reading the log file using the correct encoding and then converts it to UTF-8. The LineReader directive of this module sets the encoding type to be used. The xm_charconv extension instance references the InputType directive of the from_file input instance by the name _charconv.

This NXLog Agent configuration defines two constants. UAL_REGEX stores the regular expression used for parsing specific fields from each event record, while the LOG_PATH is the absolute path to the log file.

The Exec block of the im_file module compares each event entry with UAL_REGEX. If an events matches, the parsedate() function converts the captured timestamp to a Datetime value and assigns it to the $EventTime field. The remaining fields are captured and assigned to fields named according to the named capturing groups defined in UAL_REGEX. If an event does not match UAL_REGEX, entries are discarded using the drop() procedure.

The final result is converted to JSON format using the to_json() procedure of the xm_json module.

nxlog.conf

# A regular expression defined as a constant to read the content of the logs
define UAL_REGEX  /(?x)^(?<Number>\d+)\,(?<Event_ID>\d+)\,\
                  (\d+\-\d+\-\d+\s+\d+\:\d+\:\d+\.\d+)\,\
                  (?<Event_Text>.*?)\,(?<User>.*?)\,(?<Severity>\d+)\,\
                  (?<Source>.*?)\,(?<Sender_IP>.*?)\,(?<ExtraInfo>.*)/

# Part of the log path defined as a constant
define LOG_PATH   C:\sc\apl\main\apl_

<Extension json>
    Module        xm_json
</Extension>

<Extension _charconv>
    Module        xm_charconv
    # Conversion from UCS-2LE to UTF-8
    LineReader    UCS-2LE
</Extension>

<Input from_file>
    Module        im_file
    File          '%LOG_PATH%\APL_UAL2020_UAL.txt'
    InputType     _charconv
    <Exec>
        # Matches the events with a regular expression
        if $raw_event =~ %UAL_REGEX%
        {
            # Creates the timestamp
            $EventTime = parsedate($3);
            # Formats the result as JSON
            to_json();
        }
        # Discard event if it doesn't match a/the regular expression
        else drop();
    </Exec>
</Input>

Below is the processed record in JSON format.

Output sample in JSON

{
  "EventReceivedTime": "2021-08-24T17:05:23.655971+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Event_ID": "1110",
  "Event_Text": "Log-in successful",
  "ExtraInfo": "Administrator,WIN-5RU7GP5MI4V(127.0.0.1)",
  "Number": "26",
  "Sender_IP": "",
  "Severity": "0",
  "Source": "MAIN",
  "User": "Administrator",
  "EventTime": "2020-07-10T15:13:42.223000+03:00"
}

CET (Communication Engineering Tool) for IEC 61850 OPC Server log

Communication Engineering Tool for IEC 61850 OPC Server is a MicroSCADA Pro SYS600 component used for creating and configuring a hierarchical model of a substation or a system for the IEC 61850 OPC Server.

CET for IEC 61850 OPC Server activity events is stored in the Common_MS-CET_IEC61850_Log_yyyymmdd.txt log file in the …\PCMDatabases\LogFiles\ directory by default. However, the filename and file location can be changed in the configuration settings of CET for the IEC 61850 OPC Server.

The following event sample is taken from Common_MS-CET_IEC61850_Log_yyyymmdd.txt and presents the next set of fields:

Timestamp (string format: mm/dd/yyyy hh:mm:ss:fff AM/PM)
Severity
Source
Message

Input sample

7/2/2020  3:10:20.552 PM  |  Security warning  |  System  |  The 'Always trust IED security certificates' option was disabled.

The constants CET_REGEX and LOG_PATH in this NXLog Agent configuration specify the regular expression used for parsing specific fields and the absolute path to the log file, respectively.

The Exec block of the im_file module compares the event record to CET_REGEX. If an event matches the regular expression, the first three captured fields are concatenated to construct the timestamp, which is then converted to a Datetime value using the parsedate() function. The remaining captured fields are assigned to the field names defined by the named capturing groups of CET_REGEX.

Once the event record has been processed, it is converted to JSON format using the to_json() procedure of the xm_json module. Messages that do not match CET_REGEX are discarded using the drop() procedure.

nxlog.conf

# A regular expression defined as a constant to read the content of the logs
define CET_REGEX  /(?x)^(\d+.\d+.\d+.)\s*(\d+.\d+.\d+)\.\d+(.\w+)\s*\
                  [\|\;]\s*(?<Severity>.*?)\s*[\|\;]\s*(.*)\s*\
                  [\|\;]\s*(?<Source>.*?)\s*[\|\;]\s*(?<Message>.*)/

# Part of the log path defined as a constant
define LOG_PATH   C:\PCMDatabases\LogFiles

<Extension json>
    Module    xm_json
</Extension>

<Input from_file>
    Module    im_file
    File      '%LOG_PATH%\Common_MS-CET_IEC61850_Log_*.txt'
    <Exec>
        # Matches the events with a regular expression
        if $raw_event =~ %CET_REGEX%
        {
            # Creates the timestamp
            $EventTime = parsedate($1 + $2 + $3);
            # Formats the result as JSON
            to_json();
        }
        # Discard event if it doesn't match a/the regular expression
        else drop();
    </Exec>
</Input>

The sample below represents a processed event in JSON format.

Output sample in JSON

{
  "EventReceivedTime": "2021-08-24T23:19:52.610594+03:00",
  "SourceModuleName": "from_file",
  "SourceModuleType": "im_file",
  "Message": "The 'Always trust IED security certificates' option was disabled.",
  "Severity": "Security warning",
  "Source": "System",
  "EventTime": "2020-07-02T15:10:20.000000+03:00"
}

Passive network monitoring

MicroSCADA possesses rather unique communication capabilities that allows it to link the SYS600 system server with various process devices, such as IEDs, RTUs, PLCs, etc. Communication with process devices relies on multiple protocols, such as IEC 60870-5-104, IEC 61850, DNP3, and Modbus. MicroSCADA provides a communication software interface for each supported protocol according to its characteristics. The following communication units handle the software interface:

PC-NET - used with most of the SYS600 protocols
IEC 61850 OPC Server/External OPC DA Client - used for IEC 61850 protocols standard
OPC Server/External OPC DA Client - used to establish communication with slave devices managed by the OPC server

The traffic samples in the following examples are taken from the interaction between the External OPC DA Client and a third-party OPC server. These examples describe how to monitor network traffic passively using the following industrial protocols:

Modbus TCP
BACnet
DNP3
IEC 60870-5-104

Modbus TCP

Modbus is an asynchronous, byte-packaged communication protocol used for the master stations' interaction with Intelligent Electronic Devices (IEDs), Remote Terminal Units (RTUs), or Programmable Logic Controllers (PLCs). It supports a single master station and up to 247 RTUs on a multi-drop line and can use a serial line, Ethernet, or TCP/IP as a transport layer.

The Modbus TCP protocol is used in LAN and WAN networks to connect central stations and outstations.

Example 3. Capturing Modbus TCP packets

In the following NXLog Agent configuration, the Dev directive of the im_pcap module specifies the network interface for capturing packets. The Protocol group directive defines the exact protocol name for monitoring.

The Exec block calls the to_json() procedure of the xm_json module to convert messages to JSON format.

nxlog.conf

<Extension _json>
    Module    xm_json
</Extension>

<Input pcap>
    Module    im_pcap
    # Specifies the name of a network device/interface
    Dev       \Device\NPF_{159289BE-CE80-47DB-A659-2F8BF277C9C6}
    # Specifies the protocol type
    <Protocol>
        Type  modbus
    </Protocol>
    # Formats the result as JSON
    Exec      to_json();
</Input>

<Output file>
    Module    om_file
    File      "C:\output.txt"
</Output>

Modbus TCP query sample

{
  "modbus.function_code": "Read Holding Registers (03)",
  "modbus.length": "6",
  "modbus.prot_id": "0",
  "modbus.query.read_holding_regs.qty_of_regs": "3",
  "modbus.query.read_holding_regs.starting_address": "0",
  "modbus.trans_id": "39303",
  "modbus.unit_id": "1",
  "EventTime": "2022-02-16T12:53:51.926342+02:00",
  "EventReceivedTime": "2022-02-16T12:53:52.984280+02:00",
  "SourceModuleName": "pcap",
  "SourceModuleType": "im_pcap"
}

Modbus TCP response sample

{
  "modbus.function_code": "Read Holding Registers (03)",
  "modbus.length": "9",
  "modbus.prot_id": "0",
  "modbus.response.read_holding_regs.byte_count": "6",
  "modbus.response.read_holding_regs.registers": "2061, 2058, 2055",
  "modbus.trans_id": "39303",
  "modbus.unit_id": "1",
  "EventTime": "2022-02-16T12:53:51.947327+02:00",
  "EventReceivedTime": "2022-02-16T12:53:52.984280+02:00",
  "SourceModuleName": "pcap",
  "SourceModuleType": "im_pcap"
}

BACnet

BACnet (Building Automation Control Network) is a standardized communication protocol for building automation. BACnet is widely used in HVAC, lighting control, safety, and fire alarm technology applications. BACnet/IP uses UDP/IP for compatibility with an existing IP infrastructure.

Example 4. Capturing BACnet packets

NXLog Agent uses the im_pcap module for passive network monitoring and the Dev directive to specify the network interface. The Protocol group directive specifies the BACnet protocol for packet capturing.

The Exec directive is required for invoking a statement, such as a function or procedure. The to_json() procedure of the xm_json module to convert messages to JSON format.

nxlog.conf

<Extension _json>
    Module    xm_json
</Extension>

<Input pcap>
    Module    im_pcap
    # Specifies the name of a network device/interface
    Dev       \Device\NPF_{159289BE-CE80-47DB-A659-2F8BF277C9C6}
    # Specifies the protocol type
    <Protocol>
        Type  bacnet
    </Protocol>
    # Formats the result as JSON
    Exec      to_json();
</Input>

<Output file>
    Module    om_file
    File      "C:\output.txt"
</Output>

BACnet request

{
  "bacnet.apdu.bacnet_confirmed_request.invoke_id": "169",
  "bacnet.apdu.bacnet_confirmed_request.max_resp": "1476",
  "bacnet.apdu.bacnet_confirmed_request.max_segs": "Unspecified",
  "bacnet.apdu.bacnet_confirmed_request.more_segments_follow": "false",
  "bacnet.apdu.bacnet_confirmed_request.segmented": "false",
  "bacnet.apdu.bacnet_confirmed_request.segmented_accepted": "true",
  "bacnet.apdu.bacnet_confirmed_request.service_choice": "Read Property (12)",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.0.object_identifier.instance_number": "0",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.0.object_identifier.type": "analog-input (0)",
  "bacnet.apdu.bacnet_confirmed_request.service_request.records.1.property_identifier": "present-value (85)",
  "bacnet.apdu.pdu_type": "BACnet-Confirmed-Request-PDU (0x00)",
  "bacnet.bvlc.function": "Original-Unicast-NPDU (0x0A)",
  "bacnet.bvlc.length": "27",
  "bacnet.bvlc.type": "BACnet/IP (Annex J) (0x81)",
  "bacnet.npdu.control": "0x0024",
  "bacnet.npdu.control.contains": "BACnet APDU message (0)",
  "bacnet.npdu.control.dst_spec": "DNET, DLEN, Hop Count present (1)",
  "bacnet.npdu.control.prio": "Normal message",
  "bacnet.npdu.control.reply_expected": "Yes (1)",
  "bacnet.npdu.control.src_spec": "SNET, SLEN, SADR absent (0)",
  "bacnet.npdu.version": "0x0001",
  "EventTime": "2019-02-16T16:42:43.339909+02:00",
  "EventReceivedTime": "2019-02-16T16:42:44.070577+02:00",
  "SourceModuleName": "pcap",
  "SourceModuleType": "im_pcap"
}

BACnet acknowledgment

{
  "bacnet.apdu.bacnet_complexack.more_segments_follow": "false",
  "bacnet.apdu.bacnet_complexack.original_invoke_id": "169",
  "bacnet.apdu.bacnet_complexack.segmented": "false",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.object_identifier.instance_number": "0",
  "bacnet.apdu.bacnet_complexack.service_ack.records.0.object_identifier.type": "analog-input (0)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.1.property_identifier": "present-value (85)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.2.records.0": "Opening Tag (3)",
  "bacnet.apdu.bacnet_complexack.service_ack.records.2.records.1": "368999.875000",
  "bacnet.apdu.bacnet_complexack.service_ack.records.2.records.2": "Closing Tag (3)",
  "bacnet.apdu.bacnet_complexack.service_choice": "Read Property (12)",
  "bacnet.apdu.pdu_type": "BACnet-Complex-ACK-PDU (0x03)",
  "bacnet.bvlc.function": "Original-Unicast-NPDU (0x0A)",
  "bacnet.bvlc.length": "32",
  "bacnet.bvlc.type": "BACnet/IP (Annex J) (0x81)",
  "bacnet.npdu.control": "0x0008",
  "bacnet.npdu.control.contains": "BACnet APDU message (0)",
  "bacnet.npdu.control.dst_spec": "DNET, DLEN, DADR, Hop Count absent (0)",
  "bacnet.npdu.control.prio": "Normal message",
  "bacnet.npdu.control.reply_expected": "No (0)",
  "bacnet.npdu.control.src_spec": "SNET, SLEN, SADR present (1)",
  "bacnet.npdu.version": "0x0001",
  "EventTime": "2019-02-16T16:42:43.361376+02:00",
  "EventReceivedTime": "2019-02-16T16:42:44.070577+02:00",
  "SourceModuleName": "pcap",
  "SourceModuleType": "im_pcap"
}

DNP3

DNP3 is a protocol for interoperability between RTUs, IEDs, and SCADA systems and acts according to the master-slave architecture. A standard DNP3 network includes a single master station and multiple outstations. DNP3 is widely used in applications specific to the electric utility, oil & gas, and water/wastewater industries.

The main advantage of the DNP3 protocol is that outstations can store timestamped event data. In order to optimize network performance, DNP3 has the master poll the outstations at regular intervals to fetch any new events they have logged since the last poll, which it stores in its database. Outstations can also be configured to operate in unsolicited response mode. This mode allows outstations to send their event logs immediately to the master, without any need for them to wait for a poll request. This is useful when the master requires notification as soon as possible after changing parameter values.

Example 5. Capturing DNP3 packets

To passively monitor network traffic, NXLog Agent uses the im_pcap module. The Dev directive in the following configuration defines the network interface used for capturing the network traffic. The traffic to be captured can be filtered by a specific protocol of interest, in this case dnp3, which is defined using the Protocol group directive.

The Exec directive enables the to_json() procedure of the xm_json module to be invoked. It converts event records to JSON format.

nxlog.conf

<Extension _json>
    Module    xm_json
</Extension>

<Input pcap>
    Module    im_pcap
    # Specifies the name of a network device/interface
    Dev       \Device\NPF_{159289BE-CE80-47DB-A659-2F8BF277C9C6}
    # Specifies the protocol type
    <Protocol>
        Type  dnp3
    </Protocol>
    # Formats the result as JSON
    Exec      to_json();
</Input>

<Output file>
    Module    om_file
    File      "C:\output.txt"
</Output>

DNP3 read data objects request

{
  "dnp3.application_layer.control.con": "0",
  "dnp3.application_layer.control.fin": "1",
  "dnp3.application_layer.control.fir": "1",
  "dnp3.application_layer.control.sequence": "8",
  "dnp3.application_layer.control.uns": "0",
  "dnp3.application_layer.function_code": "Read",
  "dnp3.application_layer.object0.count": "0",
  "dnp3.application_layer.object0.group": "60",
  "dnp3.application_layer.object0.name": "Class objects - Class 1 data",
  "dnp3.application_layer.object0.variation": "2",
  "dnp3.application_layer.object1.count": "0",
  "dnp3.application_layer.object1.group": "60",
  "dnp3.application_layer.object1.name": "Class objects - Class 2 data",
  "dnp3.application_layer.object1.variation": "3",
  "dnp3.application_layer.object2.count": "0",
  "dnp3.application_layer.object2.group": "60",
  "dnp3.application_layer.object2.name": "Class objects - Class 3 data",
  "dnp3.application_layer.object2.variation": "4",
  "dnp3.application_layer.object3.count": "0",
  "dnp3.application_layer.object3.group": "60",
  "dnp3.application_layer.object3.name": "Class objects - Class 0 data",
  "dnp3.application_layer.object3.variation": "1",
  "dnp3.data_layer.control": "0xC4",
  "dnp3.data_layer.control.dir": "1",
  "dnp3.data_layer.control.fcb": "0",
  "dnp3.data_layer.control.fcv": "0",
  "dnp3.data_layer.control.function_code": "Unconfirmed User Data",
  "dnp3.data_layer.control.prm": "1",
  "dnp3.data_layer.destination": "4",
  "dnp3.data_layer.length": "20",
  "dnp3.data_layer.source": "3",
  "dnp3.data_layer.start_bytes": "0x0564",
  "dnp3.transport.fin": "1",
  "dnp3.transport.fir": "1",
  "dnp3.transport.sequence": "62",
  "EventTime": "2019-02-16T23:12:10.893268+02:00",
  "EventReceivedTime": "2019-02-16T23:12:11.531055+02:00",
  "SourceModuleName": "pcap",
  "SourceModuleType": "im_pcap"
}

DNP3 data objects response

{
  "dnp3.application_layer.control.con": "0",
  "dnp3.application_layer.control.fin": "1",
  "dnp3.application_layer.control.fir": "1",
  "dnp3.application_layer.control.sequence": "8",
  "dnp3.application_layer.control.uns": "0",
  "dnp3.application_layer.function_code": "Response",
  "dnp3.application_layer.internal_indications.already_executing": "0",
  "dnp3.application_layer.internal_indications.broadcast": "0",
  "dnp3.application_layer.internal_indications.class1_events": "0",
  "dnp3.application_layer.internal_indications.class2_events": "0",
  "dnp3.application_layer.internal_indications.class3_events": "0",
  "dnp3.application_layer.internal_indications.config_corrupt": "0",
  "dnp3.application_layer.internal_indications.device_restart": "0",
  "dnp3.application_layer.internal_indications.device_trouble": "0",
  "dnp3.application_layer.internal_indications.events_buffer_overflow": "0",
  "dnp3.application_layer.internal_indications.local_control": "0",
  "dnp3.application_layer.internal_indications.need_time": "0",
  "dnp3.application_layer.internal_indications.no_func_code_support": "0",
  "dnp3.application_layer.internal_indications.object_unknown": "0",
  "dnp3.application_layer.internal_indications.parameter_error": "0",
  "dnp3.application_layer.internal_indications.reserved": "0 (expected 0)",
  "dnp3.application_layer.object0.count": "1",
  "dnp3.application_layer.object0.group": "30",
  "dnp3.application_layer.object0.name": "Analog input - single-precision, floating-point with flag",
  "dnp3.application_layer.object0.point0.flags": "[ONLINE]",
  "dnp3.application_layer.object0.point0.value": "3594.477295",
  "dnp3.application_layer.object0.variation": "5",
  "dnp3.data_layer.control": "0x44",
  "dnp3.data_layer.control.dir": "0",
  "dnp3.data_layer.control.fcb": "0",
  "dnp3.data_layer.control.fcv": "0",
  "dnp3.data_layer.control.function_code": "Unconfirmed User Data",
  "dnp3.data_layer.control.prm": "1",
  "dnp3.data_layer.destination": "3",
  "dnp3.data_layer.length": "20",
  "dnp3.data_layer.source": "4",
  "dnp3.data_layer.start_bytes": "0x0564",
  "dnp3.transport.fin": "1",
  "dnp3.transport.fir": "1",
  "dnp3.transport.sequence": "61",
  "EventTime": "2019-02-16T23:12:10.893719+02:00",
  "EventReceivedTime": "2019-02-16T23:12:11.531055+02:00",
  "SourceModuleName": "pcap",
  "SourceModuleType": "im_pcap"
}

IEC 60870-5-104

IEC 60870-5-104 is an international standard used for telecontrol in electrical engineering and power system applications. It is widely used for monitoring and managing power utility devices. IEC 60870 5-104 extends the IEC 60870-5-101 protocol, sending important telecontrol messages over a standard TCP/IP network.

Example 6. Capturing IEC 60870-5-104 packets

To passively monitor network traffic, NXLog Agent uses the im_pcap module. To specify the desired protocol and network interface to be used for capturing network traffic, the Protocol and Dev directives of the im_pcap module are invoked, respectively.

The result is converted to JSON using the to_json() procedure of the xm_json module.

nxlog.conf

<Extension _json>
    Module    xm_json
</Extension>

<Input pcap>
    Module    im_pcap
    # Specifies the name of a network device/interface
    Dev       \Device\NPF_{159289BE-CE80-47DB-A659-2F8BF277C9C6}
    # Specifies the protocol type
    <Protocol>
        # Protocol types
        Type  iec104apci
    </Protocol>
    <Protocol>
        Type  iec104asdu
    </Protocol>
    # Formats the result as JSON
    Exec      to_json();
</Input>

<Output file>
    Module    om_file
    File      "C:\output.txt"
</Output>

IEC 60870-5-104 response sample

{
  "iec104.apci.receive_sequence_number": "14",
  "iec104.apci.send_sequence_number": "126",
  "iec104.apci.type": "Information (I)",
  "iec104.asdu.data": {
    "io": [
      {
        "ioa": 1000,
        "ie": [
          {
            "type": "NVA",
            "value": "0.266693 (8739)"
          },
          {
            "type": "QDS",
            "invalid": false,
            "not-topical": false,
            "substituted": false,
            "blocked": false,
            "overflow": false
          },
          {
            "type": "CP56Time2A",
            "milliseconds": 50547,
            "minutes": 46,
            "hours": 23,
            "day-of-week": 0,
            "day-of-month": 16,
            "month": 2,
            "year": 22
          }
        ],
        "ies": 3
      }
    ],
    "ios": 1
  },
  "iec104.asdu.dui.cause_of_transmission": "Spontaneous (3)",
  "iec104.asdu.dui.coa": "1",
  "iec104.asdu.dui.num_records": "1",
  "iec104.asdu.dui.org": "0",
  "iec104.asdu.dui.pn": "0",
  "iec104.asdu.dui.sq": "FALSE",
  "iec104.asdu.dui.test_bit": "0",
  "iec104.asdu.dui.type": "M_ME_TD_1",
  "EventTime": "2019-02-16T23:46:45.744212+02:00",
  "EventReceivedTime": "2019-02-16T23:46:46.453211+02:00",
  "SourceModuleName": "pcap",
  "SourceModuleType": "im_pcap"
}

Disclaimer

While we endeavor to keep the information in our guides up to date and correct, NXLog makes no representations or warranties of any kind, express or implied about the completeness, accuracy, reliability, suitability, or availability of the content represented here. We update our screenshots and instructions on a best-effort basis.

The accurateness of the content was tested and proved to be working in our lab environment at the time of the last revision with the following software versions:

MicroSCADA Pro SYS600 9.4
NXLog Agent version 5.4.7313
Microsoft Windows Server 2016 Standard

Last revision: 31 March 2022