Schneider Electric EcoStruxure Process Expert
Schneider Electric’s EcoStruxure Process Expert is a modern, distributed control system (DCS) designed to reduce resources spent on engineering and commissioning. It also provides cost-effective solutions for mitigating engineering and integration risks. EcoStruxure Process Expert finds deployment in a variety of industries, the most common being oil and gas, chemicals, utilities, cement, and mining.
NXLog Agent can collect and process EcoStruxure Process Expert logs from the following sources:
Logs from Windows Event Log
There are two ways to collect logs from Windows Event Log: by Event ID or by log source.
NXLog Agent can be configured to process EcoStruxure Process Expert logs that Windows Event Log collects as shown in the examples below.
This table lists the events generated by EcoStruxure Process Expert with their corresponding Event IDs.
Event ID | Description |
---|---|
0 |
|
260 |
|
The following table lists each EcoStruxure Process Expert service that generates events in Windows Event Log, along with its source name and the path to its executable.
Service Name/ Display Name | Source | Path to Executable | Description |
---|---|---|---|
EPEAuditTrail |
EPEAuditTrail |
|
|
Cache_c-_program files_intersystems_struxurewarepev46 / Cache Controller for STRUXUREWAREPEv461 |
Cache Config STRUXUREWAREPEv461 |
|
|
Schneider Electric SUT Service |
Schneider Electric SUT Service |
|
SUT Service for Schneider Electric Software Update |
This example contains an NXLog Agent configuration for reading and processing EcoStruxure Process Expert logs from Windows Event Log. Filtering is based on the value of the Event ID field.
Windows Event Log collected the following sample from the Cache Config STRUXUREWAREPEv461 log source, which has Event ID 260.
Log Name: Application
Source: Cache Config STRUXUREWAREPEv461
Date: 9/9/2021 1:13:16 PM
Event ID: 260
Task Category: None
Level: Information
Keywords: Classic
User: SYSTEM
Computer: WIN-0NDMK54PLPR
Description:
Startup Requested Cache configuration STRUXUREWAREPEv461.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Cache Config STRUXUREWAREPEv461" />
<EventID Qualifiers="16641">260</EventID>
<Level>4</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2021-09-09T20:13:16.376032300Z" />
<EventRecordID>1330</EventRecordID>
<Channel>Application</Channel>
<Computer>WIN-0NDMK54PLPR</Computer>
<Security UserID="S-1-5-18" />
</System>
<EventData>
<Data>Startup Requested</Data>
<Data>Cache configuration STRUXUREWAREPEv461. </Data>
</EventData>
</Event>
In this configuration, the im_msvistalog module is used to collect Windows Event Log data. The QueryXML block contains a query that selects only those event records having an Event ID of 260 or 0. Data is then converted to JSON using the to_json() procedure of the xm_json module.
<Extension json>
Module xm_json
</Extension>
<Input eventlog>
Module im_msvistalog
#XML query to read Windows Event Logs based on EventID
<QueryXML>
<QueryList>
<Query Id="0" Path="Application">
<Select Path="Application">
*[System[(EventID=260 or EventID=0)]]
</Select>
</Query>
</QueryList>
</QueryXML>
# Conversion to JSON
Exec to_json();
</Input>
{
"EventTime": "2021-09-09T13:13:16.376032-07:00",
"Hostname": "WIN-0NDMK54PLPR",
"Keywords": "36028797018963968",
"EventType": "INFO",
"SeverityValue": 2,
"Severity": "INFO",
"EventID": 260,
"SourceName": "Cache Config STRUXUREWAREPEv461",
"TaskValue": 0,
"RecordNumber": 1330,
"ExecutionProcessID": 0,
"ExecutionThreadID": 0,
"Channel": "Application",
"Domain": "NT AUTHORITY",
"AccountName": "SYSTEM",
"UserID": "S-1-5-18",
"AccountType": "Well Known Group",
"Message": "Startup Requested Cache configuration STRUXUREWAREPEv461. ",
"Data": "Startup Requested",
"Data_1": "Cache configuration STRUXUREWAREPEv461. ",
"EventReceivedTime": "2021-09-09T13:13:16.982131-07:00",
"SourceModuleName": "eventlog",
"SourceModuleType": "im_msvistalog"
}
This example demonstrates how to configure NXLog Agent to parse and process EcoStruxure Process Expert log sources.
This event sample was taken from Windows Event Viewer.
Log Name: Application
Source: Schneider Electric SUT Service
Date: 9/9/2021 1:26:11 PM
Event ID: 0
Task Category: None
Level: Information
Keywords: Classic
User: N/A
Computer: WIN-0NDMK54PLPR
Description:
Service started successfully.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Schneider Electric SUT Service" />
<EventID Qualifiers="0">0</EventID>
<Level>4</Level>
<Task>0</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2021-09-09T20:26:11.596342200Z" />
<EventRecordID>1336</EventRecordID>
<Channel>Application</Channel>
<Computer>WIN-0NDMK54PLPR</Computer>
<Security />
</System>
<EventData>
<Data>Service started successfully.</Data>
</EventData>
</Event>
Unlike the im_msvistalog instance from the previous example that uses a QueryXML block to filter by Event ID, this query only accepts event records from the following three Providers:
-
Cache Config STRUXUREWAREPEv461
-
EPEAuditTrail
-
Schneider Electric SUT Service
To provide more flexibility in choosing a post-processing solution, the xm_json module’s to_json() procedure is invoked, which not only converts the newly parsed fields to JSON, but also enriches each event record with the core fields.
<Extension json>
Module xm_json
</Extension>
<Input eventlog>
Module im_msvistalog
# XML query to read Windows Event Log data
<QueryXML>
<QueryList>
<Query Id="0" Path="Application">
<Select Path="Application">*
[System[Provider[@Name='Cache Config STRUXUREWAREPEv461' or
@Name='EPEAuditTrail' or
@Name='Schneider Electric SUT Service']]]
</Select>
</Query>
</QueryList>
</QueryXML>
# Conversion to JSON
Exec to_json();
</Input>
The following JSON-formatted event record represents the original Schneider Electric SUT Service event sample after NXLog Agent parsing and processing.
{
"EventTime": "2021-09-09T13:26:11.596342-07:00",
"Hostname": "WIN-0NDMK54PLPR",
"Keywords": "36028797018963968",
"EventType": "INFO",
"SeverityValue": 2,
"Severity": "INFO",
"EventID": 0,
"SourceName": "Schneider Electric SUT Service",
"TaskValue": 0,
"RecordNumber": 1336,
"ExecutionProcessID": 0,
"ExecutionThreadID": 0,
"Channel": "Application",
"Message": "Service started successfully.",
"Opcode": "Info",
"Data": "Service started successfully.",
"EventReceivedTime": "2021-09-09T13:26:11.652134-07:00",
"SourceModuleName": "eventlog",
"SourceModuleType": "im_msvistalog"
}
Collecting diagnostic data from log files
EcoStruxure Process Expert produces several types of file-based logs that NXLog Agent can parse and process. This table lists the file-based logs that Schneider Electric generates and their locations.
Log type | File ext. | Location |
---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Activity log files
Schneider Electric EcoStruxure Process Expert records the activity of each system server, operation client, and engineering client.
It then stores these activity logs in the C:\Users\<Username>\AppData\Roaming\Schneider Electric\Process Expert 2020 R2\Logs
directory.
Server log and server trace log
Server.log
and ServerTrace.log
contain the information related to the EcoStruxure Process Expert system server component.
Both logs exhibit similar data structures and can be processed using similar configurations.
The following event sample is taken from Server.log
.
2021-09-14 06:21:55.2041 INFO [T 29] Session [Id:21e80e02-694f-431c-8ac1-73c19f270a01 System:System_1 User:Administrator (Administrator)] Command (5f6fd43f-453f-40be-8251-3250f8e964cb) BuildAll (SchneiderElectric.ProcessExpert.Unity.Executable.UnityExecutable) completed in 30196.7928 ms SchneiderElectric.ProcessExpert.FoundationServices.CommandShell.ExecuteCommand
System server logs are comprised of records that contain predictable data patterns, from which a set of fields can be parsed that are common to all logs. Compare the values in the event sample above with these fields that can be parsed from it:
Field | Data sample |
---|---|
Timestamp |
|
Severity |
|
Event ID |
|
Message |
|
Service |
|
Like many other types of log sources with a Message
field — for example, syslog records — the Message
field often contains multiple data values that can be parsed as additional fields, which are common among some records but missing from others.
Because these logs exhibit consistent data patterns, 14 regular expressions are defined and assigned to constants, all ending in _REGEX
, for parsing all of the possible fields this log source might contain.
Some events can span multiple lines.
This configuration uses the im_file module to collect file-based logs and the multiline module to read multiline log records.
To correctly process multiline event logs, a pattern needs to be defined as a regular expression that describes the header line of an event.
In the following xm_multiline extension instance, the HeaderLine
directive specifies the regular expression to be used for finding the header line of each event.
<Extension multiline_srv>
Module xm_multiline
# Parsing the header line
HeaderLine /\d+.\d+.\d+\s*\d+.\d+.\d+.\d+\s*[A-Z]+\s*\[.*?\]/
</Extension>
In the from_file
input instance, the InputType
directive is used for referencing the xm_multiline extension instance by name, multiline_srv
, which will enable the instance to establish the beginning and end of each event.
Therefore, we use the xm_multiline module to process them.
The File directive used in the from_file
instance of the im_file module defines the path and filename(s) of the log file(s) to be parsed.
Since NXLog Agent supports the use of wildcards with this directive, the wildcard character (*
) used in Server*.log
means that both Server.log
and ServerTrace.log
will be parsed.
To aid in parsing, the regular expression substitution operator is invoked s/\\r\\n/ /g
to replace all occurrences of the \r\n
pair (CR/LF, i.e., hex 0D 0A) to a space (hex 20), thus converting multiline event records to single-line records.
The logic for parsing and filtering is defined within the Exec block of the from_file
instance of the im_file input module.
All parsed fields that do not require additional processing are captured using the named capturing groups defined and stored as the 14 *_REGEX
constants.
Within these regular expressions, field names are defined within angle brackets (< >
) that will determine which event field the captured value will be assigned to.
By invoking these regular expressions as part of the complex conditional statement that begins with if $raw_event =~
, the parsing takes place automatically.
The parsedate() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime
field.
The drop procedure discards records that do not match any of the regular expressions used in the conditional statement.
Then, by calling the to_json() procedure of the JSON (xm_json) extension module, the newly parsed fields along with the core fields are added to the event record and formatted as JSON before being routed to any output instances.
The following NXLog Agent configuration combines all the steps described above.
# A regular expression defined as a constant to read the content of the logs
define HEADER_REGEX (?x)^(\d+.\d+.\d+\s*\d+.\d+.\d+.\d+)\s*(?<Severity>[A-Z]+)\s*\[(?<EventId>.*?)\]\s*
define FOOTER_REGEX (?<Message>.*?)\s*(?<Service>SchneiderElectric.*?)\s+
define SESSION_REGEX (?:Session\s*\[Id\:(?<SessionId>.*?)\s*System\:(?<System>.*?)\s*User\:(?<User>.*?)\])\s*
define COMMAND_REGEX Command\s*\((?<CommandId>.*?)\)\s*(?<Command>[\w\s\(]*SchneiderElectric[\w\.\)]+)\s*
define QUERY_REGEX Query\s*\((?<QueryId>.*?)\)\s*(?<Query>SchneiderElectric[\w\.]+)\s*
define COMPILE_REGEX Compiled\:\s*(?<Compiled>[\$\w]+\s*[\w\.]+)\s*
define EXCEPTN_REGEX (?<Exception>[\w\.]+)\:\s+(?<Exception_Comment>.*?)\s+(?<AddInfo>at\s+.*)
define SUBTASK_REGEX SubTask\:\s*(?<SubTask>.*?)\((?<SubTaskId>.*?)\)\s*
define COMSRVTRANS_REGEX (?<Action>CommitServiceTransaction)\s*\((?<CommitServiceTransactionId>.*?)\)\s*
define EXEC_REGEX (?<Action>Execution)\s+of\s+(?<Command>SchneiderElectric.*?)\s+on\s+\
(?<ExecutionTool>.*?)\s+on\s+(?<VM>.*?)\s+on\s+(?<Host>.*?)\s+
define CALLBACK_REGEX From\s*\[(?<CallbackFromIP>\d+.\d+.\d+.\d+)\]\s*(?<CallbackTimestamp>[\d\-]+\s*[\d\:\.]+)\s*\
(?<CallbackSeverity>[A-Z]+)\s*Vm\s*\[(?<CallbackFromVM>.*?)\]\s*
define DURTRANSCALL_REGEX Duration\:(?<Duration>\d+\.\d+\w+)\,\s*Transaction\:(?<Transaction>.*?)\,\s*Call\:(?<Call>.*?)\s+\
(?<Method>.*?)\s+
define FILETRANS_REGEX FileTransfer\s*(?<FileTransfer>\w+)\s*(?<File>.*?)\s*\<(?<FileSize>[\d\.\s]+.B)\>\s*
define SERVICE_REGEX (?<Action>[\w\s]+\:\s*\w+)\s*(?:Id\:(?<Id>.*?))\s*(?:ClientId\:(?<ClientId>.*?))\s*\
(?:ClientHost\:(?<ClientHost>.*?))\s*(?:TotalSessions\:(?<TotalSessions>\d+))\s*
# Part of the log path defined as a constant
define PE_LOG_PATH C:\Users\Administrator\AppData\Roaming\Schneider Electric\Process Expert 2020 R2\Logs
<Extension json>
Module xm_json
</Extension>
<Extension multiline_srv>
Module xm_multiline
# Parsing the header line
HeaderLine /\d+.\d+.\d+\s*\d+.\d+.\d+.\d+\s*[A-Z]+\s*\[.*?\]/
</Extension>
<Input from_file>
Module im_file
File '%PE_LOG_PATH%\Server*.log'
InputType multiline_srv
<Exec>
# Replaces unwanted characters
$raw_event =~ s/\\r\\n/ /g;
$raw_event =~ s/\s{2,}/ /g;
</Exec>
<Exec>
# Matches the events with a regular expression
if $raw_event =~ /%HEADER_REGEX% %SESSION_REGEX% %COMMAND_REGEX% %FOOTER_REGEX% %EXCEPTN_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %SESSION_REGEX% %COMMAND_REGEX% %FOOTER_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %SESSION_REGEX% %QUERY_REGEX% %FOOTER_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %SESSION_REGEX% %COMPILE_REGEX% %FOOTER_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %SESSION_REGEX% %SUBTASK_REGEX% %FOOTER_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %SESSION_REGEX% %COMSRVTRANS_REGEX% %FOOTER_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %SESSION_REGEX% %EXEC_REGEX% %FOOTER_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %SESSION_REGEX% %FOOTER_REGEX% %EXCEPTN_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %SESSION_REGEX% %FOOTER_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %FILETRANS_REGEX% %FOOTER_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %SERVICE_REGEX% %FOOTER_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %CALLBACK_REGEX% %DURTRANSCALL_REGEX% %FOOTER_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %CALLBACK_REGEX% %FOOTER_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %FOOTER_REGEX%/
{
# Creates the timestamp
$EventTime = parsedate($1);
# Formats the result as JSON
to_json();
}
# Discard event if it doesn't match a/the regular expression
else drop();
</Exec>
</Input>
After NXLog Agent has read, parsed, and processed the original Event sample, the following event is output. See Server log and server trace log.
{
"EventReceivedTime": "2021-09-14T06:21:55.924549-07:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"Command": "BuildAll (SchneiderElectric.ProcessExpert.Unity.Executable.UnityExecutable)",
"CommandId": "5f6fd43f-453f-40be-8251-3250f8e964cb",
"EventId": "T 29",
"Message": "completed in 30196.7928 ms",
"Service": "SchneiderElectric.ProcessExpert.FoundationServices.CommandShell.ExecuteCommand",
"SessionId": "21e80e02-694f-431c-8ac1-73c19f270a01",
"Severity": "INFO",
"System": "System_1",
"User": "Administrator (Administrator)",
"EventTime": "2021-09-14T06:21:55.204100-07:00"
}
Engineering client logs and Operation client logs
Log files generated by engineering and operation clients store information about previous client activity. All of these log files exhibit similar data patterns and can be processed using a single NXLog Agent configuration.
The following sample is taken from the EngClient.log
file.
Its event record structure is identical to that of OpClient.log
.
2021-09-16 01:53:02.6169 TRACE Query Invocation SchneiderElectric.ProcessExpert.Core.Queries.GetLicenseCategoryDetails completed in 39 ms SchneiderElectric.ProcessExpert.ClientEssentials.UIFramework.NLogLogger.Trace
The following sample exhibits the event record structure also shared by OpClientCommunication.log
.
2021-09-16 01:52:40.2137 TRACE Host: localhost, Port: 9950 SchneiderElectric.ProcessExpert.ServiceCommunication.CommunicationServiceObject.IsServerReachable
The following sample is taken from EngClientNotification.log
.
2021-09-16 00:15:15.3425 INFO Waiting for notification SEQ 12 SchneiderElectric.ProcessExpert.ServiceCommunication.NotificationReceiver.EnsureMessageOrder
The sample below is taken from EngClientObjectStore.log
.
2021-09-16 01:53:05.1818 TRACE ObjectStore.GetObjects(F.Type:Node Count:1 Fetch:1 Time:63ms) SchneiderElectric.ProcessExpert.ServiceCommunication.Caching.ObjectStore.GetObjects
Like many other types of log sources with a Message
field — for example, syslog records — the Message
field often contains multiple data values that can be parsed as additional fields, which are common among some records but missing from others.
Despite this wide variety of fields, the following fields are common to all client logs. Compare the values in the previous event sample above with these fields that can be parsed from it:
Field | Data sample |
---|---|
Timestamp |
|
Severity |
|
Message |
|
Service |
|
These fields can be parsed using the following regular expressions:
define HEADER_REGEX (?x)^(\d+.\d+.\d+\s+\d+.\d+.\d+.\d+)\s+(?<Severity>[A-Z]+)\s+
define MESSAGE_REGEX (?<Message>.*?)\s*
define SERVICE_REGEX (?<Service>SchneiderElectric.*?)\s+
Because these logs exhibit consistent data patterns, 16 regular expressions are defined and assigned to constants, all ending in _REGEX
, for parsing all of the possible fields this log source might contain.
define COMMAND_REGEX (?<Action>Command.*?)\s+(?<Command>SchneiderElectric.*?)\s+
define QUERY_REGEX (?<Action>Query.*?)\s+(?<Query>SchneiderElectric.*?)\s+
define COMPONENT_REGEX (?<Action>COMMIT|ONCLOSE)?\s*(?<Component>\w+)\#(?<ComponentNum>\d+)\s+
define ID_REGEX Id\:\s*(?<Id>.*?)\s+
define EXCEPTN_REGEX (?<Exception>[\w\.]+)\:\s+(?<Exception_Comment>.*?)[\s\-\>]*\
(?:(?<ExceptionAdd>[\w\.]+)\:\s+(?<ExceptionAdd_Comment>.*))?$
define HOSTPORT_REGEX Host\:\s+(?<Host>.*?)\,\s+Port\:\s+(?<Port>\d+)\s+
define SUCCONN_REGEX Success\:\s+(?<Success>\w+)\,\s+Connected\:\s+(?<Connected>\w+)\s+
define SERVURI_REGEX ServiceURI\:\s+(?<ServiceURI>.*?)\s+
define HOSTNAME_REGEX HostName\:\s+(?<HostName>.*?)\s+
define PROP_REGEX (?<Properties>[A-Z]+\s*\d+)\s+
define GETOBJ_REGEX \(ID\:(?<Id>\d+)\s+Time\:(?<Time>\d+\w+)\)\s+
define GETOBJX_REGEX \(F\.Type\:(?<F_Type>\w+)\s+Count\:(?<Count>\d+)\s+\
Fetch\:(?<Fetch>\d+)\s+Time\:(?<Time>\d+\w+)\)\s+
define NOTIFCN_REGEX NOTIFICATION\s+\[ID\:(?<NotificationId>[\w\-]+)\s+DESC\:\
(?<NotificationDesc>.*?)\]\s+
The File directive used in the from_file
instance of the im_file module defines the path and filename(s) of the log file(s) to be parsed.
Since NXLog Agent supports the use of wildcards with this directive, the wildcard character (*
) used in *Client*.log
means that any file with a .log
file extension in the specified directory having Client
as part of its filename will be read and parsed.
The logic for parsing and filtering is defined within the Exec block of the from_file
instance of the im_file input module.
All parsed fields that do not require additional processing are captured using the named capturing groups defined and stored as the 16 *_REGEX
constants.
Within these regular expressions, field names are defined within angle brackets (< >
) that will determine which event field the captured value will be assigned to.
By invoking these regular expressions as part of the complex conditional statement that begins with if $raw_event =~
, the parsing takes place automatically.
The parsedate() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime
field.
The drop procedure discards records that do not match any of the regular expressions used in the conditional statement.
Then, by calling the to_json() procedure of the JSON (xm_json) extension module, the newly parsed fields along with the core fields are added to the event record and formatted as JSON before being routed to any output instances.
The following NXLog Agent configuration combines all the steps described above.
# Regular expressions defined as a constants to read the content of the logs
define HEADER_REGEX (?x)^(\d+.\d+.\d+\s+\d+.\d+.\d+.\d+)\s+(?<Severity>[A-Z]+)\s+
define MESSAGE_REGEX (?<Message>.*?)\s*
define SERVICE_REGEX (?<Service>SchneiderElectric.*?)\s+
define COMMAND_REGEX (?<Action>Command.*?)\s+(?<Command>SchneiderElectric.*?)\s+
define QUERY_REGEX (?<Action>Query.*?)\s+(?<Query>SchneiderElectric.*?)\s+
define COMPONENT_REGEX (?<Action>COMMIT|ONCLOSE)?\s*(?<Component>\w+)\#(?<ComponentNum>\d+)\s+
define ID_REGEX Id\:\s*(?<Id>.*?)\s+
define EXCEPTN_REGEX (?<Exception>[\w\.]+)\:\s+(?<Exception_Comment>.*?)[\s\-\>]*\
(?:(?<ExceptionAdd>[\w\.]+)\:\s+(?<ExceptionAdd_Comment>.*))?$
define HOSTPORT_REGEX Host\:\s+(?<Host>.*?)\,\s+Port\:\s+(?<Port>\d+)\s+
define SUCCONN_REGEX Success\:\s+(?<Success>\w+)\,\s+Connected\:\s+(?<Connected>\w+)\s+
define SERVURI_REGEX ServiceURI\:\s+(?<ServiceURI>.*?)\s+
define HOSTNAME_REGEX HostName\:\s+(?<HostName>.*?)\s+
define PROP_REGEX (?<Properties>[A-Z]+\s*\d+)\s+
define GETOBJ_REGEX \(ID\:(?<Id>\d+)\s+Time\:(?<Time>\d+\w+)\)\s+
define GETOBJX_REGEX \(F\.Type\:(?<F_Type>\w+)\s+Count\:(?<Count>\d+)\s+\
Fetch\:(?<Fetch>\d+)\s+Time\:(?<Time>\d+\w+)\)\s+
define NOTIFCN_REGEX NOTIFICATION\s+\[ID\:(?<NotificationId>[\w\-]+)\s+DESC\:\
(?<NotificationDesc>.*?)\]\s+
# Part of the log path defined as a constant
define PE_LOG_PATH C:\Users\Administrator\AppData\Roaming\Schneider Electric\Process Expert 2020 R2\Logs
<Extension json>
Module xm_json
</Extension>
<Input from_file>
Module im_file
File '%PE_LOG_PATH%\*Client*.log'
<Exec>
# Matches the events with a regular expression
if $raw_event =~ /%HEADER_REGEX% %COMMAND_REGEX% %MESSAGE_REGEX% %SERVICE_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %QUERY_REGEX% %MESSAGE_REGEX% %SERVICE_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %COMPONENT_REGEX% %SERVICE_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %MESSAGE_REGEX% %ID_REGEX% %SERVICE_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %MESSAGE_REGEX% %SERVICE_REGEX% %EXCEPTN_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %HOSTPORT_REGEX% %SERVICE_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %SUCCONN_REGEX% %SERVICE_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %SERVURI_REGEX% %SERVICE_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %HOSTNAME_REGEX% %SERVICE_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %MESSAGE_REGEX% %PROP_REGEX% %SERVICE_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %MESSAGE_REGEX% %GETOBJ_REGEX% %SERVICE_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %MESSAGE_REGEX% %GETOBJX_REGEX% %SERVICE_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %NOTIFCN_REGEX% %MESSAGE_REGEX% %SERVICE_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %MESSAGE_REGEX% %SERVICE_REGEX%/
{
# Creates the timestamp
$EventTime = parsedate($1);
# Formats the result as JSON
to_json();
}
# Discard event if it doesn't match a/the regular expression
else drop();
</Exec>
</Input>
The following output samples show the processed messages in JSON format.
This event was parsed from the EngClient.log event sample.
EngClient.log
in JSON format{
"EventReceivedTime": "2021-09-16T01:53:03.318546-07:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"Action": "Query Invocation",
"Message": "completed in 39 ms",
"Query": "SchneiderElectric.ProcessExpert.Core.Queries.GetLicenseCategoryDetails",
"Service": "SchneiderElectric.ProcessExpert.ClientEssentials.UIFramework.NLogLogger.Trace",
"Severity": "TRACE",
"EventTime": "2021-09-16T01:53:02.616900-07:00"
}
This event was parsed from the EngClientCommunication.log event sample.
EngClientCommunication.log
in JSON{
"EventReceivedTime": "2021-09-16T01:52:40.306515-07:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"Host": "localhost",
"Port": "9950",
"Service": "SchneiderElectric.ProcessExpert.ServiceCommunication.CommunicationServiceObject.IsServerReachable",
"Severity": "TRACE",
"EventTime": "2021-09-16T01:52:40.213700-07:00"
}
This event was parsed from the EngClientNotification.log event sample.
EngClientNotification.log
in JSON{
"EventReceivedTime": "2021-09-16T00:15:16.342500-07:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"Message": "Waiting for notification",
"Properties": "SEQ 12",
"Service": "SchneiderElectric.ProcessExpert.ServiceCommunication.NotificationReceiver.EnsureMessageOrder",
"Severity": "INFO",
"EventTime": "2021-09-16T00:15:15.342500-07:00"
}
This event was parsed from the EngClientObjectStore.log event sample.
EngClientObjectStore.log
in JSON{
"EventReceivedTime": "2021-09-16T01:53:05.869078-07:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"Count": "1",
"F_Type": "Node",
"Fetch": "1",
"Message": "ObjectStore.GetObjects",
"Service": "SchneiderElectric.ProcessExpert.ServiceCommunication.Caching.ObjectStore.GetObjects",
"Severity": "TRACE",
"Time": "63ms",
"EventTime": "2021-09-16T01:53:05.181800-07:00"
}
Object store log
The following event sample is taken from ObjectStore.log
.
ALthough its event record structure is similar to those from previous examples, it lacks a Message
field and exhibits no variation in fields.
2021-09-17 11:40:17.8680 TRACE [T 38] Session [Id:e18a1ce1-cd3b-484b-a963-e00ca5a61035 System:Global User:Administrator (Administrator)] IObjectStoreService.GetObjects(F.Type:SchneiderElectric.ProcessExpert.Systems.Node, Count:1, Time:0ms) SchneiderElectric.ProcessExpert.FoundationServices.ObjectStoreService.GetObjects
Because these logs exhibit consistent data patterns, six regular expressions are defined and assigned to constants, all ending in _REGEX
, for parsing all of the possible fields this log source might contain.
The logic for parsing and filtering is defined within the Exec block of the from_file
instance of the im_file input module.
All parsed fields that do not require additional processing are captured using the named capturing groups defined and stored as the six *_REGEX
constants.
Within these regular expressions, field names are defined within angle brackets (< >
) that will determine which event field the captured value will be assigned to.
By invoking these regular expressions as part of the complex conditional statement that begins with if $raw_event =~
, the parsing takes place automatically.
The parsedate() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime
field.
The drop procedure discards records that do not match any of the regular expressions used in the conditional statement.
Then, by calling the to_json() procedure of the JSON (xm_json) extension module, the newly parsed fields along with the core fields are added to the event record and formatted as JSON before being routed to any output instances.
The following NXLog Agent configuration combines all the steps described above.
# Regular expressions defined as a constants to read the content of the logs
# Regular expressions for the main part
define HEADER_REGEX (?x)^(\d+.\d+.\d+\s*\d+.\d+.\d+.\d+)\s*(?<Severity>[A-Z]+)\s*\[(?<EventId>.*?)\]\s+
define MESSAGE_REGEX (?<Message>.*?)\s*
define SERVICE_REGEX (?<Service>SchneiderElectric.*?)\s+
define SESSION_REGEX (?:Session\s*\[Id\:(?<SessionId>.*?)\s*System\:(?<System>.*?)\s*User\:(?<User>.*?)\])\s+
define GETOBJ_REGEX (?<Method>\w+\.\w+)\((?:(?<Property_1>.*?)\,\s*(?<Property_2>[\w\.]+\:\d+))?\)\s+
define GETOBJX_REGEX (?<Method>\w+\.\w+)\(F\.Type\:(?<F_Type>.*?)\,\s+Count\:(?<Count>\d+)\,\s+Time\:(?<Time>\d+\w+)\)\s+
# Part of the log path defined as a constant
define PE_LOG_PATH C:\Users\Administrator\AppData\Roaming\Schneider Electric\Process Expert 2020 R2\Logs
<Extension json>
Module xm_json
</Extension>
<Input from_file>
Module im_file
File '%PE_LOG_PATH%\ObjectStore.log'
<Exec>
# Matches the events with a regular expression
if $raw_event =~ /%HEADER_REGEX% %SESSION_REGEX% %GETOBJX_REGEX% %SERVICE_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %SESSION_REGEX% %GETOBJ_REGEX% %SERVICE_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %MESSAGE_REGEX% %SERVICE_REGEX%/
{
# Creates the timestamp
$EventTime = parsedate($1);
# Formats the result as JSON
to_json();
}
# Discard event if it doesn't match a/the regular expression
else drop();
</Exec>
</Input>
This event was parsed from the Object store log event sample.
{
"EventReceivedTime": "2021-09-17T11:40:17.927585-07:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"Count": "1",
"EventId": "T 38",
"F_Type": "SchneiderElectric.ProcessExpert.Systems.Node",
"Method": "IObjectStoreService.GetObjects",
"Service": "SchneiderElectric.ProcessExpert.FoundationServices.ObjectStoreService.GetObjects",
"SessionId": "e18a1ce1-cd3b-484b-a963-e00ca5a61035",
"Severity": "TRACE",
"System": "Global",
"Time": "0ms",
"User": "Administrator (Administrator)",
"EventTime": "2021-09-17T11:40:17.868000-07:00"
}
Audit trail log
Audit trail functionality allows the system server to connect to the syslog server and send audit trail messages.
Messages contain information that correspond to user actions performed by using engineering clients or the system server.
Typically, audit trail messages refer to a user action. These entries can be found in the notification panel of the engineering client.
The AuditTrail.log
file stores status events from the audit trail service.
These include starting/stopping the audit trail service, and connecting to the syslog server, among others.
The sample below represents the event structure of the audit trail log message, which is quite similar to the previously considered message types.
2021-09-19 11:38:00.2321 DEBUG [T 43] Session [Id:bf761b9b-d47e-42f5-b374-4284b1f0bd53 System:Global User:Administrator (Administrator)] Starting Audit Trail Windows Service... SchneiderElectric.ProcessExpert.FoundationServices.AuditTrailConfigurationService.ServerStatusChanged
Field | Data sample |
---|---|
Timestamp |
|
Severity |
|
EventId |
|
Message |
|
Service |
|
At the same time, messages may contain the following additional fields:
Field | Data sample |
---|---|
SessionId |
|
System |
|
User |
|
Because these logs exhibit consistent data patterns, four regular expressions are defined and assigned to constants, all ending in _REGEX
, for parsing all of the possible fields this log source might contain.
The logic for parsing and filtering is defined within the Exec block of the from_file
instance of the im_file input module.
All parsed fields that do not require additional processing are captured using the named capturing groups defined and stored as the four *_REGEX
constants.
Within these regular expressions, field names are defined within angle brackets (< >
) that will determine which event field the captured value will be assigned to.
By invoking these regular expressions as part of the complex conditional statement that begins with if $raw_event =~
, the parsing takes place automatically.
The parsedate() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime
field.
The drop procedure discards records that do not match any of the regular expressions used in the conditional statement.
Then, by calling the to_json() procedure of the JSON (xm_json) extension module, the newly parsed fields along with the core fields are added to the event record and formatted as JSON before being routed to any output instances.
The following NXLog Agent configuration combines all the steps described above.
# Regular expressions defined as a constants to read the content of the logs
define HEADER_REGEX (?x)^(\d+.\d+.\d+\s*\d+.\d+.\d+.\d+)\s*(?<Severity>[A-Z]+)\s*\[(?<EventId>.*?)\]\s+
define MESSAGE_REGEX (?<Message>.*?)\s*
define SERVICE_REGEX (?<Service>SchneiderElectric.*?)\s+
define SESSION_REGEX (?:Session\s*\[Id\:(?<SessionId>.*?)\s*System\:(?<System>.*?)\s*User\:(?<User>.*?)\])\s*
# Part of the log path defined as a constant
define PE_LOG_PATH C:\Users\Administrator\AppData\Roaming\Schneider Electric\Process Expert 2020 R2\Logs
<Extension json>
Module xm_json
</Extension>
<Input from_file>
Module im_file
File '%PE_LOG_PATH%\AuditTrail.log'
<Exec>
# Matches the events with a regular expression
if $raw_event =~ /%HEADER_REGEX% %SESSION_REGEX% %MESSAGE_REGEX% %SERVICE_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %MESSAGE_REGEX% %SERVICE_REGEX%/
{
# Creates the timestamp
$EventTime = parsedate($1);
# Formats the result as JSON
to_json();
}
# Discard event if it doesn't match a/the regular expression
else drop();
</Exec>
</Input>
This event was parsed from the Object store log event sample.
{
"EventReceivedTime": "2021-09-19T11:38:00.333899-07:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"EventId": "T 43",
"Message": "Starting Audit Trail Windows Service...",
"Service": "SchneiderElectric.ProcessExpert.FoundationServices.AuditTrailConfigurationService.ServerStatusChanged",
"SessionId": "bf761b9b-d47e-42f5-b374-4284b1f0bd53",
"Severity": "DEBUG",
"System": "Global",
"User": "Administrator (Administrator)",
"EventTime": "2021-09-19T11:38:00.232100-07:00"
}
Communication log
The following event is taken from EcoStruxure Process Expert Communication.log
.
2021-09-19 12:12:59.5253 DEBUG [T 16] ServiceURI: net.tcp://127.0.0.1:9950/StruxureWarePE/HostingService SchneiderElectric.ProcessExpert.ServiceCommunication.HostNameResolver.ReplaceHostNameWithIpAddress
Because these logs exhibit consistent data patterns, six regular expressions are defined and assigned to constants, all ending in _REGEX
, for parsing all of the possible fields this log source might contain.
The logic for parsing and filtering is defined within the Exec block of the from_file
instance of the im_file input module.
All parsed fields that do not require additional processing are captured using the named capturing groups defined and stored as the six *_REGEX
constants.
Within these regular expressions, field names are defined within angle brackets (< >
) that will determine which event field the captured value will be assigned to.
By invoking these regular expressions as part of the complex conditional statement that begins with if $raw_event =~
, the parsing takes place automatically.
The parsedate() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime
field.
The drop procedure discards records that do not match any of the regular expressions used in the conditional statement.
Then, by calling the to_json() procedure of the JSON (xm_json) extension module, the newly parsed fields along with the core fields are added to the event record and formatted as JSON before being routed to any output instances.
The following NXLog Agent configuration combines all the steps described above.
# Regular expressions defined as a constants to read the content of the logs
define HEADER_REGEX (?x)^(\d+.\d+.\d+\s*\d+.\d+.\d+.\d+)\s*(?<Severity>[A-Z]+)\s*\[(?<EventId>.*?)\]\s+
define MESSAGE_REGEX (?<Message>.*?)\s*
define SERVICE_REGEX (?<Service>SchneiderElectric.*?)\s+
define ACTION_REGEX (?<Action>\w+\.\w+)\s+
define SERVURI_REGEX ServiceURI\:\s+(?<ServiceURI>.*?)\s+
define HOSTNAME_REGEX HostName\:\s+(?<HostName>.*?)\s+
# Part of the log path defined as a constant
define PE_LOG_PATH C:\Users\Administrator\AppData\Roaming\Schneider Electric\Process Expert 2020 R2\Logs
<Extension json>
Module xm_json
</Extension>
<Input from_file>
Module im_file
File '%PE_LOG_PATH%\Communication.log'
<Exec>
# Matches the events with a regular expression
if $raw_event =~ /%HEADER_REGEX% %ACTION_REGEX% %SERVICE_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %SERVURI_REGEX% %SERVICE_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %HOSTNAME_REGEX% %SERVICE_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %MESSAGE_REGEX% %SERVICE_REGEX%/
{
# Creates the timestamp
$EventTime = parsedate($1);
# Formats the result as JSON
to_json();
}
# Discard event if it doesn't match a/the regular expression
else drop();
</Exec>
</Input>
The output sample below shows the processed result in JSON format.
This event was parsed from the Communication log event sample.
{
"EventReceivedTime": "2021-09-19T12:13:00.095747-07:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"EventId": "T 16",
"Service": "SchneiderElectric.ProcessExpert.ServiceCommunication.HostNameResolver.ReplaceHostNameWithIpAddress",
"ServiceURI": "net.tcp://127.0.0.1:9950/StruxureWarePE/HostingService",
"Severity": "DEBUG",
"EventTime": "2021-09-19T12:12:59.525300-07:00"
}
Data access log
The following event sample is taken from DataAccess.log
.
2021-09-20 04:51:24.1155 TRACE [T 36] Session [Id:ab0d3e11-7a7d-4532-8d89-b5ec9ed00e44 System:Global User:Administrator (Administrator)] Transaction SubTask: <InitializeAfterStartup>b__3(3e09ce6d-4aac-4703-b4d9-26cdd4158066) took 0.079ms for saving SchneiderElectric.ProcessExpert.Data.Context.Commit
Because these logs exhibit consistent data patterns, 10 regular expressions are defined and assigned to constants, all ending in _REGEX
, for parsing all of the possible fields this log source might contain.
The logic for parsing and filtering is defined within the Exec block of the from_file
instance of the im_file input module.
All parsed fields that do not require additional processing are captured using the named capturing groups defined and stored as the 10 *_REGEX
constants.
Within these regular expressions, field names are defined within angle brackets (< >
) that will determine which event field the captured value will be assigned to.
By invoking these regular expressions as part of the complex conditional statement that begins with if $raw_event =~
, the parsing takes place automatically.
The parsedate() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime
field.
The drop procedure discards records that do not match any of the regular expressions used in the conditional statement.
Then, by calling the to_json() procedure of the JSON (xm_json) extension module, the newly parsed fields along with the core fields are added to the event record and formatted as JSON before being routed to any output instances.
The following NXLog Agent configuration combines all the steps described above.
# Regular expressions defined as a constants to read the content of the logs
define HEADER_REGEX (?x)^(\d+.\d+.\d+\s*\d+.\d+.\d+.\d+)\s*(?<Severity>[A-Z]+)\s*\[(?<EventId>.*?)\]\s+
define MESSAGE_REGEX (?<Message>.*?)\s*
define SERVICE_REGEX (?<Service>SchneiderElectric.*?)\s+
define SESSION_REGEX (?:Session\s*\[Id\:(?<SessionId>.*?)\s*System\:(?<System>.*?)\s*User\:(?<User>.*?)\])\s+
define TRANSID_REGEX Transaction\:(?<TransactionId>.*?)\,\s+
define LOADED_REGEX Loaded\:(?<Loaded>\d+)\[Time\:(?<Loaded_Time>[\d\.]+\w+)\,\s+Data\:(?<Loaded_Data>.*?)\]\,?\s+
define SAVED_REGEX Saved\:(?<Saved>\d+)\[Time\:(?<Saved_Time>[\d\.]+\w+)\,\s+Data\:(?<Saved_Data>.*?)\,\s+New\:(?<Saved_New>\d+)\]\s+
define SUBTASK_REGEX Transaction\s+(?<Transaction>.*?)\:\s+(?<SubTask>(?<=SubTask\:\s).*?)\((?<Id>.*?)\)\s+
define COMMAND_REGEX Transaction\s+(?<Transaction>\w+\.\w+.*?)\((?<Id>.*?)\)\s+
define PROP_REGEX Transaction\s+(?<Transaction>.*?)\s+\((?<Property>.*?)\)\((?<Id>.*?)\)\s+
# Part of the log path defined as a constant
define PE_LOG_PATH C:\Users\Administrator\AppData\Roaming\Schneider Electric\Process Expert 2020 R2\Logs
<Extension json>
Module xm_json
</Extension>
<Input from_file>
Module im_file
File '%PE_LOG_PATH%\DataAccess.log'
<Exec>
# Matches the events with a regular expression
if $raw_event =~ /%HEADER_REGEX% %SESSION_REGEX% %TRANSID_REGEX% %LOADED_REGEX% %SAVED_REGEX% %SERVICE_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %SESSION_REGEX% %TRANSID_REGEX% %LOADED_REGEX% %SERVICE_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %SESSION_REGEX% %SUBTASK_REGEX% %MESSAGE_REGEX% %SERVICE_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %SESSION_REGEX% %COMMAND_REGEX% %MESSAGE_REGEX% %SERVICE_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %SESSION_REGEX% %PROP_REGEX% %MESSAGE_REGEX% %SERVICE_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %TRANSID_REGEX% %LOADED_REGEX% %SERVICE_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %MESSAGE_REGEX% %SERVICE_REGEX%/
{
# Creates the timestamps
$EventTime = parsedate($1);
# Formats the result as JSON
to_json();
}
# Discard event if it doesn't match a/the regular expression
else drop();
</Exec>
</Input>
The following output sample shows the processed event in JSON format.
This event was parsed from the Data access log event sample.
{
"EventReceivedTime": "2021-09-20T04:51:24.750590-07:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"EventId": "T 36",
"Id": "3e09ce6d-4aac-4703-b4d9-26cdd4158066",
"Message": "took 0.079ms for saving",
"Service": "SchneiderElectric.ProcessExpert.Data.Context.Commit",
"SessionId": "ab0d3e11-7a7d-4532-8d89-b5ec9ed00e44",
"Severity": "TRACE",
"SubTask": "<InitializeAfterStartup>b__3",
"System": "Global",
"Transaction": "SubTask",
"User": "Administrator (Administrator)",
"EventTime": "2021-09-20T04:51:24.115500-07:00"
}
Virtual machine log
The following event sample is taken from HybridDCS.Vm.2020R2#0.log
.
2021-09-20 11:36:04.9881 : Launch 5000 "c:\Pes\Vm\VBoxWrap.Guest.exe" 192.168.101.2 39181 Ok
Because these logs exhibit consistent data patterns, eight regular expressions are defined and assigned to constants, all ending in _REGEX
, for parsing all of the possible fields this log source might contain.
This configuration uses the im_file module to collect file-based logs and the multiline module to read multiline log records.
To correctly process multiline event logs, a pattern needs to be defined as a regular expression that describes the header line of an event.
In the following xm_multiline extension instance, the HeaderLine
directive specifies the regular expression to be used for finding the header line of each event.
<Extension multiline_srv>
Module xm_multiline
# Parsing the header line
HeaderLine /\d+.\d+.\d+\s*\d+.\d+.\d+.\d+/
</Extension>
In the from_file
input instance, the InputType
directive is used for referencing the xm_multiline extension instance by name, multiline_srv
, which will enable the instance to establish the beginning and end of each event.
To aid in parsing, the regular expression substitution operator is invoked s/\\r\\n/ /g
to replace all occurrences of the \r\n
pair (CR/LF, i.e., hex 0D 0A) to a space (hex 20), thus converting multiline event records to single-line records.
The logic for parsing and filtering is defined within the Exec block of the from_file
instance of the im_file input module.
All parsed fields that do not require additional processing are captured using the named capturing groups defined and stored as the eight *_REGEX
constants.
Within these regular expressions, field names are defined within angle brackets (< >
) that will determine which event field the captured value will be assigned to.
By invoking these regular expressions as part of the complex conditional statement that begins with if $raw_event =~
, the parsing takes place automatically.
The parsedate() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime
field.
The drop procedure discards records that do not match any of the regular expressions used in the conditional statement.
Then, by calling the to_json() procedure of the JSON (xm_json) extension module, the newly parsed fields along with the core fields are added to the event record and formatted as JSON before being routed to any output instances.
The following NXLog Agent configuration combines all the steps described above.
# Regular expressions defined as a constants to read the content of the logs
define HEADER_REGEX (?x)^(\d+.\d+.\d+\s*\d+.\d+.\d+.\d+)[\s\:]*
define MESSAGE_REGEX (?<Message>.*)
define COPY_REGEX (?<Action>.*?)\s+\"(?<File>.*?)\"\s+\"(?<Destination>.*?)\"\s+(?<Status>.*)
define LAUNCH_REGEX (?<Action>.*?)\s+\"(?<Exec>.*?)\"\s+(?<IP>\d+\.\d+\.\d+\.\d+)\s+(?<Port>\d+)\s+(?<Status>.*)
define FILEEXITS_REGEX (?<Action>.*?)\s+\"(?<FilePath>.*?)\"\s+(?<Status>.*)
define EXCEPTION_REGEX (?<Exception>.*Exception.*?)\:\s+(?<ExceptionComment>.*?)\s+
define OBJNAME_REGEX Object\s+name[\:\s\']*(?<ObjectName>.*?)[\'\.]*\s+
define ADDINFO_REGEX (?<AddInfo>at\s+.*)
# Part of the log path defined as a constant
define PE_LOG_PATH C:\Users\Administrator\AppData\Roaming\Schneider Electric\Process Expert 2020 R2\Logs
<Extension multiline_vm>
Module xm_multiline
# Regular expression to look for the header of the message
HeaderLine /\d+.\d+.\d+\s*\d+.\d+.\d+.\d+/
</Extension>
<Extension json>
Module xm_json
</Extension>
<Input from_file>
Module im_file
File '%PE_LOG_PATH%\HybridDCS.Vm.2020R2#0.log'
InputType multiline_vm
<Exec>
# Replaces unwanted characters
$raw_event =~ s/\\r\\n/ /g;
$raw_event =~ s/\s{2,}/ /g;
</Exec>
<Exec>
# Matches the events with a regular expression
if $raw_event =~ /%HEADER_REGEX% %EXCEPTION_REGEX% %OBJNAME_REGEX% %ADDINFO_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %COPY_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %LAUNCH_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %FILEEXITS_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %MESSAGE_REGEX%/
{
# Creates the timestamps
$EventTime = parsedate($1);
# Formats the result as JSON
to_json();
}
# Discard event if it doesn't match a/the regular expression
else drop();
</Exec>
</Input>
The following output sample shows the processed event in JSON format.
This event was parsed from the Virtual machine log event sample.
{
"EventReceivedTime": "2021-09-20T11:45:15.464379-07:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"Action": "Launch 5000",
"Exec": "c:\\Pes\\Vm\\VBoxWrap.Guest.exe",
"IP": "192.168.101.2",
"Port": "39181",
"Status": "Ok",
"EventTime": "2021-09-20T11:36:04.988100-07:00"
}
Performance log
The event sample below is taken from Performance.log
and exhibits the log file message structure, which is similar to the other EcoStruxure Process Expert log files.
2021-09-20 13:32:44.1836 TRACE [T 16] Session [Id:25d6030c-f7aa-419d-89af-181bfa1e4dd1 System:System_1 User:Administrator (Administrator)] BUILD ALL> Total 60.719s - Built 'Executable_1' SchneiderElectric.ProcessExpert.FoundationUtils.StopwatchLogger.Total
The event contains fields that are common for almost all EcoStruxure Process Expert log files:
Field | Data sample |
---|---|
Timestamp |
|
Severity |
|
EventId |
|
Message |
|
The fields listed above can be parsed and processed by NXLog Agent using the following regular expressions
define HEADER_REGEX (?x)^(\d+.\d+.\d+\s*\d+.\d+.\d+.\d+)\s*(?<Severity>[A-Z]+)\s*\[(?<EventId>.*?)\]\s+
define MESSAGE_REGEX (?<Message>.*?)\s*
define SERVICE_REGEX (?<Service>SchneiderElectric.*?)\s+
At the same time, messages may contain the following additional fields:
Field | Data sample |
---|---|
SessionId |
|
System |
|
User |
|
Action |
|
TotalTime |
|
define SESSION_REGEX (?:Session\s*\[Id\:(?<SessionId>.*?)\s*System\:(?<System>.*?)\s*User\:(?<User>.*?)\])\s+
define ACTION_REGEX (?<Action>.*?)\>\s+
define ELAPSED_REGEX (?:Elapsed\s+(?<ElapsedTime>[\w\.]+))[\s\-]+
define TOTAL_REGEX (?:Total\s+(?<TotalTime>[\w\.]+))[\s\-]+
Because these logs exhibit consistent data patterns, seven regular expressions are defined and assigned to constants, all ending in _REGEX
, for parsing all of the possible fields this log source might contain.
The logic for parsing and filtering is defined within the Exec block of the from_file
instance of the im_file input module.
All parsed fields that do not require additional processing are captured using the named capturing groups defined and stored as the seven *_REGEX
constants.
Within these regular expressions, field names are defined within angle brackets (< >
) that will determine which event field the captured value will be assigned to.
By invoking these regular expressions as part of the complex conditional statement that begins with if $raw_event =~
, the parsing takes place automatically.
The parsedate() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime
field.
The drop procedure discards records that do not match any of the regular expressions used in the conditional statement.
Then, by calling the to_json() procedure of the JSON (xm_json) extension module, the newly parsed fields along with the core fields are added to the event record and formatted as JSON before being routed to any output instances.
The following NXLog Agent configuration combines all the steps described above.
# Regular expressions defined as a constants to read the content of the logs
define HEADER_REGEX (?x)^(\d+.\d+.\d+\s*\d+.\d+.\d+.\d+)\s*(?<Severity>[A-Z]+)\s*\[(?<EventId>.*?)\]\s+
define MESSAGE_REGEX (?<Message>.*?)\s*
define SERVICE_REGEX (?<Service>SchneiderElectric.*?)\s+
define SESSION_REGEX (?:Session\s*\[Id\:(?<SessionId>.*?)\s*System\:(?<System>.*?)\s*User\:(?<User>.*?)\])\s+
define ACTION_REGEX (?<Action>.*?)\>\s+
define ELAPSED_REGEX (?:Elapsed\s+(?<ElapsedTime>[\w\.]+))[\s\-]+
define TOTAL_REGEX (?:Total\s+(?<TotalTime>[\w\.]+))[\s\-]+
# Part of the log path defined as a constant
define PE_LOG_PATH C:\Users\Administrator\AppData\Roaming\Schneider Electric\Process Expert 2020 R2\Logs
<Extension json>
Module xm_json
</Extension>
<Input from_file>
Module im_file
File '%PE_LOG_PATH%\Performance.log'
<Exec>
# Matches the events with a regular expression
if $raw_event =~ /%HEADER_REGEX% %SESSION_REGEX% %ACTION_REGEX% %ELAPSED_REGEX% %MESSAGE_REGEX% %SERVICE_REGEX%/ or
$raw_event =~ /%HEADER_REGEX% %SESSION_REGEX% %ACTION_REGEX% %TOTAL_REGEX% %MESSAGE_REGEX% %SERVICE_REGEX%/
{
# Creates the timestamps
$EventTime = parsedate($1);
# Formats the result as JSON
to_json();
}
# Discard event if it doesn't match a/the regular expression
else drop();
</Exec>
</Input>
The following output sample shows the processed event in JSON format.
This event was parsed from the Performance log event sample.
{
"EventReceivedTime": "2021-09-20T13:32:44.916056-07:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"Action": "BUILD ALL",
"EventId": "T 16",
"Message": "Built 'Executable_1'",
"Service": "SchneiderElectric.ProcessExpert.FoundationUtils.StopwatchLogger.Total",
"SessionId": "25d6030c-f7aa-419d-89af-181bfa1e4dd1",
"Severity": "TRACE",
"System": "System_1",
"TotalTime": "60.719s",
"User": "Administrator (Administrator)",
"EventTime": "2021-09-20T13:32:44.183600-07:00"
}
System information log
The EcoStruxure Process Expert SysInfo.log
file provides a detailed overview of the main system components such as operating system, CPUs, physical memory, disk controller, video controller, etc.
The log file is updated once the system server is started.
The following event samples are taken from SysInfo.log
and represent different entries, single-line and multiline.
Because these logs exhibit consistent data patterns, specific fields can be parsed using the following regular expressions defined by HEADER_REGEX
, PROPVAL_REGEX
, and DRIVE_REGEX
:
2021-09-25 22:07:47.0444 TRACE [T 10] ENVIRONMENT PARAM SystemDirectory = C:\Windows\system32
define HEADER_REGEX (?x)^(\d+.\d+.\d+\s*\d+.\d+.\d+.\d+)\s*(?<Severity>[A-Z]+)\s*\[(?<EventId>.*?)\]\s+(?<EventType>.*?)\t+\
(?<Property>.*?)\s+\=\s+(?<Value>.*)
Using HEADER_REGEX
, the following fields can be parsed.
Field | Data sample |
---|---|
Timestamp |
|
Severity |
|
EventId |
|
EventType |
|
Property |
|
Value |
|
2021-09-25 22:21:50.1003 TRACE [T 40] ENVIRONMENT PARAM Drive Z:\ Properties = Volume Label : Shared Folders, Drive Type : Network, Drive Format : HGFS, Total Size : 208784060416, Available Free Space : 108368785408
define DRIVE_REGEX Volume\s+Label[\s\:]+(?<VolumeLabel>.*?)\,\s+Drive\s+Type[\s\:]+(?<DriveType>.*?)\,\s+\
Drive\s+Format[\s\:]+(?<DriveFormat>.*?)\,\s+Total\s+Size[\s\:]+(?<TotalSize>.*?)\,\s+\
Available\s+Free\s+Space[\s\:]+(?<AvailableFreeSpace>.*)
Using DRIVE_REGEX
, the following fields can be parsed.
Field | Data sample |
---|---|
VolumeLabel |
|
DriveType |
|
DriveFormat |
|
TotalSize |
|
AvailableFreeSpace |
|
SysInfo.log
can also contain multiline events like the following one.
2021-09-25 22:21:50.1927 TRACE [T 40] ENVIRONMENT PARAM Operating System =
BootDevice : \Device\HarddiskVolume2
BuildNumber : 14393
BuildType : Multiprocessor Free
Caption : Microsoft Windows Server 2016 Standard
CodeSet : 1252
CountryCode : 1
CreationClassName : Win32_OperatingSystem
CSCreationClassName : Win32_ComputerSystem
CSDVersion :
CSName : WIN-0NDMK54PLPR
CurrentTimeZone : -420
DataExecutionPrevention_32BitApplications : True
DataExecutionPrevention_Available : True
DataExecutionPrevention_Drivers : True
DataExecutionPrevention_SupportPolicy : 3
Debug : False
Description :
Distributed : False
EncryptionLevel : 256
ForegroundApplicationBoost : 2
FreePhysicalMemory : 5790392
FreeSpaceInPagingFiles : 1228800
FreeVirtualMemory : 7034056
InstallDate : 20210909051158.000000-420
LargeSystemCache :
LastBootUpTime : 20210925205239.486214-420
LocalDateTime : 20210925222150.137000-420
Locale : 0409
Manufacturer : Microsoft Corporation
MaxNumberOfProcesses : 4294967295
MaxProcessMemorySize : 137438953344
MUILanguages : System.String[]
Name : Microsoft Windows Server 2016 Standard|C:\Windows|\Device\Harddisk0\Partition4
NumberOfLicensedUsers : 0
NumberOfProcesses : 92
NumberOfUsers : 1
OperatingSystemSKU : 7
Organization :
OSArchitecture : 64-bit
OSLanguage : 1033
OSProductSuite : 272
OSType : 18
OtherTypeDescription :
PAEEnabled :
PlusProductID :
PlusVersionNumber :
PortableOperatingSystem : False
Primary : True
ProductType : 3
RegisteredUser : Windows User
SerialNumber : 00377-60000-00000-AA027
ServicePackMajorVersion : 0
ServicePackMinorVersion : 0
SizeStoredInPagingFiles : 1310720
Status : OK
SuiteMask : 272
SystemDevice : \Device\HarddiskVolume4
SystemDirectory : C:\Windows\system32
SystemDrive : C:
TotalSwapSpaceSize :
TotalVirtualMemorySize : 9698268
TotalVisibleMemorySize : 8387548
Version : 10.0.14393
WindowsDirectory : C:\Windows
Normally, a long and complex regular expression would be required to parse such a large number of fields.
However, a simple but effective approach is to use PROPVAL_REGEX
and parse each line of this multiline event as individual events having only two fields, $Property
and $Value
.
define PROPVAL_REGEX (?x)^\t+(?<Property>.*?)\s+\:\s+(?<Value>.*)
Using PROPVAL_REGEX
, the following fields can be parsed from multiline event records without any dependence on HEADER_REGEX
or DRIVE_REGEX
.
Field | Data sample |
---|---|
Property |
|
Value |
|
The logic for parsing and filtering is defined within the Exec block of the from_file
instance of the im_file input module.
All parsed fields that do not require additional processing are captured using the named capturing groups contained defined by HEADER_REGEX
, PROPVAL_REGEX
or DRIVE_REGEX
.
Each field name is defined within angle brackets (< >
) that will determine which event field the captured value will be assigned to.
In this example, depending on which of the three regular expressions an event matches, the event record will be enriched with a subset of the following fields:
$EventId
, $EventType
, $Severity
, $Property
, and $Value
.
If the event contains drive information, the following fields will also be added: $VolumeLabel
, $DriveType
, $DriveFormat
, $TotalSize
, and $AvailableFreeSpace
.
The strptime() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime
field.
The drop procedure discards records that match neither the HEADER_REGEX
nor the PROPVAL_REGEX
nor the DRIVE_REGEX
regular expressions.
Then, by calling the to_json() procedure of the JSON (xm_json) extension module, the newly parsed fields along with the core fields are added to the event record and formatted as JSON before being routed to any output instances.
The following NXLog Agent configuration combines all the steps described above.
# Regular expressions defined as a constants to read the content of the logs
define HEADER_REGEX (?x)^(\d+.\d+.\d+\s*\d+.\d+.\d+.\d+)\s*(?<Severity>[A-Z]+)\s*\[(?<EventId>.*?)\]\s+(?<EventType>.*?)\t+\
(?<Property>.*?)\s+\=\s+(?<Value>.*)
define PROPVAL_REGEX (?x)^\t+(?<Property>.*?)\s+\:\s+(?<Value>.*)
define DRIVE_REGEX Volume\s+Label[\s\:]+(?<VolumeLabel>.*?)\,\s+Drive\s+Type[\s\:]+(?<DriveType>.*?)\,\s+\
Drive\s+Format[\s\:]+(?<DriveFormat>.*?)\,\s+Total\s+Size[\s\:]+(?<TotalSize>.*?)\,\s+\
Available\s+Free\s+Space[\s\:]+(?<AvailableFreeSpace>.*)
# Part of the log path defined as a constant
define PE_LOG_PATH C:\Users\Administrator\AppData\Roaming\Schneider Electric\Process Expert 2020 R2\Logs
<Extension json>
Module xm_json
</Extension>
<Input from_file>
Module im_file
File '%PE_LOG_PATH%\SysInfo.log'
<Exec>
# Matches the events with a regular expression
if $raw_event =~ /%HEADER_REGEX% %DRIVE_REGEX%/ or
$raw_event =~ /%HEADER_REGEX%/
{
# Creates the timestamps
$EventTime = parsedate($1);
# Formats the result as JSON
to_json();
}
else if $raw_event =~ /%PROPVAL_REGEX%/
{
# Formats the result as JSON
to_json();
}
# Discard event if it doesn't match a/the regular expression
else drop();
</Exec>
</Input>
The following output samples depict the processed events in JSON format.
This event was parsed from the System information log input sample, single-line event.
{
"EventReceivedTime": "2021-09-25T22:07:47.758428-07:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"EventId": "T 10",
"EventType": "ENVIRONMENT PARAM",
"Property": "SystemDirectory",
"Severity": "TRACE",
"Value": "C:\\Windows\\system32",
"EventTime": "2021-09-25T22:07:47.044400-07:00"
}
This event was parsed from the System information log input sample with drive information, single-line event.
{
"EventReceivedTime": "2021-09-25T22:21:50.733777-07:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"AvailableFreeSpace": "108368785408",
"DriveFormat": "HGFS",
"DriveType": "Network",
"EventId": "T 40",
"EventType": "ENVIRONMENT PARAM",
"Property": "Drive Z:\\ Properties",
"Severity": "TRACE",
"TotalSize": "208784060416",
"Value": "",
"VolumeLabel": "Shared Folders",
"EventTime": "2021-09-25T22:21:50.100300-07:00"
}
This event was parsed from the System information log input sample, event.
{
"EventReceivedTime": "2021-09-25T22:21:50.734408-07:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"EventId": "T 40",
"EventType": "ENVIRONMENT PARAM",
"Property": "Operating System",
"Severity": "TRACE",
"Value": "",
"EventTime": "2021-09-25T22:21:50.192700-07:00"
}
This event was parsed from the System information log input sample, event.
{
"EventReceivedTime": "2021-09-25T22:21:50.734408-07:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"Property": "BootDevice",
"Value": "\\Device\\HarddiskVolume2"
}
Migration log
The MigrationLogFile.txt
contains the information related to the database migration process.
The log file is located in C:\ProgramData\Schneider Electric\Process Expert 2020 R2\Db
folder.
The following event samples are taken from MigrationLogFile.txt
and represent two types of input entries.
The first event type contains the following list of fields:
Field | Data sample |
---|---|
Severity |
|
Timestamp |
|
Message |
|
[Info] (09/27/2021 04:51:53) Database Global : Message 0 of 1 Global Templates converted successfully : Domain : ContentRepository
The following fields can be parsed from the second event type:
Field | Data sample |
---|---|
Timestamp |
|
Message |
|
9/27/2021 4:51:53 AM Database migration for database System_1 sucessful : 0 Error(s) , 0 Warning(s) and 3 Information
Because these logs exhibit consistent data patterns, specific fields can be parsed using the following regular expressions defined by MIGR_TYPE1_REGEX
and MIGR_TYPE2_REGEX
:
define MIGR_TYPE1_REGEX /(?x)^\[(?<Severity>\w+)?\]\s+\((\d+.\d+.\d+\s+\d+.\d+.\d+)\)\s*(?<Message>.*)/
define MIGR_TYPE2_REGEX /(?x)^(\d+\/\d+\/\d+\s+\d+\:\d+\:\d+\s+\w+)\s+(?<Message>.*)/
The logic for parsing and filtering is defined within the Exec block of the from_file
instance of the im_file input module.
All parsed fields that do not require additional processing are captured using the named capturing groups defined in MIGR_TYPE1_REGEX
or MIGR_TYPE2_REGEX
.
Each field name is defined within angle brackets (< >
) that will determine which event field the captured value will be assigned to.
In this example, depending on which of the two regular expressions an event matches, the event record will be enriched with a subset of the following fields:
$Severity
and $Message
.
The strptime() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime
field.
The drop procedure discards records that match neither the MIGR_TYPE1_REGEX
nor the MIGR_TYPE2_REGEX
regular expressions.
Then, by calling the to_json() procedure of the JSON (xm_json) extension module, the newly parsed fields along with the core fields are added to the event record and formatted as JSON before being routed to any output instances.
The following NXLog Agent configuration combines all the steps described above.
# Regular expressions defined as a constants to read the content of the logs
define MIGR_TYPE1_REGEX /(?x)^\[(?<Severity>\w+)?\]\s+\((\d+.\d+.\d+\s+\d+.\d+.\d+)\)\s*(?<Message>.*)/
define MIGR_TYPE2_REGEX /(?x)^(\d+\/\d+\/\d+\s+\d+\:\d+\:\d+\s+\w+)\s+(?<Message>.*)/
# Part of the log path defined as a constant
define PE_LOG_PATH C:\ProgramData\Schneider Electric\Process Expert 2020 R2\Db
<Extension json>
Module xm_json
</Extension>
<Input from_file>
Module im_file
File '%PE_LOG_PATH%\MigrationLogFile.txt'
<Exec>
# Matches the events with a regular expression
if $raw_event =~ %MIGR_TYPE1_REGEX%
{
# Creates the timestamps
$EventTime = strptime($2,'%m/%d/%Y %T');
# Formats the result as JSON
to_json();
}
else if $raw_event =~ %MIGR_TYPE2_REGEX%
{
# Creates the timestamps
$EventTime = parsedate($1);
# Formats the result as JSON
to_json();
}
# Discard event if it doesn't match a/the regular expression
else drop();
</Exec>
</Input>
The following output sample represents a processed event in JSON format.
This event was parsed from the Input sample of event type 1.
{
"EventReceivedTime": "2021-09-27T04:51:54.372805-07:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"Message": "Database Global : Message 0 of 1 Global Templates converted successfully : Domain : ContentRepository",
"Severity": "Info",
"EventTime": "2021-09-27T04:51:53.000000-07:00"
}
This event was parsed from the Input sample of event type 2.
{
"EventReceivedTime": "2021-09-27T04:51:54.372805-07:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"Message": "Database migration for database System_1 sucessful : 0 Error(s) , 0 Warning(s) and 3 Information",
"EventTime": "2021-09-27T04:51:53.000000-07:00"
}
Passive Network monitoring
EcoStruxure Process Expert DCS allows building the control system using the predefined objects integrated into tested, validated, and documented libraries.
The communication functionality provided by the integrated libraries allows the integration of Schneider Electric and third-party devices into the EcoStruxure Process Expert DCS using a variety of open and propriety protocols. It makes the data of connected devices available to other control systems.
Modbus TCP
Modbus is an open, simple, and robust industrial communication standard. It has been combined with Ethernet to create the Modbus TCP communication protocol.
Modbus TCP is an open Ethernet protocol that does not require any license to develop a connection and can be used with any product supporting TCP/IP. Vendor support for Modbus TCP is well-established and there are numerous products currently using this protocol.
Since the Modbus TCP application layer is identical to the Modbus serial link, there is no need to convert protocols for routing between networks. Like the TCP/IP protocol it was built upon, Modbus TCP also supports IP routing.
The Dev directive in the im_pcap module specifies the network interface.
The Protocol group directive defines the modbus
protocol for monitoring.
The Exec block of im_pcap calls the to_json() procedure of the xm_json module to convert messages to JSON.
<Extension _json>
Module xm_json
</Extension>
<Input pcap_modbus>
Module im_pcap
# Name of a network device/interface
Dev \Device\NPF_{B89D13DD-37BB-4799-BFCD-6C9D326C481A}
<Protocol>
# Protocol type
Type modbus
</Protocol>
# Conversion to JSON
Exec to_json();
</Input>
{
"modbus.function_code": "Read Holding Registers (03)",
"modbus.length": "6",
"modbus.prot_id": "0",
"modbus.query.read_holding_regs.qty_of_regs": "5",
"modbus.query.read_holding_regs.starting_address": "0",
"modbus.trans_id": "80",
"modbus.unit_id": "1",
"EventTime": "2021-10-07T05:16:15.251715-07:00",
"EventReceivedTime": "2021-10-07T05:16:15.293361-07:00",
"SourceModuleName": "pcap_modbus",
"SourceModuleType": "im_pcap"
}
{
"modbus.function_code": "Read Holding Registers (03)",
"modbus.length": "13",
"modbus.prot_id": "0",
"modbus.response.read_holding_regs.byte_count": "10",
"modbus.response.read_holding_regs.registers": "216, 199, 193, 179, 162",
"modbus.trans_id": "80",
"modbus.unit_id": "1",
"EventTime": "2021-10-07T05:16:15.262300-07:00",
"EventReceivedTime": "2021-10-07T05:16:16.280934-07:00",
"SourceModuleName": "pcap_modbus",
"SourceModuleType": "im_pcap"
}
DNP3
The distributed network protocol (DNP3) is an open standard for interoperability between master stations, substation devices, Remote Terminal Units (RTUs), and Intelligent Electronic Devices (IEDs). DNP3 is widely used in North America by the oil and gas, utilities, and transportation industries.
The protocol was originally developed for serial communications, but the current DNP3 IP version supports TCP/IP-based networking.
The current NXLog Agent configuration uses the im_pcap module for passive network monitoring. The Dev directive of this module specifies the network interface. The Protocol group directive defines the protocol for capturing. The Exec block of im_pcap calls the to_json() procedure of the xm_json module to convert messages to JSON.
<Extension _json>
Module xm_json
</Extension>
<Input pcap_dnp3>
Module im_pcap
# Name of a network device/interface
Dev \Device\NPF_{B89D13DD-37BB-4799-BFCD-6C9D326C481A}
<Protocol>
# Protocol type
Type dnp3
</Protocol>
# Conversion to JSON
Exec to_json();
</Input>
{
"dnp3.application_layer.control.con": "1",
"dnp3.application_layer.control.fin": "1",
"dnp3.application_layer.control.fir": "1",
"dnp3.application_layer.control.sequence": "7",
"dnp3.application_layer.control.uns": "1",
"dnp3.application_layer.function_code": "Unsolicited Response",
"dnp3.application_layer.internal_indications.already_executing": "0",
"dnp3.application_layer.internal_indications.broadcast": "0",
"dnp3.application_layer.internal_indications.class1_events": "0",
"dnp3.application_layer.internal_indications.class2_events": "0",
"dnp3.application_layer.internal_indications.class3_events": "0",
"dnp3.application_layer.internal_indications.config_corrupt": "0",
"dnp3.application_layer.internal_indications.device_restart": "0",
"dnp3.application_layer.internal_indications.device_trouble": "0",
"dnp3.application_layer.internal_indications.events_buffer_overflow": "0",
"dnp3.application_layer.internal_indications.local_control": "0",
"dnp3.application_layer.internal_indications.need_time": "0",
"dnp3.application_layer.internal_indications.no_func_code_support": "0",
"dnp3.application_layer.internal_indications.object_unknown": "0",
"dnp3.application_layer.internal_indications.parameter_error": "0",
"dnp3.application_layer.internal_indications.reserved": "0 (expected 0)",
"dnp3.application_layer.object0.count": "1",
"dnp3.application_layer.object0.group": "32",
"dnp3.application_layer.object0.name": "Analog input event - single-precision, floating-point with time",
"dnp3.application_layer.object0.point0.flags": "[ONLINE]",
"dnp3.application_layer.object0.point0.index": "1",
"dnp3.application_layer.object0.point0.time_of_occurance": "1633867737071",
"dnp3.application_layer.object0.point0.value": "11876.329102",
"dnp3.application_layer.object0.range": "2-octet count of objects",
"dnp3.application_layer.object0.variation": "7",
"dnp3.data_layer.control": "0x44",
"dnp3.data_layer.control.dir": "0",
"dnp3.data_layer.control.fcb": "0",
"dnp3.data_layer.control.fcv": "0",
"dnp3.data_layer.control.function_code": "Unconfirmed User Data",
"dnp3.data_layer.control.prm": "1",
"dnp3.data_layer.destination": "3",
"dnp3.data_layer.length": "28",
"dnp3.data_layer.source": "4",
"dnp3.data_layer.start_bytes": "0x0564",
"dnp3.transport.fin": "1",
"dnp3.transport.fir": "1",
"dnp3.transport.sequence": "12",
"EventTime": "2021-10-10T02:08:57.076909-07:00",
"EventReceivedTime": "2021-10-10T02:08:57.210239-07:00",
"SourceModuleName": "pcap_dnp3",
"SourceModuleType": "im_pcap"
}
{
"dnp3.application_layer.control.con": "0",
"dnp3.application_layer.control.fin": "1",
"dnp3.application_layer.control.fir": "1",
"dnp3.application_layer.control.sequence": "7",
"dnp3.application_layer.control.uns": "1",
"dnp3.application_layer.function_code": "Confirm",
"dnp3.data_layer.control": "0xC4",
"dnp3.data_layer.control.dir": "1",
"dnp3.data_layer.control.fcb": "0",
"dnp3.data_layer.control.fcv": "0",
"dnp3.data_layer.control.function_code": "Unconfirmed User Data",
"dnp3.data_layer.control.prm": "1",
"dnp3.data_layer.destination": "4",
"dnp3.data_layer.length": "8",
"dnp3.data_layer.source": "3",
"dnp3.data_layer.start_bytes": "0x0564",
"dnp3.transport.fin": "1",
"dnp3.transport.fir": "1",
"dnp3.transport.sequence": "7",
"EventTime": "2021-10-10T02:08:57.186236-07:00",
"EventReceivedTime": "2021-10-10T02:08:57.211453-07:00",
"SourceModuleName": "pcap_dnp3",
"SourceModuleType": "im_pcap"
}
{
"dnp3.application_layer.control.con": "0",
"dnp3.application_layer.control.fin": "1",
"dnp3.application_layer.control.fir": "1",
"dnp3.application_layer.control.sequence": "14",
"dnp3.application_layer.control.uns": "0",
"dnp3.application_layer.function_code": "Read",
"dnp3.application_layer.object0.count": "0",
"dnp3.application_layer.object0.group": "60",
"dnp3.application_layer.object0.name": "Class objects - Class 1 data",
"dnp3.application_layer.object0.variation": "2",
"dnp3.application_layer.object1.count": "0",
"dnp3.application_layer.object1.group": "60",
"dnp3.application_layer.object1.name": "Class objects - Class 2 data",
"dnp3.application_layer.object1.variation": "3",
"dnp3.application_layer.object2.count": "0",
"dnp3.application_layer.object2.group": "60",
"dnp3.application_layer.object2.name": "Class objects - Class 3 data",
"dnp3.application_layer.object2.variation": "4",
"dnp3.data_layer.control": "0xC4",
"dnp3.data_layer.control.dir": "1",
"dnp3.data_layer.control.fcb": "0",
"dnp3.data_layer.control.fcv": "0",
"dnp3.data_layer.control.function_code": "Unconfirmed User Data",
"dnp3.data_layer.control.prm": "1",
"dnp3.data_layer.destination": "4",
"dnp3.data_layer.length": "17",
"dnp3.data_layer.source": "3",
"dnp3.data_layer.start_bytes": "0x0564",
"dnp3.transport.fin": "1",
"dnp3.transport.fir": "1",
"dnp3.transport.sequence": "6",
"EventTime": "2021-10-10T02:08:56.986131-07:00",
"EventReceivedTime": "2021-10-10T02:08:57.210239-07:00",
"SourceModuleName": "pcap_dnp3",
"SourceModuleType": "im_pcap"
}
{
"dnp3.application_layer.control.con": "0",
"dnp3.application_layer.control.fin": "1",
"dnp3.application_layer.control.fir": "1",
"dnp3.application_layer.control.sequence": "14",
"dnp3.application_layer.control.uns": "0",
"dnp3.application_layer.function_code": "Response",
"dnp3.application_layer.internal_indications.already_executing": "0",
"dnp3.application_layer.internal_indications.broadcast": "0",
"dnp3.application_layer.internal_indications.class1_events": "0",
"dnp3.application_layer.internal_indications.class2_events": "0",
"dnp3.application_layer.internal_indications.class3_events": "0",
"dnp3.application_layer.internal_indications.config_corrupt": "0",
"dnp3.application_layer.internal_indications.device_restart": "0",
"dnp3.application_layer.internal_indications.device_trouble": "0",
"dnp3.application_layer.internal_indications.events_buffer_overflow": "0",
"dnp3.application_layer.internal_indications.local_control": "0",
"dnp3.application_layer.internal_indications.need_time": "0",
"dnp3.application_layer.internal_indications.no_func_code_support": "0",
"dnp3.application_layer.internal_indications.object_unknown": "0",
"dnp3.application_layer.internal_indications.parameter_error": "0",
"dnp3.application_layer.internal_indications.reserved": "0 (expected 0)",
"dnp3.data_layer.control": "0x44",
"dnp3.data_layer.control.dir": "0",
"dnp3.data_layer.control.fcb": "0",
"dnp3.data_layer.control.fcv": "0",
"dnp3.data_layer.control.function_code": "Unconfirmed User Data",
"dnp3.data_layer.control.prm": "1",
"dnp3.data_layer.destination": "3",
"dnp3.data_layer.length": "10",
"dnp3.data_layer.source": "4",
"dnp3.data_layer.start_bytes": "0x0564",
"dnp3.transport.fin": "1",
"dnp3.transport.fir": "1",
"dnp3.transport.sequence": "11",
"EventTime": "2021-10-10T02:08:56.986861-07:00",
"EventReceivedTime": "2021-10-10T02:08:57.210239-07:00",
"SourceModuleName": "pcap_dnp3",
"SourceModuleType": "im_pcap"
}
IEC 60870-5-104
The IEC 60870-5-104 protocol extends the IEC 60870-5-101 protocol and combines transport, network, link, and physical layers to enable communication between control stations and substations using TCP/IP. NXLog Agent can be configured to monitor network traffic that uses this protocol.
To passively monitor network traffic, NXLog Agent uses the im_pcap module.
The Dev directive of this module specifies the network interface.
The Protocol group directive is then defined twice to simultaneously monitor the iec104apci
and iec104asdu
protocols.
The Exec block of this module calls the to_json() procedure of the xm_json module to convert messages to JSON.
<Extension _json>
Module xm_json
</Extension>
<Input pcap_iec>
Module im_pcap
# Name of a network device/interface
Dev \Device\NPF_{B89D13DD-37BB-4799-BFCD-6C9D326C481A}
<Protocol>
# Protocol types
Type iec104apci
</Protocol>
<Protocol>
Type iec104asdu
</Protocol>
# Conversion to JSON
Exec to_json();
</Input>
{
"iec104.apci.receive_sequence_number": "2",
"iec104.apci.send_sequence_number": "39",
"iec104.apci.type": "Information (I)",
"iec104.asdu.data": {
"io": [
{
"ioa": 1020,
"ie": [
{
"type": "R32",
"value": 29899.640625
},
{
"type": "QDS",
"invalid": false,
"not-topical": false,
"substituted": false,
"blocked": false,
"overflow": false
},
{
"type": "CP56Time2A",
"milliseconds": 35304,
"minutes": 59,
"hours": 13,
"day-of-week": 0,
"day-of-month": 10,
"month": 10,
"year": 21
}
],
"ies": 3
}
],
"ios": 1
},
"iec104.asdu.dui.cause_of_transmission": "Spontaneous (3)",
"iec104.asdu.dui.coa": "1",
"iec104.asdu.dui.num_records": "1",
"iec104.asdu.dui.org": "0",
"iec104.asdu.dui.pn": "0",
"iec104.asdu.dui.sq": "FALSE",
"iec104.asdu.dui.test_bit": "0",
"iec104.asdu.dui.type": "M_ME_TF_1",
"EventTime": "2021-10-10T03:59:35.681739-07:00",
"EventReceivedTime": "2021-10-10T03:59:35.871039-07:00",
"SourceModuleName": "pcap_iec",
"SourceModuleType": "im_pcap"
}