AVEVA System Platform
AVEVA System Platform is a modular and scalable industrial software platform for software solutions focused on the needs of industrial automation and engineering personnel, including SCADA, HMI, IIoT, Manufacturing Execution Systems (MES).
System Platform supports both the supervisory control layer and the manufacturing execution system (MES) layer, presenting them as a single information source.
NXLog can be configured to collect and process all types of logs produced by the AVEVA System Platform.
Logs from Windows Event Log
This table contains the AVEVA System Platform services, which generate Windows Event Log data along with their display names and executables.
Service Name | Display Name | Source Name | Path to Executable |
---|---|---|---|
AsbCertificateRenewalService |
ArchestrA Certificate Renewal Service |
AsbCertificateRenewalService |
C:\Program Files (x86)\AVEVA\Platform Common Services\CoreServices\Asb.CertificateRenewalService.exe |
aaGR |
ArchestrA GalaxyRepository |
aaGR |
C:\Program Files (x86)\ArchestrA\Framework\Bin\aaGR.exe |
aahInSight |
AVEVA Historian Client Web |
Browser Client |
C:\Program Files (x86)\Wonderware\HistorianInsight\Server\aahInSightHost.exev |
InSQLConfiguration |
AVEVA Historian Configuration |
Historian |
C:\Program Files (x86)\Wonderware\Historian\aahCfgSvc.exe |
nSQLIndexing |
AVEVA Historian Indexing |
InSQLIndexing |
C:\Program Files (x86)\Wonderware\Historian\aahIndexSvc.exe |
AIGWebServer |
AVEVA Industrial Graphics Service |
Browser Client |
C:\Program Files (x86)\Wonderware\InTouchWeb\Server\Aveva.Web.Host.exe |
EricomAuthenticationServer |
InTouch Access Anywhere Authentication Server |
EricomAuthenticationServer / Ericom Authentication Server 9.2 |
C:\Program Files (x86)\Wonderware\InTouch Access Anywhere Secure Gateway\InTouch Access Anywhere Authentication Server\EricomAuthenticationServer.exe |
EricomSecureGateway |
InTouch Access Anywhere Secure Gateway |
Ericom Secure Gateway 9.2 |
C:\Program Files (x86)\Wonderware\InTouch Access Anywhere Secure Gateway\InTouch Access Anywhere Secure Gateway\EricomSecureGateway.exe |
InTouchDataService |
InTouch IData Service |
InTouch IData Service |
C:\Program Files (x86)\Common Files\ArchestrA\Services\InTouchDataService.exe |
Product License Service |
Product License Service |
Service1 |
C:\Program Files (x86)\Common Files\ArchestrA\Licensing Framework\License API2\ProductLicenseWindowsService.exe |
adpHostSrv |
Sentinel Agent Service |
PSMS Agent |
C:\Program Files (x86)\Sentinel System Monitor\Sentinel Agent\adpHostSrv.exe |
psmsConsoleSrv |
Sentinel Console Service |
PSMS Console Service |
C:\Program Files (x86)\Sentinel System Monitor\Sentinel Manager\Console Service\psmsConsoleSrv.exe |
simHostSrv |
Sentinel Install Manager Service |
Sentinel Install Manager Service |
C:\Program Files (x86)\Sentinel System Monitor\Sentinel AIM\simHostSrv.exe |
Wonderware InTouch Access Anywhere |
Wonderware InTouch Access Anywhere |
Service1 |
C:\Program Files (x86)\Wonderware\InTouch Access Anywhere Server\WebServer\AccessAnywhere\InTouchAccessAnywhereService.exe |
Log data can be collected using the Event IDs of log entries or the event source names.
This table lists the Event IDs related to the AVEVA Historian Configuration service.
Event ID | Event text |
---|---|
13019 |
Historian configuration management service starting |
6058 |
Configuration manager started |
6104 |
Starting system (auto start) |
13044 |
Configuration service shutting down |
1669 |
Cannot initialize server for status object |
This sample event is produced with Event ID 6058 from Windows Event Log.
Log Name: Application
Source: Historian
Date: 10/27/2021 12:51:36 PM
Event ID: 6058
Task Category: Configuration
Level: Information
Keywords: Classic
User: N/A
Computer: WIN-5RU7GP5MI4V
Description:
Configuration manager started
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Historian" />
<EventID Qualifiers="0">6058</EventID>
<Level>4</Level>
<Task>7</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2021-10-27T19:51:36.273337900Z" />
<EventRecordID>50636</EventRecordID>
<Channel>Application</Channel>
<Computer>WIN-5RU7GP5MI4V</Computer>
<Security />
</System>
<EventData>
<Binary>3200200073003B00</Binary>
</EventData>
</Event>
This NXLog configuration uses the im_msvistalog module to read data with the listed Event IDs from the Application channel of Windows Event Log using Xpath filtering.
All listed Event IDs are related to the AVEVA Historian Configuration service. The Exec directive uses the to_json() procedure to format the output as JSON.
<Extension json>
Module xm_json
</Extension>
<Input eventlog>
Module im_msvistalog
#XML query to read Windows Event Logs based on EventID
<QueryXML>
<QueryList>
<Query Id="0" Path="Application">
<Select Path="Application">
*[System[(EventID=13019 or
EventID=6058 or EventID=6104 or
EventID=13044 or EventID=1669)]]
</Select>
</Query>
</QueryList>
</QueryXML>
# Conversion to JSON
Exec to_json();
</Input>
{
"EventTime": "2021-10-27T12:51:36.273337-07:00",
"Hostname": "WIN-5RU7GP5MI4V",
"Keywords": "36028797018963968",
"EventType": "INFO",
"SeverityValue": 2,
"Severity": "INFO",
"EventID": 6058,
"SourceName": "Historian",
"TaskValue": 7,
"RecordNumber": 50636,
"ExecutionProcessID": 0,
"ExecutionThreadID": 0,
"Channel": "Application",
"Message": "Configuration manager started",
"Category": "Configuration",
"Opcode": "Info",
"EventData.Binary": "3200200073003B00",
"EventReceivedTime": "2021-10-27T12:51:36.461050-07:00",
"SourceModuleName": "eventlog",
"SourceModuleType": "im_msvistalog"
}
As previously mentioned, NXLog can filter Windows Event Log entries by their event source names.
The following sample event was collected from the Historian log source.
Log Name: Application
Source: Historian
Date: 10/28/2021 4:04:39 AM
Event ID: 13019
Task Category: Configuration
Level: Information
Keywords: Classic
User: N/A
Computer: WIN-5RU7GP5MI4V
Description:
Historian configuration management service starting
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Historian" />
<EventID Qualifiers="0">13019</EventID>
<Level>4</Level>
<Task>7</Task>
<Keywords>0x80000000000000</Keywords>
<TimeCreated SystemTime="2021-10-28T11:04:39.409946300Z" />
<EventRecordID>51099</EventRecordID>
<Channel>Application</Channel>
<Computer>WIN-5RU7GP5MI4V</Computer>
<Security />
</System>
<EventData>
</EventData>
</Event>
This NXLog configuration uses the im_msvistalog module to read Windows Event Log entries. The QueryXML directive of this module specifies the Application channel and the following sources to filter data by:
-
AsbCertificateRenewalService
-
aaGR
-
Browser Client
-
Historian
-
InSQLIndexing
-
EricomAuthenticationServer
-
Ericom Authentication Server 9.2
-
Ericom Secure Gateway 9.2
-
InTouch IData Service
-
Service1
-
PSMS Agent
-
PSMS Console Service
-
Sentinel Install Manager Service
<Extension json>
Module xm_json
</Extension>
<Input eventlog>
Module im_msvistalog
# XML query to read Windows Event Log data
<QueryXML>
<QueryList>
<Query Id="0" Path="Application">
<Select Path="Application">*
[System[Provider[@Name='AsbCertificateRenewalService' or
@Name='aaGR' or
@Name='Browser Client' or
@Name='Historian' or
@Name='InSQLIndexing' or
@Name='EricomAuthenticationServer' or
@Name='Ericom Authentication Server 9.2' or
@Name='Ericom Secure Gateway 9.2' or
@Name='InTouch IData Service' or
@Name='Service1' or
@Name='PSMS Agent' or
@Name='PSMS Console Service' or
@Name='Sentinel Install Manager Service']]]
</Select>
</Query>
</QueryList>
</QueryXML>
# Conversion to JSON
Exec to_json();
</Input>
The following output sample shows the Historian
event source message processed and converted to JSON by NXLog.
{
"EventTime": "2021-10-28T04:04:39.409946-07:00",
"Hostname": "WIN-5RU7GP5MI4V",
"Keywords": "36028797018963968",
"EventType": "INFO",
"SeverityValue": 2,
"Severity": "INFO",
"EventID": 13019,
"SourceName": "Historian",
"TaskValue": 7,
"RecordNumber": 51099,
"ExecutionProcessID": 0,
"ExecutionThreadID": 0,
"Channel": "Application",
"Message": "Historian configuration management service starting",
"Category": "Configuration",
"Opcode": "Info",
"EventReceivedTime": "2021-10-28T04:04:39.753417-07:00",
"SourceModuleName": "eventlog",
"SourceModuleType": "im_msvistalog"
}
File-based logs
AVEVA System Platform produces several file-based logs that NXLog can parse and process. The following table lists the file-based logs and their locations.
Log type | File ext. |
Location |
---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
System logs
ArchestrA-related components send their messages to ArchestrA Logger, a background process that stores messages in the system log.
System logs are stored in files that use the system-specific file extension .aaLog
, which can be viewed with the ArchestrA Log Viewer.
The ArchestrA Log Viewer is part of the ArchestrA System Management Console (SMC) and saves logged messages to a file.
This is very useful since .aaLog
files can’t be processed directly.
System log files are located in the C:\ProgramData\ArchestrA\LogFiles
directory by default.
The location can be changed in the ArchestrA Log Viewer.
The ArchestrA Log Flag Editor can be used to customize logged messages.
The following example shows how to process the system log, which has been saved as
a .txt
file.
The following event samples are taken from the exported ArchestrA system log and represent the single-line and multiline events.
NXLog can parse and process the following fields:
-
Number
-
Date/Time
-
Process ID
-
Thread ID
-
Process Name
-
Component Name
-
Log Flag
-
Message
6302696 2021/11/03 15:23:20.356/-0:2AE4:37D8/adpHostSrv/adpHostSrv /Info - Unable to reach Sentinel Manager https://WIN-5RU7GP5MI4V:443/systemmonitor/api/psms/
This log file is designed to be human-readable.
The information after the Message
field will be added without parsing due to the high complexity and absence of a definite structure.
6302697 2021/11/03 15:23:20.356/-0:2AE4:0FEC/adpHostSrv/adpHostSrv /Trace - Exception Type: HttpRequestException
Message: An error occurred while sending the request.
Stack trace: at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at SentinelSystemMonitor.Utilities.Net.PsmsHttpClient.<GetAsync>d__a.MoveNext()
Inner Exception Message: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.
Inner Exception Stack trace: at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
This NXLog configuration uses regular expressions for parsing the fields.
The regular expression is defined by the SYS_REGEX
constant.
The LOG_PATH
constant stores the absolute path to the log file.
Since the log file is UTF-16LE encoded, the xm_charconv extension module enables the data to be read using the correct encoding.
The LineReader
directive sets the encoding type to be used.
The InputType
directive of the from_file
input instance references the xm_charconv extension instance by name charconv
.
The Exec block contains instructions to replace unwanted characters and parse the event using the SYS_REGEX
expression.
Each captured field is added to the event record per named capturing groups defined in the regular expression’s angle brackets (< >
).
The strptime() function is called to convert the timestamp captured as $2
to a datetime value that it assigns to the $EventTime
field.
The processed event record is formatted to JSON using the to_json() procedure of the xm_json module.
# Regular expressions defined to read the contents of log entries
define SYS_REGEX /(?x)^(?<No>\d+)\s+(\d+\/\d+\/\d+\s+\d+\:\d+\:\d+).*?\:\
(?<ProcessID>\w+)\:(?<ThreadID>\w+)\/(?<ProcessName>.*?)\/\
(?<Component>.*?)\s*\/(?<LogFlag>.*?)\s+\-\s+(?<Message>.*)/
# Part of the log path defined as a constant
define LOG_PATH C:\ProgramData\ArchestrA\LogFiles
<Extension json>
Module xm_json
</Extension>
<Extension charconv>
Module xm_charconv
LineReader UTF-16LE
</Extension>
<Input from_file>
Module im_file
File '%LOG_PATH%\LogExport*.txt'
InputType charconv
<Exec>
# Parsing multi-line messages with module variables
if $raw_event =~ /^\d+\s+\d+\/\d+\/\d+\s+\d+\:\d+\:\d+\.\d+/
{
if defined(get_var('saved'))
{
$tmp = $raw_event;
$raw_event = get_var('saved');
set_var('saved', $tmp);
delete($tmp);
}
else
{
set_var('saved', $raw_event);
drop();
}
}
else
{
set_var('saved', get_var('saved') + " >> " + $raw_event);
drop();
}
# Replaces unwanted characters
$raw_event =~ s/\s{2,}/ /g;
if $raw_event =~ %SYS_REGEX%
{
# Creates the timestamp
$EventTime = strptime($2, "%Y/%m/%d %T");
# Converts to JSON
to_json();
}
# Discard event if it doesn't match a/the regular expression
else drop();
</Exec>
</Input>
The following samples represent processed events in JSON format.
{
"EventReceivedTime": "2021-11-04T13:24:44.152760-07:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"Component": "adpHostSrv",
"LogFlag": "Info",
"Message": "Unable to reach Sentinel Manager https://WIN-5RU7GP5MI4V:443/systemmonitor/api/psms/",
"No": "6302696",
"ProcessID": "2AE4",
"ProcessName": "adpHostSrv",
"ThreadID": "37D8",
"EventTime": "2021-11-03T15:23:20.000000-07:00"
}
{
"EventReceivedTime": "2021-11-04T13:24:44.152760-07:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"Component": "adpHostSrv",
"LogFlag": "Trace",
"Message": "Exception Type: HttpRequestException >> Message: An error occurred while sending the request. >> Stack trace: at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) >> at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) >> at SentinelSystemMonitor.Utilities.Net.PsmsHttpClient.<GetAsync>d__a.MoveNext() >> Inner Exception Message: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. >> Inner Exception Stack trace: at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult) >> at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)",
"No": "6302697",
"ProcessID": "2AE4",
"ProcessName": "adpHostSrv",
"ThreadID": "0FEC",
"EventTime": "2021-11-03T15:23:20.000000-07:00"
}
ArchestrA Logger and Log Viewer logs
Messages related to ArchestrA Logger service activity are stored in the Logger.log
file.
ArchestrA Log Viewer activity is registered in the LogReader.log
file.
Both files are located in the C:\ProgramData\ArchestrA\Logger
directory and follow the same message structure, which means only a single configuration is needed to process them.
The following input sample is taken from LogReader.log
, which contains the following fields that can be parsed and processed:
-
Date/Time
-
Severity
-
Property
-
Command
-
Message
-
Start Index
-
End Index
-
Requested Count
-
Messages Count
11/10/2021 13:42.07.310 Info Reader:StartExport succeeded for C:\Users\Administrator\Desktop\LogExport11102021.txt
This NXLog configuration contains three constants, TYPE1_REGEX
, TYPE2_REGEX
, and TYPE3_REGEX
that store the regular expressions used for parsing these fields.
The fourth constant, LOG_PATH
, defines the absolute path to the log file directory.
In the Exec block of the im_file module, the conditional if
block determines if an event matches any of the regular expressions.
If an event does match, the parsedate() function is called to convert the captured timestamp to a datetime value that it assigns to the $EventTime
field.
The rest of the event is parsed and saved to fields according to the names of the named capturing groups defined in angle brackets (< >
) of the regular expressions.
The result is then converted to JSON using the to_json() procedure of the xm_json module. The drop procedure discards records that don’t match any of the regular expressions.
# Regular expressions defined as a constants to read the content of the logs
define TYPE1_REGEX /(?x)^(\d+\/\d+\/\d+\s+\d+\:\d+\.\d+)\.\d+\s+(?<Severity>\w+)\s+\
(?<Property>[\w\s]+)\:\s*(?<Message>.*)/
define TYPE2_REGEX /(?x)^(\d+\/\d+\/\d+\s+\d+\:\d+\.\d+)\.\d+\s+(?<Severity>\w+)\s+\
(?<Property>[\w\s]+)\:\s*(?<Command>\w+)\:\s*(?<Message>.*)/
define TYPE3_REGEX /(?x)^(\d+\/\d+\/\d+\s+\d+\:\d+\.\d+)\.\d+\s+(?<Severity>\w+)\s+\
(?<Property>[\w\s]+)\:\s*(?<Command>\w+)\s+\
StartIndex\s+(?<StartIndex>\d+)[\,\s]+EndIndex\s+(?<EndIndex>\d+)[\,\s]+\
Requested\s+Count\s+(?<RequestedCount>\d+)[\w\s]+\
Messages\s+Count\s+(?<MessagesCount>\d+)/
# Part of the log path defined as a constant
define LOG_PATH C:\ProgramData\ArchestrA\Logger
<Extension json>
Module xm_json
</Extension>
<Input from_file>
Module im_file
File '%LOG_PATH%\Log*.log'
<Exec>
# Matches the events with a regular expression
if $raw_event =~ %TYPE3_REGEX% or
$raw_event =~ %TYPE2_REGEX% or
$raw_event =~ %TYPE1_REGEX%
{
# Creates the timestamp
$EventTime = strptime($1, '%m/%d/%Y %H:%M.%S');
# Formats the result as JSON
to_json();
}
# Discard event if it doesn't match a/the regular expression
else drop();
</Exec>
</Input>
The following sample shows the input event after NXLog has processed it and converted it to JSON.
{
"EventReceivedTime": "2021-11-10T13:42:07.435240-08:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"Message": "StartExport succeeded for C:\\Users\\Administrator\\Desktop\\LogExport11102021.txt",
"Property": "Reader",
"Severity": "Info",
"EventTime": "2021-11-10T13:42:07.000000-08:00"
}
Historian Search logs
AVEVA Historian Search is a search process used to store and retrieve tags, saved content, and keywords.
The process produces several log files located in the C:\ProgramData\ArchestrA\HistorianSearch\Logs
directory.
HistorianSearch-x64.yyyy-mm-dd.log
logThe following input sample is taken from the HistorianSearch-x64.yyyy-mm-dd.log
and contains the set of fields that can be parsed and processed with NXLog:
-
Date/Time
-
Severity
-
Thread ID
-
Message
[2021-11-11 06:01:22] [info] [ 6120] Running 'HistorianSearch-x64' Service...
The following NXLog configuration uses the regular expression defined as a constant HISSRCH_REGEX
to parse input messages.
Another constant, LOG_PATH
, specifies the full path to the log file directory.
In the Exec block of the im_file module, each message is compared to HISSRCH_REGEX
.
In case of a match, all defined fields are created according to the named capturing groups of HISSRCH_REGEX
.
The captured timestamp data is converted to a datetime value using the parsedate() function and assigned to the $EventTime
field.
The result is converted to JSON using the to_json() procedure of the xm_json module.
The drop procedure discards records that do not match the HISSRCH_REGEX
regular expression.
# Regular expressions are defined as a constants to read the content of the logs
define HISSRCH_REGEX /(?x)^\[(\d+\-\d+\-\d+\s+\d+\:\d+\:\d+)[\[\]\s]+\
(?<Severity>\w+)[\[\]\s]+(?<TID>\d+)[\[\]\s]+(?<Message>.*)/
# Part of the log path defined as a constant
define LOG_PATH C:\ProgramData\ArchestrA\HistorianSearch\Logs
<Extension json>
Module xm_json
</Extension>
<Input from_file>
Module im_file
File '%LOG_PATH%\HistorianSearch-x64.*.log'
<Exec>
# Matches the events with a regular expression
if $raw_event =~ %HISSRCH_REGEX%
{
# Creates the timestamp
$EventTime = parsedate($1);
# Formats the result as JSON
to_json();
}
# Discard event if it doesn't match a/the regular expression
else drop();
</Exec>
</Input>
The following sample shows the input event after NXLog has processed it and converted it to JSON.
{
"EventReceivedTime": "2021-11-11T06:01:23.195020-08:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"Message": "Running 'HistorianSearch-x64' Service...",
"Severity": "info",
"TID": "6120",
"EventTime": "2021-11-11T06:01:22.000000-08:00"
}
The input sample below represents the structure of both
historiansearch-x64-stdout.yyyy-mm-dd.log
and
historiansearch-x64-stderr.yyyy-mm-dd.log
log files with the following fields:
-
Date/Time
-
Message
2021-11-11 06:36:25 Commons Daemon procrun stdout initialized
In this NXLog configuration, two constants are defined.
The HISSRCH_STD_REGEX
constant stores the regular expression for parsing fields from each message, while the LOG_PATH
is the absolute path to the log files.
In the Exec block of the im_file module, each message is compared to HISSRCH_STD_REGEX
.
If the message matches, the parsedate() function converts the captured timestamp to a datetime value and assigns it to the $EventTime
field.
All other fields are created using the named capturing groups.
The parsed fields are converted to JSON using the to_json() procedure of the xm_json module.
The drop procedure discards records that do not match the HISSRCH_STD_REGEX
regular expression.
# Regular expressions defined as a constants to read the content of the logs
define HISSRCH_STD_REGEX /(?x)^(\d+\-\d+\-\d+\s+\d+\:\d+\:\d+)\s+(?<Message>.*)/
# Part of the log path defined as a constant
define LOG_PATH C:\ProgramData\ArchestrA\HistorianSearch\Logs
<Extension json>
Module xm_json
</Extension>
<Input from_file>
Module im_file
File '%LOG_PATH%\historiansearch-x64*.log'
<Exec>
# Matches the events with a regular expression
if $raw_event =~ %HISSRCH_STD_REGEX%
{
# Creates the timestamp
$EventTime = parsedate($1);
# Formats the result as JSON
to_json();
}
# Discard event if it doesn't match a/the regular expression
else drop();
</Exec>
</Input>
The following sample shows the input event after NXLog has processed it and converted it to JSON.
{
"EventReceivedTime": "2021-11-11T06:36:26.057635-08:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"Message": "Commons Daemon procrun stderr initialized",
"EventTime": "2021-11-11T06:36:25.000000-08:00"
}
Historian Configuration Exporter error log
The Historian Database Export/Import Utility is a standalone utility that uses a text file for importing or exporting AVEVA Historian configuration information.
This utility keeps track of errors that occur during import and logs progress along with any errors it encounters to an error log file named aahDBDumpLog.txt
.
Its default location is C:\Users\<UserName>\Documents
.
The following input sample was taken from aahDBDumpLog.txt
.
2021/11/28 12:15:26.846 Connecting to Historian localhost
The following NXLog configuration defines two constants to avoid reusing lengthy expressions.
The first constant, HISCONFEXP_REGEX
, stores the regular expression used for parsing input messages.
The second constant, LOG_PATH
, specifies the absolute path to the log file.
The Exec block of the im_file module compares each message to HISCONFEXP_REGEX
.
Once a match occurs, the respective fields are created according to the named capturing groups.
The timestamp fields are converted to datetime using the strptime() function and assigned to the $EventTime
field.
The result is finally converted to JSON using the to_json() procedure of xm_json.
The drop procedure discards records that do not match the HISCONFEXP_REGEX
regular expression.
# Regular expressions defined as a constants to read the content of the logs
define HISCONFEXP_REGEX /(?x)^(\d+\/\d+\/\d+\s+\d+\:\d+\:\d+\.\d+)\s+\
(?<Message>\w+.*)/
# Part of the log path defined as a constant
define LOG_PATH C:\Users\Administrator\Documents
<Extension json>
Module xm_json
</Extension>
<Input from_file>
Module im_file
File '%LOG_PATH%\aahDBDumpLog.txt'
<Exec>
# Matches the events with a regular expression
if $raw_event =~ %HISCONFEXP_REGEX%
{
# Creates the timestamp
$EventTime = strptime($1, "%Y/%m/%d %T");
# Formats the result as JSON
to_json();
}
# Discard event if it doesn't match a/the regular expression
else drop();
</Exec>
</Input>
The following sample shows the input event after NXLog has processed it and converted it to JSON.
{
"EventReceivedTime": "2021-11-28T12:15:28.082729-08:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"Message": "Connecting to Historian localhost",
"EventTime": "2021-11-28T12:15:26.000000-08:00"
}
InTouch Access Anywhere logs
AVEVA InTouch Access Anywhere provides remote access to a running InTouch application through a secure HTML5-compliant web browser without any need for a separate client application.
InTouch Access Anywhere consists of the following components:
-
The InTouch Access Anywhere server (WebSocket server) includes a collection of web resources (HTML files, CSS, JavaScript, images, etc.). It is installed on the same Remote Desktop Services host where InTouch WindowViewer runs InTouch applications.
-
The Authentication Server authenticates InTouch Access Anywhere users before granting them access to InTouch applications.
-
The InTouch Access Anywhere Secure Gateway is an optional server installed separately on a computer in a DMZ to access InTouch applications protected by a firewall.
Log type | Location |
---|---|
Access Anywhere Server logs |
|
Access Anywhere Secure Gateway logs |
|
Access Anywhere Authentication Server logs |
|
The following input samples are taken from the Access Server.LOG
file, representing the data that can be parsed and processed with NXLog.
Every new instance of the Access Server log starts with a header containing current information about the software, context, process, and the log file itself.
**********************************************************************
C:/Program Files (x86)/Wonderware/InTouch Access Anywhere Server/Logs/Access Server.LOG
Started at : 21/12/06 23:47:42
Terminated at:
Elapsed time :
Software:
Execute: C:/Program Files (x86)/Wonderware/InTouch Access Anywhere Server/AccessServer64.exe
Version: 9.2.0.46379
Built : 9.2.0.46379.20191215-_Release_AccessServer920-
19/12/15 13:25:06
Context:
Account: \SYSTEM
Profile:
Machine: WORKGROUP\WIN-5RU7GP5MI4V
IP addr: 192.168.10.126
IP name: 192.168.10.126
OS ver.: Windows Server 2016 Standard, 10.0.14393, x64
CPU : Dual-Core
OrgName:
Process:
PID : 2840
Thread : 7760
CmdLine: "C:\Program Files (x86)\Wonderware\InTouch Access Anywhere Server\AccessServer64.exe"
NOTE: Date format is YY/MM/DD
**********************************************************************
Each message in Access Server.LOG
can consist of a single line or span multiple lines.
Since the log file is designed to be human-readable, the number of lines that comprise multiline events can vary making them difficult to parse.
The most effective approach is to parse each line separately as if it were a single-line event.
The following sample represents a multiline event.
21/12/06 23:47:43.504 | 7760 | Configuration was loaded from registry.
| Path: HKEY_LOCAL_MACHINE\SOFTWARE\Ericom Software\Access Server
The constants ACCESSSRV_HEADER_REGEX
, ACCESSSRV_REGEX
, ACCESSSRV_PROP_REGEX
, and ACCESSSRV_MSG_REGEX
in the NXLog configuration define the regular expressions for parsing messages.
The LOG_PATH
constant stores the absolute path to the log file.
The xm_multiline module is used to read the multiline header. The HeaderLine and the EndLine directives define the regular expressions to detect the header boundaries.
Separate Exec blocks are used for parsing and processing events: one in the from_file_multi
input instance for processing the header data of multiline events, and another in the from_file
input instance for parsing single-line events.
The first Exec
block removes redundant whitespace characters, as well as carriage return and line feed characters, in order to more effectively parse the header data.
Event records are then compared to their respective regular expressions in both blocks. If they match, the timestamps are converted to the datetime values using the strptime() function and assigned to their respective fields. The remaining fields are created according to the named capturing groups.
Processed event records are finally formatted to JSON by calling the to_json() procedure of the xm_json module or discarded by the drop() procedure if the input record doesn’t match any of the regular expressions.
# Regular expressions defined as a constants to read the content of the logs
define ACCESSSRV_HEADER_REGEX /(?x)^\*+\s*(?<LogFilePath>[A-Z]\:\/.*?)\s*Started\s*at[\s\:]*(.*?)\s*\
Terminated\s*at[\s\:]*(.*?)\s*Elapsed\s*time[\s\:]*(?<Elapsed_time>\d+\:\d+\:\d+)*\s*\
(?<Message>.*?)\s*Software[\s\:]*Execute[\s\:]*(?<SoftwareExecute>.*?)\s*\
Version[\s\:]*(?<SoftwareVersion>.*?)\s*Built[\s\:]*(?<SoftwareBuilt>[\s\S]*)\s*\
Context[\s\:]*Account[\s\:]*(?<Account>.*?)\s*Profile[\s\:]*(?<Profile>.*?)\s*\
Machine[\s\:]*(?<Machine>.*?)\s*IP\s*addr[\s\:]*(?<IP_addr>.*?)\s*\
IP\s*name[\s\:]*(?<IP_name>.*?)\s*OS\s*ver\.[\s\:]*(?<OS_ver>.*?)\s*\
CPU[\s\:]*(?<CPU>.*?)\s*OrgName[\s\:]*(?<OrgName>.*?)\s*Process[\s\:]*\
PID[\s\:]*(?<PID>.*?)\s*Thread[\s\:]*(?<Thread>.*?)\s*CmdLine[\s\:]*(?<CmdLine>.*?)\s*\
NOTE[\s\:]*(?<NOTE>.*?)\s*\*+/
define ACCESSSRV_REGEX /(?x)^(\d+.\d+.\d+\s+\d+.\d+.\d+)\.\d+[\s\|]*(?<TID>\d+)[\s\|]*(?<Message>[^~*]*)/
define ACCESSSRV_PROP_REGEX /(?x)^\s+\|\s*(?<Property>.*?)\s*\:\s+(?<Value>.*)/
define ACCESSSRV_MSG_REGEX /(?x)^\s+\|\s*(?<Message>[^~*]*)/
# Part of the log path defined as a constant
define LOG_PATH C:\Program Files (x86)\Wonderware\InTouch Access Anywhere Server\logs
<Extension json>
Module xm_json
</Extension>
<Extension multiline_header>
Module xm_multiline
# Regular expression to look for the header of the message
HeaderLine /^\*+/
EndLine /^\*+/
</Extension>
<Input from_file_multi>
Module im_file
File '%LOG_PATH%\Access Server.LOG'
InputType multiline_header
<Exec>
$raw_event = replace($raw_event, "\r", "");
$raw_event = replace($raw_event, "\n", "");
$raw_event =~ s/\s{2,}/ /g;
# Matches the events with a regular expression
if $raw_event =~ %ACCESSSRV_HEADER_REGEX%
{
# Creates the timestamp
$Started_at = strptime($2, "%y/%m/%d %T");
$Terminated_at = strptime($3, "%y/%m/%d %T");
# Formats the result as JSON
to_json();
}
# Discard event if it doesn't match a/the regular expression
else drop();
</Exec>
</Input>
<Input from_file>
Module im_file
File '%LOG_PATH%\Access Server.LOG'
<Exec>
if $raw_event =~ %ACCESSSRV_REGEX%
{
# Creates the timestamp
$EventTime = strptime($1, "%y/%m/%d %T");
# Formats the result as JSON
to_json();
}
else if $raw_event =~ %ACCESSSRV_PROP_REGEX% or
$raw_event =~ %ACCESSSRV_MSG_REGEX%
{
# Formats the result as JSON
to_json();
}
# Discard event if it doesn't match a/the regular expression
else drop();
</Exec>
</Input>
The following samples show the input events after NXLog has processed them and converted them to JSON.
{
"EventReceivedTime": "2021-12-06T23:47:43.359141-08:00",
"SourceModuleName": "from_file_multi",
"SourceModuleType": "im_file",
"Account": "\\SYSTEM",
"CPU": "Dual-Core",
"CmdLine": "\"C:\\Program Files (x86)\\Wonderware\\InTouch Access Anywhere Server\\AccessServer64.exe\"",
"Elapsed_time": "",
"IP_addr": "192.168.10.126",
"IP_name": "192.168.10.126",
"LogFilePath": "C:/Program Files (x86)/Wonderware/InTouch Access Anywhere Server/Logs/Access Server.LOG",
"Machine": "WORKGROUP\\WIN-5RU7GP5MI4V",
"Message": "",
"NOTE": "Date format is YY/MM/DD",
"OS_ver": "Windows Server 2016 Standard, 10.0.14393, x64",
"OrgName": "",
"PID": "2840",
"Profile": "",
"SoftwareBuilt": "9.2.0.46379.20191215-_Release_AccessServer920- 19/12/15 13:25:06",
"SoftwareExecute": "C:/Program Files (x86)/Wonderware/InTouch Access Anywhere Server/AccessServer64.exe",
"SoftwareVersion": "9.2.0.46379",
"Thread": "7760",
"Started_at": "2021-12-06T23:47:42.000000-08:00",
"Terminated_at": null
}
{
"EventReceivedTime": "2021-12-06T23:47:44.374534-08:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"Message": "Configuration was loaded from registry.",
"TID": "7760",
"EventTime": "2021-12-06T23:47:43.000000-08:00"
}
{
"EventReceivedTime": "2021-12-06T23:47:44.374534-08:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"Property": "Path",
"Value": "HKEY_LOCAL_MACHINE\\SOFTWARE\\Ericom Software\\Access Server"
}
The License Manager log consists of the header data and events as well as the Access Server log considered above. The processing approach is similar to the Access Server log.
The following samples are taken from LicenseManager.log
and show the header along with an event.
*******************************************************************************
C:/Program Files (x86)/Wonderware/InTouch Access Anywhere Server/logs/LicenseManager.log
Started at: Fri Dec 10 09:28:30 2021
Software:
Execute: C:\Program Files (x86)\Wonderware\InTouch Access Anywhere Server\LicenseServer.exe
32 bit executable.
Version: 9.2.0.46379
Build: 20191215-_Release_AccessServer920-
Build Time: Sun Dec 15 14:25:18 2019
Context:
Account:
Machine: WIN-5RU7GP5MI4V
IP Addresses: 192.168.10.127, 127.0.0.1,
OS ver.: Windows Server 2016 Standard, 10.0.14393, x64
Log File Settings:
Max Log Files: 100
Max File Size KB: 16384
Log Level: 1
*******************************************************************************
2021/12/10 09:43:40 | 4840 | [:1] File C:\Program Files (x86)\Wonderware\InTouch Access Anywhere Server\licenses\Blaze.lf loaded with status 2
The constants ACSSRV_LICHDR_REGEX
, ACSSRV_LIC_REGEX
, ACSSRV_LICPROP_REGEX
, ACSSRV_LICMSG_REGEX
, and LOG_PATH
in the NXLog configuration define the regular expressions for parsing messages and the absolute path to the log file, respectively.
The xm_multiline reads the multiline header. The HeaderLine and the EndLine directives define the regular expressions to identify the header boundaries.
Separate Exec blocks are used for parsing and processing events: one in the from_file_multi
input instance for processing the header data of multiline events, and another in the from_file
input instance for parsing single-line events.
The first Exec
block removes redundant whitespace characters, as well as carriage return and line feed characters, in order to more effectively parse the header data.
Event records are compared to the respective regular expressions in both blocks. If a match occurs, the timestamp fields are converted to datetime using the parsedate() function or the strptime() function if a custom date/time format is used. The remaining fields are created according to the named capturing groups.
Processed event records are finally formatted to JSON using the to_json() procedure of the xm_json module or discarded by the drop() procedure if the input record doesn’t match any of the regular expressions.
# Regular expressions defined as a constants to read the content of the logs
define ACSSRV_LICHDR_REGEX /(?x)^\*+\s*(?<LogFilePath>[A-Z]\:\/.*?)\s*Started\s*at[\s\:]*(.*?)\s*\
Software[\s\:]*Execute[\s\:]*(?<Execute>.*?)\s*Version[\s\:]*(?<Version>.*?)\s*\
Build[\s\:]*(?<Build>.*?)\s*Build\s*Time[\s\:]*(.*?)\s*\
Context[\s\:]*Account[\s\:]*(?<Account>.*?)\s*Machine[\s\:]*(?<Machine>.*?)\s*\
IP\s*Addresses[\s\:]*(?<IPAddresses>.*?)\s*OS\s*ver[\.\s\:]*(?<OSVersion>.*?)\s*\
Log\s*File\s*Settings[\s\:]*Max\s*Log\s*Files[\s\:]*(?<MaxLogFiles>.*?)\s*\
Max\s*File\s*Size\s*KB[\s\:]*(?<MaxFileSize_KB>.*?)\s*\
Log\s*Level[\s\:]*(?<LogLevel>.*?)\s*\*+/
define ACSSRV_LIC_REGEX /(?x)^(\d+.\d+.\d+\s+\d+.\d+.\d+)[\s\|]*(?<TID>\d+)[\s\|]*(?<Message>[^~*]*)/
define ACSSRV_LICPROP_REGEX /(?x)^\s+\|\s*(?<Property>.*?)\s*\:\s+(?<Value>.*)/
define ACSSRV_LICMSG_REGEX /(?x)^\s+\|\s*(?<Message>[^~*]*)/
# Part of the log path defined as a constant
define LOG_PATH C:\Program Files (x86)\Wonderware\InTouch Access Anywhere Server\logs
<Extension json>
Module xm_json
</Extension>
<Extension multiline_header>
Module xm_multiline
# Regular expression to look for the header of the message
HeaderLine /^\*+/
EndLine /^\*+/
</Extension>
<Input from_file_multi>
Module im_file
File '%LOG_PATH%\LicenseManager.log'
InputType multiline_header
<Exec>
$raw_event = replace($raw_event, "\r\n", " ");
$raw_event = replace($raw_event, "\t", " ");
$raw_event =~ s/\s{2,}/ /g;
# Matches the events with a regular expression
if $raw_event =~ %ACSSRV_LICHDR_REGEX%
{
# Creates the timestamp
$Started_at = parsedate($2);
$BuildTime = parsedate($6);
# Formats the result as JSON
to_json();
}
# Discard event if it doesn't match a/the regular expression
else drop();
</Exec>
</Input>
<Input from_file>
Module im_file
File '%LOG_PATH%\LicenseManager.log'
<Exec>
if $raw_event =~ %ACSSRV_LIC_REGEX%
{
# Creates the timestamp
$EventTime = strptime($1, "%Y/%m/%d %T");
# Formats the result as JSON
to_json();
}
else if $raw_event =~ %ACSSRV_LICPROP_REGEX% or
$raw_event =~ %ACSSRV_LICMSG_REGEX%
{
# Formats the result as JSON
to_json();
}
# Discard event if it doesn't match a/the regular expression
else drop();
</Exec>
</Input>
The following samples shows the input events after NXLog has processed them and converted them to JSON.
{
"EventReceivedTime": "2021-12-10T09:30:31.820051-08:00",
"SourceModuleName": "from_file_multi",
"SourceModuleType": "im_file",
"Account": "",
"Build": "20191215-_Release_AccessServer920-",
"Execute": "C:\\Program Files (x86)\\Wonderware\\InTouch Access Anywhere Server\\LicenseServer.exe 32 bit executable.",
"IPAddresses": "192.168.10.127, 127.0.0.1,",
"LogFilePath": "C:/Program Files (x86)/Wonderware/InTouch Access Anywhere Server/logs/LicenseManager.log",
"LogLevel": "1",
"Machine": "WIN-5RU7GP5MI4V",
"MaxFileSize_KB": "16384",
"MaxLogFiles": "100",
"OSVersion": "Windows Server 2016 Standard, 10.0.14393, x64",
"Version": "9.2.0.46379",
"Started_at": "2021-12-10T09:28:30.000000-08:00",
"BuildTime": "2019-12-15T14:25:18.000000-08:00"
}
{
"EventReceivedTime": "2021-12-10T09:43:40.894013-08:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"Message": "[:1] File C:\\Program Files (x86)\\Wonderware\\InTouch Access Anywhere Server\\licenses\\Blaze.lf loaded with status 2",
"TID": "4840",
"EventTime": "2021-12-10T09:43:40.000000-08:00"
}
Access Server Errors.csv
logIn rare cases, log events can span several lines.
Thus the NXLog configuration is designed to be able to process multiline events.
The following input sample represents the message structure of the Access Server Errors.csv
log file.
Date & Time,Error ID,Error Type,Sub-Error ID,Error Description,Error Location,Elapsed from Start,Delta Commited VM,Delta Reserved VM,Delta Total VM,Commited VM,Reserved VM,Total VM,Biggest VM Fragment (MB),Delta Working Set,Working Set,Delta Kernel CPU,Delta Kernel CPU %,Delta User CPU,Delta User CPU %,Delta CPU,Delta CPU %,Kernel CPU %,User CPU,User CPU %,CPU,CPU %,USERobjs,GDIobjs,Handles,Threads,Suspended / Canceled Threads,Occur. Count,Prev. Occur. D&T,Elapsed from Prev. Occur.,"C:/Program Files (x86)/Wonderware/InTouch Access Anywhere Server/AccessServer64.exe ver. 9.2.0.46379 (19/12/15 14:25:06)"
"2021.12.20 06:01:15.211"," 0",""," 0","Buffer OnCompletionFailed #5, #995. ",""," 0:01:23"," 536","18446744073709551500"," 420"," 151852","2147503788","2147655640","129871074"," 156"," 32292","0","0.104%","0","0.000%","0","0.104%","","0.262%","","0.300%","","0.563%"," 0"," 4"," 390"," 27",""," 1"
Regular expressions and the log file directory path are defined as constants ACSSRV_ERR_REGEX
and LOG_PATH
, respectively, to make the configuration more readable.
The xm_multiline extension module processes multiline events as a single message. The HeaderLine directive of this module specifies a regular expression that determines the multiline event boundary.
The Exec block of the input module instance compares each event sample with the corresponding regular expression. Once a match occurs, the respective fields are created according to the named capturing groups.
The captured group $1
string is converted to datetime using the strptime(), which is then assigned to the $EventTime
field.
If the event sample does not match the regular expression, it is discarded using the drop() procedure.
# Regular expressions defined as a constants to read the content of the logs
define ACSSRV_ERR_REGEX /(?x)^(?:\"\s*(\d+\.\d+\.\d+\s+\d+\:\d+\:\d+)\.\d+\")\,\
(?:\"\s*(?<ErrorID>.*?)\")\,(?:\"\s*(?<ErrorType>.*?)\")\,\
(?:\"\s*(?<SubErrorID>.*?)\")\,(?:\"\s*(?<ErrorDescription>.*?)\")\,\
(?:\"\s*(?<ErrorLocation>.*?)\")\,(?:\"\s*(?<ElapsedFromStart>.*?)\")\,\
(?:\"\s*(?<DeltaCommitedVM>.*?)\")\,(?:\"\s*(?<DeltaReservedVM>.*?)\")\,\
(?:\"\s*(?<DeltaTotalVM>.*?)\")\,(?:\"\s*(?<CommitedVM>.*?)\")\,\
(?:\"\s*(?<ReservedVM>.*?)\")\,(?:\"\s*(?<TotalVM>.*?)\")\,\
(?:\"\s*(?<BiggestVMFragment_MB>.*?)\")\,(?:\"\s*(?<DeltaWorkingSet>.*?)\")\,\
(?:\"\s*(?<WorkingSet>.*?)\")\,(?:\"\s*(?<DeltaKernelCPU>.*?)\")\,\
(?:\"\s*(?<DeltaKernelCPU_pct>.*?)\")\,(?:\"\s*(?<DeltaUserCPU>.*?)\")\,\
(?:\"\s*(?<DeltaUserCPU_pct>.*?)\")\,(?:\"\s*(?<DeltaCPU>.*?)\")\,\
(?:\"\s*(?<DeltaCPU_pct>.*?)\")\,(?:\"\s*(?<KernelCPU>.*?)\")\,\
(?:\"\s*(?<KernelCPU_pct>.*?)\")\,(?:\"\s*(?<UserCPU>.*?)\")\,\
(?:\"\s*(?<UserCPU_pct>.*?)\")\,(?:\"\s*(?<CPU>.*?)\")\,\
(?:\"\s*(?<CPU_pct>.*?)\")\,(?:\"\s*(?<USERobjs>.*?)\")\,\
(?:\"\s*(?<GDIobjs>.*?)\")\,(?:\"\s*(?<Handles>.*?)\")\,\
(?:\"\s*(?<Threads>.*?)\")\,(?:\"\s*(?<SuspendedCanceled_Threads>.*?)\")\,\
(?:\"\s*(?<OccurCount>.*?)\")\,*(?:\"\s*(?<PrevOccurDT>.*?)\")*\,*\
(?:\"\s*(?<ElapsedFromPrevOccur>.*?)\")*/
# Part of the log path defined as a constant
define LOG_PATH C:\Program Files (x86)\Wonderware\InTouch Access Anywhere Server\logs
<Extension json>
Module xm_json
</Extension>
<Extension multiline>
Module xm_multiline
# Regular expression to look for the header of the message
HeaderLine /^\"\d+\.\d+\.\d+\s+\d+\:\d+\:\d+\.\d+\"\,\"\s*\d+\"/
</Extension>
<Input from_file>
Module im_file
File '%LOG_PATH%\Access Server Errors.csv'
InputType multiline
<Exec>
# Replaces unwanted characters
$raw_event = replace($raw_event, "\r", "");
$raw_event = replace($raw_event, "\n", "");
$raw_event =~ s/\s{2,}/ /g;
</Exec>
<Exec>
if $raw_event =~ %ACSSRV_ERR_REGEX%
{
# Creates the timestamp
$EventTime = strptime($1, "%Y.%m.%d %T");
# Formats the result as JSON
to_json();
}
# Discard event if it doesn't match a/the regular expression
else drop();
</Exec>
</Input>
The following output sample depicts the processed events in JSON format.
{
"EventReceivedTime": "2021-12-20T06:01:16.551982-08:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"BiggestVMFragment_MB": "129871074",
"CPU": "",
"CPU_pct": "0.563%",
"CommitedVM": "151852",
"DeltaCPU": "0",
"DeltaCPU_pct": "0.104%",
"DeltaCommitedVM": "536",
"DeltaKernelCPU": "0",
"DeltaKernelCPU_pct": "0.104%",
"DeltaReservedVM": "18446744073709551500",
"DeltaTotalVM": "420",
"DeltaUserCPU": "0",
"DeltaUserCPU_pct": "0.000%",
"DeltaWorkingSet": "156",
"ElapsedFromStart": "0:01:23",
"ErrorDescription": "Buffer OnCompletionFailed #5, #995. ",
"ErrorID": "0",
"ErrorLocation": "",
"ErrorType": "",
"GDIobjs": "4",
"Handles": "390",
"KernelCPU": "",
"KernelCPU_pct": "0.262%",
"OccurCount": "1",
"ReservedVM": "2147503788",
"SubErrorID": "0",
"SuspendedCanceled_Threads": "",
"Threads": "27",
"TotalVM": "2147655640",
"USERobjs": "0",
"UserCPU": "",
"UserCPU_pct": "0.300%",
"WorkingSet": "32292",
"EventTime": "2021-12-20T06:01:15.000000-08:00"
}
Both InTouch Access Anywhere Secure Gateway and InTouch Access Anywhere Authentication Server collect their system events and store them in their designated log files, respectively:
-
EricomSecureGateway_xx.logX
-
EricomAuthenticationServer_xx.logX
These files can be viewed and exported as .csv
files using the integrated TracerX-Viewer
.
This example shows how to process these exported versions of the log files.
Both log files have the same message structure, which consists of the following fields:
-
Session
-
Line Number
-
Level
-
Logger
-
Thread Number
-
Thread Name
-
Date/Time
-
Method
-
Text
The following input sample is taken from EricomSecureGateway_xx.logX
.
1,4,Info,Service Control,1,ESG Main Thread," 12/22/21 10:36:19.791",,"Service started."
The following NXLog configuration defines several constants to improve readability.
The first constant, ERICOM_REGEX
, specifies the Regular expression to parse incoming messages, while the second constant stores the path to the log file directory.
To adapt the current configuration to process events from EricomAuthenticationServer_xx.logX
, specify the current location path to the exported version of this log file in the LOG_PATH
constant definition line.
Each incoming message is compared to ERICOM_REGEX
in the Exec block of the im_file module.
If it matches, all fields will be created according to the named capturing groups of ERICOM_REGEX
.
The event’s original timestamp is captured and converted to a datetime value using the strptime() function.
The parsed fields are converted to JSON using the to_json() procedure of the xm_json module. If an event does not match the regular expression, it is discarded using the drop() procedure.
# Regular expressions defined as a constants to read the content of the logs
define ERICOM_REGEX /(?x)^(?<Session>\d+)\,(?<LineNumber>[\d.]+)\,(?<Level>\w+)\,\
(?<Logger>.*?)\,(?<ThreadNumber>\d+)\,(?<ThreadName>.*?)\,\
[\"\s]*(\d+\/\d+\/\d+\s*\d+\:\d+\:\d+)\.\d+\"*\,\
[\"\s]*(?<Method>.*?)\"*\,\"\s*(?<Text>.*?)\"/
# Part of the log path defined as a constant
define LOG_PATH C:\Program Files (x86)\Wonderware\InTouch Access Anywhere Secure Gateway\InTouch Access Anywhere Secure Gateway\Logs
<Extension json>
Module xm_json
</Extension>
<Input from_file>
Module im_file
File '%LOG_PATH%\Ericom*.csv'
<Exec>
# Matches the events with a regular expression
if $raw_event =~ %ERICOM_REGEX%
{
# Creates the timestamp
$EventTime = strptime($7, "%m/%d/%y %T");
# Formats the result as JSON
to_json();
}
# Discard event if it doesn't match a/the regular expression
else drop();
</Exec>
</Input>
The following output sample depicts the processed events in JSON format.
{
"EventReceivedTime": "2021-12-22T11:37:57.433642-08:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"Level": "Info",
"LineNumber": "4",
"Logger": "Service Control",
"Method": "",
"Session": "1",
"Text": "Service started.",
"ThreadName": "ESG Main Thread",
"ThreadNumber": "1",
"EventTime": "2021-12-22T10:36:19.000000-08:00"
}
This example show how to process the following log files:
-
C:\Program Files (x86)\Wonderware\InTouch Access Anywhere Server\Logs
-
Access Server System Information.csv
-
Access Server CommLog.csv
-
-
C:\Program Files (x86)\Wonderware\InTouch Access Anywhere Secure Gateway\InTouch Access Anywhere Secure Gateway\Logs
-
EricomSecureGateway System Information_xx.CSV`
-
-
C:\Program Files (x86)\Wonderware\InTouch Access Anywhere Secure Gateway\InTouch Access Anywhere Authentication Server\Logs
-
EricomAuthenticationServer System Information_xx.CSV
-
These log files exhibit a similar event structure and can be handled in a single example.
To adapt the NXLog configuration to each particular log file, it is necessary to:
-
Update the existing regular expression with the appropriate one.
-
Update the
LOG_PATH
constant with the correct log file location path. -
Update the
File
directive of the input module with the correct log file name. -
Update the
if
statement in the input module with the correct constant name for the regular expression.
The following input sample is taken from the Access Server System Information.csv
file.
Date & Time,Elapsed from Start,CPU USAGE >>>,Delta Kernel CPU,Delta Kernel CPU %,Delta User CPU,Delta User CPU %,Delta CPU,Delta CPU %,Kernel CPU,Kernel CPU %,User CPU,User CPU %,CPU,CPU %,MEMORY >>>,Delta Commited VM,Delta Reserved VM,Delta Total VM,Commited VM,Reserved VM,Total VM,Biggest VM Fragment (MB),Delta Working Set,Working Set,Delta Page Faults,Page Faults,QuotaPagedPoolUsage,QuotaNonPagedPoolUsage,PagefileUsage,SYSTEM RESOURCES >>>,USERobjs,GDIobjs,Handles,Threads,Suspended / Canceled Threads,Memory / Threads,Free Disk Space,Error
"2021.12.07 15:46:50.710"," 0:02:00",,"0","0.052%","0","0.000%","0","0.052%","","0.252%","","0.213%"," 0:00:01","0.465%",," 84"," 1964"," 2048"," 130924","2147504264","2147635188","130336799"," 64"," 18864"," 1993"," 97572"," 227928"," 28944"," 17694720",," 0"," 4"," 345"," 26",""," 82601353","64370171904",""
This NXLog configuration has two constants, ACSSRV_SYSINFO_REGEX
and LOG_PATH
, that define the regular expression and the absolute path to the log file directory, respectively.
The Exec block of the im_file module compares each message to the regular expression. The respective fields are created according to the named capturing groups.
Captured group $1
contains an event timestamp in a custom date/time format.
Therefore it is converted to datetime using the strptime() function and then assigned to the $EventTime
field.
The result is converted to JSON using the to_json() procedure of xm_json. Messages are discarded using the drop() procedure if the match fails.
# Regular expressions defined as a constants to read the content of the logs
define ACSSRV_SYSINFO_REGEX /(?x)^(?:\"\s*(\d+\.\d+\.\d+\s+\d+\:\d+\:\d+)\.\d+\")\,\
(?:\"\s*(?<ElapsedFromStart>\d+\:\d+\:\d+)\")\,+\
(?:\"\s*(?<CPU_USAGE_DeltaKernelCPU>.*?)\")\,(?:\"\s*(?<CPU_USAGE_DeltaKernelCPU_pct>.*?)\")\,\
(?:\"\s*(?<CPU_USAGE_DeltaUserCPU>.*?)\")\,(?:\"\s*(?<CPU_USAGE_DeltaUserCPU_pct>.*?)\")\,\
(?:\"\s*(?<CPU_USAGE_DeltaCPU>.*?)\")\,(?:\"\s*(?<CPU_USAGE_DeltaCPU_pct>.*?)\")\,\
(?:\"\s*(?<CPU_USAGE_KernelCPU>.*?)\")\,(?:\"\s*(?<CPU_USAGE_KernelCPU_pct>.*?)\")\,\
(?:\"\s*(?<CPU_USAGE_UserCPU>.*?)\")\,(?:\"\s*(?<CPU_USAGE_UserCPU_pct>.*?)\")\,\
(?:\"\s*(?<CPU_USAGE_CPU>.*?)\")\,(?:\"\s*(?<CPU_USAGE_CPU_pct>.*?)\")\,+\
(?:\"\s*(?<MEMORY_DeltaCommitedVM>.*?)\")\,(?:\"\s*(?<MEMORY_DeltaReservedVM>.*?)\")\,\
(?:\"\s*(?<MEMORY_DeltaTotalVM>.*?)\")\,(?:\"\s*(?<MEMORY_CommitedVM>.*?)\")\,\
(?:\"\s*(?<MEMORY_ReservedVM>.*?)\")\,(?:\"\s*(?<MEMORY_TotalVM>.*?)\")\,\
(?:\"\s*(?<MEMORY_BiggestVMFragment_MB>.*?)\")\,(?:\"\s*(?<MEMORY_DeltaWorkingSet>.*?)\")\,\
(?:\"\s*(?<MEMORY_WorkingSet>.*?)\")\,(?:\"\s*(?<MEMORY_DeltaPageFaults>.*?)\")\,\
(?:\"\s*(?<MEMORY_PageFaults>.*?)\")\,(?:\"\s*(?<MEMORY_QuotaPagedPoolUsage>.*?)\")\,\
(?:\"\s*(?<MEMORY_QuotaNonPagedPoolUsage>.*?)\")\,(?:\"\s*(?<MEMORY_PagefileUsage>.*?)\")\,+\
(?:\"\s*(?<SYSRES_USERobjs>.*?)\")\,(?:\"\s*(?<SYSRES_GDIobjs>.*?)\")\,\
(?:\"\s*(?<SYSRES_Handles>.*?)\")\,(?:\"\s*(?<SYSRES_Threads>.*?)\")\,\
(?:\"\s*(?<SYSRES_SuspendedCanceled_Threads>.*?)\")\,(?:\"\s*(?<SYSRES_Memory_Threads>.*?)\")\,\
(?:\"\s*(?<SYSRES_FreeDiskSpace>.*?)\")\,(?:\"\s*(?<SYSRES_Error>.*?)\")/
# Part of the log path defined as a constant
define LOG_PATH C:\Program Files (x86)\Wonderware\InTouch Access Anywhere Server\logs
<Extension json>
Module xm_json
</Extension>
<Input from_file>
Module im_file
File '%LOG_PATH%\Access Server System Information.csv'
<Exec>
if $raw_event =~ %ACSSRV_SYSINFO_REGEX%
{
# Creates the timestamp
$EventTime = strptime($1, "%Y.%m.%d %T");
# Formats the result as JSON
to_json();
}
# Discard event if it doesn't match a/the regular expression
else drop();
</Exec>
</Input>
The output sample below depicts the processed messages in JSON format.
{
"EventReceivedTime": "2021-12-07T15:46:51.178596-08:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"CPU_USAGE_CPU": "0:00:01",
"CPU_USAGE_CPU_pct": "0.465%",
"CPU_USAGE_DeltaCPU": "0",
"CPU_USAGE_DeltaCPU_pct": "0.052%",
"CPU_USAGE_DeltaKernelCPU": "0",
"CPU_USAGE_DeltaKernelCPU_pct": "0.052%",
"CPU_USAGE_DeltaUserCPU": "0",
"CPU_USAGE_DeltaUserCPU_pct": "0.000%",
"CPU_USAGE_KernelCPU": "",
"CPU_USAGE_KernelCPU_pct": "0.252%",
"CPU_USAGE_UserCPU": "",
"CPU_USAGE_UserCPU_pct": "0.213%",
"ElapsedFromStart": "0:02:00",
"MEMORY_BiggestVMFragment_MB": "130336799",
"MEMORY_CommitedVM": "130924",
"MEMORY_DeltaCommitedVM": "84",
"MEMORY_DeltaPageFaults": "1993",
"MEMORY_DeltaReservedVM": "1964",
"MEMORY_DeltaTotalVM": "2048",
"MEMORY_DeltaWorkingSet": "64",
"MEMORY_PageFaults": "97572",
"MEMORY_PagefileUsage": "17694720",
"MEMORY_QuotaNonPagedPoolUsage": "28944",
"MEMORY_QuotaPagedPoolUsage": "227928",
"MEMORY_ReservedVM": "2147504264",
"MEMORY_TotalVM": "2147635188",
"MEMORY_WorkingSet": "18864",
"SYSRES_Error": "",
"SYSRES_FreeDiskSpace": "64370171904",
"SYSRES_GDIobjs": "4",
"SYSRES_Handles": "345",
"SYSRES_Memory_Threads": "82601353",
"SYSRES_SuspendedCanceled_Threads": "",
"SYSRES_Threads": "26",
"SYSRES_USERobjs": "0",
"EventTime": "2021-12-07T15:46:50.000000-08:00"
}
The following are regular expressions for parsing events from the corresponding log files.
Access Server CommLog.csv
define ACSSRV_ERR_REGEX /(?x)^(?:["\s]*(\d+\.\d+\.\d+\s+\d+\:\d+\:\d+)\.\d+\")\,\
(?<Id>.*?)\,(?<Protocol>.*?)\,(?<State>.*?)\,\s*(?<Elapsed>[\d:.]*)\,\,\
["\s]*(?<ClntSrv_LocalAddress>[\d.:]*)\"*\,["\s]*(?<ClntSrv_RemoteAddress>[\d\.\:]*)\"*\,\
(?<ClntSrv_BufferSize_KB>\d+)\,["\s]*(?<ClntSrv_SendBuffers>.*?)\"*\,\
["\s]*(?<ClntSrv_StopReceivingCount>.*?)\"*\,["\s]*(?<ClntSrv_MaxStopReceivingTicks>.*?)\"*\,\
["\s]*(?<ClntSrv_AvgStopReceivingTicks>.*?)\"*\,["\s]*(?<ClntSrv_HighStopReceivingCount>.*?)\"*\,\
["\s]*(?<ClntSrv_PacketsReceived>.*?)\"*\,["\s]*(?<ClntSrv_BytesReceived>\d+\,\d+|)\"*\,\
["\s]*(?<ClntSrv_ReceivedPacketsPerSecond>.*?)\"*\,["\s]*(?<ClntSrv_ReceivedBytesPerSecond>.*?)\"*\,\
["\s]*(?<ClntSrv_MaxReceivedPacket>\d+\,\d+|)\"*\,["\s]*(?<ClntSrv_AvgReceivedPacket>.*?)\"*\,\
["\s]*(?<ClntSrv_MaxReceivedAckTicks>.*?)\"*\,["\s]*(?<ClntSrv_HighReceivedAckTicksCnt>.*?)\"*\,\
["\s]*(?<ClntSrv_LastReceivedIssuedTime>[\d\s.:]*)\"*\,["\s]*(?<ClntSrv_LastReceivedAckTime>[\d\s.:]*)\"*\,\
["\s]*(?<ClntSrv_PacketsSent>.*?)\"*\,["\s]*(?<ClntSrv_BytesSent>\d+\,\d+|)\"*\,\
["\s]*(?<ClntSrv_SentPacketsPerSecond>.*?)\"*\,["\s]*(?<ClntSrv_SentBytesPerSecond>.*?)\"*\,\
["\s]*(?<ClntSrv_MaxSentPacket>\d+\,\d+|)\"*\,["\s]*(?<ClntSrv_AvgSentPacket>.*?)\"*\,\
["\s]*(?<ClntSrv_MaxSentAckTicks>.*?)\"*\,["\s]*(?<ClntSrv_HighSentAckTicksCount>.*?)\"*\,\
["\s]*(?<ClntSrv_LastSentIssuedTime>[\d\s.:]*)\"*\,["\s]*(?<ClntSrv_LastSentAckTime>[\d\s.:]*)\"*\,\
["\s]*(?<ClntSrv_CompressionRatio>.*?)\"*\,["\s]*(?<ClntSrv_ConcatenationRatio>.*?)\"*\,\,\
["\s]*(?<SrvClnt_LocalAddress>[\d.:]*)\"*\,["\s]*(?<SrvClnt_RemoteAddress>[\d.:]*)\"*\,\
["\s]*(?<SrvClnt_BufferSize_KB>\d+)\,["\s]*(?<SrvClnt_SendBuffers>.*?)\"*\,\
["\s]*(?<SrvClnt_StopReceivingCount>.*?)\"*\,["\s]*(?<SrvClnt_MaxStopReceivingTicks>.*?)\"*\,\
["\s]*(?<SrvClnt_AvgStopReceivingTicks>.*?)\"*\,["\s]*(?<SrvClnt_HighStopReceivingCount>.*?)\"*\,\
["\s]*(?<SrvClnt_PacketsReceived>.*?)\"*\,["\s]*(?<SrvClnt_BytesReceived>.*?)\"*\,\
["\s]*(?<SrvClnt_ReceivedPacketsPerSecond>.*?)\"*\,["\s]*(?<SrvClnt_ReceivedBytesPerSecond>.*?)\"*\,\
["\s]*(?<SrvClnt_MaxReceivedPacket>.*?)\"*\,["\s]*(?<SrvClnt_AvgReceivedPacket>.*?)\"*\,\
["\s]*(?<SrvClnt_MaxReceivedAckTicks>.*?)\"*\,["\s]*(?<SrvClnt_HighReceivedAckTicksCnt>.*?)\"*\,\
["\s]*(?<SrvClnt_LastReceivedIssuedTime>[\d\s.:]*)\"*\,["\s]*(?<SrvClnt_LastReceivedAckTime>[\d\s.:]*)\"*\,\
["\s]*(?<SrvClnt_PacketsSent>.*?)\"*\,["\s]*(?<SrvClnt_BytesSent>\d+\,\d+|)\"*\,\
["\s]*(?<SrvClnt_SentPacketsPerSecond>.*?)\"*\,["\s]*(?<SrvClnt_SentBytesPerSecond>.*?)\"*\,\
["\s]*(?<SrvClnt_MaxSentPacket>\d+\,\d+|)\"*\,["\s]*(?<SrvClnt_AvgSentPacket>.*?)\"*\,\
["\s]*(?<SrvClnt_MaxSentAckTicks>.*?)\"*\,["\s]*(?<SrvClnt_HighSentAckTicksCount>.*?)\"*\,\
["\s]*(?<SrvClnt_LastSentIssuedTime>[\d\s.:]*)\"*\,["\s]*(?<SrvClnt_LastSentAckTime>[\d\s.:]*)\"*\,\
["\s]*(?<SrvClnt_CompressionRatio>.*?)\"*\,["\s]*(?<SrvClnt_ConcatenationRatio>.*?)\"*\,\
["\s]*(?<SrvClnt_PendingBuffers>.*?)\"*\,["\s]*(?<SrvClnt_ActiveSessions>.*?)\"*\,\
["\s]*(?<SrvClnt_CP_Threads>.*?)\"*\,/
EricomSecureGateway System Information_xx.CSV
and Authentication Server System Information Log EricomAuthenticationServer System Information_xx.CSV
define SECGTW_SYSINF_REGEX /(?x)^(?:\s*(\d+\.\d+\.\d+\s+\d+\:\d+\:\d+)\.\d+)\,\"\s*\
(?<ElapsedFromStart>\d+\:\d+\:\d+)\"\,+\"\s*(?<CPU_USAGE_DeltaKernelCPU>.*?)\"\,\s*\
(?<CPU_USAGE_DeltaKernelCPU_pct>.*?)\,\"\s*(?<CPU_USAGE_DeltaUserCPU>.*?)\"\,\s*\
(?<CPU_USAGE_DeltaUserCPU_pct>.*?)\,\"\s*(?<CPU_USAGE_DeltaCPU>.*?)\"\,\s*\
(?<CPU_USAGE_DeltaCPU_pct>.*?)\,\"\s*(?<CPU_USAGE_KernelCPU>.*?)\"\,\s*\
(?<CPU_USAGE_KernelCPU_pct>.*?)\,\"\s*(?<CPU_USAGE_UserCPU>.*?)\"\,\s*\
(?<CPU_USAGE_UserCPU_pct>.*?)\,\"\s*(?<CPU_USAGE_CPU>.*?)\"\,\s*\
(?<CPU_USAGE_CPU_pct>.*?)\,\s*(?<CPU_USAGE_MachineCPU_pct>.*?)\,+\"\s*\
(?<MEMORY_DeltaPageFaults>.*?)\"\,\"\s*(?<MEMORY_PageFaults>.*?)\"\,\"\s*\
(?<MEMORY_DeltaCommitSize>.*?)\"\,\"\s*(?<MEMORY_CommitSize>.*?)\"\,\"\s*\
(?<MEMORY_DeltaWorkingSet>.*?)\"\,\"\s*(?<MEMORY_WorkingSet>.*?)\"\,+\"\s*\
(?<SYSRES_Handles>.*?)\"\,\"\s*(?<SYSRES_Threads>.*?)\"\,\"\s*\
(?<SYSRES_UserObjects>.*?)\"\,\"\s*(?<SYSRES_GDIObjects>.*?)\"\,+\"\s*\
(?<IO_Reads>.*?)\"\,\"\s*(?<IO_ReadBytes>.*?)\"\,\"\s*\
(?<IO_Writes>.*?)\"\,\"\s*(?<IO_WriteBytes>.*?)\"\,\"\s*\
(?<IO_Other>.*?)\"\,\"\s*(?<IO_OtherBytes>.*?)\"/
License Server logs
License Server provides all the functionality needed for acquiring, storing, maintaining, and serving licenses to AVEVA Enterprise software.
The input samples below represent the event structure of AVEVA License Server logs. This example shows how to process various logs related to the License Server in a single NXLog configuration.
The following fields can be parsed from fne-error.log
:
-
Severity
-
Date/Time
-
Message
fne-error.log
Info : [12/Nov/2021:04:48:57 -0800] Server environment tolerance interval set to 1296000.
The following fields can be parsed from fne-access.log
:
-
HostAddress
-
Identity
-
Username
-
Date/Time
-
Request
-
StatusCode
-
Size
fne-access.log
fe80::f8f9:6a54:1a96:441c%2 - - [12/Nov/2021:04:45:10 -0800] "GET /fne/xml/diagnostics HTTP/1.1" 200 703
The regular expressions for parsing events from each log file are defined in the configuration as the constants LICSRV_ERR_REGEX
and LICSRV_ACCESS_REGEX
.
The path to the log file directory is defined as the LOG_PATH
constant.
In the Exec block of the im_file, each incoming message is compared to the corresponding regular expression.
If the input event matches a regular expression, all fields are created according to the named capturing groups of the related regular expression.
The timestamp captured as group $1
or $4
is converted to a datetime value and assigned to the $EventTime
field.
Each event record is then converted to JSON using the to_json() procedure of the xm_json module. The drop() procedure discards any message that doesn’t match either of these two regular expressions.
# Regular expressions defined as a constants to read the content of the logs
define LICSRV_ERR_REGEX /(?x)^(?<Severity>\w+)\s+\:\s+\[\
(\d+\/\w+\/\d+\:\d+\:\d+\:\d+\s+\-\d+)\]\s+(?<Message>.*)/
define LICSRV_ACCESS_REGEX /(?x)^(?<HostAddress>[\w\:\%]+)\s+(?<Identity>.*?)\s+\
(?<Username>.*?)\s+\[(\d+\/\w+\/\d+\:\d+\:\d+\:\d+\s+\-\d+)\]\s+\"\
(?<Request>.*?)\"\s+(?<StatusCode>\d+)\s+(?<Size>\d+)/
# Part of the log path defined as a constant
define LOG_PATH C:\ProgramData\AVEVA\Licensing\License Server
<Extension json>
Module xm_json
</Extension>
<Input from_file>
Module im_file
File '%LOG_PATH%\fne-*.log'
<Exec>
# Matches the events with a regular expression
if $raw_event =~ %LICSRV_ERR_REGEX%
{
# Creates the timestamp
$EventTime = parsedate($2);
# Formats the result as JSON
to_json();
}
else if $raw_event =~ %LICSRV_ACCESS_REGEX%
{
# Creates the timestamp
$EventTime = parsedate($4);
# Formats the result as JSON
to_json();
}
# Discard event if it doesn't match a/the regular expression
else drop();
</Exec>
</Input>
The following output samples represent the processed events in JSON format.
fne-error.log
in JSON{
"EventReceivedTime": "2021-11-12T04:48:58.523964-08:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"Message": "Server environment tolerance interval set to 1296000.",
"Severity": "Info",
"EventTime": "2021-11-12T04:48:57.000000-08:00"
}
fne-access.log
in JSON{
"EventReceivedTime": "2021-11-12T04:45:11.000374-08:00",
"SourceModuleName": "from_file",
"SourceModuleType": "im_file",
"HostAddress": "fe80::f8f9:6a54:1a96:441c%2",
"Identity": "-",
"Request": "GET /fne/xml/diagnostics HTTP/1.1",
"Size": "703",
"StatusCode": "200",
"Username": "-",
"EventTime": "2021-11-12T04:45:10.000000-08:00"
}
Database tables
AVEVA System Platform operates on immense volumes of data for different purposes. This data can be acquired, stored in local or remote databases, and processed using powerful software components such as AVEVA System Monitor, AVEVA Historian, etc.
Reading information from database tables can provide insights into AVEVA System Platform internal components and process-related data such as tag data, process alarms, and process events.
Database Name | File ext. |
Location |
---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
System Monitor database
AVEVA System Monitor is an application that monitors the AVEVA System Platform core software components, applications, hardware, and network infrastructure.
It is responsible for detecting and processing system performance issues, monitoring key system attributes, generating alerts, and storing them in the local database.
The following CSV-formatted record represents the data from a row in the dbo.Alert
table of the System Monitor database after it was saved to file.
Id,MonitoredMachineId,SeverityStatusId,RuleId,AlertStatusId,Path,Source,Name,CreatedDate,LastUpdatedBy,LastUpdatedDateTime,OccuranceCount,EventReportedDate,LastBadStateChangeDate,LastGoodReportedDate,CategoryId,SubCategoryId,AlertContext
1,1,1,9,1,WIN-5RU7GP5MI4V,,aaBootstrap,2021-10-19 13:22:10.980,1,2021-10-19 13:24:25.860,2,2021-10-19 13:22:10.207,2021-10-19 13:22:10.207,NULL,274,1087,"<Event xmlns=""http://schemas.microsoft.com/win/2004/08/events/event""><System><Provider Name=""PSMSWinSrvDataProvider""/><EventID Qualifiers=""0"">1</EventID><Level>4</Level><Task>2</Task><Keywords>0x80000000000000</Keywords><TimeCreated SystemTime=""2021-10-19T13:24:09.069701200Z""/><EventRecordID>3</EventRecordID><Channel>PSMSWinSrvLogs</Channel><Computer>WIN-5RU7GP5MI4V</Computer><Security/></System><EventData><Data>WIN-5RU7GP5MI4V</Data><Data>aaBootstrap</Data><Data>ArchestrA Bootstrap</Data><Data>Stopped</Data><Data>10/19/2021 1:24:09 PM</Data></EventData></Event>"
This example reads data from a SQL Server database. The NXLog im_odbc module is used for connecting to the System Monitor database. The ConnectionString directive specifies the ODBC data source information. The im_odbc SQL directive accepts an SQL query that facilitates reading data from the table.
<Extension json>
Module xm_json
</Extension>
<Input from_odbc>
Module im_odbc
ConnectionString Driver={ODBC Driver 17 for SQL Server}; \
Server=WIN-5RU7GP5MI4V; \
Trusted_Connection=yes; \
Database=SentinelDB;
SQL SELECT * FROM dbo.Alert WHERE Id > ?
# Converts to JSON
Exec to_json();
</Input>
Below is the processed record in JSON format.
{
"Id": 1,
"MonitoredMachineId": 1,
"SeverityStatusId": 1,
"RuleId": 9,
"AlertStatusId": 1,
"Path": "WIN-5RU7GP5MI4V",
"Source": "",
"Name": "aaBootstrap",
"CreatedDate": "2021-10-19T13:22:10.980000-07:00",
"LastUpdatedBy": 1,
"LastUpdatedDateTime": "2021-10-19T13:24:25.860000-07:00",
"OccuranceCount": 2,
"EventReportedDate": "2021-10-19T13:22:10.207000-07:00",
"LastBadStateChangeDate": "2021-10-19T13:22:10.207000-07:00",
"LastGoodReportedDate": null,
"CategoryId": 274,
"SubCategoryId": 1087,
"AlertContext": "<Event xmlns=\"http://schemas.microsoft.com/win/2004/08/events/event\"><System><Provider Name=\"PSMSWinSrvDataProvider\"/><EventID Qualifiers=\"0\">1</EventID><Level>4</Level><Task>2</Task><Keywords>0x80000000000000</Keywords><TimeCreated SystemTime=\"2021-10-19T13:24:09.069701200Z\"/><EventRecordID>3</EventRecordID><Channel>PSMSWinSrvLogs</Channel><Computer>WIN-5RU7GP5MI4V</Computer><Security/></System><EventData><Data>WIN-5RU7GP5MI4V</Data><Data>aaBootstrap</Data><Data>ArchestrA Bootstrap</Data><Data>Stopped</Data><Data>10/19/2021 1:24:09 PM</Data></EventData></Event>",
"EventReceivedTime": "2021-11-25T13:06:20.064071-08:00",
"SourceModuleName": "from_odbc",
"SourceModuleType": "im_odbc"
}
Runtime database
The Runtime database is an online database for storing configuration information such as system configuration, tag definitions, InTouch integration information, system namespaces and grouping information, Classic Event subsystem configuration information, and user-entered annotations.
AVEVA Historian runs with the Runtime database and logically stores historical tag values.
The Runtime database file is named according to this convention:
RuntimeDat_<version_number>_<original_server_name>.mdf
The dbo._Tag
table of the Runtime database is used for storing basic definitions of tags.
The following CSV-formatted record represents the data from a row in the dbo._Tag
table after it was saved to file.
ShardId,TagId,TagName,IOServerKey,TopicKey,Description,AcquisitionType,StorageType,StorageRate,ItemName,TagType,DeadbandType,TimeDeadband,ServerTimeStamp,ChannelStatus,MessageKey,EUKey,MinEU,MaxEU,MinRaw,MaxRaw,Scaling,RawType,ValueDeadband,IntegerSize,SignedInteger,RateDeadband,InterpolationType,RolloverValue,MaxLength,DoubleByte,StructureId,SourceTag,SourceServer,SourceTagId,CurrentEditor,wwTagKey,AIHistory,DateCreated,CreatedBy,ChangeVersion,CEVersion,Status
00000000-0000-0000-0000-000000000000,37976E02-9127-4BE7-AC5A-BB9D96160D3F,MB_9,2,2,Modbus TCP variable #9,2,3,0,,1,1,0,0,1,NULL,2,-32768,32767,-32768,32767,0,3,0,32,1,0,254,0,NULL,NULL,NULL,NULL,NULL,NULL,2,206,0,2022-01-09 22:41:15.0620000,MDAS,0x0000000000002EE1,0,0
The following NXLog configuration uses the im_odbc module to connect to the Runtime database.
The ConnectionString directive specifies the data source information.
The im_odbc SQL directive accepts an SQL query that facilitates reading data from the table.
However, one of its requirements is a column of unique values either named id
or aliased as id
.
The other requirement is a WHERE
clause that references this id
column (or the column that id
is aliasing).
This is used for the next query, in order to prevent reading the same events repetitively.
The im_odbc IdType directive specifies how the value of id
will be interpreted.
In this example it should be treated as a timestamp
because the DateCreated
column it is aliasing contains datetime2 values.
The Id
column is deleted in the Exec block of the im_odbc module to prevent duplicating data in multiple columns.
The result is converted to JSON using the to_json() procedure of the xm_json module.
<Extension json>
Module xm_json
</Extension>
<Input from_odbc>
Module im_odbc
IdType timestamp
ConnectionString Driver={ODBC Driver 17 for SQL Server}; \
Server=WIN-5RU7GP5MI4V; \
Trusted_Connection=yes; \
Database=Runtime;
SQL SELECT *, DateCreated AS Id \
FROM dbo._Tag WHERE DateCreated > ?
<Exec>
# Deletes the "Id" field
delete($Id);
# Converts to JSON
to_json();
</Exec>
</Input>
Below is the processed record in JSON format.
{
"ShardId": "00000000-0000-0000-0000-000000000000",
"TagId": "37976E02-9127-4BE7-AC5A-BB9D96160D3F",
"TagName": "MB_9",
"IOServerKey": 2,
"TopicKey": 2,
"Description": "Modbus TCP variable #9",
"AcquisitionType": 2,
"StorageType": 3,
"StorageRate": 0,
"ItemName": "",
"TagType": 1,
"DeadbandType": 1,
"TimeDeadband": 0,
"ServerTimeStamp": false,
"ChannelStatus": 1,
"MessageKey": null,
"EUKey": 2,
"MinEU": "-32768.0",
"MaxEU": "32767.0",
"MinRaw": "-32768.0",
"MaxRaw": "32767.0",
"Scaling": 0,
"RawType": 3,
"ValueDeadband": "0.0",
"IntegerSize": 32,
"SignedInteger": true,
"RateDeadband": "0.0",
"InterpolationType": 254,
"RolloverValue": "0.0",
"MaxLength": null,
"DoubleByte": null,
"StructureId": null,
"SourceTag": null,
"SourceServer": null,
"SourceTagId": null,
"CurrentEditor": 2,
"wwTagKey": 206,
"AIHistory": false,
"DateCreated": "2022-01-09T22:41:15.062000-08:00",
"CreatedBy": "MDAS",
"ChangeVersion": "0000000000002ee1",
"CEVersion": 0,
"Status": 0,
"EventReceivedTime": "2022-01-09T22:41:16.577412-08:00",
"SourceModuleName": "from_odbc",
"SourceModuleType": "im_odbc"
}
Reading records from history blocks
AVEVA Historian stores processing data, replication data, and auto-summary data in history blocks using a proprietary file format. These files are located in subdirectories of the main historian storage directory.
History blocks are remote data sources stored in special history files using OLE DB technology.
They exist outside of a SQL Server database file (.MDF
).
All tag data stored in history blocks appears in remote tables of the Runtime database.
The sample below is a history block entry saved as a .csv
file with column names.
DateTime,TagName,Value,vValue,Quality,QualityDetail,OPCQuality,wwTagKey,wwRowCount,wwResolution,wwEdgeDetection,wwRetrievalMode,wwTimeDeadband,wwValueDeadband,wwTimeZone,wwVersion,wwCycleCount,wwTimeStampRule,wwInterpolationType,wwQualityRule,wwStateCalc,StateTime,PercentGood,wwParameters,StartDateTime,SourceTag,SourceServer,wwFilter,wwValueSelector,wwMaxStates,wwOption,wwExpression,wwUnit
2022-01-09 21:35:02.9990000,MB_1,312,312,0,192,192,202,-65536,1047,NONE,DELTA,0,0,Pacific Standard Time,LATEST,-65536,END,STAIRSTEP,EXTENDED,TOTAL,0,100,1,2022-01-09 21:35:02.9990000,NULL,NULL,NoFilter,Auto,0,PRIMARYDATA,NULL,None
The following NXLog configuration uses the im_odbc module to connect to the Runtime database.
The ConnectionString directive specifies the data source information.
The im_odbc SQL directive accepts an SQL query that facilitates reading data from the table.
However, one of its requirements is a column of unique values either named id
or aliased as id
.
The other requirement is a WHERE
clause that references this id
column (or the column that id
is aliasing).
This is used for the next query, in order to prevent reading the same events repetitively.
The im_odbc IdType directive specifies how the value of id
will be interpreted.
In this example it should be treated as a timestamp
because the DateCreated
column it is aliasing contains datetime2 values.
The Id
column is deleted in the Exec block to prevent duplicating data in different columns.
The result is then converted to JSON using the to_json() procedure of the xm_json module.
<Extension json>
Module xm_json
</Extension>
<Input from_odbc>
Module im_odbc
IdType timestamp
ConnectionString Driver={ODBC Driver 17 for SQL Server}; \
Server=WIN-5RU7GP5MI4V; \
Trusted_Connection=yes; \
Database=Runtime;
SQL SELECT *, DateTime AS Id \
FROM dbo.History WHERE TagName = 'MB_1' AND DateTime > ?
<Exec>
# Deletes the "Id" field
delete($Id);
# Converts to JSON
to_json();
</Exec>
</Input>
The following output sample shows the result of processing in JSON format.
{
"DateTime": "2022-01-09T21:35:02.999000-08:00",
"TagName": "MB_1",
"Value": "312.0",
"vValue": "312",
"Quality": 0,
"QualityDetail": 192,
"OPCQuality": 192,
"wwTagKey": 202,
"wwRowCount": -65536,
"wwResolution": null,
"wwEdgeDetection": "NONE",
"wwRetrievalMode": "DELTA",
"wwTimeDeadband": 0,
"wwValueDeadband": "0.0",
"wwTimeZone": "Pacific Standard Time",
"wwVersion": "LATEST",
"wwCycleCount": -65536,
"wwTimeStampRule": "END",
"wwInterpolationType": "STAIRSTEP",
"wwQualityRule": "EXTENDED",
"wwStateCalc": "TOTAL",
"StateTime": "0.0",
"PercentGood": "100.0",
"wwParameters": "1",
"StartDateTime": "2022-01-09T21:35:02.999000-08:00",
"SourceTag": null,
"SourceServer": null,
"wwFilter": "NoFilter",
"wwValueSelector": "Auto",
"wwMaxStates": 0,
"wwOption": "PRIMARYDATA",
"wwExpression": null,
"wwUnit": "None",
"EventReceivedTime": "2022-01-09T21:35:03.265155-08:00",
"SourceModuleName": "from_odbc",
"SourceModuleType": "im_odbc"
}
Holding database
The Holding database is used internally by the historian to temporarily store topic and configuration data imported from an InTouch application to AVEVA Historian.
The file name for the Holding database is HoldingDat_<version_number>_<original_server_name>.mdf
This sample represents a Holding database table record in .csv
format.
DdeSourceKey,NodeKey,SourceName,ApplicationName,TopicName,RequestInitialData,AlwaysAdvise,StorageType,StorageRate,TimeDeadband,ValueDeadband,RateDeadband,DeadbandType,Edited,ToInSQL,ProtocolType,IODriverKey,R_IOServerKey,R_TopicKey
5,1,"KEPServerEX ","server_runtime ","CH1_DEV1 ",1,1,2,10000,0,0,0,2," ",1,2,2,6,7
The following NXLog configuration uses the im_odbc module to connect to the Holding database . The ConnectionString directive specifies the data source information. The im_odbc SQL directive accepts an SQL query that facilitates reading data from the table.
The dbo.DdeCfg
table does not contain a column with data that can be used as an identifier for subsequent queries.
Therefore, an SQL query uses a row number in a WHERE
clause as an identifier.
In the Exec block, the result is converted to JSON using the to_json() procedure of the xm_json module, and the excessive characters are removed.
<Extension json>
Module xm_json
</Extension>
<Input from_odbc>
Module im_odbc
ConnectionString Driver={ODBC Driver 17 for SQL Server}; \
Server=WIN-5RU7GP5MI4V; \
Trusted_Connection=yes; \
Database=Holding;
SQL WITH DdeCfgData AS \
(SELECT *, ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) AS Id \
FROM dbo.DdeCfg) \
SELECT * FROM DdeCfgData WHERE Id > ?
<Exec>
# Converts to JSON
to_json();
# Replaces unwanted characters
$raw_event =~ s/\s{2,}//g;
</Exec>
</Input>
The following output sample shows a processed record in JSON format.
{
"DdeSourceKey": 5,
"NodeKey": 1,
"SourceName": "KEPServerEX",
"ApplicationName": "server_runtime",
"TopicName": "CH1_DEV1",
"RequestInitialData": true,
"AlwaysAdvise": true,
"StorageType": 2,
"StorageRate": 10000,
"TimeDeadband": 0,
"ValueDeadband": "0.0",
"RateDeadband": "0.0",
"DeadbandType": 2,
"Edited": " ",
"ToInSQL": true,
"ProtocolType": 2,
"IODriverKey": 2,
"R_IOServerKey": 6,
"R_TopicKey": 7,
"Id": 7,
"EventReceivedTime": "2021-11-25T12:19:49.231239-08:00",
"SourceModuleName": "from_odbc",
"SourceModuleType": "im_odbc"
}
Alarm database
Any InTouch application can be configured to generate alarms and events for notifying operators about process activity status.
Alarms represent the process conditions that could potentially cause problems. Events are standard system status messages exposing typical system conditions such as operator logging on. Alarm and event data is saved to the alarm database.
The InTouch Distributed Alarm system includes the Alarm DB Logger, storing historical alarms and events in the alarm database. The alarm database is a SQL Server database.
The sample below is the Alarm database table entry depicted in .csv
format.
EventId,EventGuid,ProviderId,GroupName,TagName,EventClass,EventType,EventState,EventPriority,EventValue,EventLimit,ValueString,LimitString,EventTime,EventTimeFracSec,EventTimeZoneOffset,EventDaylightAdjustment,OperatorNode,OperatorName,Comment,User1,User2,User3,OperatorID,EventStamp
45721,C94E61BD6DB54E249FFBFBEC6D665EAF,1,USER,MB_1,"EVENT ","DDE "," ",9,11,10,11,10,2022-01-10 09:16:40.343,3440,-480,0,WIN-5RU7GP5MI4V,None,MB_1 alarm,0,0,,NULL,2022-01-10 01:16:40.343
The following NXLog configuration uses the from_odbc
instance of the im_odbc module to connect to the Alarm database.
The ConnectionString directive specifies the data source information.
The im_odbc SQL directive accepts an SQL query that facilitates reading data from the table.
However, one of its requirements is a column of unique values either named id
or aliased as id
.
The other requirement is a WHERE
clause that references this id
column (or the column that id
is aliasing).
This is used for the next query, in order to prevent reading the same events repetitively.
In this example, Id
will act as an alias for the EventID
column.
The Exec block executes removing the Id
column to prevent duplicating data in different columns.
After removing unwanted characters, each event record is converted to JSON using the to_json() procedure of the xm_json module.
<Extension json>
Module xm_json
</Extension>
<Input from_odbc>
Module im_odbc
ConnectionString Driver={ODBC Driver 17 for SQL Server}; \
Server=WIN-5RU7GP5MI4V; \
Trusted_Connection=yes; \
Database=WWALMDB;
SQL SELECT EventId AS Id, * FROM dbo.Events WHERE EventId > ?
<Exec>
# Deletes the "Id" field
delete($Id);
# Converts to JSON
to_json();
# Replaces unwanted characters
$raw_event =~ s/\s+\"/"/g;
</Exec>
</Input>
Below is the processed record in JSON format.
{
"EventId": 45721,
"EventGuid": "C94E61BD6DB54E249FFBFBEC6D665EAF",
"ProviderId": 1,
"GroupName": "USER",
"TagName": "MB_1",
"EventClass": "EVENT",
"EventType": "DDE",
"EventState": "",
"EventPriority": 9,
"EventValue": "11.0",
"EventLimit": "10.0",
"ValueString": "11",
"LimitString": "10",
"EventTime": "2022-01-10T09:16:40.343000-08:00",
"EventTimeFracSec": 3440,
"EventTimeZoneOffset": -480,
"EventDaylightAdjustment": 0,
"OperatorNode": "WIN-5RU7GP5MI4V",
"OperatorName": "None",
"Comment": "MB_1 alarm",
"User1": "0.0",
"User2": "0.0",
"User3": "",
"OperatorID": null,
"EventStamp": "2022-01-10T01:16:40.343000-08:00",
"EventReceivedTime": "2022-01-10T01:16:41.329046-08:00",
"SourceModuleName": "from_odbc",
"SourceModuleType": "im_odbc"
}
Passive network monitoring
InTouch HMI allows creating distributed applications where the functional components are located on different nodes. These software components can interact with each other using the following communication protocols:
-
OPC
-
SuiteLink
-
DDE/FastDDE
-
ArchestrA Message Exchange
A Communication Driver is a software system component that connects the software application with data sources on the production floor. AVEVA offers various communication drivers to support the most commonly used industrial protocols.
This section describes how NXLog can passively monitor network traffic of the following industrial communication protocols:
Modbus TCP
Modbus is an application layer messaging protocol that provides client/server communication between devices connected to different types of buses or networks. Modbus TCP is simply the Modbus protocol with a TCP interface running on Ethernet. Essentially, Modbus TCP combines the Ethernet physical network with the TCP/IP networking standard and Modbus application protocol for data representation.
The interaction between InTouch HMI and the family of Modicon controllers or generic Modbus TCP devices is provided by the MBTCP Communication Driver.
In this example, the im_pcap module is configured to listen to the network traffic from the network interface specified in the Dev directive. The Protocol group directive provides network traffic filtering to capture only Modbus messages.
In the Exec block of im_pcap, all input messages will be converted to JSON using the to_json() procedure of the xm_json module.
<Extension _json>
Module xm_json
</Extension>
<Input pcap_modbus>
Module im_pcap
# Name of a network device/interface
Dev \Device\NPF_{159289BE-CE80-47DB-A659-2F8BF277C9C6}
<Protocol>
# Protocol type
Type modbus
</Protocol>
# Conversion to JSON
Exec to_json();
</Input>
{
"modbus.function_code": "Read Holding Registers (03)",
"modbus.length": "6",
"modbus.prot_id": "0",
"modbus.query.read_holding_regs.qty_of_regs": "3",
"modbus.query.read_holding_regs.starting_address": "0",
"modbus.trans_id": "2274",
"modbus.unit_id": "255",
"EventTime": "2021-11-29T18:32:08.129672-08:00",
"EventReceivedTime": "2021-11-29T18:32:08.423233-08:00",
"SourceModuleName": "pcap_modbus",
"SourceModuleType": "im_pcap"
}
{
"modbus.function_code": "Read Holding Registers (03)",
"modbus.length": "9",
"modbus.prot_id": "0",
"modbus.response.read_holding_regs.byte_count": "6",
"modbus.response.read_holding_regs.registers": "314, 310, 304",
"modbus.trans_id": "2274",
"modbus.unit_id": "255",
"EventTime": "2021-11-29T18:32:08.141036-08:00",
"EventReceivedTime": "2021-11-29T18:32:08.423233-08:00",
"SourceModuleName": "pcap_modbus",
"SourceModuleType": "im_pcap"
}
BACnet
BACnet is a widely accepted standard for building automation and control systems that integrate building management systems for heating and air conditioning, ventilation, lighting, security, and fire detection.
AVEVA InTouch HMI uses the BACLITE Communication Driver to interact via UDP/IP with all devices that are compatible with the BACnet/IP protocol.
The following example demonstrates passive network monitoring of InTouch HMI communications with a BACnet device using the BACnet/IP option.
NXLog uses the im_pcap module for passive network monitoring. The Dev directive of this module defines a specific network interface used for capturing packets. The Protocol group directive helps filter network traffic and capture packets of a particular protocol.
The Exec block of im_pcap calls the to_json() procedure of the xm_json module to convert input messages to JSON format.
<Extension _json>
Module xm_json
</Extension>
<Input pcap_bacnet>
Module im_pcap
# Name of a network device/interface
Dev \Device\NPF_{159289BE-CE80-47DB-A659-2F8BF277C9C6}
<Protocol>
# Protocol type
Type bacnet
</Protocol>
# Conversion to JSON
Exec to_json();
</Input>
{
"bacnet.apdu.bacnet_confirmed_request.invoke_id": "16",
"bacnet.apdu.bacnet_confirmed_request.max_resp": "1476",
"bacnet.apdu.bacnet_confirmed_request.max_segs": "64",
"bacnet.apdu.bacnet_confirmed_request.more_segments_follow": "false",
"bacnet.apdu.bacnet_confirmed_request.segmented": "false",
"bacnet.apdu.bacnet_confirmed_request.segmented_accepted": "true",
"bacnet.apdu.bacnet_confirmed_request.service_choice": "Confirmed COV Notification (1)",
"bacnet.apdu.bacnet_confirmed_request.service_request.records.0.subscriber_process_id": "53335072",
"bacnet.apdu.bacnet_confirmed_request.service_request.records.1.initiating_device_identifier.instance_number": "2",
"bacnet.apdu.bacnet_confirmed_request.service_request.records.1.initiating_device_identifier.object_id": "device (8)",
"bacnet.apdu.bacnet_confirmed_request.service_request.records.2.monitored_device_identifier.instance_number": "0",
"bacnet.apdu.bacnet_confirmed_request.service_request.records.2.monitored_device_identifier.object_id": "analog-input (0)",
"bacnet.apdu.bacnet_confirmed_request.service_request.records.3.time_remaining": "2441",
"bacnet.apdu.bacnet_confirmed_request.service_request.records.4": "Opening Tag (4)",
"bacnet.apdu.bacnet_confirmed_request.service_request.records.5.0.property_identifier": "present-value (85)",
"bacnet.apdu.bacnet_confirmed_request.service_request.records.5.0.property_value.records.0": "Opening Tag (2)",
"bacnet.apdu.bacnet_confirmed_request.service_request.records.5.0.property_value.records.1": "-471.500000",
"bacnet.apdu.bacnet_confirmed_request.service_request.records.5.0.property_value.records.2": "Closing Tag (2)",
"bacnet.apdu.bacnet_confirmed_request.service_request.records.5.1.property_identifier": "status-flags (111)",
"bacnet.apdu.bacnet_confirmed_request.service_request.records.5.1.property_value.records.0": "Opening Tag (2)",
"bacnet.apdu.bacnet_confirmed_request.service_request.records.5.1.property_value.records.1": "in-alarm (0): false, fault (1): false, overriden (2): false, out-of-service (3): false",
"bacnet.apdu.bacnet_confirmed_request.service_request.records.5.1.property_value.records.2": "Closing Tag (2)",
"bacnet.apdu.bacnet_confirmed_request.service_request.records.6": "Closing Tag (4)",
"bacnet.apdu.pdu_type": "BACnet-Confirmed-Request-PDU (0x00)",
"bacnet.bvlc.function": "Original-Unicast-NPDU (0x0A)",
"bacnet.bvlc.length": "55",
"bacnet.bvlc.type": "BACnet/IP (Annex J) (0x81)",
"bacnet.npdu.control": "0x000C",
"bacnet.npdu.control.contains": "BACnet APDU message (0)",
"bacnet.npdu.control.dst_spec": "DNET, DLEN, DADR, Hop Count absent (0)",
"bacnet.npdu.control.prio": "Normal message",
"bacnet.npdu.control.reply_expected": "Yes (1)",
"bacnet.npdu.control.src_spec": "SNET, SLEN, SADR present (1)",
"bacnet.npdu.version": "0x0001",
"EventTime": "2021-11-30T19:00:19.866513-08:00",
"EventReceivedTime": "2021-11-30T19:00:20.736612-08:00",
"SourceModuleName": "pcap_bacnet",
"SourceModuleType": "im_pcap"
}
{
"bacnet.apdu.bacnet_simpleack.original_invoke_id": "16",
"bacnet.apdu.bacnet_simpleack.service_ack_choice": "Confirmed COV Notification (1)",
"bacnet.apdu.pdu_type": "BACnet-Simple-ACK-PDU (0x02)",
"bacnet.bvlc.function": "Original-Unicast-NPDU (0x0A)",
"bacnet.bvlc.length": "19",
"bacnet.bvlc.type": "BACnet/IP (Annex J) (0x81)",
"bacnet.npdu.control": "0x0020",
"bacnet.npdu.control.contains": "BACnet APDU message (0)",
"bacnet.npdu.control.dst_spec": "DNET, DLEN, Hop Count present (1)",
"bacnet.npdu.control.prio": "Normal message",
"bacnet.npdu.control.reply_expected": "No (0)",
"bacnet.npdu.control.src_spec": "SNET, SLEN, SADR absent (0)",
"bacnet.npdu.version": "0x0001",
"EventTime": "2021-11-30T19:00:19.932049-08:00",
"EventReceivedTime": "2021-11-30T19:00:20.736612-08:00",
"SourceModuleName": "pcap_bacnet",
"SourceModuleType": "im_pcap"
}
{
"bacnet.apdu.bacnet_confirmed_request.invoke_id": "248",
"bacnet.apdu.bacnet_confirmed_request.max_resp": "1476",
"bacnet.apdu.bacnet_confirmed_request.max_segs": "Unspecified",
"bacnet.apdu.bacnet_confirmed_request.more_segments_follow": "false",
"bacnet.apdu.bacnet_confirmed_request.segmented": "false",
"bacnet.apdu.bacnet_confirmed_request.segmented_accepted": "true",
"bacnet.apdu.bacnet_confirmed_request.service_choice": "Read Property (12)",
"bacnet.apdu.bacnet_confirmed_request.service_request.records.0.object_identifier.instance_number": "0",
"bacnet.apdu.bacnet_confirmed_request.service_request.records.0.object_identifier.type": "analog-input (0)",
"bacnet.apdu.bacnet_confirmed_request.service_request.records.1.property_identifier": "description (28)",
"bacnet.apdu.pdu_type": "BACnet-Confirmed-Request-PDU (0x00)",
"bacnet.bvlc.function": "Original-Unicast-NPDU (0x0A)",
"bacnet.bvlc.length": "27",
"bacnet.bvlc.type": "BACnet/IP (Annex J) (0x81)",
"bacnet.npdu.control": "0x0024",
"bacnet.npdu.control.contains": "BACnet APDU message (0)",
"bacnet.npdu.control.dst_spec": "DNET, DLEN, Hop Count present (1)",
"bacnet.npdu.control.prio": "Normal message",
"bacnet.npdu.control.reply_expected": "Yes (1)",
"bacnet.npdu.control.src_spec": "SNET, SLEN, SADR absent (0)",
"bacnet.npdu.version": "0x0001",
"EventTime": "2021-11-30T19:00:19.990638-08:00",
"EventReceivedTime": "2021-11-30T19:00:20.736612-08:00",
"SourceModuleName": "pcap_bacnet",
"SourceModuleType": "im_pcap"
}
{
"bacnet.apdu.bacnet_complexack.more_segments_follow": "false",
"bacnet.apdu.bacnet_complexack.original_invoke_id": "248",
"bacnet.apdu.bacnet_complexack.segmented": "false",
"bacnet.apdu.bacnet_complexack.service_ack.records.0.object_identifier.instance_number": "0",
"bacnet.apdu.bacnet_complexack.service_ack.records.0.object_identifier.type": "analog-input (0)",
"bacnet.apdu.bacnet_complexack.service_ack.records.1.property_identifier": "description (28)",
"bacnet.apdu.bacnet_complexack.service_ack.records.2.records.0": "Opening Tag (3)",
"bacnet.apdu.bacnet_complexack.service_ack.records.2.records.1": "NXLog Enterprise Edition 5.4.7313 (ANSI X3.4/UTF-8 (0))",
"bacnet.apdu.bacnet_complexack.service_ack.records.2.records.2": "Closing Tag (3)",
"bacnet.apdu.bacnet_complexack.service_choice": "Read Property (12)",
"bacnet.apdu.pdu_type": "BACnet-Complex-ACK-PDU (0x03)",
"bacnet.bvlc.function": "Original-Unicast-NPDU (0x0A)",
"bacnet.bvlc.length": "63",
"bacnet.bvlc.type": "BACnet/IP (Annex J) (0x81)",
"bacnet.npdu.control": "0x0008",
"bacnet.npdu.control.contains": "BACnet APDU message (0)",
"bacnet.npdu.control.dst_spec": "DNET, DLEN, DADR, Hop Count absent (0)",
"bacnet.npdu.control.prio": "Normal message",
"bacnet.npdu.control.reply_expected": "No (0)",
"bacnet.npdu.control.src_spec": "SNET, SLEN, SADR present (1)",
"bacnet.npdu.version": "0x0001",
"EventTime": "2021-11-30T19:00:20.016958-08:00",
"EventReceivedTime": "2021-11-30T19:00:20.736612-08:00",
"SourceModuleName": "pcap_bacnet",
"SourceModuleType": "im_pcap"
}
DNP3
DNP3 (Distributed Network Protocol) is a communication protocol mainly used to provide system interoperability in the electric utility, oil & gas, water/wastewater industries, etc. It is highly suitable for implementation in the SCADA environment facilitating communications between various types of data acquisition and control equipment.
AVEVA InTouch HMI can communicate with DNP3 devices using the OI Gateway Communication Driver in conjunction with the OPC server. OI Gateway allows InTouch leveraging OPC and OPC UA functionality to communicate between automation and control applications, field systems and devices, etc.
In this example, the im_pcap module passively listens to the network interface specified in the Dev directive for network traffic filtered by the protocol defined in the Protocol group directive. The result is formatted as JSON using the to_json() procedure of the xm_json module, then saved to file.
<Extension _json>
Module xm_json
</Extension>
<Input pcap_dnp3>
Module im_pcap
# Name of a network device/interface
Dev \Device\NPF_{159289BE-CE80-47DB-A659-2F8BF277C9C6}
<Protocol>
# Protocol type
Type dnp3
</Protocol>
# Conversion to JSON
Exec to_json();
</Input>
{
"dnp3.application_layer.control.con": "0",
"dnp3.application_layer.control.fin": "1",
"dnp3.application_layer.control.fir": "1",
"dnp3.application_layer.control.sequence": "11",
"dnp3.application_layer.control.uns": "0",
"dnp3.application_layer.function_code": "Read",
"dnp3.application_layer.object0.count": "0",
"dnp3.application_layer.object0.group": "60",
"dnp3.application_layer.object0.name": "Class objects - Class 1 data",
"dnp3.application_layer.object0.variation": "2",
"dnp3.application_layer.object1.count": "0",
"dnp3.application_layer.object1.group": "60",
"dnp3.application_layer.object1.name": "Class objects - Class 2 data",
"dnp3.application_layer.object1.variation": "3",
"dnp3.application_layer.object2.count": "0",
"dnp3.application_layer.object2.group": "60",
"dnp3.application_layer.object2.name": "Class objects - Class 3 data",
"dnp3.application_layer.object2.variation": "4",
"dnp3.data_layer.control": "0xC4",
"dnp3.data_layer.control.dir": "1",
"dnp3.data_layer.control.fcb": "0",
"dnp3.data_layer.control.fcv": "0",
"dnp3.data_layer.control.function_code": "Unconfirmed User Data",
"dnp3.data_layer.control.prm": "1",
"dnp3.data_layer.destination": "4",
"dnp3.data_layer.length": "17",
"dnp3.data_layer.source": "3",
"dnp3.data_layer.start_bytes": "0x0564",
"dnp3.transport.fin": "1",
"dnp3.transport.fir": "1",
"dnp3.transport.sequence": "51",
"EventTime": "2021-12-01T00:40:56.974559-08:00",
"EventReceivedTime": "2021-12-01T00:40:57.167446-08:00",
"SourceModuleName": "pcap_dnp3",
"SourceModuleType": "im_pcap"
}
{
"dnp3.application_layer.control.con": "1",
"dnp3.application_layer.control.fin": "1",
"dnp3.application_layer.control.fir": "1",
"dnp3.application_layer.control.sequence": "11",
"dnp3.application_layer.control.uns": "0",
"dnp3.application_layer.function_code": "Response",
"dnp3.application_layer.internal_indications.already_executing": "0",
"dnp3.application_layer.internal_indications.broadcast": "0",
"dnp3.application_layer.internal_indications.class1_events": "0",
"dnp3.application_layer.internal_indications.class2_events": "0",
"dnp3.application_layer.internal_indications.class3_events": "0",
"dnp3.application_layer.internal_indications.config_corrupt": "0",
"dnp3.application_layer.internal_indications.device_restart": "0",
"dnp3.application_layer.internal_indications.device_trouble": "0",
"dnp3.application_layer.internal_indications.events_buffer_overflow": "0",
"dnp3.application_layer.internal_indications.local_control": "0",
"dnp3.application_layer.internal_indications.need_time": "0",
"dnp3.application_layer.internal_indications.no_func_code_support": "0",
"dnp3.application_layer.internal_indications.object_unknown": "0",
"dnp3.application_layer.internal_indications.parameter_error": "0",
"dnp3.application_layer.internal_indications.reserved": "0 (expected 0)",
"dnp3.application_layer.object0.count": "2",
"dnp3.application_layer.object0.group": "32",
"dnp3.application_layer.object0.name": "Analog input event - single-precision, floating-point with time",
"dnp3.application_layer.object0.point0.flags": "[ONLINE]",
"dnp3.application_layer.object0.point0.index": "0",
"dnp3.application_layer.object0.point0.time_of_occurance": "1638319253902",
"dnp3.application_layer.object0.point0.value": "15660.298828",
"dnp3.application_layer.object0.point1.flags": "[ONLINE]",
"dnp3.application_layer.object0.point1.index": "0",
"dnp3.application_layer.object0.point1.time_of_occurance": "1638319255919",
"dnp3.application_layer.object0.point1.value": "5458.169434",
"dnp3.application_layer.object0.range": "2-octet count of objects",
"dnp3.application_layer.object0.variation": "7",
"dnp3.data_layer.control": "0x44",
"dnp3.data_layer.control.dir": "0",
"dnp3.data_layer.control.fcb": "0",
"dnp3.data_layer.control.fcv": "0",
"dnp3.data_layer.control.function_code": "Unconfirmed User Data",
"dnp3.data_layer.control.prm": "1",
"dnp3.data_layer.destination": "3",
"dnp3.data_layer.length": "41",
"dnp3.data_layer.source": "4",
"dnp3.data_layer.start_bytes": "0x0564",
"dnp3.transport.fin": "1",
"dnp3.transport.fir": "1",
"dnp3.transport.sequence": "27",
"EventTime": "2021-12-01T00:40:56.976879-08:00",
"EventReceivedTime": "2021-12-01T00:40:57.167446-08:00",
"SourceModuleName": "pcap_dnp3",
"SourceModuleType": "im_pcap"
}
{
"dnp3.application_layer.control.con": "0",
"dnp3.application_layer.control.fin": "1",
"dnp3.application_layer.control.fir": "1",
"dnp3.application_layer.control.sequence": "11",
"dnp3.application_layer.control.uns": "0",
"dnp3.application_layer.function_code": "Confirm",
"dnp3.data_layer.control": "0xC4",
"dnp3.data_layer.control.dir": "1",
"dnp3.data_layer.control.fcb": "0",
"dnp3.data_layer.control.fcv": "0",
"dnp3.data_layer.control.function_code": "Unconfirmed User Data",
"dnp3.data_layer.control.prm": "1",
"dnp3.data_layer.destination": "4",
"dnp3.data_layer.length": "8",
"dnp3.data_layer.source": "3",
"dnp3.data_layer.start_bytes": "0x0564",
"dnp3.transport.fin": "1",
"dnp3.transport.fir": "1",
"dnp3.transport.sequence": "52",
"EventTime": "2021-12-01T00:40:57.001038-08:00",
"EventReceivedTime": "2021-12-01T00:40:57.167446-08:00",
"SourceModuleName": "pcap_dnp3",
"SourceModuleType": "im_pcap"
}
IEC 60870-5-104
IEC 60870-5-104 is a part of the IEC60870-5 standard that provides a communication profile for sending basic telecontrol messages between two electrical engineering and power system automation systems.
IEC 60870-5-104 provides network access to IEC 60870-5-101, delivering its messages as application data over TCP.
AVEVA InTouch HMI can communicate with IEC 60870-5-104 devices using the OI Gateway Communication Driver in conjunction with the OPC server. OI Gateway allows InTouch to leverage OPC and OPC UA functionality in order to communicate between automation and control applications, field systems and devices, etc.
NXLog uses the im_pcap module to monitor network traffic. The Dev directive of this module defines the network interface used to capture packets. The Protocol group directive specifies which protocols will be monitored for capturing their messages.
The result is formatted as JSON using the to_json() procedure of the xm_json module, then saved to file.
<Extension _json>
Module xm_json
</Extension>
<Input pcap_iec104>
Module im_pcap
# Name of a network device/interface
Dev \Device\NPF_{159289BE-CE80-47DB-A659-2F8BF277C9C6}
<Protocol>
# Protocol type
Type iec104apci
</Protocol>
<Protocol>
# Protocol type
Type iec104asdu
</Protocol>
# Conversion to JSON
Exec to_json();
</Input>
{
"iec104.apci.receive_sequence_number": "8",
"iec104.apci.send_sequence_number": "73",
"iec104.apci.type": "Information (I)",
"iec104.asdu.data": {
"io": [
{
"ioa": 1000,
"ie": [
{
"type": "NVA",
"value": "0.477905 (15660)"
},
{
"type": "QDS",
"invalid": false,
"not-topical": false,
"substituted": false,
"blocked": false,
"overflow": false
},
{
"type": "CP56Time2A",
"milliseconds": 44454,
"minutes": 7,
"hours": 15,
"day-of-week": 0,
"day-of-month": 1,
"month": 12,
"year": 21
}
],
"ies": 3
}
],
"ios": 1
},
"iec104.asdu.dui.cause_of_transmission": "Spontaneous (3)",
"iec104.asdu.dui.coa": "1",
"iec104.asdu.dui.num_records": "1",
"iec104.asdu.dui.org": "0",
"iec104.asdu.dui.pn": "0",
"iec104.asdu.dui.sq": "FALSE",
"iec104.asdu.dui.test_bit": "0",
"iec104.asdu.dui.type": "M_ME_TD_1",
"EventTime": "2021-12-01T15:07:43.011872-08:00",
"EventReceivedTime": "2021-12-01T15:07:43.790386-08:00",
"SourceModuleName": "pcap_iec104",
"SourceModuleType": "im_pcap"
}