SAP
SAP is a provider of enterprise software for the management of business operations and customer relations. The SAP Business Suite and newer S/4HANA are leading Enterprise Resource Planning (ERP) solutions, but also include products for Supplier Relationship Management (SRM), Customer Relatonship Management (CRM), Supply Chain Management (SCM), Product Lifecycle Management (PLM), and Transportation Management (TMS).
The SAP platform is made up of various components, at the center of which are the SAP Netweaver Application Server and the SAP HANA relational database management system. Logging from these components can assist in troubleshooting issues or mitigating security threats, while certain logging is required for security auditing and compliance. NXLog Agent can integrate with the SAP platform to collect and streamline logs from different components, and save them in a central repository or forward them to a SIEM. This guide demonstrates how to collect and parse SAP logs commonly used for troubleshooting and auditing.
SAP NetWeaver Application Server logs
The SAP NetWeaver Application Server is the runtime environment for SAP web applications. One of the most important logs created by the Application Server is the Security Audit Log (SAL), which is used to record security-related events such as changes to master records and user access. The SAP audit log is an important tool used by auditors and is often required for compliance. Another important log is the System Log, where system errors, warnings, user locks, and process messages are recorded. The system log is useful for troubleshooting and resolving issues that arise within the SAP system.
Both the Security Audit Log and the System Log write to a file in a format that cannot be easily read by humans and not supported out of the box by most SIEM solutions. With NXLog Agent, you can collect these logs and convert them into a format that is easier to process by humans and SIEM solutions alike.
SAP NetWeaver Application Security Audit Log
The Security Audit Log is not enabled by default.
Follow the instructions below to configure security auditing from SAP GUI.
By default, log files are written to the /usr/sap/<SID>/<instno>/log/ directory and filenames follow the format audit_<SAP_instance_number>.
Users must be a member of the sapsys OS group to read the log files.
These steps apply to release 750 and later.
See transaction SM19 for earlier versions.
Set Profile parameters:
-
Call transaction
RSAU_CONFIG. -
Under Configuration, expand Security Audit Log Configuration and double-click on Parameter.
-
Switch to editing mode by clicking the glasses icon.
-
Enable Static security audit active and select Record in File System from the Recording Target drop-down.
-
Set a value for Number of Filters per Profile e.g. 10.
-
Enable Generic user selection.
-
Configure the audit file system settings according to your requirements.
-
Save settings.
More information on the audit log parameters can be found in the SAP documentation on Preparing the Security Audit Log.
Next, configure audit filters:
-
Call transaction
RSAU_CONFIG. -
Under Configuration, right click on Static Configuration and select Create Profile.
-
Enter a name for the profile and save.
-
Right click on the new profile and select Create Filter.
-
Configure the filter settings according to your requirements. For example, to audit events from all clients and users, enter
*in the Client and User fields under Standard Selection. -
Save settings and activate the profile by clicking the lit match icon.
-
Restart the SAP system.
See the SAP documentation on Defining Filters for more information on filter settings.
NXLog Agent provides the xm_sap extension module to parse SAP Security Audit Log records into structured data. An xm_sap module instance can be used as an input reader function with the im_file input module.
This configuration reads Security Audit Log files from the /usr/sap/NPL/D00/log/ directory.
The InputType directive of the im_file module instance specifies the name of the xm_sap module instance.
Log records are converted to JSON using the to_json() procedure of the xm_json module.
<Extension sap>
Module xm_sap
</Extension>
<Extension json>
Module xm_json
</Extension>
<Input sap_audit_log>
Module im_file
File '/usr/sap/NPL/D00/log/audit_*'
InputType sap
Exec to_json();
</Input>Below is an example of a Security Audit Log record for a successful logon. This sample depicts the kind of data and format NXLog Agent will collect.
2AU120220329171501000779800001D1SV01 SAP* SESSION_MANAGER SAPMSYST 0011A&0&P 10.0.0.103The following JSON object shows the same record after it was processed by NXLog Agent.
{
"EntryType": "2",
"MessageID": "AU1",
"EventTime": "2022-03-29 17:15:01",
"ProcessID": "07798",
"Task": "00001",
"ProcessType": "D1",
"Terminal": "SV0110.0.0.103",
"Username": "SAP*",
"TransmissionCode": "SESSION_MANAGER",
"Program": "SAPMSYST",
"Client": "001",
"DialogMode": "1",
"MessageData": "A&0&P",
"EventReceivedTime": "2022-03-29T17:15:06.927043+02:00",
"SourceModuleName": "sap_audit_log",
"SourceModuleType": "im_file"
}
For a list of SAP Message IDs and their corresponding message text, execute transaction SE92.
|
SAP NetWeaver System Log
The System Log is enabled by default.
Each SAP Application Server instance maintains a local log file.
When the log file reaches its file size limit, the system overwrites the oldest data.
Files are written to the /usr/sap/<SID>/<instno>/log/ directory and filenames follow the format SLOG<SAP_instance_number>.
Users must be a member of the sapsys OS group to read the log files.
The following profile parameters are available for configuration:
| Parameter | Description | Default Value |
|---|---|---|
rslg/max_diskspace/local |
File size limit (bytes) |
10000000 (10MB) |
rsl/local/file |
Name of the local log file |
SLOG<SAP_instance_number> |
Execute transaction RZ11 on the relevant instance to configure the parameters listed above.
See the SAP documentation on Changing the Name or Size of the System Log File for more information.
System log records consist of 320 characters and the log file does not contain line endings. Each record is divided into 24 fields as follows:
-
Syslog type (1 character)
-
Message ID (3 characters)
-
Timestamp (14 characters)
-
Year - YYYY (4 characters)
-
Month - MM (2 characters)
-
Day - DD (2 characters)
-
Hour - HH (2 characters)
-
Minute - mm (2 characters)
-
Seconds - ss (2 characters)
-
-
Process ID (5 characters)
-
Unused (2 characters)
-
Task (5 characters)
-
ProcessType (2 characters)
-
Terminal (8 characters)
-
Username (12 characters)
-
Transaction code (20 characters)
-
Program (40 characters)
-
Client (3 characters)
-
Session (1 character)
-
Message text (64 characters)
-
EPP transaction ID (32 characters)
-
EPP root context ID (32 characters)
-
EPP connection ID (32 characters)
-
Unknown (22 characters)
-
End of record (22 characters)
The following Python script can read system log files and output individual log records to the standard output. The script expects one argument which specifies the path of the folder to monitor.
# Copyright © NXLog Ltd. All Rights Reserved.
#
# This software is the proprietary product of NXLog Ltd and NXLog Ltd
# retains all copyright and ownership interest in the software.
#
# This file is part of the nxlog log collector tools.
# For licensing terms questions please contact support@nxlog.org.
# Website: http://nxlog.org
# Author: Seth Stenzel <seth.stenzel@nxlog.org>
import os
import sys
import glob
import time
#event character length for System Logs
event_length = 320
try:
if len(sys.argv) <2:
raise Exception("No directory argument provided.")
# directory to monitor
directory = str(sys.argv[1])
previous_file = ''
previous_size = 0
new_bytes = 0
while True:
try:
# get most recent file which contains name part
# if needing a specific file ext then add ext after '\*' (ex: '\*.csv')
dir_files = glob.glob(directory + '/SLOG*') (1)
if len(dir_files) > 0:
latest_file = max(dir_files, key=os.path.getmtime)
else:
time.sleep(0.01)
continue
# check if most recent event file name has changed
if latest_file != '' and previous_file != '' and latest_file != previous_file:
# checks if there is still new data to be read from previous file when new latest file is found
# this is rough handling with the assumption that new files could not be made so quickly
# that between one iteration, and another that multiple new files will not be created
if previous_size == os.stat(previous_file).st_size:
previous_size = 0
previous_file = latest_file
else:
latest_file = previous_file
# get the current event file size
latest_size = os.stat(latest_file).st_size
# if empty or unchanged, sleep for 10ms then go to next loop interation
if latest_size == 0 or latest_size == previous_size:
time.sleep(0.01)
continue
# else if file has content and has grown, parse new events
elif latest_size != previous_size and latest_size > previous_size:
new_bytes = latest_size - previous_size
# open the file with context manager
with open(latest_file, 'rb') as lf:
# seeks to the point in the file where new bytes should start instead of reading whole file
lf.seek(latest_size - new_bytes)
# reads from seek point for the length of new_bytes
new_events = lf.read(new_bytes)
# decodes utf-16 data
new_events = new_events.decode('utf-16')
# splits the string at n length where n length is the variable event_length at the top of script;
new_events_list = [new_events[i:i+event_length] for i in range(0, len(new_events), event_length)]
# print each new event
for event in new_events_list:
print(event.replace("\n", " "))
previous_file = latest_file
previous_size = latest_size
else:
print("WARNING - FILE TRUNCATION DETECTED")
previous_file = latest_size
except Exception as e:
print(f"ERROR - ERROR PARSING FILE ({e})", repr(e))
except Exception as e:
print(f"ERROR - ERROR ({e})", repr(e))| 1 | Specify the name of the files to read. |
|
The script does not keep a record of the last read event, therefore it will process all files available in the specified folder every time NXLog Agent is started. To avoid duplicate log records, it is recommended to only process complete log files and remove the files from the folder once they are processed by NXLog Agent. Using log files with a smaller file size can help to reduce the risk of duplicate data. This script is provided "AS IS" without warranty of any kind, either expressed or implied. Use at your own risk. |
This configuration uses the im_exec input module to execute the above Python script and read logs from its standard output. Once NXLog Agent receives a log record, it uses a regular expression to parse it into structured data, and then converts the record to JSON using the xm_json module.
define SLOG_REGEX /(?x)^(?<SyslogType>[a-z]|\s)(?<MessageID>.{3})(?<Year>.{4})\
(?<Month>.{2})(?<Day>.{2})(?<Hour>.{2})(?<Minutes>.{2})\
(?<Seconds>.{2})(?<ProcessID>.{5})(?<Unused>.{2})(?<Task>.{5})\
(?<ProcessType>.{2})(?<Terminal>.{8})(?<Username>.{12})\
(?<TransactionCode>.{20})(?<Program>.{40})(?<Client>.{3})\
(?<Session>.{1})(?<MessageText>.{64})(?<EPPTransactionID>.{32})\
(?<EPPRootContextID>.{32})(?<EPPConnectionID>.{32})\
(?<UNKNOWN>.{22})\*{22}/
<Extension json>
Module xm_json
</Extension>
<Input sap_system_log>
Module im_exec
# Path to python interpreter executable.
Command '/usr/bin/python3'
# Force the stdout and stderr streams to be unbuffered with -u argument.
Arg -u
# Path to python script .py file to run.
Arg /opt/nxlog/etc/sap-log-reader.py
# Path to logs folder to monitor.
Arg /usr/sap/NPL/D00/log
<Exec>
if $raw_event =~ %SLOG_REGEX%
{
$EventTime = $Year + "-" + $Month + "-" + $Day + " " + $Hour + \
":" + $Minutes + ":" + $Seconds;
delete($Year);
delete($Month);
delete($Day);
delete($Hour);
delete($Minutes);
delete($Seconds);
delete($Unused);
# Trim trailing whitespace
if $Terminal =~ s/\s+$//g ;
if $Username =~ s/\s+$//g ;
if $TransactionCode =~ s/\s+$//g ;
if $Program =~ s/\s+$//g ;
if $MessageText =~ s/\s+$//g ;
if $EPPTransactionID =~ s/\s+$//g ;
if $EPPRootContextID =~ s/\s+$//g ;
if $EPPConnectionID =~ s/\s+$//g ;
if $UNKNOWN =~ s/\s+$//g ;
}
else
{
$Message = $raw_event;
}
to_json();
</Exec>
</Input>Below is an example of a System Log record for a logon failure. This sample depicts the kind of data and format NXLog Agent will collect.
BY020220402101601000786500000D0SV01 DEVELOPER DBACOCKPIT SAPLSDBACCMS 0011Enterprise]Login failed.
9 8280D1D62E110000E0060E3261179BAB1D712E74046E1EDBB7B5641611EEA254000000000000000000000000000000000000000000 **********************The following JSON object shows the same record after it was processed by NXLog Agent.
{
"EventReceivedTime": "2022-04-02T10:16:40.502746+02:00",
"SourceModuleName": "sap_system_log",
"SourceModuleType": "im_exec",
"Client": "001",
"EPPConnectionID": "00000000000000000000000000000000",
"EPPRootContextID": "1D712E74046E1EDBB7B5641611EEA254",
"EPPTransactionID": "8280D1D62E110000E0060E3261179BAB",
"MessageID": "BY0",
"MessageText": "Enterprise]Login failed. 9",
"ProcessID": "00078",
"ProcessType": "D0",
"Program": "SAPLSDBACCMS",
"Session": "1",
"SyslogType": " ",
"Task": "00000",
"Terminal": "SV01",
"TransactionCode": "DBACOCKPIT",
"UNKNOWN": "0000000000",
"Username": "DEVELOPER",
"EventTime": "2022-04-02 10:16:01"
}
For a list of SAP Message IDs and their corresponding message text, execute transaction SE92.
|
SAP HANA logs
SAP HANA is an in-memory relational database management system that serves as a database server for SAP applications. SAP HANA auditing provides a log of database transactions and can answer the question of who did what and when. Audit logging may be required for security audits and compliance, and can help tighten security practices by uncovering security holes, such as users with unnecessary privileges or attempts to breach the system.
Auditing is not enabled by default. Activating it requires that you first configure audit parameters, and then create an audit policy to define the actions that should be audited. SAP HANA supports three audit trail targets, syslog, SAP HANA database table, and CSV text file. NXLog Agent can be configured to collect logs from all these targets. Note that the CSV target should only be enabled for testing purposes and is not recommended for use in production. For more information, see the SAP documentation on Audit Trails.
To configure auditing and create audit policies from the UI, refer to the SAP documentation on Managing Auditing in the SAP HANA Cockpit. Alternatively, auditing can be configured by executing the ALTER SYSTEM ALTER CONFIGURATION and CREATE AUDIT POLICY SQL statements.
The following SQL statement enables auditing in the system database.
ALTER SYSTEM ALTER CONFIGURATION ('nameserver.ini', 'system') set ('auditing configuration', 'global_auditing_state') = 'true';The next statement sets the audit target to a database table (the default is SYSLOGPROTOCOL).
ALTER SYSTEM ALTER CONFIGURATION ('nameserver.ini', 'system') set ('auditing configuration', 'default_audit_trail_type') = 'CSTABLE';See System Properties for Configuring Auditing in the SAP documentation for a complete list of available properties.
The CREATE AUDIT POLICY statement has the following format:
CREATE AUDIT POLICY <policy_name> [ FOR <database_name> ] AUDITING
<audit_status> <audit_actions> LEVEL <audit_level> [ <opt_audit_trail_type> ]For example, this statement audits INSERT operations on a table
HOTEL.CUSTOMERS by USER1.
CREATE AUDIT POLICY CUSTOMERSAUDIT AUDITING SUCCESSFUL INSERT ON
HOTEL.CUSTOMERS FOR USER1 LEVEL INFO;| Auditing can also be enabled for SAP HANA XSA applications. For more information, see Audit Log Services in XS Advanced in the SAP documentation. |
SAP HANA syslog audit trail
SAP HANA does not provide log forwarding capabilities. However, the operating system syslog daemon can be configured to forward audit logs coming from SAP HANA to NXLog Agent. The following examples show how this can be achieved with Rsyslog.
To forward all logs coming from a SAP HANA instance:
:msg, contains, "<sid>;<instanceno>" @@<ip>:<port>
For example, this configuration forwards all logs from SAP HANA database server HXE instance 90 to 192.168.0.123 on port 514.
:msg, contains, "HXE;90" @@192.168.0.123:514
This configuration uses a control structure to only forward logs with syslog facility authpriv.
if ($syslogfacility-text == "authpriv" and \
$msg contains "HXE;90")
then @@192.168.0.123:514
The forwarding configuration needs to be added to the /etc/rsyslog.conf file.
For configuration options related to other syslog daemons, refer to the documentation of your respective OS.
When SAP HANA auditing is configured with the syslog trail target, log records consist of a set of fields delimited by the ; character.
See Audit Trail Layout for Trail Target CSV and SYSLOG in the SAP documentation for a complete list of fields.
The configuration below uses the im_tcp input module to listen for IPv4 connections on all available network interfaces. It uses the xm_syslog module to parse the syslog message, and the xm_csv module to parse the audit record into structured data. The record is then converted to JSON using the xm_json module.
<Extension json>
Module xm_json
</Extension>
<Extension syslog>
Module xm_syslog
</Extension>
<Extension hdb_audit_parser>
Module xm_csv
Fields $EventTime, $ServiceName, $Hostname, $SID, $InstanceNumber, \
$PortNumber, $DatabaseName, $ClientIP, $ClientName, \
$ClientProcessID, $ClientPortNumber, $PolicyName, \
$AuditLevel, $AuditAction, $SessionUser, $TargetSchema, \
$TargetObject, $PrivilegeName, $Grantable, $RoleName, \
$TargetPrinciple, $ActionStatus, $Component, $Section, \
$Parameter, $OldValue, $NewValue, $Comment, \
$ExecutedStatement, $SessionID, $ApplicationUserName, \
$RoleSchemaName, $GranteeSchemaName, $OriginDatabaseName, \
$OriginUserName, $XSApplicationUserName, $ApplicationName, \
$StatementUserName, b$CreateTime, $XSA_MESSAGE_IP, \
$XSA_TENANT, $XSA_UUID, $XSA_CHANNEL, $XSA_ATTACHMENT_ID, \
$XSA_ATTACHMENT_NAME, $XSA_ORGANIZATION_ID, $XSA_SPACE_ID, \
$XSA_INSTANCE_ID, $XSA_BINDING_ID, $XSA_OBJECT, \
$XSA_DATA_SUBJECT
Delimiter ;
</Extension>
<Input tcp_listen>
Module im_tcp
ListenAddr 0.0.0.0:514
<Exec>
parse_syslog();
hdb_audit_parser->parse_csv($Message);
delete($Message);
to_json();
</Exec>
</Input>The following JSON object shows a SAP HANA database audit record after it was processed by NXLog Agent.
{
"MessageSourceAddress": "192.168.1.120",
"EventReceivedTime": "2021-07-19T10:32:45.857855+02:00",
"SourceModuleName": "tcp_listen",
"SourceModuleType": "im_tcp",
"SyslogFacilityValue": 10,
"SyslogFacility": "AUTHPRIV",
"SyslogSeverityValue": 6,
"SyslogSeverity": "INFO",
"SeverityValue": 2,
"Severity": "INFO",
"Hostname": "hxehost",
"EventTime": "2021-07-19T10:32:37.327360Z",
"SourceName": "HDB_SYSTEMDB",
"ProcessID": "1788",
"ServiceName": "nameserver",
"SID": "HXE",
"InstanceNumber": "90",
"PortNumber": "39001",
"DatabaseName": "SYSTEMDB",
"ClientIP": "192.168.1.122",
"ClientName": "JANE-PC",
"ClientProcessID": "6592",
"ClientPortNumber": "51254",
"PolicyName": "CUSTOMERSAUDIT",
"AuditLevel": "INFO",
"AuditAction": "INSERT",
"SessionUser": "USER1",
"TargetSchema": "HOTEL",
"TargetObject": "CUSTOMERS",
"ActionStatus": "SUCCESSFUL",
"ExecutedStatement": "INSERT INTO HOTEL.CUSTOMERS VALUES(1234, 'John', 'Doe', '123 8th Avenue', 'New York', 'NY', '10019', 'US');",
"SessionID": "100050",
"ApplicationUserName": "jane",
"XSApplicationUserName": "USER1",
"ApplicationName": "hdbsql",
"StatementUserName": "USER1"
}SAP HANA Database table audit trail
NXLog Agent can read records from a SAP HANA database using the im_odbc input module. To establish a connection to the database, this module requires an ODBC client to be installed. For ODBC setup instructions, see the SAP documentation on how to Connect to SAP HANA via ODBC. The ODBC driver is installed as part of the SAP HANA Client.
Audit entries are accessible through the AUDIT_LOG public system view.
Only SELECT operations can be performed on this view by users with the AUDIT OPERATOR or AUDIT ADMIN system privilege.
See Audit Trail Layout for Trail Target Database Table in the SAP documentation for a complete list of available fields.
The configuration below uses the im_odbc input module to read records from the AUDIT_LOG view. im_odbc requires the ConnectionString directive to be specified.
The SQL directive must specify a SELECT statement that returns an incremental id field and contains a WHERE clause specifying the same field to filter the result.
Since the AUDIT_LOG view does not contain an integer-base unique ID, this configuration uses the timestamp as the identifier field and specifies the IdType directive.
Once a record is processed, the id field is removed and the record is converted to JSON format.
<Extension json>
Module xm_json
</Extension>
<Input hdb_audit>
Module im_odbc
ConnectionString DRIVER={HDBODBC};UID=<user>;PWD=<password>; \
SERVERNODE=hxehost:39013;
SQL SELECT TIMESTAMP as id, * FROM AUDIT_LOG WHERE \
TIMESTAMP > ?
IdType timestamp
Exec delete($id); to_json();
</Input>The following JSON object shows a SAP HANA database audit record after it was processed by NXLog Agent.
{
"TIMESTAMP": "2021-07-18T16:33:32.482068+02:00",
"HOST": "hxehost",
"PORT": 39001,
"SERVICE_NAME": "nameserver",
"CONNECTION_ID": 100127,
"CLIENT_HOST": "hxehost",
"CLIENT_IP": "127.0.0.1",
"CLIENT_PID": 2475,
"CLIENT_PORT": 40401,
"USER_NAME": "USER1",
"STATEMENT_USER_NAME": "USER1",
"APPLICATION_NAME": "hdbsql",
"APPLICATION_USER_NAME": "hxeadm",
"XS_APPLICATION_USER_NAME": "USER1",
"AUDIT_POLICY_NAME": "CUSTOMERSAUDIT",
"EVENT_STATUS": "SUCCESSFUL",
"EVENT_LEVEL": "INFO",
"EVENT_ACTION": "INSERT",
"SCHEMA_NAME": "HOTEL",
"OBJECT_NAME": "CUSTOMERS",
"PRIVILEGE_NAME": null,
"ROLE_SCHEMA_NAME": null,
"ROLE_NAME": null,
"GRANTEE_SCHEMA_NAME": null,
"GRANTEE": null,
"GRANTABLE": null,
"FILE_NAME": null,
"SECTION": null,
"KEY": null,
"PREV_VALUE": null,
"VALUE": null,
"STATEMENT_STRING": "INSERT INTO HOTEL.CUSTOMERS VALUES(1234, 'John', 'Doe', '123 8th Avenue', 'New York', 'NY', '10019', 'US');",
"COMMENT": null,
"ORIGIN_DATABASE_NAME": null,
"ORIGIN_USER_NAME": null,
"EventReceivedTime": "2021-07-18T16:34:21.618841+02:00",
"SourceModuleName": "hdb_audit",
"SourceModuleType": "im_odbc"
}SAP Internet Communication Manager (ICM) and Web Dispatcher logs
SAP ICM is a component of the SAP NetWeaver Application Server and its job is to ensure that communication between the SAP system and external sources works properly. The SAP Web Dispatcher uses the same codebase as the ICM but it performs load balancing and passes requests to the ICM. Both of these components are configured using profile parameters.
To configure ICM logs, execute transaction RZ10 to add the required parameters to the current profile.
By default, logs will be written to the /usr/sap/<sap_system_id>/<instance>/work directory.
To configure Web Dispatcher logs, add the required parameters to the [profile] section of the webdispatcher.ini configuration file.
SAP ICM HTTP(S) access log
To enable HTTP(S) access logging, the following parameters need to be configured:
-
icm/HTTP/logging_<xx> for inbound requests
-
icm/HTTP/logging_client_<xx> for outbound requests
This example value configures HTTP(S) access log file names to http_access_log-*, sets the maximum file size to 10MB, specifies that a new log file should be created per day, and sets the log format to Common Log Format (CLF).
PREFIX=/, LOGFILE=http_access_log-%y-%m, MAXSIZEKB=10000, SWITCHTF=day, LOGFORMAT=CLF
This configuration uses the im_file input module to read logs from files with name starting with http_access_log. It uses a regular expression to parse CLF records into structured data, and converts the records to JSON using the xm_json module.
<Extension json>
Module xm_json
</Extension>
<Input http_access>
Module im_file
File '/path/to/http_access_log*'
<Exec>
if $raw_event =~ /(?x)^(\S+)\ \S+\ (\S+)\ \[([^\]]+)\]\ \"(\S+)\ (.+)
\ HTTP\/\d\.\d\"\ (\S+)\ (\S+)/
{
$Hostname = $1;
if $2 != '-' $AccountName = $2;
$EventTime = parsedate($3);
$HTTPMethod = $4;
$HTTPURL = $5;
$HTTPResponseStatus = $6;
if $7 != '-' $FileSize = $7;
}
else
{
$RawEvent = $raw_event;
}
to_json();
</Exec>
</Input>The following is an HTTP access log record. This input sample depicts the kind of data and format NXLog Agent will collect.
192.168.1.122 - - [19/Jul/2021:17:40:23 +0200] "GET /sap/public/icf_info/logon_groups HTTP/1.0" 200 13The following JSON object shows the same log record after it was processed by NXLog Agent.
{
"EventReceivedTime": "2021-07-19T17:41:48.369931+02:00",
"SourceModuleName": "http_access",
"SourceModuleType": "im_file",
"Hostname": "192.168.1.122",
"EventTime": "2021-07-19T17:40:23.000000+02:00",
"HTTPMethod": "GET",
"HTTPURL": "/sap/public/icf_info/logon_groups",
"HTTPResponseStatus": "200",
"FileSize": "13"
}SAP ICM TCP log
To enable TCP logging, the icm/TCP/logging parameter needs to be configured. This parameter only applies to SAP ICM.
This example value configures TCP log file names to tcp_log-*, sets the maximum file size to 10MB, and specifies that a new log file should be created per day.
LOGFILE=tcp_log-%y-%m, MAXSIZEKB=10000, SWITCHTF=day
The configuration below uses the im_file input module to read logs from files with name starting with tcp_log. It uses the xm_csv module to parse log records into structured data, and converts the records to JSON using the xm_json module.
<Extension json>
Module xm_json
</Extension>
<Extension icm_tcp_parser>
Module xm_csv
Fields $EventTime, $EventType, $Direction, $User, $Client, \
$ApplicationID, $LocalIP, $LocalPort, $PeerIP, $PeerPort, \
$DestinationHost, $DestinationPort, $WSHandle, \
$RootContextID, $ConnectionID, $ConnectionCounter, \
$LargeMessage, $PayloadLength, $Payload
Delimiter |
<Exec>
# Drop the record if it's the file header
if $raw_event =~ /^Date and Time.*/
{
drop();
}
</Exec>
</Extension>
<Input icm_tcp_log>
Module im_file
File '/path/to/tcp_log*'
<Exec>
icm_tcp_parser->parse_csv($raw_event);
to_json();
</Exec>
</Input>The following is a TCP log record. This input sample depicts the kind of data and format NXLog Agent will collect.
2021/07/21:10:02:30.090133|ERROR|IN||||192.168.1.120|12345|192.168.1.122|62722|||NO_HANDLE|||0000||0083|Protocol switch to TCP rejected, close connection (HTTP status code 403, Forbidden)The following JSON object shows the same log record after it was processed by NXLog Agent.
{
"EventReceivedTime": "2021-07-21T10:28:02.957226+02:00",
"SourceModuleName": "icm_tcp_log",
"SourceModuleType": "im_file",
"EventTime": "2021/07/21:10:02:30.090133",
"EventType": "ERROR",
"Direction": "IN",
"LocalIP": "192.168.1.120",
"LocalPort": "12345",
"PeerIP": "192.168.1.122",
"PeerPort": "62722",
"WSHandle": "NO_HANDLE",
"ConnectionCounter": "0000",
"PayloadLength": "0083",
"Payload": "Protocol switch to TCP rejected, close connection (HTTP status code 403, Forbidden)"
}SAP ICM Security log
The security log is useful for administrators and can help in identifying unauthorized access to the system. It logs irregular behavior such as logon errors, requests for data with invalid format, and attempted access to objects that do not exist. To enable security logging, the icm/security_log parameter needs to be configured.
This example value configures security log file names to dev_icm_sec-*, sets the maximum file size to 10MB, and specifies that a new log file should be created per day.
LOGFILE=dev_icm_sec-%y-%m, MAXSIZEKB=10000, SWITCHTF=day
The configuration below uses the im_file input module to read logs from files with name starting with dev_icm_sec. It uses the xm_multiline module to read log records that span multiple lines, and parses the log records into structured data using a regular expression. This configuration is specifically collecting security warnings and drops any other type of record.
define REGEX /\*{6}\s+(S.*\s*W.*G)\s+\*{6}\n\*{83}\n(.+)\nError: (.+)\n([^\*]*)/
<Extension json>
Module xm_json
</Extension>
<Extension icm_security>
Module xm_multiline
HeaderLine /^\*{6}[^\*]*\*{6}$/
</Extension>
<Input icm_sec_log>
Module im_file
File '/path/to/dev_icm_sec*'
InputType icm_security
<Exec>
if $raw_event =~ %REGEX%
{
$EventType = $1;
$EventTime = parsedate($2);
$Error = $3;
$ConnectionInfo = $4;
}
else
{
drop();
}
to_json();
</Exec>
</Input>The following is a security warning from the security log. This input sample depicts the kind of data and format NXLog Agent will collect.
***********************************************************************************
****** SECURITY WARNING ******
***********************************************************************************
Wed Jul 21 10:02:30 2021
Error: Protocol error (-21), Protocol switch to TCP rejected, close connection (HTTP status code 403, Forbidden) [http_plg.c 5678]
CONNECTION (id=1/22):
used: 1, type: default, role: Server(1), stateful: 0
nihdl: 116, ssl: (nil), protocol: TCP(29)
local host: 192.168.1.120:12345 ()
remote host: 192.168.1.122:62722 ()
***********************************************************************************The following JSON object shows the same log record after it was processed by NXLog Agent.
{
"EventReceivedTime": "2021-07-21T10:02:55.698402+02:00",
"SourceModuleName": "icm_sec_log",
"SourceModuleType": "im_file",
"EventType": "SECURITY WARNING",
"EventTime": "2021-07-21T10:02:30.000000+02:00",
"Error": "Protocol error (-21), Protocol switch to TCP rejected, close connection (HTTP status code 403, Forbidden) [http_plg.c 5678]",
"ConnectionInfo": "CONNECTION (id=1/22):\n used: 1, type: default, role: Server(1), stateful: 0\n nihdl: 116, ssl: (nil), protocol: TCP(29)\n local host: 192.168.1.120:12345 ()\n remote host: 192.168.1.122:62722 ()\n"
}