Microsoft Azure (im_azure)
This module can be used to collect logs from Microsoft Azure applications. It can operate in three different modes to connect to the following data sources:
-
Azure Table Storage
-
Azure Blob Storage
-
Azure Log Analytics workspace
| To examine the supported platforms, see the list of installation packages. |
Storage setup
For Blob or Table modes, Azure web application logging and storage can be configured from the Azure Management Portal by following these steps:
-
After logging in to the Portal, open the left portal menu, select Storage accounts, and then select Create.
-
Create the new storage account, providing a storage account name, region, and redundancy type. Take note of the storage name, you will need to specify this value in the NXLog Agent configuration. More information on the storage settings can be found in the Microsoft documentation on how to Create a storage account.
-
On the Review + create tab, click Create and wait for storage setup to complete.
-
Navigate to your app and select App Service logs.
-
Select On for Application Logging (Blob).
-
Configure Storage Settings corresponding with the storage account created above. More information on the configuration settings can be found in the Microsoft documentation on how to Enable diagnostic logging for apps.
-
Confirm the changes by clicking Save, then restart the service. Note that it may take a while for Azure to create the table and/or blob in the storage.
Log Analytics workspace setup
For Analytics mode, an application needs to be registered and
granted the necessary permissions to read from the Log Analytics workspace.
Follow the instructions to Register an application in Microsoft Entra ID.
Once you have created the new application, open the settings for your new app and take note of the Application (client) ID. You will need to specify this value for the ClientID directive.
Grant the application permission to use the Log Analytics API and create a client secret:
-
From the left menu, select API permissions and then + Add a permission. Select APIs my organization uses and choose Log Analytics API.
-
Select Application permissions for the type of permission required, and Data.Read for the permission.
-
Click Add permissions to create the permission.
-
From the left menu, select Certificates & secrets and then + New client secret. Enter a description and expiration period, then click Add.
-
Take note of the Value of the new secret, you will need to specify this for the SharedKey directive.
Add the app to your Log Analytics workspace:
-
Go to your Log Analytics workspace and take note of the Workspace ID. You will need to specify this value for the WorkspaceID directive.
-
From the left menu, select Access control (IAM). Click + Add and select Add role assignments.
-
Choose Log Analytics Reader for role, select the application registered above, and click Save.
Configuration
The im_azure module accepts the following directives in addition to the common module directives.
Required directives
The following directive is required for the module to start.
This mandatory directive specifies the type of service the module should connect to.
It accepts one of the following values: |
Table mode directives
The following are mandatory directives when using Table mode.
This directive specifies the authentication key to use for connecting to the Azure Storage account. |
|
This directive specifies the name of the storage account to connect to. |
|
This directive specifies the storage table from which to collect logs. |
Blob mode directives
The following are mandatory directives when using Blob mode.
This directive specifies either the name of the blob container or the path of a single blob (formatted as |
|
This directive specifies the authentication key to connect to the Azure Storage account. |
|
This directive specifies the name of the storage account to connect to. |
Analytics mode directives
The following are mandatory directives when using Analytics mode.
The |
|
The Microsoft Entra ID application client secret. |
|
This directive specifies the storage table from which to collect logs. |
|
The ID of the Microsoft Entra tenant to connect to. |
|
This directive specifies the workspace ID of the Log Analytics account from which to collect logs. |
HTTPS directives
The following directives configure secure data transfer via HTTPS.
Set this directive to |
||||||||||||||||
Set this directive to |
||||||||||||||||
Set this directive to |
||||||||||||||||
Set this directive to the path of a directory containing Certificate Authority (CA) certificates.
The module uses these certificates to verify the certificate presented by the remote host.
Name the certificate files using the OpenSSL hashed format: the hash of the certificate followed by
For example, if the certificate hash is To trust a remote host’s self-signed certificate, include a copy of it in this directory. If this directive isn’t specified, NXLog Agent falls back to the default operating system root certificate store.
Unix-like operating systems commonly store root certificates in In addition, Microsoft’s PKI repository contains root certificates for Microsoft services. This directive is mutually exclusive with the HTTPSCAThumbprint, HTTPSCAPattern, and HTTPSSearchAllCertStores directives. |
||||||||||||||||
Set this directive to the path of the Certificate Authority (CA) certificate used to verify the certificate presented by the remote host. To trust a remote host’s self-signed certificate, specify the remote host’s certificate itself. For certificates signed by an intermediate CA, the specified certificate must contain the complete certificate chain (certificate bundle). This directive is mutually exclusive with the HTTPSCAThumbprint, HTTPSCAPattern, and HTTPSSearchAllCertStores directives. |
||||||||||||||||
Set this directive to a PCRE2-compliant regular expression to locate a suitable Certificate Authority (CA) certificate. This directive is supported on Windows and macOS. On Windows, the certificate must be in the Windows Certificate Store. On macOS, the certificate must be in the system keychain. NXLog Agent matches the pattern against a string built from the certificate’s fields in this order:
The pattern doesn’t need to match every field.
Include only the fields you want to match, and use For example, the following pattern matches only the
This produces the following output in the NXLog Agent log file:
If the pattern matches multiple certificates, NXLog Agent uses the first match. If the certificate changes, restart NXLog Agent to apply the new certificate. This directive is mutually exclusive with the HTTPSCAThumbprint, HTTPSCADir, HTTPSCAFile, and HTTPSSearchAllCertStores directives.
|
||||||||||||||||
Set this directive to the thumbprint of the Certificate Authority (CA) certificate used to verify the certificate presented by the remote host. This directive is supported on Windows and macOS. On Windows, copy the hexadecimal fingerprint string from Windows Certificate Manager (certmgr.msc). The certificate must be in a Windows certificate store accessible by NXLog Agent. On macOS, obtain the hexadecimal fingerprint string from the Keychain Access application. The certificate must be accessible in the system keychain. Whitespaces and colons are automatically removed. This directive is mutually exclusive with the HTTPSCADir, HTTPSCAFile, HTTPSCAPattern, and HTTPSSearchAllCertStores directives.
|
||||||||||||||||
Set this directive to the path of the certificate file to present to the remote host during the HTTPS handshake. |
||||||||||||||||
Set this directive to the path of the private key file corresponding to the certificate specified by HTTPSCertFile. |
||||||||||||||||
Set this directive to a PCRE2-compliant pattern to identify the certificate and its corresponding private key. This directive is supported on Windows and macOS. NXLog Agent matches the pattern against a string built from the certificate’s fields in this order:
The pattern doesn’t need to match every field.
Include only the fields you want to match, and use On Windows, import the certificate in PFX format into the On macOS, ensure the certificate and its private key are present in the System Keychain ( During configuration, NXLog Agent resolves this directive into the corresponding HTTPSCertThumbprint value. If the pattern matches multiple certificates, NXLog Agent uses the first match. This directive is not dynamic; restart NXLog Agent to apply a changed certificate. We recommend keeping the certificate storage referenced by this directive well-maintained for optimal performance. This directive is mutually exclusive with the HTTPSCertThumbprint, HTTPSCertFile and HTTPSCertKeyFile directives. Configuration examples:
or
This produces the following output in the NXLog Agent log file:
|
||||||||||||||||
Set this directive to the thumbprint of the certificate to present to the remote host during the HTTPS handshake. Whitespaces and colons are automatically removed. This directive is supported on Windows and macOS. On Windows, copy the hexadecimal fingerprint string from Windows Certificate Manager (certmgr.msc).
Import the certificate to the
When the global directive UseCNGCertificates is set to
When UseCNGCertificates is set to On macOS, obtain the hexadecimal fingerprint string from the Keychain Access application.
The certificate and its associated private key must be present in the System Keychain ( For example:
|
||||||||||||||||
Set this directive to the path of a directory containing certificate revocation list (CRL) files.
The module uses these files to reject connections from hosts presenting a revoked certificate.
Name the files using the OpenSSL hashed format: the hash of the issuer followed by
For example, if the hash is |
||||||||||||||||
Set this directive to the path of the certificate revocation list (CRL) file used to reject connections from hosts presenting a revoked certificate. To generate a CRL file using OpenSSL:
|
||||||||||||||||
Set this directive to the path of a file containing Diffie-Hellman parameters for key exchange. Generate the parameters using dhparam(1ssl). If this directive is not set, the module uses default parameters. See the OpenSSL Wiki for further details. |
||||||||||||||||
Set this directive to the passphrase of the private key specified by HTTPSCertKeyFile. This directive applies only to encrypted private keys. To generate a private key with Triple DES encryption using OpenSSL:
|
||||||||||||||||
Set this directive to This directive is only supported on Windows. |
||||||||||||||||
Set this directive to |
||||||||||||||||
Set this directive to |
||||||||||||||||
Set this directive to override the permitted cipher list for TLSv1.2 and below.
Use the format described in the ciphers(1ssl) man page.
For example, specify
|
||||||||||||||||
Set this directive to override the permitted cipher list for TLSv1.3. Use the same format as HTTPSSSLCipher. Refer to the OpenSSL documentation for a list of valid TLS v1.3 cipher suites. The default is:
|
||||||||||||||||
Set this directive to
|
||||||||||||||||
Set this directive to restrict the TLS/SSL protocols the module accepts.
Specify a comma-separated list of the following values: |
||||||||||||||||
Set this directive to This directive depends on the values of the HTTPSAllowUntrusted and HTTPSRequireCert directives:
|
||||||||||||||||
Set this directive to the signature algorithm parameter to pass to the Windows SSL library. Accepted values depend on the available encryption providers. This directive is only supported on Windows. |
||||||||||||||||
Set this directive to the hostname to use for Server Name Indication (SNI) in HTTPS mode. If not set, the module uses the hostname specified in the URL directive. |
||||||||||||||||
Set this directive to |
Optional directives
This optional directive specifies the local port number of the connection. If not specified, a random high port number will be used, which may be unsuitable for firewalled network environments. |
|||||||||||||||||||||||||||||||||||||
This directive specifies how frequently the module will check for new events, in seconds.
If this directive is not specified, it defaults to 1 second.
Fractional seconds may be specified ( |
|||||||||||||||||||||||||||||||||||||
Set this directive to the number of seconds to wait between reconnection attempts. If you do not set this directive, the module starts with a 1 second interval and doubles it after each failed attempt, resetting to 1 second once it reconnects successfully.
|
|||||||||||||||||||||||||||||||||||||
Set this directive to |
|||||||||||||||||||||||||||||||||||||
This boolean directive instructs the module on where to start reading events from the log source when NXLog Agent starts. When When The default is The following matrix shows the outcome of this directive in conjunction with the SavePos directive:
If the NoCache directive is |
|||||||||||||||||||||||||||||||||||||
Set this directive to The default is If the NoCache directive is |
|||||||||||||||||||||||||||||||||||||
This directive specifies the URL for connecting to the storage account. The module works in a failover configuration if additional URL directives are specified on new lines. If the first URL is unreachable, the module automatically fails over to the next one. If the last URL is unreachable, the module fails over to the first URL. If this directive is not specified, it defaults to
|
Examples
When in Table mode, im_azure expects a table containing a Message field.
The value of this field is made available in the $raw_event field.
This configuration collects logs from a table and converts the data to JSON format using the xm_json extension module. Since the configuration uses a secure URL, the HTTPSCADir is specified and points to a folder that contains the Trusted Root CA certificates on the machine. Alternatively, the HTTPSCAFile directive can be used to specify a file containing the complete certificate chain for the Azure server, or the HTTPSAllowUntrusted directive to accept all certificates.
<Extension json>
Module xm_json
</Extension>
<Input azure_table>
Module im_azure
Mode Table
URL https://<storage_name>.table.core.windows.net/
HTTPSCADir /path/to/trusted/ca/cert/store
SharedKey storage_access_key
StorageName storage_name
TableName storage_table_name
Exec $Message = $raw_event; to_json();
</Input>
The following is a log record stored in an Azure table. It contains a Message
field and thus conforms to im_azure's data format requirements.
| PartitionKey | RowKey | Tiestamp | Message |
|---|---|---|---|
PK1 |
RK1 |
2021-08-10T14:14:59.0689266Z |
This is test message 1 |
The following is the same log message in JSON format after it was processed by NXLog Agent.
{
"ProcessID": 0,
"ThreadID": 0,
"EventTime": "2021-08-10T16:14:59.068926+02:00",
"EventReceivedTime": "2021-08-18T16:01:59.531635+02:00",
"SourceModuleName": "azure_table",
"SourceModuleType": "im_azure",
"Message": "This is test message 1"
}
When in Blob mode, im_azure expects data to be in CSV format and contain
the following header:
date,level,applicationName,instanceId,eventTickCount,eventId,pid,tid,message,activityId
This configuration collects logs from a blob container and converts the data to JSON format using the xm_json extension module. Since the configuration uses a secure URL, the HTTPSCADir is specified and points to a folder which contains the Trusted Root CA certificates on the machine. Alternatively, the HTTPSCAFile directive can be used to specify a file containing the complete certificate chain for the Azure server, or the HTTPSAllowUntrusted directive to accept all certifcates.
<Extension json>
Module xm_json
</Extension>
<Input azure_blob>
Module im_azure
Mode Blob
URL https://<storage_name>.blob.core.windows.net/
HTTPSCADir /path/to/trusted/ca/cert/store
BlobName blob_container_name
SharedKey storage_access_key
StorageName storage_name
Exec $Message = $raw_event; to_json();
</Input>
The following is a log record conforming to the CSV data format expected by im_azure.
2021-08-17T18:44:23.369931,2,MyWebApp,003,2,405,1001,1009,This is test message 1,3131
The following is the same log record in JSON format after it was processed by NXLog Agent.
{
"SourceName": "MyWebApp",
"SeverityValue": 4,
"Severity": "ERROR",
"ProcessID": 1001,
"ThreadID": 1009,
"EventTime": "2021-08-17T20:44:23.369931+02:00",
"EventReceivedTime": "2021-08-18T15:30:42.290532+02:00",
"SourceModuleName": "azure_blob",
"SourceModuleType": "im_azure",
"Message": "This is test message 1"
}
This configuration collects logs from an AuditLogs table in Azure Log
Analytics workspace. im_azure receives records from the Log Analytics API in
JSON format, which it then parses into structured data and writes each record
as a list of key-value pairs to the $raw_event field.
<Input azure_workspace>
Module im_azure
Mode Analytics
# Since the API uses HTTPS, SSL must be configured
HTTPSAllowUntrusted TRUE
ClientID ms_entra_app_id
SharedKey ms_entra_app_client_secret
TenantID ms_entra_tenant_id
WorkspaceID workspace_id
TableName AuditLogs
</Input>
The following is a record from the AuditLog table after it was processed by NXLog Agent.
2021-08-19 12:23:15 nxlog-server INFO TenantId="c1580b88-581a-4cd0-a5c8-a3dd46241740" SourceSystem="Azure AD" TimeGenerated="2021-08-11 18:30:30" ResourceId="/tenants/50fc51f4-477d-4a3c-8ea2-d9306b08461f/providers/Microsoft.aadiam" OperationName="Update application" OperationVersion="1.0" Category="ApplicationManagement" ResultType="" ResultSignature="None" ResultDescription="" DurationMs="0" CorrelationId="947f8943-691f-4393-833f-e64ca47dfd49" Resource="Microsoft.aadiam" ResourceGroup="Microsoft.aadiam" ResourceProvider="" Identity="" Level="4" Location="" AdditionalDetails="[{\"key\":\"User-Agent\",\"value\":\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36 Edg/92.0.902.67\"}]" Id="Directory_947f8943-691f-4393-833f-e64ca47dfd49_H1F9O_140032957" InitiatedBy="{\"user\":{\"id\":\"f6385f8d-9c20-4235-a2df-17b76774f75f\",\"displayName\":null,\"userPrincipalName\":\"admin@example.com\",\"ipAddress\":null,\"roles\":[]}}" LoggedByService="Core Directory" Result="success" ResultReason="" TargetResources="[{\"id\":\"c118f7ea-f476-478a-9b7a-e6ad2d7b6d77\",\"displayName\":\"nxlog-agent\",\"type\":\"Application\",\"modifiedProperties\":[],\"administrativeUnits\":[]}]" AADTenantId="50fc51f4-477d-4a3c-8ea2-d9306b08461f" ActivityDisplayName="Update application" ActivityDateTime="2021-08-11 18:30:30" AADOperationType="Update" Type="AuditLogs"