NXLog Legacy Documentation

Microsoft Azure and Office 365

This add-on has been replaced by im_ms365 as of NXLog Enterprise Edition version 5.7 and is no longer supported.

This NXLog add-on can retrieve information about various user, admin, system, and policy actions and events from Microsoft Azure and Office 365. Once configured, the add-on prints syslog events, each with a JSON payload, to standard output for processing by NXLog.

The add-on supports collecting logs from the following reports corresponding to the supported Microsoft REST-based APIs:

Azure Active Directory reports (based on Microsoft Graph API) — Sign-In events and directory audit log events.
Office 365 Management Activity API — Azure Active Directory Audit events, Exchange Audit events and SharePoint Audit logs using the Audit.AzureActiveDirectory, Audit.Exchange, and Audit.SharePoint parameters.

For more information about the log sources, see the links below:

Prerequisites

In order to complete the steps in this section and collect logs from the above mentioned APIs, the following prerequisites laid out in this section will need to be met.

During the steps explained in this section you need to make a note of the following data:

client-id
tenant-id
tenant-domain <domainname>.onmicrosoft.com
cert-thumbprint
cert-keyfile <certkey.pem>

Azure requirements and permissions

Access to the Azure Management Portal with a tenant user having the necessary permissions to make the configuration changes
An Azure Active Directory application with appropriate permissions and licences
Audit log search to be turned on, for audit logging to work

Some of the add-on arguments (parameters) require certain permissions set in Microsoft Azure. They are listed in the table below with a reference to the Microsoft documentation. Their configuration is detailed in the Parameters section below.

Table 1. Required permissions
Parameter	API used	Azure AD permissions required	Microsoft Docs link
`--enable-azure-ad-reports`	Microsoft Graph API v1.0	AuditLog.Read.All and Directory.Read.All	See reference link, and reference link
`--management-activity-sources`	Office 365 Management APIs	ActivityFeed.Read	See reference link

Required Microsoft licenses

Depending on the arguments in use, certain Microsoft licenses or service plans need to be active.

Parameter License required Reference link

Parameter	License required	Reference link
`--enable-azure-ad-reports`	An Azure Active Directory Premium license (either `AAD_PREMIUM` or `AAD_PREMIUM_P2`), or a license that includes it	See reference link
`--management-activity-sources`	An Office 365 license, or a license that includes it	See reference link

--enable-azure-ad-reports

An Azure Active Directory Premium license (either AAD_PREMIUM or AAD_PREMIUM_P2), or a license that includes it

See reference link

--management-activity-sources

An Office 365 license, or a license that includes it

See reference link

As Microsoft’s licensing information can be subject to change at any time, always double-check your current requirements with the licensing/service plan documentation. The required licenses can be managed in the Microsoft 365 admin center.

The above table with the licensing requirements are for informational purposes only.

Setup procedure

The complete procedure includes installing the NXLog Microsoft Azure and Office 365 add-on, setting up a Microsoft Azure AD application with its required permissions, and generating a certificate.

Installing the Microsoft Azure and Office 365 NXLog add-on

Install the add-on with dpkg:
```
# dpkg -i nxlog-msazure-<version>.deb
```
If the previous command exits non-zero, resolve any missing dependencies:
```
# apt-get -f install
```

The installation can be found under /opt/nxlog-msazure.

The nxlog-msazure add-on depends on nxlog or nxlog-trial.

Create an Azure AD application to access the APIs

To access information from your directory, you must create an application in your Azure Active Directory tenant and grant the appropriate read permissions to access the data.

Carry out the steps described in Register an application section.

Once the new application has been registered, make note of the Application (client) ID (this will be the client-id), as well as the Directory (tenant) ID (this will be tenant-id) on the Overview page for the new application.

Recording the Application and Directory ID

Grant permissions to the application

Grant the required permissions and grant admin consent to the above created application by following the steps in the Grant permissions section of the Microsoft documentation.

Grant the following permissions as described previously:

For the Microsoft Graph API:

AuditLog.Read.All
Directory.Read.All

For the Office 365 Management APIs:

ActivityFeed.Read

Once your permissions are set up and the Admin consent is granted, your permission list should look like the one below.

Generate and set up an X.509 certificate

The log collection process uses service-to-service calls via the Microsoft REST-based APIs, so it is important to generate and set up an X.509 certificate for authenticating to the service. A gencertkey.sh script is provided for Linux that can be used to simplify the process. It creates the private key-pair in a certkey.pem file in the working directory. The script is located in the /opt/nxlog-msazure/bin/ directory.

The gencertkey.sh script depends on the openssl toolkit and the uuidgen program. Install the corresponding packages if necessary.

On Debian-based platforms:

# apt install openssl uuid-runtime

On Centos/RedHat platforms:

# yum install openssl util-linux

Follow the steps below to generate the X.509 certificate and insert the relevant portion into the manifest file in Microsoft Azure:

Generate the certificate with the gencertkey.sh script on the computer where the add-on is installed.

$ ./gencertkey.sh
Generating a RSA private key
............+++++
................................................+++++
writing new private key to 'certkey.pem'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:
Email Address []:
ThumbPrint:0nFt3fB0JP7zuSmHaRQtmsFNYqo=
"keyCredentials": [
{
    "customKeyIdentifier":"0nFt3fB0JP7zuSmHaRQtmsFNYqo=",
    "keyId":"629ab88d-1059-454b-b258-4ca05b46dee4",
    "type":"AsymmetricX509Cert",
    "usage":"Verify",
    "value":"MIIDXTCCAkWgAwIBAgIJAP+XrnwhAxjOMA0GCSqGSIb3DQEBCwUAMEUxCzAJB..."
}
],

Make note of the base64-encoded certificate fingerprint value after ThumbPrint: (cert-thumbprint), and the KeyCredentials portion (which will be used in the following steps).

In the App registration page in Microsoft Azure, select Manifest on the left side and click Download.

Edit the downloaded manifest file and replace the "empty" KeyCredentials section with the previously generated output.

From

"keyCredentials": [],

"keyCredentials": [
{
    "customKeyIdentifier":"0nFt3fB0JP7zuSmHaRQtmsFNYqo=",
    "keyId":"629ab88d-1059-454b-b258-4ca05b46dee4",
    "type":"AsymmetricX509Cert",
    "usage":"Verify",
    "value":"MIIDXTCCAkWgAwIBAgIJAP+XrnwhAxjOMA0GCSqGSIb3DQEBCwUAMEUxCzAJB..."
}
],

Save the modified manifest and upload it.

Follow the steps below to move the generated certificate files to their intended directory as well as make the required permission changes:

Move the certificates you generated into the /opt/nxlog-msazure/conf directory. This directory is used later on as a value for the --working-dir parameter.
```
$ mv cert* /opt/nxlog-msazure/conf/
```
Set the file ownership and permissions to be in agreement with the User and Group directives (NXLog runs under the nxlog user and nxlog group by default).
```
$ chown nxlog:nxlog /opt/nxlog-msazure/conf/*
$ chmod 750 /opt/nxlog-msazure/conf/cert*
```

Parameters

Certain parameters need to be passed to the NXLog Microsoft Azure and Office 365 add-on as arguments in order to achieve the desired outcome. These parameters can be passed to the add-on by using the Arg directive.

Mandatory parameters

The add-on requires the following mandatory parameters. Details about these parameters and their values are listed in the Prerequisites section.

--client-id=

The Azure App registration Application (client) ID

--tenant-id=

The Azure App registration Directory (tenant) ID

--tenant-domain=

The domain name created in Microsoft Azure AD <domainname>.onmicrosoft.com

--cert-thumbprint=

The certificate fingerprint value

--cert-keyfile=

The certificate key file certkey.pem

--working-dir=

The path where the add-on is run, which is /opt/nxlog-msazure/conf by default

The --cert-keyfile path is always relative to the --working-dir.

Source parameters

To specify the data sources, use the following parameters.

--enable-azure-ad-reports: Active Directory sign-in events and directory audit logs ( based on Microsoft Graph API ). This parameter does not require any value to be passed to it.
--management-activity-sources=: Office 365 Management Activity API

The available values are: Audit.Exchange, Audit.SharePoint, Audit.AzureActiveDirectory

Optional parameters

These parameters are already defined in the built-in configuration of the add-on (default configuration can be printed with --print-default-config parameter), therefore they are not mandatory. However, the default parameters can be overridden by defining any parameters that might require non-default values.

--top=n: The top parameter works only with Azure Active Directory reports and events. It returns a subset of the entries for the given report, consisting of the first n entries, where n is a positive integer. top=5 returns the 5 most recent audit report events. top will be overridden where start-date and end-date can be used—top is lower priority.
--start-date=YYYY-MM-DDTHH:MM:SSZ|amonthago|aweekago|yesterday
--end-date=YYYY-MM-DDTHH:MM:SSZ|amonthago|aweekago|yesterday|now: The start-date and end-date parameters specify the time range of content to return. These parameters work with all Office 365 reports and most of the Azure Active Directory reports. Where start/end ranges are not supported, the add-on uses top. The amonthago, aweekago, yesterday, and now values are dynamic and calculated in every loop.

To pull reports from the last 24 hours, use: --start-date=yesterday --end-date=now
--sleep=n: The script will sleep n seconds between loops.

NXLog configuration examples

Once all the details have been collected, the NXLog configuration file /opt/nxlog/etc/nxlog.conf needs to be edited and augmented with the relevant details.

Example 1. Azure Active Directory events

This configuration collects all the Azure Active Directory report events, such as user creation, group membership, permission changes and so on. The output provided by Microsoft is in JSON format.

nxlog.conf

<Extension _json>
    Module  xm_json
</Extension>

<Input msazure>
    Module  im_exec
    Command /opt/nxlog-msazure/bin/msazure
    Arg     --client-id=912497ba-9780-46bc-a6a6-3a56a4c14278
    Arg     --tenant-id=e681b493-14a8-438b-8bbf-d65abdc826c2
    Arg     --tenant-domain=contoso.onmicrosoft.com
    Arg     --cert-thumbprint=D64Rm2IkRQxp26XK4Da7Bcbqu2o=
    Arg     --cert-keyfile=certkey.pem
    Arg     --working-dir=/opt/nxlog-msazure/conf
    Arg     --enable-azure-ad-reports
    Exec    parse_json();
</Input>

Output of a delete user event in JSON format

{
  "activityDateTime": "2020-05-21T10:27:24.7742514Z",
  "activityDisplayName": "Delete user",
  "additionalDetails": [],
  "category": "UserManagement",
  "correlationId": "3fc2e655-491b-4edd-a450-a7d60ec3aff2",
  "id": "Directory_3fc2e655-491b-4edd-a450-a7d60ec3aff2_S3OF7_28513191",
  "initiatedBy": {
    "app": null,
    "user": {
      "displayName": null,
      "id": "6a304e04-3ebd-4190-b128-efe4d5c7e664",
      "ipAddress": "51.105.112.41",
      "userPrincipalName": "nxlogadmin@testnxlog.onmicrosoft.com"
    }
  },
  "loggedByService": "Core Directory",
  "operationType": "Delete",
  "result": "success",
  "resultReason": "",
  "targetResources": [
    {
      "displayName": null,
      "groupType": null,
      "id": "de80979d-026b-4282-91ac-eb1925b94718",
      "modifiedProperties": [
        {
          "displayName": "Is Hard Deleted",
          "newValue": "\"False\"",
          "oldValue": null
        }
      ],
      "type": "User",
      "userPrincipalName": "de80979d026b428291aceb1925b94718johndoe@testnxlog.onmicrosoft.com"
    }
  ]
}

Example 2. Office 365 events

This configuration collects Office 365 related events, such as document creation, deletion, permission changes and so on. The output provided by Microsoft is in JSON format.

nxlog.conf

<Extension _json>
    Module  xm_json
</Extension>

<Input msazure>
    Module  im_exec
    Command /opt/nxlog-msazure/bin/msazure-pull
    Arg     --client-id=912497ba-9780-46bc-a6a6-3a56a4c14278
    Arg     --tenant-id=e681b493-14a8-438b-8bbf-d65abdc826c2
    Arg     --tenant-domain=contoso.onmicrosoft.com
    Arg     --cert-thumbprint=D64Rm2IkRQxp26XK4Da7Bcbqu2o=
    Arg     --cert-keyfile=certkey.pem
    Arg     --working-dir=/opt/nxlog-msazure/conf
    Arg     --management-activity-sources=Audit.Exchange,Audit.SharePoint,Audit.AzureActiveDirectory
    Exec    parse_json();
</Input>

Output of a modified document event in JSON format

{
  "ClientIP": "20.40.136.153",
  "CorrelationId": "0b98549f-0056-2000-baa9-211499d2b0e1",
  "CreationTime": "2020-05-21T13:02:22",
  "EventSource": "SharePoint",
  "Id": "1afa6393-6d9f-44dd-72a6-08d7fd8733d5",
  "ItemType": "File",
  "ListId": "d1df9d8a-25ad-4173-a9e3-c0dce2675f9a",
  "ListItemUniqueId": "ebcd6f01-564a-467f-a24a-b1a20c44b907",
  "ObjectId": "https://testnxlog.sharepoint.com/sites/nxlogtest/Shared Documents/Secret.xlsx",
  "Operation": "FileModified",
  "OrganizationId": "a78f0974-05ea-44c8-9ba3-3edaee870793",
  "RecordType": 6,
  "Site": "0ba16f09-d3b9-4827-b8bb-77f00694d6af",
  "SiteUrl": "https://testnxlog.sharepoint.com/sites/nxlogtest/",
  "SourceFileExtension": "xlsx",
  "SourceFileName": "Secret.xlsx",
  "SourceRelativeUrl": "Shared Documents",
  "UserAgent": "MSWAC",
  "UserId": "nxlogadmin@testnxlog.onmicrosoft.com",
  "UserKey": "i:0h.f|membership|10032000ba9b0c07@live.com",
  "UserType": 0,
  "Version": 1,
  "WebId": "c7b9cd18-c3a0-437e-99be-eba97cf33f09",
  "Workload": "SharePoint"
}

Running in standalone mode

Although the Microsoft Azure and Office 365 add-on is designed to work and collect logs as part of NXLog, it can be run in standalone mode from a Linux terminal.

The first NXLog configuration example above would look like the one below if it were invoked from a terminal console. In this case, the received events would be continuously printed to the terminal.

Example 3. Azure Active Directory events in standalone mode

$ /opt/nxlog-msazure/bin/msazure                       \
      --client-id=912497ba-9780-46bc-a6a6-3a56a4c14278 \
      --tenant-id=e681b493-14a8-438b-8bbf-d65abdc826c2 \
      --tenant-domain=contoso.onmicrosoft.com          \
      --cert-thumbprint=D64Rm2IkRQxp26XK4Da7Bcbqu2o=   \
      --cert-keyfile=certkey.pem                       \
      --working-dir=/opt/nxlog-msazure/conf            \
      --enable-azure-ad-reports