NXLog Legacy Documentation

Google Logging API

This add-on has been replaced by im_googlelogging and om_googlelogging as of NXLog Enterprise Edition version 5.7 and is no longer supported.

This NXLog add-on can exchange log data with the Google Logging API using gRPC and protocol buffers. The input part of the add-on retrieves logs, and the output component pushes them to the Logging API files.

This add-on runs on Windows, Linux, and macOS operating systems.

The setup is a two-part procedure:

Configuring the Google Cloud account
Installing and configuring NXLog

Configuring the Google Cloud account

This subsection provides the minimum steps for configuring a Google Cloud account.

If you don’t already have an account, create one on the Google Account page of the Google website.
While logged in to Google, go to the Manage resources page.
On the Manage resources page, click CREATE PROJECT.
On the New Project page, specify the project name and organization, then click CREATE.

On the Manage resources page, click the Overflow menu (vertical ellipsis) to the right of the project name, then click Settings from the drop-down menu.
On the IAM & Admin page, click the Navigation menu (three-line menu) in the upper left-hand corner of the window and navigate to APIs & Services > Credentials.
On the Credentials page, click CREATE CREDENTIALS, then select Service account from the drop-down list.
On the Create service account page, specify the service account name and click CREATE. Select the Owner role from the Role drop-down list, then click DONE.
Under the Service accounts table, click the service account link to open its configuration page.
On the service account page, click the KEYS tab, expand the ADD KEY drop-down list and select Create new key.
In the Create private key for "<YOURACCOUNTNAME>" dialog window, select JSON and click CREATE to download the private key. Save the private key file as credentials.json. This file will be used later in the configuration.

Installing and configuring NXLog

The add-on consists of two files for handling the input and output log traffic respectively. Each add-on file reads its configuration from the config.json file stored in the same directory.

NXLog is configured separately for collecting log data and forwarding them to the Logging API.

For ease of configuration and setup the following script can be used to create some of the required folders for the examples. Note that other folders can be used as required.

Bash script for installing the Google Logging API add-on on Linux platforms. Run this script in the directory that the credentials.json file is saved.

Installation script for Google add-on

#!/bin/bash
if [[ -f "./credentials.json" ]]
then
        echo "Beginning installation of NXLog Google Add-On"
else
        echo "Credentials file missing, ensure that credentials.json file \
        is on the same folder as the installer"
        exit
fi

echo "Installing package"
if [[ -f "/etc/debian_version" ]]
then
        echo "Installing Debian based package"
        dpkg -i nxlog-google_1.0.1_amd64.deb
elif [[ -f "/etc/redhat-release" ]]
then
        echo "Installing RHEL based package"
        rpm -i nxlog-google-1.0.1-1.x86_64.rpm
fi

echo "Creating folders"
mkdir /opt/nxlog-google-cloud/{conf,log,input,output}
chown nxlog /opt/nxlog-google-cloud
echo "Credentials permissions"
cp ./credentials.json /opt/nxlog-google-cloud/conf/
chown nxlog /opt/nxlog-google-cloud/conf/credentials.json
chmod 400 /opt/nxlog-google-cloud/conf/credentials.json
echo "Finished setup"

The script can be skipped if the add-on has already been installed manually. However, you must manually set up the folders and permissions as in the script to follow the below examples.

You can run the script even if you manually installed the Google Logging API add-on. It will skip the installation step, and set up the folders and permissions correctly.

Collecting data from the Google Logging API

Example 1. Collecting data from the Google Logging API

Below is a sample add-on configuration for collecting data from the Google Cloud logging infrastructure. This configuration should be saved as a config.json file and placed in the same directory as the input add-on file.

The configuration accepts relative path names in case the application directory is specified in the NXLog configuration. Otherwise, absolute path names can be used instead.

config.json

{
    "credentials": "./conf/credentials.json",
    "resourceNames": [
        "projects/nxlog"
    ],
    "filter": "logName=\"projects/nxlog/logs/0\"",
    "pollInterval": 30,
    "verbose": {
        "active": true,
        "file": "./log/input.log"
    },

    "cursor": {
        "active": true,
        "file": "./input-cursor.json"
    }
}

The following table explains each configuration parameter from the configuration above.

Table 1. Input add-on configuration
Parameter name	Format	Description
`credentials`	String	Path to the file containing a private key and other information needed for successful authorization with the Google Cloud account, see the Configuring the Google Cloud account section.
`resourceNames`	Array of strings	Names of resources to be monitored. Each element is a string of the form `projects/<YOURPROJECTNAME>`, where `<YOURPROJECTNAME>` is the placeholder for the project name.
`filter`	String	Expression used for querying log files from the cloud. String of the form `logName=\"project/<YOURPROJECTNAME>/logs/<LOG_ID>\"`, where `<YOURPROJECTNAME>` is the placeholder for the actual project name, and the `<LOG_ID>` should be replaced with the actual log source name. For more details, see the Advanced log queries page on the Google Cloud website.
`verbose`	JSON object	Object for configuring the history of on-premises recording function calls. The `active` boolean key indicates whether logging should be enabled, and the `file` key specifies the file to write the data to. Normally, this parameter should be disabled by setting the `active` key to `FALSE`.
`pollInterval`	Integer	Specifies the interval in seconds to check for new entries.
`cursor`	JSON object	This parameter provides additional filtering of entries through recording the `timestamp` and `insertId` values of the last downloaded message to a separate JSON file. After starting the add-on, these values are read and used as a point to continue collecting the entries from. The `cursor` parameter consists of the `active` and `file` keys. The `active` boolean key indicates whether the cursor should be applied, and the `file` specifies the file to write the `timestamp` and `insertId` values to. In case this parameter is enabled along with the timestamp explicitly specified in the `filter` parameter, the `cursor` takes precedence and the timestamp of the `filter` parameter will be overwritten.

This NXLog configuration uses the im_exec module with several arguments. For clarity, all directives and arguments are supplied with comments.

nxlog.conf

# Defines part of the directory paths used in this configuration
define GBASE /opt/nxlog-google-cloud

<Extension _json>
    Module    xm_json
</Extension>

<Input from_exec>
    Module    im_exec
    # Path to the input add-on file
    Command   %GBASE%/bin/google-cloud-input
    # Denotes the configuration file usage
    Arg       -c
    # Path to the configuration file
    Arg       %GBASE%/input/config.json
    # Enables using the application directory and relative paths to files
    Arg       -e
    # Path to the application directory
    Arg       %GBASE%
</Input>

Forwarding logs to the Google Logging API

This example explains how to use the output add-on to forward system logs to the Google Logging API.

Example 2. Forwarding logs to the Google Logging API

Below is the add-on configuration for pushing data to the Google Logging API. This configuration should be saved as a config.json file and placed in the same directory as the output add-on file.

The configuration allows using relative path names in case the application directory is specified in the NXLog configuration. Otherwise, absolute file paths should be used instead.

config.json

{
    "credentials": "./conf/credentials.json",
    "logName": "projects/nxlog/logs/0",
    "resourceType": "global",
    "labels": {
        "LabelOne": "ValueOne",
        "LabelTwo": "ValueTwo"
    },
    "verbose": {
        "active": true,
        "file": "./log/output.log"
    }
}

The following table explains each configuration parameter.

Table 2. Output add-on configuration
Parameter name	Format	Description
`credentials`	String	Path to the file containing a private key and other information needed for successful authentication with the Google Cloud account, see the Configuring the Google Cloud account subsection.
`logName`	String	Name of the log source on Google Cloud. String of the form `project/<YOURPROJECTNAME>/logs/<LOG_ID>`, where `<YOURPROJECTNAME>` is a placeholder for the actual project name, and `<LOG_ID>` should be replaced with the actual log source name. For more information about querying Google Cloud, see the Advanced log queries page on the Google Cloud website.
`resourceType`	String	The resource type monitored by the Google Logging API. For more information, see the Monitored resource types section of the Google Cloud API Reference. By default, the `global` value can be used here.
`labels`	JSON object	Custom metadata that can be added to the log entries under the `labels` field.
`verbose`	JSON object	Object for configuring the history of on-premises recording function calls. The `active` boolean key indicates whether logging should be enabled, and the `file` key specifies the file to write the data to. Normally, this parameter should be disabled by setting the `active` key to `FALSE`.

The NXLog configuration below uses the im_systemd module to read systemd logs from Linux. The entries are then converted to JSON using the xm_json module. The om_exec module is used to serialize the data and forward the result to the Google Logging API.

The om_exec module configuration contains comments explaining the meaning of each directive and argument.

nxlog.conf

# Defines part of the directory paths used in this configuration
define GBASE /opt/nxlog-google-cloud

<Extension _json>
    Module    xm_json
</Extension>

<Input from_systemd>
    Module    im_systemd
</Input>

<Output to_exec>
    Module    om_exec
    # Path to the output add-on file
    Command   %GBASE%/bin/google-cloud-output
    # Denotes the configuration file usage
    Arg       -c
    # Path to the add-on configuration file
    Arg       %GBASE%/output/config.json
    # Enables using the application directory and relative paths to files
    Arg       -e
    # Path to the application directory
    Arg       %GBASE%
    Exec      to_json();
</Output>

Disclaimer

While we endeavor to keep the information in this topic up to date and correct, NXLog makes no representations or warranties of any kind, express or implied about the completeness, accuracy, reliability, suitability, or availability of the content represented here.

The accurateness of the content was tested and proved to be working in our lab environment at the time of the last revision with the following software versions:

NXLog 5.4.7313
Linux (Debian 10)
NXLog Google Add-On 1.0.1
Google Cloud available at the time of revision

Last revision: 15 February 2022