Google Logging API

This add-on is available for purchase. For more information, please contact us.

This NXLog add-on can exchange log data with the Google Logging API using gRPC and protocol buffers. The input part of the add-on retrieves logs, and the output component pushes them to the Logging API files.

This add-on runs on Windows, Linux, and macOS operating systems.

The setup is a two-part procedure:

Configuring the Google Cloud account

This subsection provides the minimum steps for configuring a Google Cloud account.

  1. If you don’t already have an account, create one on the Google Account page of the Google website.

  2. While logged in to Google, go to the Manage resources page.

  3. On the Manage resources page, click CREATE PROJECT.

    Manage resources page
  4. On the New Project page, specify the project name and organization, then click CREATE.

    The New Project page

    On the Manage resources page, click the Overflow menu (vertical ellipsis) to the right of the project name, then click Settings from the drop-down menu.

    Drop-down menu of the project
  5. On the IAM & Admin page, click the Navigation menu (three-line menu) in the upper left-hand corner of the window and navigate to APIs & Services > Credentials.

    The Credentials menu
  6. On the Credentials page, click CREATE CREDENTIALS, then select Service account from the drop-down list.

    Creating the service account
  7. On the Create service account page, specify the service account name and click CREATE. Select the Owner role from the Role drop-down list, then click DONE.

    The Create service account page
  8. Under the Service accounts table, click the service account link to open its configuration page.

    The Service accounts table
  9. On the service account page, click the KEYS tab, expand the ADD KEY drop-down list and select Create new key.

    Downloading the private key
  10. In the Create private key for "<YOURACCOUNTNAME>" dialog window, select JSON and click CREATE to download the private key. Save the private key file under the credentials.json name to a place accessible by NXLog. This file name will be used later in this section.

Configuring NXLog

The add-on consists of two files for handling the input and output log traffic respectively. Each add-on file reads its configuration from the config.json file stored in the same directory.

NXLog is configured separately for collecting log data and forwarding them to the Logging API.

For ease of configuration and setup the following script can be used to create some of the required folders for the examples. Note that other folders can be used as required.

Configuration script for Google Add-On
#!/bin/bash
if [[ -f "./credentials.json" ]]
then
        echo "Beginning installation of NXLog Google Add-On"
else
        echo "Credentials file missing, ensure that credentials.json file \
        is on the same folder as the installer"
        exit
fi

echo "Installing package"
if [[ -f "/etc/debian_version" ]]
then
        echo "Installing Debian based package"
        dpkg -i nxlog-google_1.0.1_amd64.deb
elif [[ -f "/etc/redhat-release" ]]
then
        echo "Installing RHEL based package"
        rpm -i nxlog-google-1.0.1-1.x86_64.rpm
fi

echo "Creating folders"
mkdir /opt/nxlog-google-cloud/{conf,log,input,output}
chown nxlog /opt/nxlog-google-cloud
echo "Credentials permissions"
cp ./credentials.json /opt/nxlog-google-cloud/conf/
chown nxlog /opt/nxlog-google-cloud/conf/credentials.json
chmod 400 /opt/nxlog-google-cloud/conf/credentials.json
echo "Finished setup"

Collecting data from the Google Logging API

Example 1. Collecting data from the Google Logging API

Below is a sample add-on configuration for collecting data from the Google Cloud logging infrastructure. This configuration should be saved as a config.json file and placed in the same directory as the input add-on file.

The configuration accepts relative path names in case the application directory is specified in the NXLog configuration. Otherwise, absolute path names can be used instead.
config.json
{
    "credentials": "./conf/credentials.json",
    "resourceNames": [
        "projects/nxlog"
    ],
    "filter": "logName=\"projects/nxlog/logs/0\"",
    "pollInterval": 30,
    "verbose": {
        "active": true,
        "file": "./log/input.log"
    },

    "cursor": {
        "active": true,
        "file": "./input-cursor.json"
    }
}

The following table explains each configuration parameter from the configuration above.

Table 1. Input add-on configuration

Parameter name

Format

Description

credentials

String

Path to the file containing a private key and other information needed for successful authorization with the Google Cloud account, see the Configuring the Google Cloud account section.

resourceNames

Array of strings

Names of resources to be monitored. Each element is a string of the form projects/<YOURPROJECTNAME>, where <YOURPROJECTNAME> is the placeholder for the project name.

filter

String

Expression used for querying log files from the cloud. String of the form logName=\"project/<YOURPROJECTNAME>/logs/<LOG_ID>\", where <YOURPROJECTNAME> is the placeholder for the actual project name, and the <LOG_ID> should be replaced with the actual log source name. For more details, see the Advanced log queries page on the Google Cloud website.

verbose

JSON object

Object for configuring the history of on-premises recording function calls. The active boolean key indicates whether logging should be enabled, and the file key specifies the file to write the data to.

Normally, this parameter should be disabled by setting the active key to FALSE.

pollInterval

Integer

Specifies the interval in seconds to check for new entries.

cursor

JSON object

This parameter provides additional filtering of entries through recording the timestamp and insertId values of the last downloaded message to a separate JSON file. After starting the add-on, these values are read and used as a point to continue collecting the entries from.

The cursor parameter consists of the active and file keys. The active boolean key indicates whether the cursor should be applied, and the file specifies the file to write the timestamp and insertId values to.

In case this parameter is enabled along with the timestamp explicitly specified in the filter parameter, the cursor takes precedence and the timestamp of the filter parameter will be overwritten.

This NXLog configuration uses the im_exec module with several arguments. For clarity, all directives and arguments are supplied with comments.

nxlog.conf
# Defines part of the directory paths used in this configuration
define GBASE /opt/nxlog-google-cloud

<Extension _json>
    Module    xm_json
</Extension>

<Input from_exec>
    Module    im_exec
    # Path to the input add-on file
    Command   %GBASE%/bin/google-cloud-input
    # Denotes the configuration file usage
    Arg       -c
    # Path to the configuration file
    Arg       %GBASE%/input/config.json
    # Enables using the application directory and relative paths to files
    Arg       -e
    # Path to the application directory
    Arg       %GBASE%
</Input>

Forwarding logs to the Google Logging API

This example explains how to use the output add-on to forward system logs to the Google Logging API.

Example 2. Forwarding logs to the Google Logging API

Below is the add-on configuration for pushing data to the Google Logging API. This configuration should be saved as a config.json file and placed in the same directory as the output add-on file.

The configuration allows using relative path names in case the application directory is specified in the NXLog configuration. Otherwise, absolute file paths should be used instead.
config.json
{
    "credentials": "./conf/credentials.json",
    "logName": "projects/nxlog/logs/0",
    "resourceType": "global",
    "labels": {
        "LabelOne": "ValueOne",
        "LabelTwo": "ValueTwo"
    },
    "verbose": {
        "active": true,
        "file": "./log/output.log"
    }
}

The following table explains each configuration parameter.

Table 2. Output add-on configuration

Parameter name

Format

Description

credentials

String

Path to the file containing a private key and other information needed for successful authentication with the Google Cloud account, see the Configuring the Google Cloud account subsection.

logName

String

Name of the log source on Google Cloud. String of the form project/<YOURPROJECTNAME>/logs/<LOG_ID>, where <YOURPROJECTNAME> is a placeholder for the actual project name, and <LOG_ID> should be replaced with the actual log source name. For more information about querying Google Cloud, see the Advanced log queries page on the Google Cloud website.

resourceType

String

The resource type monitored by the Google Logging API. For more information, see the Monitored resource types section of the Google Cloud API Reference.

By default, the global value can be used here.

labels

JSON object

Custom metadata that can be added to the log entries under the labels field.

verbose

JSON object

Object for configuring the history of on-premises recording function calls. The active boolean key indicates whether logging should be enabled, and the file key specifies the file to write the data to.

Normally, this parameter should be disabled by setting the active key to FALSE.

The NXLog configuration below uses the im_systemd module to read systemd logs from Linux. The entries are then converted to JSON using the xm_json module. The om_exec module is used to serialize the data and forward the result to the Google Logging API.

The om_exec module configuration contains comments explaining the meaning of each directive and argument.

nxlog.conf
# Defines part of the directory paths used in this configuration
define GBASE /opt/nxlog-google-cloud

<Extension _json>
    Module    xm_json
</Extension>

<Input from_systemd>
    Module    im_systemd
</Input>

<Output to_exec>
    Module    om_exec
    # Path to the output add-on file
    Command   %GBASE%/bin/google-cloud-output
    # Denotes the configuration file usage
    Arg       -c
    # Path to the add-on configuration file
    Arg       %GBASE%/output/config.json
    # Enables using the application directory and relative paths to files
    Arg       -e
    # Path to the application directory
    Arg       %GBASE%
    Exec      to_json();
</Output>
Disclaimer

While we endeavor to keep the information in this topic up to date and correct, NXLog makes no representations or warranties of any kind, express or implied about the completeness, accuracy, reliability, suitability, or availability of the content represented here.

The accurateness of the content was tested and proved to be working in our lab environment at the time of the last revision with the following software versions:

NXLog 5.4.7313
Linux (Debian 10)
NXLog Google Add-On 1.0.1
Google Cloud available at the time of revision

Last revision: 15 February 2022