NXLog Docs

Centralized deployment and management of NXLog agents

This guide discusses various approaches to centrally managing large deployments of NXLog agents.

Managing large-scale deployments of NXLog agents without the proper tools can be challenging. With large numbers of agents, solutions must be in place to handle new deployments, updates to existing agents and their configurations, and monitor individual agent health. In this guide, several solutions with varying degrees of functionality will be examined, two of which are native NXLog products.

NXLog Manager

NXLog Manager is a web-based tool that can manage and configure all aspects of NXLog agents. Its primary features are an intuitive web interface and the ability to define and store common assets, such as configuration templates and security certificates that are centrally located and accessible to all authorized users. You can deploy them to either specific agents or agent groups. If needed, NXLog Manager can be deployed in distributed mode as a cluster of servers to meet the future demands of rapidly growing deployments that require horizontal scaling.

NXLog Manager does not support the installation or upgrade of NXLog agents.

NXLog Manager features

Centralized management: Remotely configure NXLog agents via the xm_admin module.
Agent templates: Vast numbers of agents can be configured via custom templates with a single action, eliminating the need to configure each agent individually.
Agent monitoring: Each agent’s status and performance are visible, along with some basic logging metrics.
Web-based configuration: Although it is possible to use preexisting configuration files, creating and managing configurations can also be an intuitive, interactive experience.
Built-in public key infrastructure: Its built-in certificate management system supports X509 certificates for configuring TLS-based connections with NXLog agents.
Distributed mode: This mode provides horizontal scaling to meet the performance requirements of service areas spanning multiple geographical regions or to handle an ever-increasing need for new agent deployments. Whether you deploy NXLog Manager as a single, standalone server or a cluster of servers, the user experience remains the same. Accessing the system and regular operations are performed similarly in single-server and distributed modes.
Pattern and correlation rule editors: Create log extraction routines and correlation rules within the UI. NXLog Manager can also test regular expressions to confirm their intended operation.

You can find details on system requirements, supported platforms, installation, and configuration in the NXLog Manager User Guide.

Software Configuration Management tools

Software Configuration Management (SCM) tools, though limited in their ability to manage NXLog configurations, are another means of installing and upgrading NXLog agents. The remainder of this guide will show how to deploy NXLog using four well-known SCM solutions.

Deploy NXLog using Ansible

Ansible is an open-source software configuration management and application deployment tool. It employs a unique declarative language to represent system configuration and state. Ansible connects with remote servers via SSH but can also use other remote APIs.

Only control machines can run Ansible, which must be running a UNIX-like operating system, but Windows hosts that have the Windows Subsystem for Linux (WSL) installed are also supported. Managed nodes are agentless. However, both control machines and managed nodes require Python. For further details, see the Ansible Installation Guide.

Execute the commands below to install and configure NXLog on Ubuntu and Windows:

Create the directory structure

$ mkdir -p ansible/config
$ cd ansible
$ touch playbook.yml inventory config/{ubuntu,windows}.conf

Log in to your NXLog account. Click on Your Account > Downloads, and from the Version drop-down menu select NXLog Enterprise Edition v5, then download the files:
- nxlog-6.2.9212_ubuntu20_amd64.tar.bz2 for Ubuntu.
- nxlog-6.2.9212_windows_x64.msi for Windows.
  
  Copy both files to the ansible directory you created in the previous step. Ansible will upload and install it during playbook execution.
Compose the inventory file located in the config directory.
Example 1. inventory
The inventory file allows us to define categories or groups of servers. In this example, a single category, nxlog_servers, is used for both binaries.
[nxlog_servers] ubuntu ansible_host=192.168.1.10 windows ansible_host=192.168.1.11

Create the NXLog agent configuration.

Example 2. NXLog Ansible Ubuntu and Windows configuration examples.

This configuration example will set up NXLog to read log messages from the Linux systemd journal using the im_systemd module. Each event is then converted to syslog and sent to a remote host over TCP port 514 using the om_tcp module.

ubuntu.conf

<Extension _syslog>
    Module  xm_syslog
</Extension>

<Input systemd>
    Module    im_systemd
</Input>

<Output tcp>
    Module    om_tcp
    Host      192.168.1.10:514
    Exec      to_syslog_bsd();
</Output>

<Route systemd_to_tcp>
    Path      systemd => tcp
</Route>

In this NXLog configuration example, the im_msvistalog module collects Windows events. The om_tcp output module invokes the to_syslog_bsd() procedure to convert each event to syslog format as it is forwarded to a remote host over TCP port 514.

windows.conf

<Extension syslog>
    Module    xm_syslog
</Extension>

<Input eventlog>
    Module    im_msvistalog
    <QueryXML>
        <QueryList>
            <Query Id='0'>
                <Select Path='Application'>*</Select>
                <Select Path='Security'>*[System/Level&lt;4]</Select>
                <Select Path='System'>*</Select>
            </Query>
        </QueryList>
    </QueryXML>
</Input>

<Output tcp>
    Module    om_tcp
    Host      192.168.1.11:514
    Exec      to_syslog_bsd();
</Output>

<Route eventlog_to_tcp>
    Path      eventlog => tcp
</Route>

The Ansible playbook defines and configures the automation tasks. For more information on creating playbooks, see Ansible’s Intro to playbooks.

playbook.yaml

---
- hosts: nxlog_servers
  become: true
  become_user: root

  tasks:
    - name: Upload NXLog installer to remote Ubuntu server
      copy:
        src: nxlog-5.4.7313_ubuntu20_amd64.tar.bz2
        dest: /root/
        mode: '0755'
      when: "ansible_distribution == 'Ubuntu' and ansible_distribution_release == 'focal'"

    - name: Install NXLog on Ubuntu Focal
      shell:
        cmd: |
          mkdir -p nxlog; tar -xvf nxlog-5.4.7313_ubuntu20_amd64.tar.bz2 -C nxlog;
          apt install -y ./nxlog/*.deb;
      when: "ansible_distribution == 'Ubuntu' and ansible_distribution_release == 'focal'"

    - name: Copy the "ubuntu.conf" configuration file to the NXLog config directory
      copy:
        src: config/ubuntu.conf
        dest: /opt/nxlog/etc/nxlog.d/ubuntu.conf
        owner: nxlog
        group: nxlog
        mode: '0755'
      when: "ansible_distribution == 'Ubuntu' and ansible_distribution_release == 'focal'"

    - name: Restarting NXLog on Ubuntu Focal
      command: systemctl restart nxlog
      when: "ansible_distribution == 'Ubuntu' and ansible_distribution_release == 'focal'"

#====================================================================================================#

    - name: Upload NXLog installer to remote Windows server
      copy:
        src: nxlog-5.4.7313_windows_x64.msi
        dest: C:\Users\administrator
        owner: nxlog
        group: nxlog
      when: ansible_distribution == 'Windows'

    - name: Install NXLog on Windows
      win_package:
        path: C:\Users\administrator\nxlog-5.4.7313_windows_x64.msi
        state: present
      when: ansible_distribution == 'Windows'

    - name: Copy new "windows.conf" file into the NXLog default config directory
      copy:
        src: config/windows.conf
        dest: C:\Program Files\nxlog\conf\nxlog.d\windows.conf
        owner: nxlog
        group: nxlog
        mode: '750'
      when: ansible_distribution == 'Windows'

    - name: Set nxlog service startup mode to auto and ensure it's started.
      win_service:
        name: nxlog
        state: restarted

Run the playbook.
$ ansible-playbook -i inventory -u root --ask-pass --ask-become-pass playbook.yml
NOTE: According to the Setting up SSH keys section of the Ansible User Guide, "By default, Ansible assumes you are using SSH keys to connect to remote machines. SSH keys are encouraged …".

This example illustrates which ansible-playbook flags are required if the more secure SSH public key pair authentication method is not an option. The --ask-pass option enables the less secure SSH password authentication method.

The Become directives page explains that the --ask-become-pass flag (or -K for short) enables one to specify a password for sudo.

Deploy NXLog using Chef

Chef is a configuration management tool for automating operations and tasks on multiple nodes.

To install Chef Workstation, refer to the Chef installation guide for details.

This guide assumes that you have Chef Workstation installed on your machine. It will be automatically downloaded and installed if you have not installed Chef Client on the nodes you want to configure.

Chef architecture

Chef’s architecture consists of:

A workstation for creating cookbooks and managing your environment
A Chef server (complete with web UI)
A Chef client that runs on the remote server(s) for resolving and applying configurations

The Chef server is responsible for configuration management in Chef. This guide will use a Chef server that is already running. According to the recipes uploaded via Chef Workstation, it keeps the Chef clients updated.

Creating the Chef file structure

You need to run a series of commands to create the correct file structure. These commands will generate a Chef repository, the cookbooks, and the recipes. Cookbooks are a collection of recipes, and there can be many recipes within a cookbook as well as many cookbooks.

Create the required file structure by executing the following commands:

$ chef generate repo chef-repo (1)
$ cd chef-repo
$ mkdir -p .chef (2)
$ chef generate cookbook nxlog (3)
$ cd nxlog
$ chef generate recipe nxlog_install (4)

1	Initiate a Chef repository
2	Create a hidden `.chef` directory containing connection information with the Chef
3	Generate a cookbook
4	Create a recipe

Upon completion, the following directory structure and files should be available.

├── chefignore
├── cookbooks
│   ├── example
│   │   ├── attributes
│   │   │   └── default.rb
│   │   ├── metadata.rb
│   │   ├── README.md
│   │   └── recipes
│   │       └── default.rb
│   ├── nxlog
│   │   ├── CHANGELOG.md
│   │   ├── chefignore
│   │   ├── kitchen.yml
│   │   ├── LICENSE
│   │   ├── metadata.rb
│   │   ├── Policyfile.rb
│   │   ├── README.md
│   │   ├── recipes
│   │   │   ├── default.rb
│   │   │   └── nxlog_install.rb
│   │   ├── spec
│   │   │   ├── spec_helper.rb
│   │   │   └── unit
│   │   │       └── recipes
│   │   │           └── nxlog_install_spec.rb
│   │   └── test
│   │       └── integration
│   │           └── default
│   │               ├── default_test.rb
│   │               └── nxlog_install_test.rb
│   └── README.md
├── data_bags
│   ├── example
│   │   └── example_item.json
│   └── README.md
├── LICENSE
├── policyfiles
│   └── README.md
└── README.md

nxlog_install.rb is a Ruby script that we will use to describe all our tasks sent to the Chef server.

Visit Chef Managed Server and create an account.
Navigate to Administration > Organizations > Create to create an organization.
Click on the newly created organization; from the left column, click on Starter Kit and then click on the Download Start Kit button.
Unzip the chef-starter.zip and copy the files config.rb and <your_username>.pem to the .chef directory. These two files contain the connection details and credentials for the Chef Server.
Connect a node with the Chef Server by executing:
```
knife bootstrap NODE_IP --ssh-user root --ssh-password NODE_ROOT_PASSWORD --node-name NODE_NAME
```
After you execute the command, you will notice that chef-client automatically installs on the node.
Execute the following command to create a directory for the NXLog tar archive and configuration files. The required files are nxlog-6.2.9212_debian10_amd64.tar.bz2 and ubuntu.conf.
$ mkdir -p cookbooks/nxlog/files
The files directory in Chef cookbooks stores files used in your cookbook, with the cookbook_file resource used later on.
Test connectivity with the Chef Server by running:
```
$ knife upload cookbook nxlog
```
When uploading cookbooks and recipes, ensure your current working directory is chef-repo/cookbooks.

Add the recipe to Chef’s run list:

$ knife node run_list add NODE_NAME recipe[nxlog::nxlog_install]

NXLog recipe file

The NXLog recipe file resides in the cookbooks/nxlog/recipes directory. Copy the contents below to nxlog_install.rb.

nxlog_install.rb

# Cookbook:: nxlog
# Recipe:: nxlog_install
#
# Copyright:: 2022, The Authors, All Rights Reserved.

cookbook_file "Copy NXLog archive to remote node" do
    source "nxlog-5.4.7313_ubuntu20_amd64.tar.bz2"
    path "/root/nxlog-5.4.7313_ubuntu20_amd64.tar.bz2"
    owner "root"
    group "root"
    mode "0755"
  end

archive_file "Extract NXLog archive" do
   path        "/root/nxlog-5.4.7313_ubuntu20_amd64.tar.bz2"
   destination "/root/nxlog/"
   owner       "root"
   group       "root"
   mode        "755"
   action      :extract
 end

execute "Install missing NXLog dependencies" do
  command "apt update; apt upgrade -y; apt-get install ruby libdbi1 librdkafka1 libodbc1 libzmq5; apt install -y -f; apt autoremove -y"
end

Dir.glob("/root/nxlog/*.deb") do |nxlog_deb|
   puts "installing : #{nxlog_deb}"
   dpkg_package 'Install NXLog' do
    package_name  "#{nxlog_deb}"
    source        "#{nxlog_deb}"
    action        :install
  end
 end

cookbook_file "Copy NXLog archive to remote node" do
   source "ubuntu.conf"
   path "/opt/nxlog/etc/nxlog.d/ubuntu.conf"
   owner "nxlog"
   group "nxlog"
   action :create
 end

service 'nxlog' do
  subscribes :reload, 'file[/opt/nxlog/etc/nxlog.d/ubuntu.conf]', :immediately
 end

Finally, two steps remain:

Execute the command below to upload the recipe to the Chef server, that in turn, will communicate with the Chef client on the target node and run the tasks:
```
cd ~/chef-repo/cookbooks
knife upload cookbook nxlog
```
On the target node, run the command:
$ chef-client -i 300 -d
This will daemonize the Chef Infra Client and automatically run it every 300 seconds (5 minutes).

Deploy NXLog using Puppet Bolt

Puppet Bolt is an agentless and masterless remote task runner that you can utilize with your existing server administration scripts. It connects with Windows nodes using WinRM and Linux nodes using SSH, eliminating the need to install agents on remote hosts.

To install Puppet Bolt, refer to the Puppet Bolt installation guide, where you can find platform-specific instructions for Windows and Linux systems.

To install NXLog using Puppet Bolt, follow the instructions below:

First, create a project directory by issuing the following command:
```
$ mkdir -p nxlog_bolt; cd nxlog_bolt; bolt project init
```
It should result in the creation of three new files:
- bolt-project.yaml
- inventory.yaml
- .gitignore (a hidden file)
Next, create a module containing the NXLog directory and two subdirectories, plans and files.

Execute the following command to create the directories needed:
```
$ mkdir -p modules/nxlog/{files,plans}
```
Edit the bolt-project.yaml file:
Example 3. Puppet Bolt project file
bolt-project.yaml is created when you execute the bolt project init command. Edit the file as follows to declare the plan definition, in this case, nxlog::nxlog_install.
bolt-project.yaml
--- name: nxlog_bolt modules: [] color: true plans: - nxlog::nxlog_install

Update the inventory.yaml file:

Example 4. Puppet Bolt inventory file

Puppet Bolt uses this file to connect to the remote host and execute the specified plan.

inventory.yaml

groups:
  - name: linux
    targets:
      - uri: 192.168.1.10
        name: ubuntu
    config:
      transport: ssh
      ssh:
        user: root
        password: your_root_password
        host-key-check: false

NXLog agent configuration
Example 5. NXLog Puppet Bolt configuration example
This configuration example collects logs from im_systemd module to read log messages from the Linux systemd journal. Each event from the log source is then converted to syslog and sent via TCP port 514 using om_tcp to a remote server.
ubuntu.conf
<Extension _syslog> Module xm_syslog </Extension> <Input systemd> Module im_systemd </Input> <Output tcp> Module om_tcp Host 192.168.1.10:514 Exec to_syslog_bsd(); </Output> <Route systemd_to_tcp> Path systemd => tcp </Route>
ubuntu.conf has to exist inside the files directory. The same applies to the downloaded NXLog installation file. It is Puppet Bolt’s default technique for uploading files to remote hosts.

Next, log in to your NXLog account, click on Your Account > Downloads, and from the Version drop-down menu select NXLog Enterprise Edition v5, then download nxlog-6.2.9212_debian10_amd64.tar.bz2.

Place the NXLog tar archive in the modules/nxlog/files directory you created in step 2. Puppet Bolt will upload the tar archive, unpack it, and install the Debian packages when the plan is executed.

The Plan below executes four tasks:

Upload the compressed archive.
Unpack and install the NXLog agent and various additional .deb files.
Upload the NXLog configuration file to /opt/nxlog/etc/nxlog.d.

Restart NXLog for the new changes to take effect.

Example 6. nxlog_install.yaml

---
parameters:
  targets:
    type: TargetSpec

steps:
  - name: upload_ubuntu_nxlog_executable
    upload: nxlog/files/nxlog-5.4.7313_ubuntu20_amd64.tar.bz2
    destination: /root/
    targets: ubuntu
    description: Uploaded NXLog Archive

  - name: nxlog_ubuntu_install
    command: apt install -y libapr1 libdbi1 curl openjdk-8-jdk;
      mkdir -p nxlog; tar -xvf nxlog-5.4.7313_ubuntu20_amd64.tar.bz2 -C nxlog;
      dpkg -i nxlog/nxlog-5.4.7313_ubuntu20_amd64.deb;
      dpkg -i nxlog/*.deb;
      apt install -y -f;
    catch_errors: true
    targets: ubuntu
    description: "Install NxLog On Ubuntu using the .deb package"

  - name: nxlog_ubuntu_configure
    upload: nxlog/files/ubuntu.conf
    destination: /opt/nxlog/etc/nxlog.d/ubuntu.conf
    targets: ubuntu
    description: Uploaded Ubuntu Configuration File

  - name: restart_nxlog_ubuntu
    command: systemctl restart nxlog
    targets: ubuntu
    description: Restarting NXLog Service

Plan execution

To execute the plan, run the following command:

$ bolt plan run nxlog::nxlog_install -t all

Description of each component that comprises the command above

bolt: The executable located in /usr/local/bin
plan: Instructs Puppet Bolt to execute a plan
nxlog::nxlog_install: The plan reference definition consists of two parts, nxlog is the module name and corresponds to the name of the directory we created earlier, and nxlog_install is the file’s name in the plans directory.
-t: The target servers. We used the all option, but you can specify linux to target only Linux hosts or windows to target only Windows hosts.

Deploy NXLog with Salt

Salt is a free software configuration management tool that offers fast and scalable configuration management and a remote execution software framework. This framework runs concurrently on the minions, permitting the instantaneous configuration of an unlimited number of remote hosts by using language-specific state files.

A basic Salt setup consists of a Salt master (your workstation) handling one or more Salt minions (the target servers). A server running the salt-master service is called the Salt master. The master provides the orchestration and automation environment between managed nodes. A system or device operated by the Salt master is called a Salt minion. A minion can either run the salt-minion service or be agentless utilizing salt-ssh or salt-proxy. In our example, we will use the salt-ssh service to reduce complexity.

To install Salt, refer to the Salt installation guide.

Salt Master configuration file

Before using salt-ssh, you must configure the Salt-Master installed on your workstation. Edit /etc/salt/master and uncomment the following lines:

Example 7. master

interface: 0.0.0.0
publish_port: 4505
enable_ssh_minions: True
ret_port: 4506
conf_file: /etc/salt/master
ssh_port: 22
file_roots:
  base:
    - /srv/salt/states
    - /srv/salt/files

The most important information here is the file_roots key that defines the respective paths to state files, images, installation files, etc. It will fail if a task attempts to upload a file to any path not specified here.

Now restart the salt-master service to apply the changes by executing:

$ systemctl restart salt-master

Salt configuration file

Salt SSH connects to remote nodes using a file called roster located at /etc/salt/roster. The roster file contains information regarding the remote systems and their connection. With the roster file configured, salt-ssh can execute all Salt commands. Although the roster file does not have a file extension, it is a YAML file.

The following YAML file will be used for this example.

roster

ubuntu:
  host:         192.168.1.10
  user:         <username>
  passwd:       <password>
  sudo:         True
  tty:          True

When using sudo, ensure that tty is also enabled. Otherwise, the connection will fail.

Test the connection to the remote node:

$ salt-ssh 'ubuntu*' test.ping

The command should return:

ubuntu:
    True

NXLog configuration file

Prior to writing the Salt state file, the NXLog configuration file needs to be created and saved in /srv/salt/files/ubuntu.conf.

ubuntu.conf

<Extension _syslog>
    Module  xm_syslog
</Extension>

<Input systemd>
    Module    im_systemd
</Input>

<Output tcp>
    Module    om_tcp
    Host      192.168.1.10:514
    Exec      to_syslog_bsd();
</Output>

<Route systemd_to_tcp>
    Path      systemd => tcp
</Route>

The next step is to create the state file, which defines the steps for installing and configuring NXLog.

nxlog.sls

copy_nxlog_archive:
  file.managed:
    - name: /srv/salt/files/nxlog-5.4.7313_ubuntu20_amd64.tar.bz2
    - source: salt://nxlog-5.4.7313_ubuntu20_amd64.tar.bz2
    - user: root
    - group: root
    - makedirs: True

nxlog_archive_extraction:
  archive.extracted:
    - name: /srv/salt/files/nxlog
    - source: salt://nxlog-5.4.7313_ubuntu20_amd64.tar.bz2
    - enforce_toplevel: False

nxlog_deps_installation:
  pkg.installed:
    - pkgs:
      - libapr1
      - libdbi1
      - curl
      - openjdk-8-jdk

nxlog_instalaltion:
  pkg.installed:
    - sources:
      - nxlog: /srv/salt/files/nxlog/nxlog-5.4.7313_ubuntu20_amd64.deb
      - ruby: /srv/salt/files/nxlog/nxlog-ruby_5.4.7313_ubuntu20_amd64.deb
      - systemd: /srv/salt/files/nxlog/nxlog-systemd_5.4.7313_ubuntu20_amd64.deb
      - java: /srv/salt/files/nxlog/nxlog-java_5.4.7313_ubuntu20_amd64.deb
      - python: /srv/salt/files/nxlog/nxlog-python_5.4.7313_ubuntu20_amd64.deb
      - odbc: /srv/salt/files/nxlog/nxlog-odbc_5.4.7313_ubuntu20_amd64.deb
      - checkpoint: /srv/salt/files/nxlog/nxlog-checkpoint_5.4.7313_ubuntu20_amd64.deb 
      - pcap: /srv/salt/files/nxlog/nxlog-pcap_5.4.7313_ubuntu20_amd64.deb
      - wseventing: /srv/salt/files/nxlog/nxlog-wseventing_5.4.7313_ubuntu20_amd64.deb
      - dbi: /srv/salt/files/nxlog/nxlog-dbi_5.4.7313_ubuntu20_amd64.deb
      - perl: /srv/salt/files/nxlog/nxlog-perl_5.4.7313_ubuntu20_amd64.deb
      - zmq: /srv/salt/files/nxlog/nxlog-zmq_5.4.7313_ubuntu20_amd64.deb
      - kafka: /srv/salt/files/nxlog/nxlog-kafka_5.4.7313_ubuntu20_amd64.deb

copy_nxlog_config:
  file.managed:
    - name: /opt/nxlog/etc/nxlog.d/ubuntu.conf
    - source: salt://ubuntu.conf
    - user: nxlog
    - group: nxlog

restart_nxlog:
  service.running:
    - name: nxlog
    - enable: True
    - full_restart: True
    - watch:
      - file: /opt/nxlog/etc/nxlog.d/ubuntu.conf

The state file has the .sls extension.

To run the tasks on the remote server, execute the command:

$ salt-ssh 'ubuntu*' state.apply nxlog

Disclaimer

While we endeavor to keep the information in this topic up to date and correct, NXLog makes no representations or warranties of any kind, express or implied about the completeness, accuracy, reliability, suitability, or availability of the content represented here. We update our screenshots and instructions on a best-effort basis.

The accurateness of the content was tested and proved to be working in our lab environment at the time of the last revision with the following software versions:

NXLog EE 5.4.7313
Ansible 2.9.6
Chef Workstation version 21.10.640
Puppet Bolt 3.22.1
Salt Master 3004.1, Salt Minion 3004.1, Salt SSH 3004.1
Ubuntu 20.04.4 LTS

Last revision: 15 April 2022