NXLog Legacy Documentation

Centralized deployment and management of NXLog agents

This guide discusses various approaches to centrally managing large deployments of NXLog agents.

Managing large-scale deployments of NXLog agents without the proper tools can be challenging. With large numbers of agents, solutions must be in place to handle new deployments, updates to existing agents and their configurations, and monitor individual agent health. In this guide, several solutions with varying degrees of functionality will be examined, two of which are native NXLog products.

NXLog Manager

NXLog Manager is a web-based tool that can manage and configure all aspects of NXLog agents. Its primary features are an intuitive web interface and the ability to define and store common assets, such as configuration templates and security certificates that are centrally located and accessible to all authorized users. You can deploy them to either specific agents or agent groups. If needed, NXLog Manager can be deployed in distributed mode as a cluster of servers to meet the future demands of rapidly growing deployments that require horizontal scaling.

NXLog Manager does not support the installation or upgrade of NXLog agents.

NXLog Manager features

Centralized management

Remotely configure NXLog agents via the xm_admin module.

Agent templates

Vast numbers of agents can be configured via custom templates with a single action, eliminating the need to configure each agent individually.

Agent monitoring

Each agent’s status and performance are visible, along with some basic logging metrics.

Web-based configuration

Although it is possible to use preexisting configuration files, creating and managing configurations can also be an intuitive, interactive experience.

Built-in public key infrastructure

Its built-in certificate management system supports X509 certificates for configuring TLS-based connections with NXLog agents.

Distributed mode

This mode provides horizontal scaling to meet the performance requirements of service areas spanning multiple geographical regions or to handle an ever-increasing need for new agent deployments. Whether you deploy NXLog Manager as a single, standalone server or a cluster of servers, the user experience remains the same. Accessing the system and regular operations are performed similarly in single-server and distributed modes.

Pattern and correlation rule editors

Create log extraction routines and correlation rules within the UI. NXLog Manager can also test regular expressions to confirm their intended operation.

You can find details on system requirements, supported platforms, installation, and configuration in the NXLog Manager User Guide.

Software Configuration Management tools

Software Configuration Management (SCM) tools, though limited in their ability to manage NXLog configurations, are another means of installing and upgrading NXLog agents. The remainder of this guide will show how to deploy NXLog using four well-known SCM solutions.

Deploy NXLog using Ansible

Ansible is an open-source software configuration management and application deployment tool. It employs a unique declarative language to represent system configuration and state. Ansible connects with remote servers via SSH but can also use other remote APIs.

Only control machines can run Ansible, which must be running a UNIX-like operating system, but Windows hosts that have the Windows Subsystem for Linux (WSL) installed are also supported. Managed nodes are agentless. However, both control machines and managed nodes require Python. For further details, see the Ansible Installation Guide.

Execute the commands below to install and configure NXLog on Ubuntu and Windows:

  1. Create the directory structure

    $ mkdir -p ansible/config
    $ cd ansible
    $ touch playbook.yml inventory config/{ubuntu,windows}.conf
  2. Log in to your NXLog account. Click on Your Account > Downloads, and from the Version drop-down menu select NXLog Enterprise Edition v5, then download the files:

    • nxlog-6.5.9781_ubuntu20_amd64.tar.bz2 for Ubuntu.

    • nxlog-6.5.9781_windows_x64.msi for Windows.

      Copy both files to the ansible directory you created in the previous step. Ansible will upload and install it during playbook execution.

  3. Compose the inventory file located in the config directory.

    Example 1. inventory

    The inventory file allows us to define categories or groups of servers. In this example, a single category, nxlog_servers, is used for both binaries.

    [nxlog_servers]
    ubuntu ansible_host=192.168.1.10
    windows ansible_host=192.168.1.11
  4. Create the NXLog agent configuration.

    Example 2. NXLog Ansible Ubuntu and Windows configuration examples.

    This configuration example will set up NXLog to read log messages from the Linux systemd journal using the im_systemd module. Each event is then converted to syslog and sent to a remote host over TCP port 514 using the om_tcp module.

    ubuntu.conf
    <Extension _syslog>
        Module  xm_syslog
    </Extension>
    
    <Input systemd>
        Module    im_systemd
    </Input>
    
    <Output tcp>
        Module    om_tcp
        Host      192.168.1.10:514
        Exec      to_syslog_bsd();
    </Output>
    
    <Route systemd_to_tcp>
        Path      systemd => tcp
    </Route>

    In this NXLog configuration example, the im_msvistalog module collects Windows events. The om_tcp output module invokes the to_syslog_bsd() procedure to convert each event to syslog format as it is forwarded to a remote host over TCP port 514.

    windows.conf
    <Extension syslog>
        Module    xm_syslog
    </Extension>
    
    <Input eventlog>
        Module    im_msvistalog
        <QueryXML>
            <QueryList>
                <Query Id='0'>
                    <Select Path='Application'>*</Select>
                    <Select Path='Security'>*[System/Level&lt;4]</Select>
                    <Select Path='System'>*</Select>
                </Query>
            </QueryList>
        </QueryXML>
    </Input>
    
    <Output tcp>
        Module    om_tcp
        Host      192.168.1.11:514
        Exec      to_syslog_bsd();
    </Output>
    
    <Route eventlog_to_tcp>
        Path      eventlog => tcp
    </Route>
  5. The Ansible playbook defines and configures the automation tasks. For more information on creating playbooks, see Ansible’s Intro to playbooks.

    playbook.yaml
    ---
    - hosts: nxlog_servers
      become: true
      become_user: root
    
      tasks:
        - name: Upload NXLog installer to remote Ubuntu server
          copy:
            src: nxlog-5.4.7313_ubuntu20_amd64.tar.bz2
            dest: /root/
            mode: '0755'
          when: "ansible_distribution == 'Ubuntu' and ansible_distribution_release == 'focal'"
    
        - name: Install NXLog on Ubuntu Focal
          shell:
            cmd: |
              mkdir -p nxlog; tar -xvf nxlog-5.4.7313_ubuntu20_amd64.tar.bz2 -C nxlog;
              apt install -y ./nxlog/*.deb;
          when: "ansible_distribution == 'Ubuntu' and ansible_distribution_release == 'focal'"
    
        - name: Copy the "ubuntu.conf" configuration file to the NXLog config directory
          copy:
            src: config/ubuntu.conf
            dest: /opt/nxlog/etc/nxlog.d/ubuntu.conf
            owner: nxlog
            group: nxlog
            mode: '0755'
          when: "ansible_distribution == 'Ubuntu' and ansible_distribution_release == 'focal'"
    
        - name: Restarting NXLog on Ubuntu Focal
          command: systemctl restart nxlog
          when: "ansible_distribution == 'Ubuntu' and ansible_distribution_release == 'focal'"
    
    #====================================================================================================#
    
        - name: Upload NXLog installer to remote Windows server
          copy:
            src: nxlog-5.4.7313_windows_x64.msi
            dest: C:\Users\administrator
            owner: nxlog
            group: nxlog
          when: ansible_distribution == 'Windows'
    
        - name: Install NXLog on Windows
          win_package:
            path: C:\Users\administrator\nxlog-5.4.7313_windows_x64.msi
            state: present
          when: ansible_distribution == 'Windows'
    
        - name: Copy new "windows.conf" file into the NXLog default config directory
          copy:
            src: config/windows.conf
            dest: C:\Program Files\nxlog\conf\nxlog.d\windows.conf
            owner: nxlog
            group: nxlog
            mode: '750'
          when: ansible_distribution == 'Windows'
    
        - name: Set nxlog service startup mode to auto and ensure it's started.
          win_service:
            name: nxlog
            state: restarted
  6. Run the playbook.

    $ ansible-playbook -i inventory -u root --ask-pass --ask-become-pass playbook.yml

    NOTE: According to the Setting up SSH keys section of the Ansible User Guide, "By default, Ansible assumes you are using SSH keys to connect to remote machines. SSH keys are encouraged …​".

    This example illustrates which ansible-playbook flags are required if the more secure SSH public key pair authentication method is not an option. The --ask-pass option enables the less secure SSH password authentication method.

    The Become directives page explains that the --ask-become-pass flag (or -K for short) enables one to specify a password for sudo.

Deploy NXLog using Chef

Chef is a configuration management tool for automating operations and tasks on multiple nodes.

To install Chef Workstation, refer to the Chef installation guide for details.

This guide assumes that you have Chef Workstation installed on your machine. It will be automatically downloaded and installed if you have not installed Chef Client on the nodes you want to configure.

Chef architecture

Chef’s architecture consists of:

  • A workstation for creating cookbooks and managing your environment

  • A Chef server (complete with web UI)

  • A Chef client that runs on the remote server(s) for resolving and applying configurations

The Chef server is responsible for configuration management in Chef. This guide will use a Chef server that is already running. According to the recipes uploaded via Chef Workstation, it keeps the Chef clients updated.

Creating the Chef file structure

You need to run a series of commands to create the correct file structure. These commands will generate a Chef repository, the cookbooks, and the recipes. Cookbooks are a collection of recipes, and there can be many recipes within a cookbook as well as many cookbooks.

  1. Create the required file structure by executing the following commands:

    $ chef generate repo chef-repo (1)
    $ cd chef-repo
    $ mkdir -p .chef (2)
    $ chef generate cookbook nxlog (3)
    $ cd nxlog
    $ chef generate recipe nxlog_install (4)
    1 Initiate a Chef repository
    2 Create a hidden .chef directory containing connection information with the Chef
    3 Generate a cookbook
    4 Create a recipe

    Upon completion, the following directory structure and files should be available.

    ├── chefignore
    ├── cookbooks
    │   ├── example
    │   │   ├── attributes
    │   │   │   └── default.rb
    │   │   ├── metadata.rb
    │   │   ├── README.md
    │   │   └── recipes
    │   │       └── default.rb
    │   ├── nxlog
    │   │   ├── CHANGELOG.md
    │   │   ├── chefignore
    │   │   ├── kitchen.yml
    │   │   ├── LICENSE
    │   │   ├── metadata.rb
    │   │   ├── Policyfile.rb
    │   │   ├── README.md
    │   │   ├── recipes
    │   │   │   ├── default.rb
    │   │   │   └── nxlog_install.rb
    │   │   ├── spec
    │   │   │   ├── spec_helper.rb
    │   │   │   └── unit
    │   │   │       └── recipes
    │   │   │           └── nxlog_install_spec.rb
    │   │   └── test
    │   │       └── integration
    │   │           └── default
    │   │               ├── default_test.rb
    │   │               └── nxlog_install_test.rb
    │   └── README.md
    ├── data_bags
    │   ├── example
    │   │   └── example_item.json
    │   └── README.md
    ├── LICENSE
    ├── policyfiles
    │   └── README.md
    └── README.md

    nxlog_install.rb is a Ruby script that we will use to describe all our tasks sent to the Chef server.

  2. Visit Chef Managed Server and create an account.

  3. Navigate to Administration > Organizations > Create to create an organization.

  4. Click on the newly created organization; from the left column, click on Starter Kit and then click on the Download Start Kit button.

  5. Unzip the chef-starter.zip and copy the files config.rb and <your_username>.pem to the .chef directory. These two files contain the connection details and credentials for the Chef Server.

  6. Connect a node with the Chef Server by executing:

    knife bootstrap NODE_IP --ssh-user root --ssh-password NODE_ROOT_PASSWORD --node-name NODE_NAME

    After you execute the command, you will notice that chef-client automatically installs on the node.

  7. Execute the following command to create a directory for the NXLog tar archive and configuration files. The required files are nxlog-6.5.9781_debian10_amd64.tar.bz2 and ubuntu.conf.

    $ mkdir -p cookbooks/nxlog/files

    The files directory in Chef cookbooks stores files used in your cookbook, with the cookbook_file resource used later on.

  8. Test connectivity with the Chef Server by running:

    $ knife upload cookbook nxlog
    When uploading cookbooks and recipes, ensure your current working directory is chef-repo/cookbooks.
  9. Add the recipe to Chef’s run list:

    $ knife node run_list add NODE_NAME recipe[nxlog::nxlog_install]

NXLog recipe file

The NXLog recipe file resides in the cookbooks/nxlog/recipes directory. Copy the contents below to nxlog_install.rb.

nxlog_install.rb
# Cookbook:: nxlog
# Recipe:: nxlog_install
#
# Copyright:: 2022, The Authors, All Rights Reserved.

cookbook_file "Copy NXLog archive to remote node" do
    source "nxlog-5.4.7313_ubuntu20_amd64.tar.bz2"
    path "/root/nxlog-5.4.7313_ubuntu20_amd64.tar.bz2"
    owner "root"
    group "root"
    mode "0755"
  end

archive_file "Extract NXLog archive" do
   path        "/root/nxlog-5.4.7313_ubuntu20_amd64.tar.bz2"
   destination "/root/nxlog/"
   owner       "root"
   group       "root"
   mode        "755"
   action      :extract
 end

execute "Install missing NXLog dependencies" do
  command "apt update; apt upgrade -y; apt-get install ruby libdbi1 librdkafka1 libodbc1 libzmq5; apt install -y -f; apt autoremove -y"
end

Dir.glob("/root/nxlog/*.deb") do |nxlog_deb|
   puts "installing : #{nxlog_deb}"
   dpkg_package 'Install NXLog' do
    package_name  "#{nxlog_deb}"
    source        "#{nxlog_deb}"
    action        :install
  end
 end

cookbook_file "Copy NXLog archive to remote node" do
   source "ubuntu.conf"
   path "/opt/nxlog/etc/nxlog.d/ubuntu.conf"
   owner "nxlog"
   group "nxlog"
   action :create
 end

service 'nxlog' do
  subscribes :reload, 'file[/opt/nxlog/etc/nxlog.d/ubuntu.conf]', :immediately
 end

Finally, two steps remain:

  1. Execute the command below to upload the recipe to the Chef server, that in turn, will communicate with the Chef client on the target node and run the tasks:

    cd ~/chef-repo/cookbooks
    knife upload cookbook nxlog
  2. On the target node, run the command:

    $ chef-client -i 300 -d

    This will daemonize the Chef Infra Client and automatically run it every 300 seconds (5 minutes).

Deploy NXLog using Puppet Bolt

Puppet Bolt is an agentless and masterless remote task runner that you can utilize with your existing server administration scripts. It connects with Windows nodes using WinRM and Linux nodes using SSH, eliminating the need to install agents on remote hosts.

To install Puppet Bolt, refer to the Puppet Bolt installation guide, where you can find platform-specific instructions for Windows and Linux systems.

To install NXLog using Puppet Bolt, follow the instructions below:

  1. First, create a project directory by issuing the following command:

    $ mkdir -p nxlog_bolt; cd nxlog_bolt; bolt project init

    It should result in the creation of three new files:

    • bolt-project.yaml

    • inventory.yaml

    • .gitignore (a hidden file)

  2. Next, create a module containing the NXLog directory and two subdirectories, plans and files.

    Execute the following command to create the directories needed:

    $ mkdir -p modules/nxlog/{files,plans}
  3. Edit the bolt-project.yaml file:

    Example 3. Puppet Bolt project file

    bolt-project.yaml is created when you execute the bolt project init command. Edit the file as follows to declare the plan definition, in this case, nxlog::nxlog_install.

    bolt-project.yaml
    ---
    name: nxlog_bolt
    modules: []
    
    color: true
    
    plans:
      - nxlog::nxlog_install
  4. Update the inventory.yaml file:

    Example 4. Puppet Bolt inventory file

    Puppet Bolt uses this file to connect to the remote host and execute the specified plan.

    inventory.yaml
    groups:
      - name: linux
        targets:
          - uri: 192.168.1.10
            name: ubuntu
        config:
          transport: ssh
          ssh:
            user: root
            password: your_root_password
            host-key-check: false
  5. NXLog agent configuration

    Example 5. NXLog Puppet Bolt configuration example

    This configuration example collects logs from im_systemd module to read log messages from the Linux systemd journal. Each event from the log source is then converted to syslog and sent via TCP port 514 using om_tcp to a remote server.

    ubuntu.conf
    <Extension _syslog>
        Module  xm_syslog
    </Extension>
    
    <Input systemd>
        Module    im_systemd
    </Input>
    
    <Output tcp>
        Module    om_tcp
        Host      192.168.1.10:514
        Exec      to_syslog_bsd();
    </Output>
    
    <Route systemd_to_tcp>
        Path      systemd => tcp
    </Route>
    ubuntu.conf has to exist inside the files directory. The same applies to the downloaded NXLog installation file. It is Puppet Bolt’s default technique for uploading files to remote hosts.
  6. Next, log in to your NXLog account, click on Your Account > Downloads, and from the Version drop-down menu select NXLog Enterprise Edition v5, then download nxlog-6.5.9781_debian10_amd64.tar.bz2.

    Place the NXLog tar archive in the modules/nxlog/files directory you created in step 2. Puppet Bolt will upload the tar archive, unpack it, and install the Debian packages when the plan is executed.

    The Plan below executes four tasks:

    • Upload the compressed archive.

    • Unpack and install the NXLog agent and various additional .deb files.

    • Upload the NXLog configuration file to /opt/nxlog/etc/nxlog.d.

    • Restart NXLog for the new changes to take effect.

      Example 6. nxlog_install.yaml
      ---
      parameters:
        targets:
          type: TargetSpec
      
      steps:
        - name: upload_ubuntu_nxlog_executable
          upload: nxlog/files/nxlog-5.4.7313_ubuntu20_amd64.tar.bz2
          destination: /root/
          targets: ubuntu
          description: Uploaded NXLog Archive
      
        - name: nxlog_ubuntu_install
          command: apt install -y libapr1 libdbi1 curl openjdk-8-jdk;
            mkdir -p nxlog; tar -xvf nxlog-5.4.7313_ubuntu20_amd64.tar.bz2 -C nxlog;
            dpkg -i nxlog/nxlog-5.4.7313_ubuntu20_amd64.deb;
            dpkg -i nxlog/*.deb;
            apt install -y -f;
          catch_errors: true
          targets: ubuntu
          description: "Install NxLog On Ubuntu using the .deb package"
      
        - name: nxlog_ubuntu_configure
          upload: nxlog/files/ubuntu.conf
          destination: /opt/nxlog/etc/nxlog.d/ubuntu.conf
          targets: ubuntu
          description: Uploaded Ubuntu Configuration File
      
        - name: restart_nxlog_ubuntu
          command: systemctl restart nxlog
          targets: ubuntu
          description: Restarting NXLog Service

Plan execution

To execute the plan, run the following command:

$ bolt plan run nxlog::nxlog_install -t all
Description of each component that comprises the command above
bolt

The executable located in /usr/local/bin

plan

Instructs Puppet Bolt to execute a plan

nxlog::nxlog_install

The plan reference definition consists of two parts, nxlog is the module name and corresponds to the name of the directory we created earlier, and nxlog_install is the file’s name in the plans directory.

-t

The target servers. We used the all option, but you can specify linux to target only Linux hosts or windows to target only Windows hosts.

Deploy NXLog with Salt

Salt is a free software configuration management tool that offers fast and scalable configuration management and a remote execution software framework. This framework runs concurrently on the minions, permitting the instantaneous configuration of an unlimited number of remote hosts by using language-specific state files.

A basic Salt setup consists of a Salt master (your workstation) handling one or more Salt minions (the target servers). A server running the salt-master service is called the Salt master. The master provides the orchestration and automation environment between managed nodes. A system or device operated by the Salt master is called a Salt minion. A minion can either run the salt-minion service or be agentless utilizing salt-ssh or salt-proxy. In our example, we will use the salt-ssh service to reduce complexity.

To install Salt, refer to the Salt installation guide.

Salt Master configuration file

Before using salt-ssh, you must configure the Salt-Master installed on your workstation. Edit /etc/salt/master and uncomment the following lines:

Example 7. master
interface: 0.0.0.0
publish_port: 4505
enable_ssh_minions: True
ret_port: 4506
conf_file: /etc/salt/master
ssh_port: 22
file_roots:
  base:
    - /srv/salt/states
    - /srv/salt/files

The most important information here is the file_roots key that defines the respective paths to state files, images, installation files, etc. It will fail if a task attempts to upload a file to any path not specified here.

Now restart the salt-master service to apply the changes by executing:

$ systemctl restart salt-master

Salt configuration file

Salt SSH connects to remote nodes using a file called roster located at /etc/salt/roster. The roster file contains information regarding the remote systems and their connection. With the roster file configured, salt-ssh can execute all Salt commands. Although the roster file does not have a file extension, it is a YAML file.

The following YAML file will be used for this example.

roster
ubuntu:
  host:         192.168.1.10
  user:         <username>
  passwd:       <password>
  sudo:         True
  tty:          True
When using sudo, ensure that tty is also enabled. Otherwise, the connection will fail.

Test the connection to the remote node:

$ salt-ssh 'ubuntu*' test.ping

The command should return:

ubuntu:
    True

NXLog configuration file

Prior to writing the Salt state file, the NXLog configuration file needs to be created and saved in /srv/salt/files/ubuntu.conf.

ubuntu.conf
<Extension _syslog>
    Module  xm_syslog
</Extension>

<Input systemd>
    Module    im_systemd
</Input>

<Output tcp>
    Module    om_tcp
    Host      192.168.1.10:514
    Exec      to_syslog_bsd();
</Output>

<Route systemd_to_tcp>
    Path      systemd => tcp
</Route>

The next step is to create the state file, which defines the steps for installing and configuring NXLog.

nxlog.sls
copy_nxlog_archive:
  file.managed:
    - name: /srv/salt/files/nxlog-5.4.7313_ubuntu20_amd64.tar.bz2
    - source: salt://nxlog-5.4.7313_ubuntu20_amd64.tar.bz2
    - user: root
    - group: root
    - makedirs: True

nxlog_archive_extraction:
  archive.extracted:
    - name: /srv/salt/files/nxlog
    - source: salt://nxlog-5.4.7313_ubuntu20_amd64.tar.bz2
    - enforce_toplevel: False

nxlog_deps_installation:
  pkg.installed:
    - pkgs:
      - libapr1
      - libdbi1
      - curl
      - openjdk-8-jdk

nxlog_instalaltion:
  pkg.installed:
    - sources:
      - nxlog: /srv/salt/files/nxlog/nxlog-5.4.7313_ubuntu20_amd64.deb
      - ruby: /srv/salt/files/nxlog/nxlog-ruby_5.4.7313_ubuntu20_amd64.deb
      - systemd: /srv/salt/files/nxlog/nxlog-systemd_5.4.7313_ubuntu20_amd64.deb
      - java: /srv/salt/files/nxlog/nxlog-java_5.4.7313_ubuntu20_amd64.deb
      - python: /srv/salt/files/nxlog/nxlog-python_5.4.7313_ubuntu20_amd64.deb
      - odbc: /srv/salt/files/nxlog/nxlog-odbc_5.4.7313_ubuntu20_amd64.deb
      - checkpoint: /srv/salt/files/nxlog/nxlog-checkpoint_5.4.7313_ubuntu20_amd64.deb 
      - pcap: /srv/salt/files/nxlog/nxlog-pcap_5.4.7313_ubuntu20_amd64.deb
      - wseventing: /srv/salt/files/nxlog/nxlog-wseventing_5.4.7313_ubuntu20_amd64.deb
      - dbi: /srv/salt/files/nxlog/nxlog-dbi_5.4.7313_ubuntu20_amd64.deb
      - perl: /srv/salt/files/nxlog/nxlog-perl_5.4.7313_ubuntu20_amd64.deb
      - zmq: /srv/salt/files/nxlog/nxlog-zmq_5.4.7313_ubuntu20_amd64.deb
      - kafka: /srv/salt/files/nxlog/nxlog-kafka_5.4.7313_ubuntu20_amd64.deb

copy_nxlog_config:
  file.managed:
    - name: /opt/nxlog/etc/nxlog.d/ubuntu.conf
    - source: salt://ubuntu.conf
    - user: nxlog
    - group: nxlog

restart_nxlog:
  service.running:
    - name: nxlog
    - enable: True
    - full_restart: True
    - watch:
      - file: /opt/nxlog/etc/nxlog.d/ubuntu.conf
The state file has the .sls extension.

To run the tasks on the remote server, execute the command:

$ salt-ssh 'ubuntu*' state.apply nxlog
Disclaimer

While we endeavor to keep the information in this topic up to date and correct, NXLog makes no representations or warranties of any kind, express or implied about the completeness, accuracy, reliability, suitability, or availability of the content represented here. We update our screenshots and instructions on a best-effort basis.

The accurateness of the content was tested and proved to be working in our lab environment at the time of the last revision with the following software versions:

NXLog EE 5.4.7313
Ansible 2.9.6
Chef Workstation version 21.10.640
Puppet Bolt 3.22.1
Salt Master 3004.1, Salt Minion 3004.1, Salt SSH 3004.1
Ubuntu 20.04.4 LTS

Last revision: 15 April 2022