Troubleshooting the NXLog Platform installation in an air-gapped environment

Air-gapped deployment is currently a Beta feature available to Premium customers. Contact NXLog Support for more information.

There might be occasions when you experience issues during or right after the installation of NXLog Platform. In this section, we provide troubleshooting instructions and workarounds for the most common issues.

Container image bundle not found

Symptom

Validating the required dependencies fails with the following message:

$ sudo nxp_manage.sh wizard
[...]
Airgapped images archive not found and images directory is empty.

Possible reason

The image bundle nxp-x.x.x-images-amd64.tar.gz was not copied to /opt/nxp before running nxp_manage.sh wizard.

Investigation

Check that the image bundle is present in /opt/nxp.

Solution

Copy the image bundle to /opt/nxp and run the wizard again, replacing x.x.x with your version:

$ sudo cp nxp-x.x.x-images-amd64.tar.gz /opt/nxp/
$ sudo nxp_manage.sh wizard

Local documentation unavailable

Symptom

The wizard logs the following warning, but the installation continues:

$ sudo nxp_manage.sh wizard
[...]
Airgapped documentation archive not found (/opt/nxp/nxp-x.x.x-docs.tar.gz). Local documentation will be unavailable.

Possible reason

The offline documentation bundle nxp-x.x.x-docs.tar.gz was not copied to /opt/nxp before running nxp_manage.sh wizard. This bundle is optional: NXLog Platform continues to install and run without it, but the documentation is unavailable from the NXLog Platform UI.

Solution

Copy the offline documentation bundle to /opt/nxp and run the wizard again, replacing x.x.x with your version:

$ sudo cp nxp-x.x.x-docs.tar.gz /opt/nxp/
$ sudo nxp_manage.sh wizard

Cannot reach NXLog Platform while importing license

Symptom

Importing the license file fails with the following message:

$ sudo nxp_manage.sh license-import LICENSE_FILE=license_<timestamp>.jwt
[...]
Could not reach https://platform.nxlog.example.com — check network connectivity and DNS resolution.

Possible reason

NXLog Platform is not reachable at the configured domain, either because it is not running or because DNS is not resolving correctly.

Investigation

Check that NXLog Platform is running by executing the following command:
```
$ sudo nxp_manage.sh stats
```
This command lists the running NXLog Platform containers and their resource usage. If it returns no containers, NXLog Platform is not running.
Check that the NXLog Platform hostname resolves to the IP address of the host machine, replacing nxlog.example.com with your domain:
```
$ getent hosts platform.nxlog.example.com
```

Solution

If NXLog Platform is not running, start it by executing the following command:
```
$ sudo nxp_manage.sh start
```
If the hostname does not resolve to the host machine, verify DNS resolution.
Import the license file again, replacing license_<timestamp>.jwt with the path to your license file:
```
$ sudo nxp_manage.sh license-import LICENSE_FILE=license_<timestamp>.jwt
```

License import rejected by NXLog Platform

Symptom

Importing the license file fails with the following message:

$ sudo nxp_manage.sh license-import LICENSE_FILE=license_<timestamp>.jwt
[...]
License import failed (Status code: <http_status>) | Body: <response>

Possible reason

NXLog Platform rejected the license file because it may be malformed, expired, or issued for a different organization.

Solution

Request a new license file from NXLog Support, providing the error output and your organization ID.
Import the new license file, replacing license_<timestamp>.jwt with the path to your license file:
```
$ sudo nxp_manage.sh license-import LICENSE_FILE=license_<timestamp>.jwt
```

Error writing blob

Symptom

Deployment fails with the following outcome:

$ sudo nxp_manage.sh install
[...]
Error: writing blob: adding layer with blob "sha256:877cab0f2b20b3216be0b74c2533ce470ac014006d858f363cd33e62660022a4": Error processing tar file(exit status 1): write /usr/lib/libLLVM-15.so: no space left on device

Possible reason

The filesystem under /var/lib/containers does not have enough free space.

Investigation

Check the available space by executing the following command:

$ df -h /var/lib/containers
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       4.7G  4.7G   23M 100% /

Solution

If the available space is less than 10GB, provision more space and restart the deployment process.

Services can’t start after the first reboot

Symptom

After the first reboot, NXLog Platform services don’t start up, and logs indicate that Vault is unreachable.

Possible reason

The podman-restart service may have been deactivated.

Investigation

Check if Vault is running by executing the following command:

$ sudo podman ps --filter name=vault
CONTAINER ID  IMAGE                             COMMAND     CREATED             STATUS                           PORTS       NAMES
5d3ff406ba12  nxlogacr.azurecr.io/vault:1.13.3              3 hours ago  Up 3 hours (healthy)              nxlog-1_2_0-c16aa3e2-vault-1

Alternatively, you can execute the following ps command to check the status of the Vault process:

$ sudo ps -fC vault
UID          PID    PPID  C STIME TTY          TIME CMD
root       22110   21693  1 11:04 ?        00:02:48 vault server -config=/init/vault_server_config.hcl

If Vault is not started automatically following a system restart, the podman-restart service may be deactivated, resulting in a non-functional system. The PostgreSQL and container registry containers must also be started by the podman-restart service.

Check this by running the systemctl status podman-restart command:
```
$ sudo systemctl status podman-restart
○ podman-restart.service - Podman Start All Containers With Restart Policy Set To Always
     Loaded: loaded (/usr/lib/systemd/system/podman-restart.service; disabled; preset: disabled)
     Active: inactive (dead)
       Docs: man:podman-start(1)
```

Solution

If the output confirms that the podman-restart.service is inactive, activate it by executing the following commands:

$ sudo systemctl enable podman-restart
Created symlink /etc/systemd/system/default.target.wants/podman-restart.service → /usr/lib/systemd/system/podman-restart.service.
$ sudo systemctl status podman-restart
○ podman-restart.service - Podman Start All Containers With Restart Policy Set To Always
     Loaded: loaded (/usr/lib/systemd/system/podman-restart.service; enabled; preset: disabled)
     Active: inactive (dead)
       Docs: man:podman-start(1)

Reboot your server.

Turn on debug logging

If you encounter another issue or cannot resolve the problem with the troubleshooting tips above, you can turn on Podman debug logging by executing the following commands.

Remember to turn off debug logging when you no longer need it, or it could affect system performance.

$ sudo mkdir -p /etc/systemd/system/podman.service.d/
$ sudo cat << EOF > /etc/systemd/system/podman.service.d/debug.conf
[Service]
Environment=LOGGING="--log-level=debug"
EOF
$ sudo systemctl daemon-reload
$ sudo systemctl restart podman
$ sudo truncate -s 0 /var/log/syslog (1)

1	This command clears the system log file. Replace the path with your system’s logging path, usually `/var/log/syslog` on Ubuntu and `/var/log/messages` on other operating systems.

After enabling debug logging, retry the NXLog Platform installation and check the system log file for relevant errors.

If you need assistance installing NXLog Platform, please open a support ticket and one of our experts will contact you.