NXLog search
Let's talk Start free
Let's talk Start free
1.05

Troubleshooting the NXLog Platform installation

There might be occasions when you experience issues during or right after the installation of NXLog Platform. In this section, we provide troubleshooting instructions and workarounds for the most common issues.

Services can’t start after the first reboot

Symptom

After the first reboot, NXLog Platform services don’t start up, and logs indicate that Vault is unreachable.

Possible reason

The podman-restart service may not have been enabled.

Investigation

  1. Check if Vault is running by executing the following command:

    $ sudo podman ps --filter name=vault
CONTAINER ID  IMAGE                             COMMAND     CREATED             STATUS                           PORTS       NAMES
5d3ff406ba12  nxlogacr.azurecr.io/vault:1.13.3              3 hours ago  Up 3 hours (healthy)              nxlog-1_2_0-c16aa3e2-vault-1

    Alternatively, you can execute the following ps command to check the status of the Vault process:

    $ sudo ps -fC vault
UID          PID    PPID  C STIME TTY          TIME CMD
root       22110   21693  1 11:04 ?        00:02:48 vault server -config=/init/vault_server_config.hcl

  2. If Vault is not started automatically following a system restart, the podman-restart service may be disabled resulting in a non-functional system. The PostgreSQL and container registry containers must also be started by the podman-restart service.

    Check this by running the systemctl status podman-restart command:

    $ sudo systemctl status podman-restart
○ podman-restart.service - Podman Start All Containers With Restart Policy Set To Always
     Loaded: loaded (/usr/lib/systemd/system/podman-restart.service; disabled; preset: disabled)
     Active: inactive (dead)
       Docs: man:podman-start(1)
Solution

  1. If the output confirms that the podman-restart.service is disabled, enable it by executing the following commands:

    $ sudo systemctl enable podman-restart
Created symlink /etc/systemd/system/default.target.wants/podman-restart.service → /usr/lib/systemd/system/podman-restart.service.
$ sudo systemctl status podman-restart
○ podman-restart.service - Podman Start All Containers With Restart Policy Set To Always
     Loaded: loaded (/usr/lib/systemd/system/podman-restart.service; enabled; preset: disabled)
     Active: inactive (dead)
       Docs: man:podman-start(1)

  2. Reboot your server.

Error writing blob

Symptom

Deployment fails with the following outcome:

$ sudo nxp_manage.sh install
[...]
Error: writing blob: adding layer with blob "sha256:877cab0f2b20b3216be0b74c2533ce470ac014006d858f363cd33e62660022a4": Error processing tar file(exit status 1): write /usr/lib/libLLVM-15.so: no space left on device
Possible reason

The filesystem under /var/lib/containers does not have enough free space.

Investigation

Check the available space by executing the following command:

$ df -h /var/lib/containers
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       4.7G  4.7G   23M 100% /
Solution

If the available space is less than 10GB, provision more space and restart the deployment process.

This issue might occur in environments where a proxy is used to access resources on the internet.

Symptom

A wrong or missing proxy configuration could cause 403 forbidden or 404 not found errors when the proxy refuses to serve the request. In some instances, you may also see the following or similar error message during installation:

[ERROR] Please check if your system can reach platform.nxlog.co. If you're using a proxy setup please ensure that proxy environment variables are exported before running make targets.
Possible reason

When the http_proxy or https_proxy environment variables are defined on your machine, HTTP clients will be directed to go through the proxy to access all web services. The proxy may not allow access to the external resources listed in network requirements, and could also interfere with internal communication to the IP range 10.89.0.0/24 in NXLog Platform’s internal Podman network.

Investigation

  1. Check if your proxy is correctly configured on your machine.

    • If the proxy isn’t configured or working correctly, you will obtain a connection refused, timeout, or different type of error. For example:

      $ curl --head https://platform.nxlog.co/registry/tenant
curl: (7) Failed to connect to platform.nxlog.co port 443 after 0 ms: Connection refused

    • If the proxy is configured and working correctly, you will obtain an HTTP 401 Unauthorized from the NXLog cloud URL. For example:

      $ curl --head https://platform.nxlog.co/registry/tenant
HTTP/1.1 200 Connection established
Proxy-agent: tinyproxy/1.11.2

HTTP/1.1 401 Unauthorized
Date: Fri, 23 Aug 2024 09:42:25 GMT
Connection: keep-alive

  2. If the curl test wasn’t successful, define the following environment variables for your current shell and repeat the previous test:

    $ export https_proxy="http://user:password@PROXY_ADDRESS:PORT" (1)
$ export http_proxy="http://user:password@PROXY_ADDRESS:PORT"  (1)
$ export no_proxy="localhost,127.0.0.1,10.89.0.0/24"           (2)
    1 Replace PROXY_ADDRESS and PORT with those of your proxy server. If your proxy does not use basic authentication, remove user:password@. Otherwise, replace them with your proxy credentials.
    2 Exclude the internal NXLog Platform IP address range 10.89.0.0/24 from being routed through the proxy.

  3. Once the curl test is successful, run the following command and take note of the http_proxy, https_proxy, and no_proxy environment variables defined in your current shell:

    $ env | grep proxy
Solution

  1. Edit the file /etc/nxp.conf and ensure the lines that define the http_proxy, https_proxy, and no_proxy environment variables are uncommented and use the same values that you obtained during the investigation steps above:

    $ sudo nano /etc/nxp.conf

  2. Check if your proxy allows ping requests to reach the NXLog container registry:

    $ ping -c 1 nxlogacr.azurecr.io

    If not, configure Podman not to ping the registry before pulling containers:

    $ echo "podman.ping_registries = false" >> /etc/containers/containers.conf

  3. Ensure that the proxy allows access to the outbound service FQDNs listed in the network requirements.

Enable debug logging

If you encounter another issue or cannot resolve the problem with the troubleshooting tips above, you can enable Podman debug logging by executing the following commands.

Remember to turn off debug logging when you no longer need it, or it could affect system performance. 
$ sudo mkdir -p /etc/systemd/system/podman.service.d/
$ sudo cat << EOF > /etc/systemd/system/podman.service.d/debug.conf
[Service]
Environment=LOGGING="--log-level=debug"
EOF
$ sudo systemctl daemon-reload
$ sudo systemctl restart podman
$ sudo truncate -s 0 /var/log/syslog (1)
1 This command clears the system log file. Replace the path with your system’s logging path, usually /var/log/syslog on Ubuntu and /var/log/messages on other operating systems.

After enabling debug logging, retry the NXLog Platform installation and check the system log file for relevant errors.

If you need assistance installing NXLog Platform, please open a support ticket and one of our experts will contact you.

© Copyright 2024 NXLog Ltd.

PRIVACY POLICY GENERAL TERMS OF BUSINESS