Troubleshoot common issues

This page provides troubleshooting tips and solutions for common NXLog Platform on-premises issues. These issues typically stem from a misconfiguration and are straightforward to resolve.

Logs are not showing in Log Search but with the NXLog Platform application logs instead

Symptom

Logs from your NXLog Agent instances do not show in Log Search, but you can see them with the NXLog Platform application logs.

Possible reason

This issue occurs when the NXLog Platform data relay tries to forward logs to the database, but ClickHouse returns an error. In such cases, the data relay retries to send the data three times. If it fails the third time, it redirects the logs to the NXLog Platform application log. ClickHouse may fail to ingest data due to issues related to memory, buffering, or disk storage.

Investigation

Navigate to Administration > Application Logs from the left navigation menu.
Verify the log contains records from your NXLog Agent instances, for example, by filtering by the hostname.
Select one of these records and check its ResponseCode and ResponseBody fields. These fields indicate the response that ClickHouse returned.
Check the logs of the clickhouse-server container for errors.

Execute the following command on the machine where NXLog Platform is installed:
```
$ sudo podman logs $(sudo podman ps -a | grep clickhouse-server: | awk '{print $1}') 2>&1 | grep "Error"
```

Solution

The solution depends on the ClickHouse error and may require further troubleshooting. If you need assistance, please open a support ticket, and one of our experts will contact you.

Agents are sending logs to NXLog Platform but Log Search does not show any new logs

Symptom

You have verified that your NXLog Agent instances are sending logs and there are no errors from the agent side. However, you do not see the new logs in NXLog Platform.

You may also see errors like the following when you navigate to Log Search.

ClickHouse error: network error: error sending request for url (http://clickhouse:8123/?database=nxlog&query=DESCRIBE+TABLE+%60logs%60+FORMAT+JSONEachRowWithProgress)

Possible reason

The logs database ran out of memory and stopped processing logs or was shut down by the operating system’s OOM Killer.

Investigation

Check the database logs for any memory-related errors. Perform the following steps on the machine where NXLog Platform is installed:

Check the logs of the clickhouse-server container.

$ sudo podman logs $(sudo podman ps -a | grep clickhouse-server: | awk '{print $1}') 2>&1 | grep "memory limit exceeded"

If it’s a memory issue, you may see logging like the following:

2025.02.20 10:37:53.382959 [ 630 ] {} <Error> MergeTreeBackgroundExecutor:
    Exception while executing background task {9e5d59d5-ae29-494e-9d77-856bbc9ad088::202502_1_3455_300}:
    Code: 241. DB::Exception: (total) memory limit exceeded: would use 3.36 GiB (attempt to allocate chunk of 4361744 bytes),
    current RSS 533.26 MiB, maximum: 3.36 GiB. (MEMORY_LIMIT_EXCEEDED)

If not, check the operating system logs for out-of-memory errors.

Ubuntu

$ grep -i -E "oom|out of memory" /var/log/syslog

RHEL

$ grep -i -E "oom|Out of memory" /var/log/messages

If the operating system terminated the container, you will see logging similar to the following:

Feb 21 14:35:21 rhel kernel: Out of memory: Killed process 5678 (podman) total-vm:1048576kB, anon-rss:512000kB, file-rss:1024kB, shmem-rss:2048kB, UID:1000 pgtables:256kB oom_score_adj:0
Feb 21 14:35:21 rhel kernel: oom_reaper: reaped process 5678 (podman), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

Solution

Ensure that you provision enough memory for NXLog Platform to store logs. Check the CPU and RAM requirements in the installation guide.
Enable log storage in NXLog Platform by following these steps:
1. Open /etc/nxp.conf with a text editor.
  $ sudo nano /etc/nxp.conf
2. Set the value of NXP_STORE_LOGS to true.
3. Save your changes.
Apply the new settings and restart NXLog Platform.
```
$ sudo nxp_manage.sh apply
```