TLS/SSL errors

When forwarding logs over TCP with TLS/SSL encryption, you may encounter errors that prevent NXLog Agent from establishing a connection with the remote host.

Below, we provide tips to help troubleshoot various TLS/SSL-related errors.

Couldn’t read certificate

Symptom

NXLog Agent fails to connect to NXLog Platform with an error like the following:

ERROR [xm_admin|admin] SSL error, couldn't read cert, reason: no start line

Possible reason

One of the certificate files referenced by the Remote Management configuration doesn’t contain valid PEM-encoded data. This can happen after a new NXLog Agent instance first connects to NXLog Platform, since the installer only checks that the onboarding certificate files exist but doesn’t validate their content.

Investigation

Use the OpenSSL tool to check whether the files in the NXLog Agent certificate directory contain valid data. By default, the certificate directory is C:\Program Files\nxlog\cert on Windows and /opt/nxlog/var/lib/nxlog/cert on Linux and macOS.

$ openssl x509 -text -in onboarding-ca.pem
$ openssl x509 -text -in onboarding-cert.pem
$ openssl pkey -text -in onboarding-key.pem

An error from any of these commands confirms that the corresponding file is not valid.

Solution

Replace the file directly in the NXLog Agent certificate directory and restart NXLog Agent. Otherwise, force a reinstall of NXLog Agent with NXP_CERT_DIR set to a directory containing the correct files:

On Windows, add the REINSTALL=ALL option to the msiexec command.
On Debian/Ubuntu, add the --reinstall flag to the apt install command.
On Red Hat Enterprise Linux, use the dnf reinstall command.
On macOS, run the installation again.

Certificate verification failed

Symptom

Typical TLS/SSL certificate verification errors include:

SSL certificate problem: unable to get local issuer certificate

ERROR SSL certificate verification failed: unable to get local issuer certificate (err: 20)

Using a self-signed certificate causes the following error:

ERROR SSL certificate verification failed: self signed certificate (err: 18)

Possible reason

Certificate verification errors mean the Certificate Authority (CA) certificate could not verify the remote certificate’s validity. The necessary CA certificate may not exist on the machine, or you specified the wrong CA certificate in the NXLog Agent configuration.

Investigation

If you do not explicitly set a CA certificate in the configuration, NXLog Agent uses the default operating system root certificate store to verify remote server certificates. Unix-like operating systems commonly store root certificates in /etc/ssl/certs, Windows uses the Windows Certificate Store, and macOS uses the Keychain Access Application as the default certificate store.

You can use the OpenSSL tool to verify certificates. Execute the following command to verify a CA file:

$ openssl s_client -CAfile <path_to_CA_file> -connect <host:port>

Use the following command to verify a server certificate against a CA certificate:

$ openssl s_client -connect <host:port> -cert <path_to_cert_file> -key <path_to_key_file> -CAfile <path_to_CA_file> -verify 1

You can also verify a server certificate against a CA directory:

$ openssl s_client -connect <host:port> -cert <path_to_cert_file> -key <path_to_key_file> -CApath  <path_to_CA_dir> -verify 1

Solution

Ensure that NXLog Agent has access to the relevant CA certificate to validate the remote host’s certificate, either in the operating system’s root certificate store or in the NXLog Agent configuration.

If the server certificate is signed by an intermediate CA, the CA certificate must contain the complete certificate chain (certificate bundle).

If the remote host uses a self-signed certificate not signed by a CA, you can specify the server certificate itself instead of a CA certificate.

Certificate has expired

Symptom

NXLog Agent fails to connect to the remote host with the following error:

ERROR SSL certificate verification failed: certificate has expired (err: 10)

Possible reason

This error occurs when the remote host presents an expired certificate.

Investigation

You can use the OpenSSL tool to check a certificate’s expiry date:

$ openssl x509 -enddate -noout -in mycert.pem
notAfter=Dec 31 20:00:00 2024 GMT

On Windows, you can double-click on a .crt file and switch to the Details tab to view the expiration date.

Solution

Configure the remote host with a renewed certificate. Alternatively, configure the NXLog Agent input or output instance to accept expired certificates by setting AllowExpired to TRUE. We recommend only accepting expired certificates as a temporary measure to avoid log processing disruption. Remember to switch the setting off once the remote host’s certificate is renewed.

Remote SSL socket was reset

Symptom

NXLog Agent fails to connect to the remote host with the following error:

ERROR remote ssl socket was reset? (SSL_ERROR_SSL with errno=9); End of file found

Possible reason

This issue may occur for two reasons:

The remote host, such as NXLog Platform, cannot verify the NXLog Agent certificate.
A firewall or other network security device is terminating the connection.

Investigation

Check the remote host’s logs and network packet captures to troubleshoot this issue. The SSL stack refusing the connection may contain a more precise reason.

Solution

The remote host must have the relevant CA certificate to validate the NXLog Agent’s server certificate. See Certificate verification failed above for troubleshooting tips.

If you verify it is not a certificate issue, ensure a firewall does not block communication between the NXLog Agent machine and the remote host. See Connection refused or timed out for troubleshooting tips.

Unsupported certificate purpose

Symptom

NXLog Agent fails to connect to the remote host with the following error:

ERROR SSL certificate verification failed: unsupported certificate purpose (err: 26)

Possible reason

This error means that the certificate does not have the required key and extended/enhanced key usage properties. These are optional properties when creating the certificate. For example, see Key Usage and Extended Key Usage in the OpenSSL documentation.

Investigation

You can use the OpenSSL tool to examine the certificate details:

$ openssl x509 -text -in mycert.pem

The output may include details similar to the following:

X509v3 extensions:
    X509v3 Key Usage: critical
        Digital Signature, Key Encipherment, Key Agreement
    X509v3 Extended Key Usage:
       TLS Web Server Authentication, TLS Web Client Authentication

On Windows, you can double-click on a .crt file and switch to the Details tab to view the key and enhanced key usage.

Solution

If the local or remote host’s certificate has the key and extended/enhanced key usage fields, ensure that they include the following values:

Key Usage: Digital Signature, Key Encipherment, Key Agreement
Extended Key Usage: TLS Web Server Authentication, TLS Web Client Authentication

No shared cipher

Symptom

NXLog Agent fails to connect to the remote host with the following error:

ERROR SSL error, SSL_ERROR_SSL: retval -1, from 127.0.0.1:33240, reason: no shared cipher

Possible reason

This error occurs when your environment applies a TLS/SSL cipher restriction, and the remote host does not use any of those ciphers.

Investigation

Check your NXLog Agent configuration to verify if the module instance includes the SSLCipher or SSLCipherSuites directive. For example, the following im_batchcompress instance includes a cipher restriction:

<Input im_batchcompress>
    Module           im_batchcompress
    ListenAddr       0.0.0.0:2514
    CAFile           %CERTDIR%/agent-ca.pem
    CertFile         %CERTDIR%/agent-cert.pem
    CertKeyFile      %CERTDIR%/agent-key.pem
    SSLCipher        ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256
</Input>

You can use the OpenSSL tool to emulate the above configuration and check whether you encounter the same problem when the remote host tries to connect:

$ openssl s_server -key agent-key.pem -CAfile agent-ca.pem -cert agent-cert.pem -port 7000 -cipher ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256

You can test the above listener with the following command:

$ openssl s_client -key agent-key.pem -CAfile agent-ca.pem -cert agent-cert.pem -connect localhost:2514

Solution

NXLog Agent and the remote host must share at least one common cipher. For more information, see Configuring OpenSSL Defaults in the OpenSSL cookbook and Ciphers in the OpenSSL documentation.

The remote host does not support TLS/SSL

Symptom

NXLog Agent fails to connect to the remote host with the following error:

ERROR SSL error, SSL_ERROR_SSL: retval -1, reason: wrong version number

Possible reason

This error occurs when you configure NXLog Agent to use TLS/SSL, but the remote host does not support encryption.

Investigation

Verify whether the remote SIEM or service supports TLS/SSL encryption.

Solution

The remote host must support TLS/SSL and be configured to use it. If not, configure NXLog Agent to connect via TCP without TLS/SSL encryption. For configuration information, check the relevant module’s documentation.