NXLog Legacy Documentation

LogPoint

LogPoint is a security information and event management (SIEM) solution for monitoring and analyzing log data. It is capable of detecting and responding to threats in real time, as well as visualizing aggregate event data collected from different sources. NXLog can integrate directly with the LogPoint SIEM solution by sending log data to it in syslog format over TCP or UDP. NXLog also supports receiving logs from the LogPoint Raw Syslog Forwarder.

LogPoint Agent vs NXLog Enterprise Edition

NXLog is a LogPoint partner, providing the technology for the LogPoint Agent. The LogPoint Agent is essentially a limited functionality NXLog Enterprise Edition agent, both sharing the same technology and configuration method. This means that a full-featured NXLog Enterprise Edition can easily replace a LogPoint Agent installation to gain access to additional functionality including:

Support for additional operating systems including IBM AIX and Apple macOS. See Supported platforms for a complete list of operating systems and architectures supported by NXLog Enterprise Edition.
An extended list of modules providing functionality such as:
- Collecting logs from ETW on Windows systems with the im_etw module.
- Setting up NXLog as a Windows Event Collector with the im_wseventing module and using WEF to collect events from remote Windows systems.
- Collecting network packet captures with the im_pcap module.
- Receiving logs over HTTP(s) with the im_http module, enabling you to integrate with cloud-based platforms.
- Native support for collecting logs from Linux, macOS, and AIX.
- Customized log processing with support for Perl, Python, Ruby, Java, and Go.
- And many more. See the complete list of input and extension modules available in NXLog Enterprise Edition.

To replace a LogPoint Agent installation, simply make a backup of your configuration, uninstall it, and install NXLog Enterprise Edition. The required sections from your previous configuration can be copied over to the new NXLog installation.

Configure LogPoint to receive logs from NXLog

For LogPoint to accept logs from a source, it must first be added to its device repository. Follow these steps to add the NXLog host as a device and configure it for syslog collection.

Log in to the LogPoint web interface.
Create a new repository:
1. Navigate to Settings > Configuration > Repos.
2. Click Add.
3. Enter a name, path, and retention period for the repository.
4. Click the Submit button.
Create a routing policy:
1. Navigate to Settings > Configuration > Routing Policies
2. Click Add Policy.
3. Enter a name for the policy.
4. In the Catch All drop-down select the previously created repository.
5. Click the Submit button.
Create a normalization policy:
1. Navigate to Settings > Configuration > Normalization Policies
2. Click Add.
3. Enter a name for the policy.
4. Select the applicable normalizer for your logs, for example under Compiled Normalizer, choose JSONCompiledNormalizer for JSON-formatted logs.
5. Click the Submit button.
Create a processing policy:
1. Navigate to Settings > Configuration > Processing Policies.
2. Click Add.
3. Enter a name for the policy.
4. In the Normalization Policy drop-down select the normalization policy created above.
5. In the Routing Policy drop-down select the routing policy created above.
6. Click the Submit button.
Add a device:
1. Navigate to Settings > Configuration > Devices.
2. Click Add.
3. Enter the name and IP of the NXLog server.
4. Select a device group if desired.
5. In the Time Zone drop-down select the time zone of the NXLog server.
6. Click the Submit button.
7. In the Available Collector Fetchers dialog, click on Syslog Collector.
8. In the Parsers drop-down select SyslogParser.
9. In the Processing Policy drop-down choose the processing policy created above.
10. Click the Submit button.

Forwarding events from Windows Event Log

NXLog supports various log collection methods for Windows Event Log and caters for both local and remote collection. It can also act as a central Windows Event Collector server, with remote hosts forwarding events to it. See the Windows Event Log chapter of the NXLog User Guide for more information on the supported collection methods and log processing capabilities.

LogPoint provides the LPA_Windows and JSONCompiledNormalizer compiled normalizers to parse events from Windows Event Log, both requiring data to be in JSON format. NXLog can forward events compatible with the format required by these normalizers using the JSON (xm_json) extension module.

Example 1. Forwarding events from Windows Event Log

The following configuration uses the im_msvistalog input module to collect event logs from the Application, System, and Security channels. Event records are enriched with the $log_type field to make them easily searchable in LogPoint and converted to JSON using the xm_json module. Events are then forwarded to LogPoint as syslog over TCP using the om_tcp module.

nxlog.conf

<Extension json>
    Module    xm_json
</Extension>

<Extension syslog>
    Module    xm_syslog
</Extension>

<Input eventlog>
    Module    im_msvistalog
    <QueryXML>
        <QueryList>
            <Query Id='0'>
                <Select Path='Application'>*</Select>
                <Select Path='System'>*</Select>
                <Select Path='Security'>*</Select>
            </Query>
        </QueryList>
    </QueryXML>
    Exec      $log_type = "event_log";
    Exec      $Message = to_json();
</Input>

<Output logpoint>
    Module    om_tcp
    Host      192.168.0.123:514
    Exec      to_syslog_bsd();
</Output>

Output sample

The following is an Application event log record after it was processed by NXLog. It is comprised of the syslog (RFC 3164) header followed by the event data in JSON format.

<14>Apr 22 14:08:04 WINSERVER-1 Service_Control_Manager[548]: {"EventTime":"2021-04-22 14:08:04","Hostname":"WINSERVER-1","Keywords":"9259400833873739776","EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":7036,"SourceName":"Service Control Manager","ProviderGuid":"{555908D1-A6D7-4695-8E1E-26931D2012F4}","Version":0,"TaskValue":0,"OpcodeValue":0,"RecordNumber":8530,"ExecutionProcessID":548,"ExecutionThreadID":1744,"Channel":"System","Message":"The nxlog service entered the running state.","param1":"nxlog","param2":"running","EventData.Binary":"6E0078006C006F0067002F0034000000","EventReceivedTime":"2021-04-22 14:08:06","SourceModuleName":"eventlog","SourceModuleType":"im_msvistalog","log_type":"event_log"}

The im_msvistalog module also supports collection of events from remote hosts. See Remote collection with im_msvistalog for an example.

Example 2. Forwarding events from Event Tracing for Windows (ETW)

NXLog Enterprise Edition exclusive feature

ETW is a Windows logging mechanism designed for kernel and user-mode application events. Some log channels based on ETW cannot be collected as regular Windows Event Log channels and require reading the ETW traces. NXLog provides the im_etw input module to read ETW traces.

The configuration below collects Windows Firewall events using the im_etw module. Event records are enriched with the $log_type field to make them easily searchable in LogPoint and converted to JSON using the xm_json module. Events are then forwarded to LogPoint as syslog over TCP using the om_tcp module.

The im_etw module is not available in the LogPoint Agent and requires a full-featured NXLog Enterprise Edition installation.

nxlog.conf

<Extension json>
    Module      xm_json
</Extension>

<Extension syslog>
    Module      xm_syslog
</Extension>

<Input etw_logs>
    Module      im_etw
    Provider    Microsoft-Windows-Windows Firewall With Advanced Security
    Exec        $log_type = "event_trace";
    Exec        $Message = to_json();
</Input>

<Output logpoint>
    Module      om_tcp
    Host        192.168.0.123:514
    Exec        to_syslog_bsd();
</Output>

Output sample

The following is a Windows Firewall event sample after it was processed by NXLog. The record is comprised of the syslog (RFC 3164) header followed by the event data in JSON format.

<14>Apr 22 10:39:43 WINSERVER-1 Microsoft-Windows-Windows_Firewall_With_Advanced_Security[880]: {"SourceName":"Microsoft-Windows-Windows Firewall With Advanced Security","ProviderGuid":"{D1BC9AFF-2ABF-4D71-9146-ECB2A986EB85}","EventID":2005,"Version":0,"ChannelID":16,"OpcodeValue":0,"TaskValue":0,"Keywords":"9223374235878031360","EventTime":"2021-04-22 10:39:43","ExecutionProcessID":880,"ExecutionThreadID":1436,"EventType":"INFO","SeverityValue":2,"Severity":"INFO","Hostname":"WINSERVER-1","Domain":"NT AUTHORITY","AccountName":"LOCAL SERVICE","UserID":"S-1-5-19","AccountType":"Well Known Group","Flags":"0","RuleId":"WMI-WINMGMT-Out-TCP","RuleName":"Windows Management Instrumentation (WMI-Out)","Origin":"Local","ApplicationPath":"C:\\Windows\\system32\\svchost.exe","ServiceName":"winmgmt","Direction":"Outbound","Protocol":"TCP","LocalPorts":"*","RemotePorts":"*","Action":"Allow","Profiles":"Private,Domain, Public","LocalAddresses":"*","RemoteAddresses":"*","EmbeddedContext":"@FirewallAPI.dll,-34251","Active":"No","EdgeTraversal":"None","LooseSourceMapped":"No","SecurityOptions":"None","ModifyingUser":"S-1-5-21-774970328-3136211108-3663920140-500","ModifyingApplication":"C:\\Windows\\System32\\mmc.exe","SchemaVersion":"538","RuleStatus":"65536","LocalOnlyMapped":"No","EventReceivedTime":"2021-04-22 10:39:44","SourceModuleName":"etw_logs","SourceModuleType":"im_etw","log_type":"event_trace"}

Forwarding logs in JSON format

NXLog can parse log records in various formats through its selection of extension modules or by making use of regular expressions and other string manipulation functions available in the NXLog language. See the chapter on Parsing various log formats for more information and examples.

LogPoint supports parsing events in JSON format with its JSONCompiledNormalizer compiled normalizer. Logs collected and parsed by NXLog can easily be forwarded as JSON using the JSON (xm_json) extension module.

Example 3. Forwarding network packet captures

The configuration below uses the im_pcap input module to capture HTTP network traffic. Packet captures are enriched with the $log_type and $protocol fields to make them easily searchable in LogPoint and converted to JSON using the xm_json module. Events are then forwarded to LogPoint as syslog over TCP using the om_tcp module.

The im_pcap module is not available in the LogPoint Agent and requires a full-featured NXLog Enterprise Edition installation.

nxlog.conf

<Extension json>
    Module      xm_json
</Extension>

<Extension syslog>
    Module      xm_syslog
</Extension>

<Input http_capture>
    Module      im_pcap
    <Protocol>
        Type    http
    </Protocol>
    <Exec>
        $log_type = "pcap";
        $protocol = "http";
        to_json();
    </Exec>
</Input>

<Output logpoint>
    Module      om_tcp
    Host        192.168.0.123:514
    Exec        to_syslog_bsd();
</Output>

Output sample

The following is an HTTP packet capture after it was processed by NXLog. The record is comprised of the syslog (RFC 3164) header followed by the packet capture data in JSON format.

<13>Apr 22 10:20:03 SERVER-1 {"http.header.accept":"application/font-woff2;q=1.0,application/font-woff;q=0.9,*/*;q=0.8","http.header.accept_encoding":"gzip, deflate","http.header.accept_language":"en-US,en;q=0.5","http.header.connection":"keep-alive","http.header.host":"movie.example.com","http.header.if_range":"\"575efded-a3e4\"","http.header.origin":"http://example.com","http.header.range":"bytes=12719-","http.header.referer":"http://example.com/","http.header.user_agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0","http.request.method":"GET","http.request.size":"84","http.request.uri":"/profiles/superhero/themes/consilium/fonts/opensans-light-webfont.ttf","http.request.url":"movie.example.com/profiles/superhero/themes/consilium/fonts/opensans-light-webfont.ttf","http.request.version":"HTTP/1.1","EventTime":"2021-04-22T10:20:03.044675+02:00","EventReceivedTime":"2021-04-22T10:20:04.015584+02:00","SourceModuleName":"http_capture","SourceModuleType":"im_pcap","log_type":"pcap","protocol":"http"}

Forwarding raw plain-text logs

LogPoint can parse logs into key-value pairs, a process referred to as normalization in the LogPoint solution. A comprehensive selection of normalization packages is available, with support for various vendor and application-specific log formats. NXLog can be configured to forward the logs it collects in their raw text-based format, leaving the parsing to be done by the LogPoint normalization process.

Example 4. Forwarding file-based logs

The configuration below uses the im_file input module to read events from a file. Events are then forwarded to LogPoint as syslog over UDP using the om_udp module.

nxlog.conf

<Extension syslog>
    Module      xm_syslog
</Extension>

<Input log_file>
    Module      im_file
    File        "/path/to/log/file"
</Input>

<Output logpoint>
    Module      om_udp
    Host        192.168.0.123:514
    Exec        to_syslog_bsd();
</Output>

Output sample

The following is an Apache access log record after it was processed by NXLog. It is comprised of the syslog (RFC 3164) header followed by the raw event data. LogPoint can parse this log record with the LP_Apache HTTP Server normalization package.

<13>Apr 22 11:42:23 SERVER-1 83.149.9.216 - - [22/Apr/2021:11:31:44 +0000] "GET /presentations/nxlog/images/icon.png HTTP/1.1" 200 7697 "http://example.com/presentations/nxlog/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"

Forwarding logs over TCP using TLS/SSL

NXLog can forward logs to LogPoint over a secure channel using the om_ssl output module. By default, LogPoint’s Syslog Collector listens for secure connections on port 6514 and uses certificates located in the following path: /opt/immune/etc/remote_connection/certificates/

To use the default certificates, the CA certificate ca.crt found in this location needs to be copied to the NXLog server and specified in the configuration as the CA certificate to be used for validating the connection with the LogPoint server.

To use custom certificates the configuration of the LogPoint Syslog Collector needs to be modified. It is not recommended to simply replace the default certificates with your certificates. For more information, refer to the LogPoint documentation on How to use custom certificate on Syslog Collector.

Changing the configuration of the LogPoint Syslog Collector requires root access to the LogPoint Server. Contact LogPoint support to request a partner account with root access and provide the MACHINE ID. Command to fetch the machine ID: cat /etc/machine-id

Example 5. Forwarding logs over TCP with TLS/SSL

The configuration below uses the om_ssl output module to forward logs to LogPoint over TCP with TLS/SSL encryption. The CAFile directive specifies the path to the CA certificate that will be used to validate the client certificate presented by the LogPoint server.

nxlog.conf

<Extension syslog>
    Module      xm_syslog
</Extension>

<Output logpoint>
    Module      om_ssl
    Host        192.168.0.123:6514
    CAFile      /path/to/logpoint/ca.crt
    Exec        to_syslog_bsd();
</Output>

Receiving logs from LogPoint

The LogPoint Raw Syslog Forwarder can forward raw events to a remote server in syslog format. NXLog can receive events forwarded by LogPoint over TCP or UDP. To configure the Raw Syslog Forwarder:

Log in to the LogPoint web interface.
Navigate to Settings > Configuration > Raw Syslog Forwarder.
Click Targets and then Add IP.
Enter the name, IP, and port of the NXLog server.
Select the desired protocol.
Click the Submit button.
Back on the Raw Syslog Forwarder page click Add.
Under Device Target Information select the devices you wish to forward events for.
Under Remote Target select the newly added NXLog server.
Click the Submit button.

Example 6. Receiving logs from LogPoint

The following input instance is configured to listen on TCP port 514 using all available IPv4 addresses. It expects to receive logs in the syslog (RFC 3164) format compatible with the LogPoint Raw Syslog Forwarder.

nxlog.conf

<Extension syslog>
    Module        xm_syslog
</Extension>

<Input tcp_listen>
    Module        im_tcp
    ListenAddr    0.0.0.0:514
    Exec          parse_syslog_bsd();
</Input>

Verifying data in LogPoint

Reception of log data can be verified using the LogPoint web interface. One way to do this is to search for events by fields using the LogPoint search syntax. Navigate to the Search page, enter your query in the search bar, and click the Search button. By default, only events that occurred in the last 10 minutes are displayed. You may need to modify this setting from the time range selector.

The image below shows events filtered by the log_type field. This was a custom field added to log records by the NXLog configuration.

LogPoint supports exporting search results to file. For this, an export target must be configured:

Navigate to Settings > Configuration > Export Management.
Click on Add.
Select the communication protocol of your choice between SCP and FTP.
Enter the target details and click Save.
Once an export target is configured, search results can be exported from the Search page by clicking on the More drop-down and selecting Export Logs. The raw events will be exported as a log file, one record per line.

Disclaimer

While we endeavor to keep the information in this topic up to date and correct, NXLog makes no representations or warranties of any kind, express or implied about the completeness, accuracy, reliability, suitability, or availability of the content represented here. We update our screenshots and instructions on a best-effort basis.

The accurateness of the content was tested and proved to be working in our lab environment at the time of the last revision with the following software versions:

NXLog version 5.3.6720
LogPoint version 6.9.0

Last revision: 12 May 2021