LogPoint
LogPoint is a security information and event management (SIEM) solution for monitoring and analyzing log data. It is capable of detecting and responding to threats in real time, as well as visualizing aggregate event data collected from different sources. NXLog Agent can integrate directly with the LogPoint SIEM solution by sending log data to it in syslog format over TCP or UDP. NXLog Agent also supports receiving logs from the LogPoint Raw Syslog Forwarder.
LogPoint Agent vs NXLog Agent
NXLog is a LogPoint partner, providing the technology for the LogPoint Agent. The LogPoint Agent is essentially a limited functionality NXLog Agent agent, both sharing the same technology and configuration method. This means that a full-featured NXLog Agent can easily replace a LogPoint Agent installation to gain access to additional functionality including:
-
Support for additional operating systems including IBM AIX and Apple macOS.
-
An extended list of modules providing functionality such as:
-
Collecting logs from ETW on Windows systems with the im_etw module.
-
Setting up NXLog Agent as a Windows Event Collector with the im_wseventing module and using WEF to collect events from remote Windows systems.
-
Collecting network packet captures with the im_pcap module.
-
Receiving logs over HTTP(s) with the im_http module, enabling you to integrate with cloud-based platforms.
-
Native support for collecting logs from Linux, macOS, and AIX.
-
Customized log processing with support for Perl, Python, Ruby, Java, and Go.
-
And many more. See the complete list of input and extension modules available in NXLog Agent.
-
To replace a LogPoint Agent installation, simply make a backup of your configuration, uninstall it, and install NXLog Agent. The required sections from your previous configuration can be copied over to the new NXLog Agent installation.
Configure LogPoint to receive logs from NXLog Agent
For LogPoint to accept logs from a source, it must first be added to its device repository. Follow these steps to add the NXLog Agent host as a device and configure it for syslog collection.
-
Log in to the LogPoint web interface.
-
Create a new repository:
-
Navigate to Settings > Configuration > Repos.
-
Click Add.
-
Enter a name, path, and retention period for the repository.
-
Click the Submit button.
-
-
Create a routing policy:
-
Navigate to Settings > Configuration > Routing Policies
-
Click Add Policy.
-
Enter a name for the policy.
-
In the Catch All drop-down select the previously created repository.
-
Click the Submit button.
-
-
Create a normalization policy:
-
Navigate to Settings > Configuration > Normalization Policies
-
Click Add.
-
Enter a name for the policy.
-
Select the applicable normalizer for your logs, for example under Compiled Normalizer, choose JSONCompiledNormalizer for JSON-formatted logs.
-
Click the Submit button.
-
-
Create a processing policy:
-
Navigate to Settings > Configuration > Processing Policies.
-
Click Add.
-
Enter a name for the policy.
-
In the Normalization Policy drop-down select the normalization policy created above.
-
In the Routing Policy drop-down select the routing policy created above.
-
Click the Submit button.
-
-
Add a device:
-
Navigate to Settings > Configuration > Devices.
-
Click Add.
-
Enter the name and IP of the NXLog Agent server.
-
Select a device group if desired.
-
In the Time Zone drop-down select the time zone of the NXLog Agent server.
-
Click the Submit button.
-
In the Available Collector Fetchers dialog, click on Syslog Collector.
-
In the Parsers drop-down select SyslogParser.
-
In the Processing Policy drop-down choose the processing policy created above.
-
Click the Submit button.
-
Forwarding events from Windows Event Log
NXLog Agent supports various log collection methods for Windows Event Log and caters for both local and remote collection. It can also act as a central Windows Event Collector server, with remote hosts forwarding events to it. See the Windows Event Log chapter of the NXLog Agent User Guide for more information on the supported collection methods and log processing capabilities.
LogPoint provides the LPA_Windows and JSONCompiledNormalizer compiled normalizers to parse events from Windows Event Log, both requiring data to be in JSON format. NXLog Agent can forward events compatible with the format required by these normalizers using the JSON (xm_json) extension module.
The following configuration uses the im_msvistalog input
module to collect event logs from the Application, System, and Security
channels. Event records are enriched with the $log_type
field to
make them easily searchable in LogPoint and converted to JSON using the
xm_json module. Events are then forwarded to LogPoint
as syslog over TCP using the om_tcp module.
<Extension json>
Module xm_json
</Extension>
<Extension syslog>
Module xm_syslog
</Extension>
<Input eventlog>
Module im_msvistalog
<QueryXML>
<QueryList>
<Query Id='0'>
<Select Path='Application'>*</Select>
<Select Path='System'>*</Select>
<Select Path='Security'>*</Select>
</Query>
</QueryList>
</QueryXML>
Exec $log_type = "event_log";
Exec $Message = to_json();
</Input>
<Output logpoint>
Module om_tcp
Host 192.168.0.123:514
Exec to_syslog_bsd();
</Output>
The following is an Application event log record after it was processed by NXLog Agent. It is comprised of the syslog (RFC 3164) header followed by the event data in JSON format.
<14>Apr 22 14:08:04 WINSERVER-1 Service_Control_Manager[548]: {"EventTime":"2021-04-22 14:08:04","Hostname":"WINSERVER-1","Keywords":"9259400833873739776","EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":7036,"SourceName":"Service Control Manager","ProviderGuid":"{555908D1-A6D7-4695-8E1E-26931D2012F4}","Version":0,"TaskValue":0,"OpcodeValue":0,"RecordNumber":8530,"ExecutionProcessID":548,"ExecutionThreadID":1744,"Channel":"System","Message":"The nxlog service entered the running state.","param1":"nxlog","param2":"running","EventData.Binary":"6E0078006C006F0067002F0034000000","EventReceivedTime":"2021-04-22 14:08:06","SourceModuleName":"eventlog","SourceModuleType":"im_msvistalog","log_type":"event_log"}
The im_msvistalog module also supports collecting events from remote hosts. See Remote collection with im_msvistalog for an example. |
ETW is a Windows logging mechanism designed for kernel and user-mode application events. Some log channels based on ETW cannot be collected as regular Windows Event Log channels and require reading the ETW traces. NXLog Agent provides the im_etw input module to read ETW traces.
The configuration below collects Windows Firewall events using the im_etw
module. Event records are enriched with the $log_type
field to make them
easily searchable in LogPoint and converted to JSON using the
xm_json module. Events are then forwarded to LogPoint as syslog over
TCP using the om_tcp module.
The im_etw module is not available in the LogPoint Agent and requires a full-featured NXLog Agent installation. |
<Extension json>
Module xm_json
</Extension>
<Extension syslog>
Module xm_syslog
</Extension>
<Input etw_logs>
Module im_etw
Provider Microsoft-Windows-Windows Firewall With Advanced Security
Exec $log_type = "event_trace";
Exec $Message = to_json();
</Input>
<Output logpoint>
Module om_tcp
Host 192.168.0.123:514
Exec to_syslog_bsd();
</Output>
The following is a Windows Firewall event sample after it was processed by NXLog Agent. The record is comprised of the syslog (RFC 3164) header followed by the event data in JSON format.
<14>Apr 22 10:39:43 WINSERVER-1 Microsoft-Windows-Windows_Firewall_With_Advanced_Security[880]: {"SourceName":"Microsoft-Windows-Windows Firewall With Advanced Security","ProviderGuid":"{D1BC9AFF-2ABF-4D71-9146-ECB2A986EB85}","EventID":2005,"Version":0,"ChannelID":16,"OpcodeValue":0,"TaskValue":0,"Keywords":"9223374235878031360","EventTime":"2021-04-22 10:39:43","ExecutionProcessID":880,"ExecutionThreadID":1436,"EventType":"INFO","SeverityValue":2,"Severity":"INFO","Hostname":"WINSERVER-1","Domain":"NT AUTHORITY","AccountName":"LOCAL SERVICE","UserID":"S-1-5-19","AccountType":"Well Known Group","Flags":"0","RuleId":"WMI-WINMGMT-Out-TCP","RuleName":"Windows Management Instrumentation (WMI-Out)","Origin":"Local","ApplicationPath":"C:\\Windows\\system32\\svchost.exe","ServiceName":"winmgmt","Direction":"Outbound","Protocol":"TCP","LocalPorts":"*","RemotePorts":"*","Action":"Allow","Profiles":"Private,Domain, Public","LocalAddresses":"*","RemoteAddresses":"*","EmbeddedContext":"@FirewallAPI.dll,-34251","Active":"No","EdgeTraversal":"None","LooseSourceMapped":"No","SecurityOptions":"None","ModifyingUser":"S-1-5-21-774970328-3136211108-3663920140-500","ModifyingApplication":"C:\\Windows\\System32\\mmc.exe","SchemaVersion":"538","RuleStatus":"65536","LocalOnlyMapped":"No","EventReceivedTime":"2021-04-22 10:39:44","SourceModuleName":"etw_logs","SourceModuleType":"im_etw","log_type":"event_trace"}
Forwarding logs in JSON format
NXLog Agent can parse log records in various formats through its selection of extension modules or by making use of regular expressions and other string manipulation functions available in the NXLog Agent language.
LogPoint supports parsing events in JSON format with its JSONCompiledNormalizer compiled normalizer. Logs collected and parsed by NXLog Agent can easily be forwarded as JSON using the JSON (xm_json) extension module.
The configuration below uses the im_pcap input module to capture
HTTP network traffic. Packet captures are enriched with the $log_type
and
$protocol
fields to make them easily searchable in LogPoint and converted to
JSON using the xm_json module. Events are then forwarded to LogPoint
as syslog over TCP using the om_tcp module.
The im_pcap module is not available in the LogPoint Agent and requires a full-featured NXLog Agent installation. |
<Extension json>
Module xm_json
</Extension>
<Extension syslog>
Module xm_syslog
</Extension>
<Input http_capture>
Module im_pcap
<Protocol>
Type http
</Protocol>
<Exec>
$log_type = "pcap";
$protocol = "http";
to_json();
</Exec>
</Input>
<Output logpoint>
Module om_tcp
Host 192.168.0.123:514
Exec to_syslog_bsd();
</Output>
The following is an HTTP packet capture after it was processed by NXLog Agent. The record is comprised of the syslog (RFC 3164) header followed by the packet capture data in JSON format.
<13>Apr 22 10:20:03 SERVER-1 {"http.header.accept":"application/font-woff2;q=1.0,application/font-woff;q=0.9,*/*;q=0.8","http.header.accept_encoding":"gzip, deflate","http.header.accept_language":"en-US,en;q=0.5","http.header.connection":"keep-alive","http.header.host":"movie.example.com","http.header.if_range":"\"575efded-a3e4\"","http.header.origin":"http://example.com","http.header.range":"bytes=12719-","http.header.referer":"http://example.com/","http.header.user_agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0","http.request.method":"GET","http.request.size":"84","http.request.uri":"/profiles/superhero/themes/consilium/fonts/opensans-light-webfont.ttf","http.request.url":"movie.example.com/profiles/superhero/themes/consilium/fonts/opensans-light-webfont.ttf","http.request.version":"HTTP/1.1","EventTime":"2021-04-22T10:20:03.044675+02:00","EventReceivedTime":"2021-04-22T10:20:04.015584+02:00","SourceModuleName":"http_capture","SourceModuleType":"im_pcap","log_type":"pcap","protocol":"http"}
Forwarding raw plain-text logs
LogPoint can parse logs into key-value pairs, a process referred to as normalization in the LogPoint solution. A comprehensive selection of normalization packages is available, with support for various vendor and application-specific log formats. NXLog Agent can be configured to forward the logs it collects in their raw text-based format, leaving the parsing to be done by the LogPoint normalization process.
The configuration below uses the im_file input module to read events from a file. Events are then forwarded to LogPoint as syslog over UDP using the om_udp module.
<Extension syslog>
Module xm_syslog
</Extension>
<Input log_file>
Module im_file
File "/path/to/log/file"
</Input>
<Output logpoint>
Module om_udp
Host 192.168.0.123:514
Exec to_syslog_bsd();
</Output>
The following is an Apache access log record after it was processed by NXLog Agent. It is comprised of the syslog (RFC 3164) header followed by the raw event data. LogPoint can parse this log record with the LP_Apache HTTP Server normalization package.
<13>Apr 22 11:42:23 SERVER-1 83.149.9.216 - - [22/Apr/2021:11:31:44 +0000] "GET /presentations/nxlog/images/icon.png HTTP/1.1" 200 7697 "http://example.com/presentations/nxlog/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
Forwarding logs over TCP using TLS/SSL
NXLog Agent can forward logs to LogPoint over a secure channel using the
om_ssl output module. By default, LogPoint’s Syslog Collector listens
for secure connections on port 6514 and uses certificates located in the
following path: /opt/immune/etc/remote_connection/certificates/
To use the default certificates, the CA certificate ca.crt
found in this
location needs to be copied to the NXLog Agent server and specified in the
configuration as the CA certificate to be used for validating the
connection with the LogPoint server.
To use custom certificates the configuration of the LogPoint Syslog Collector needs to be modified. It is not recommended to simply replace the default certificates with your own certificates. For more information, refer to the LogPoint documentation on How to use custom certificate on Syslog Collector.
Changing the configuration of the LogPoint Syslog Collector requires root access to the LogPoint Server.
Contact LogPoint support to request a partner account with root access and provide the MACHINE ID.
Command to fetch the machine ID: cat /etc/machine-id
|
The configuration below uses the om_ssl output module to forward logs
to LogPoint over TCP with TLS/SSL encryption. The CAFile
directive specifies
the path to the CA certificate that will be used to validate the client
certificate presented by the LogPoint server.
<Extension syslog>
Module xm_syslog
</Extension>
<Output logpoint>
Module om_ssl
Host 192.168.0.123:6514
CAFile /path/to/logpoint/ca.crt
Exec to_syslog_bsd();
</Output>
Receiving logs from LogPoint
The LogPoint Raw Syslog Forwarder can forward raw events to a remote server in syslog format. NXLog Agent can receive events forwarded by LogPoint over TCP or UDP. To configure the Raw Syslog Forwarder:
-
Log in to the LogPoint web interface.
-
Navigate to Settings > Configuration > Raw Syslog Forwarder.
-
Click Targets and then Add IP.
-
Enter the name, IP, and port of the NXLog Agent server.
-
Select the desired protocol.
-
Click the Submit button.
-
Back on the Raw Syslog Forwarder page click Add.
-
Under Device Target Information select the devices you wish to forward events for.
-
Under Remote Target select the newly added NXLog Agent server.
-
Click the Submit button.
The following input instance is configured to listen on TCP port 514 using all available IPv4 addresses. It expects to receive logs in the syslog (RFC 3164) format compatible with the LogPoint Raw Syslog Forwarder.
<Extension syslog>
Module xm_syslog
</Extension>
<Input tcp_listen>
Module im_tcp
ListenAddr 0.0.0.0:514
Exec parse_syslog_bsd();
</Input>
Verifying data in LogPoint
Reception of log data can be verified using the LogPoint web interface. One way to do this is to search for events by fields using the LogPoint search syntax. Navigate to the Search page, enter your query in the search bar, and click the Search button. By default, only events that occurred in the last 10 minutes are displayed. You may need to modify this setting from the time range selector.
The image below shows events filtered by the log_type
field. This was a
custom field added to log records by the NXLog Agent configuration.
LogPoint supports exporting search results to a file. For this, an export target must be configured:
-
Navigate to Settings > Configuration > Export Management.
-
Click on Add.
-
Select the communication protocol of your choice between SCP and FTP.
-
Enter the target details and click Save.
-
Once an export target is configured, search results can be exported from the Search page by clicking on the More drop-down and selecting Export Logs. The raw events will be exported as a log file, one record per line.