NXLog Documentation

Google Chronicle

Google Chronicle is a cloud-based service from Google which is designed to collect and process log data. The ingested data can be searched and selected based on specific criteria, such as assets, domains, or IP addresses. This service can help alert organizations when any of their systems are compromised.

Chronicle is the foundation for an extended SIEM system called CYDERES CNAP. For more details, visit the Fishtech Group website.

Registered users can access the official Chronicle documentation through Chronicle’s user interface. To view the documentation, visit the https://<customername>.backstory.chronicle.security website and navigate to the Documentation section.

Since the documentation is only available to paid subscribers, you will need to substitute the <customername> part of the above URL with your organization’s Google Chronicle customer name.

NXLog provides various ways to send logs to Chronicle.

Forwarding log data to Chronicle

Chronicle can accept both structured (UDM-formatted) and unstructured messages. For Chronicle to accept events as structured data, it needs special formatting prior to forwarding. For unstructured events, Chronicle parses and processes them on reception.

In Chronicle, unstructured events are typically assigned to a specific log source, for instance,WINDOWS_DNS or LINUX_OS. Structured or UDM-formatted events are associated with a user or system action like USER_CREATION or PROCESS_LAUNCH. Consequently, both types of logs serve different purposes in different situations thus one type cannot be preferred without exception over the other. For more information regarding data formats, consult the Google Chronicle documentation.

NXLog can be configured to collect and forward log data to Google Chronicle using the following methods:

Sending logs via the Ingestion API is a direct forwarding method independent of any intermediary software like Chronicle Forwarder. This method is more flexible and allows Chronicle to immediately parse events as they are received, provided they are formatted according to the Unified Data Model (UDM). The only downside of this method could be the additional overhead of the JSON payload, which in most cases is negligible.

Sending logs directly to the Chronicle Ingestion API
Figure 1. Sending logs directly to the Chronicle Ingestion API

Sending logs via Chronicle Forwarder offers an easier initial configuration and built-in passive network capabilities; however, it has some significant disadvantages. First of all, it requires intermediary software to be installed on the network, which can result in additional licensing costs and resource usage. Another inflexibility of this logging tool is the requirement that the Linux version can only run in a Docker environment, which might further complicate the setup. The Chronicle Forwarder is also limited to unstructured log data, thus lacking any capability of forwarding UDM-formatted logs. Although the Chronicle Forwarder is certainly an option, it will likely complicate the logging environment while providing little in return. Google Chronicle would also require additional configuration to enable fine-grained processing of incoming data.

Sending logs via the Chronicle Forwarder
Figure 2. Sending logs via the Chronicle Forwarder

Fortunately, a central NXLog agent can replace Chronicle Forwarder for sending logs to Google Chronicle without sacrificing any functionality. This approach provides flexibility by eliminating the need to install Chronicle Forwarder on Microsoft Windows or Docker. Additionally, NXLog can be configured to process both unstructured and UDM-formatted data. It can also provide additional functionality, like Passive network monitoring by using the im_pcap module.

Sending logs via a central NXLog agent
Figure 3. Sending logs via a central NXLog agent

Forwarding logs to the Ingestion API

The Chronicle Partner Ingestion API is the universal, preferred method of delivering logs to Chronicle. This RESTful API accepts incoming data in the form of JSON payloads and uses API keys for authentication.

The Ingestion API provides endpoints for the following operations:

Listing log sources

The Ingestion API provides the logtypes endpoint for retrieving a list of over 400 unstructured log sources. You can call this endpoint with the actual API key as shown below which returns a list of JSON objects, each comprised of a logType and description field.

Retrieving log sources
$ curl --header "Content-Type: application/json" --request GET  https://malachiteingestion-pa.googleapis.com/v1/logtypes?key=<YOURAPIKEY>
Each logType in this list can only be sent as unstructured data to Chronicle.

Sending unstructured data to Chronicle

As its name suggests, the unstructuredlogentries endpoint accepts unstructured log data. However, before forwarding, log data should be contained into a JSON payload as displayed in the sample below.

JSON payload for unstructured data
{
  "log_type": "LOG_SOURCE",
  "entries": [
    {
      "log_text": "Log message"
    }
  ]
}

The sample consists of the following fields:

  • log_type specifies a supported logType, e.g. AWS_CLOUDWATCH, NGINX, WINDOWS_DHCP, etc.

  • entries specifies an array of JSON objects, each object is comprised of a single key-value pair: {"log_text":"<an entire event as a string>"}.

  • log_text specifies the entire event as a string. The value should be in the expected format of the defined log type. See the Google Chronicle documentation on Supported default parsers.

NXLog provides the om_chronicle output module to send unstructured log entries to Google Chronicle. This module takes care of creating the JSON payload according to the format described above. It uses the value specified by the LogType directive for the log_type field and the value of the $raw_event core field for the log_text field.

The om_chronicle module is available from NXLog version 5.5 onward. For older versions, you can use the om_http output module to forward logs to the Google Chronicle Ingestion API, which requires you to format the payload manually. See Forwarding logs using a central NXLog agent for an example.
Example 1. Forwarding unstructured logs in syslog format

This example demonstrates how to configure NXLog to forward logs collected from a BIND 9 DNS server. The LogType directive of the om_chronicle module specifies BIND_DNS, which requires the event to be in syslog format.

The sample DNS message below was collected from a BIND DNS server configured to write log messages to file.

Input sample
25-Jan-2022 09:56:02.187 client 10.120.20.20#4238:  query: example.com IN A + (100.90.80.102)

This NXLog configuration specifies Chronicle’s base URL, endpoint, and API key by defining three constants at the beginning of the file. For the API_KEY constant, replace <YOURAPIKEY> with the actual API key you use for authentication. The constants are used to construct the value needed for the URL directive in the om_chronicle output module instance.

The dns_logs input instance of the im_file module reads log events from file. Within the Exec block, the to_syslog_bsd() procedure of the xm_syslog module converts each event record to syslog format.

nxlog.conf
# Defining constants to compose the domain name with the API key
define BASE_URL            https://malachiteingestion-pa.googleapis.com/v1/
define ENDPOINT            unstructuredlogentries
define API_KEY             key=<YOURAPIKEY>

<Extension syslog>
    Module                 xm_syslog
    UTCTimestamp           TRUE
</Extension>

<Input dns_logs>
    Module                 im_file
    File                   '/path/to/log/file'
    <Exec>
        $Message = $raw_event;
        to_syslog_bsd();
    </Exec>
</Input>

<Output to_chronicle>
    Module                 om_chronicle
    URL                    %BASE_URL%%ENDPOINT%?%API_KEY%
    HTTPSAllowUntrusted    TRUE
    LogType                BIND_DNS
    ChronicleBatchSize     1024
</Output>
Output sample in JSON
{
  "log_type": "BIND_DNS",
  "entries": [
    {
      "log_text": "<13>Jan 25 08:56:36 DNS-SRV 25-Jan-2022 09:56:02.187 client 10.120.20.20#4238:  query: example.com IN A + (100.90.80.102)",
      "ts_rfc3339": "2022-01-25T09:56:36.265063+01:00"
    }
  ]
}
Example 2. Forwarding unstructured logs in syslog + JSON format

This example demonstrates how to configure NXLog to forward logs collected from Windows Event Log. The LogType directive of the om_chronicle module specifies WINEVTLOG, which requires the event to be in syslog + JSON format.

The sample Windows event below was collected from the System log.

Input sample
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager" />
    <EventID Qualifiers="16384">7036</EventID>
    <Version>0</Version>
    <Level>4</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8080000000000000</Keywords>
    <TimeCreated SystemTime="2022-01-25T14:44:10.864100000Z" />
    <EventRecordID>15297</EventRecordID>
    <Correlation />
    <Execution ProcessID="540" ThreadID="952" />
    <Channel>System</Channel>
    <Computer>WIN2016-2</Computer>
    <Security />
  </System>
  <EventData>
    <Data Name="param1">nxlog</Data>
    <Data Name="param2">running</Data>
    <Binary>6E0078006C006F0067002F0034000000</Binary>
  </EventData>
</Event>

This NXLog configuration specifies Chronicle’s base URL, endpoint, and API key by defining three constants at the beginning of the file. For the API_KEY constant, replace <YOURAPIKEY> with the actual API key you use for authentication. The constants are used to construct the value needed for the URL directive in the om_chronicle output module instance.

The windows_events input instance of the im_msvistalog module is configured to capture a specific set of security logs from Windows Event Log. Within the Exec block, the to_json() function of the xm_json module converts each log record to JSON. The record is then converted to syslog by calling the to_syslog_bsd() procedure of the xm_syslog module.

nxlog.conf
# Defining constants to compose the domain name with the API key
define BASE_URL            https://malachiteingestion-pa.googleapis.com/v1/
define ENDPOINT            unstructuredlogentries
define API_KEY             key=<YOURAPIKEY>

<Extension json>
    Module                 xm_json
    DateFormat             YYYY-MM-DDThh:mm:ss.sUTC
</Extension>

<Extension syslog>
    Module                 xm_syslog
    UTCTimestamp           TRUE
</Extension>

<Input windows_events>
    Module                 im_msvistalog
    <QueryXML>
        <QueryList>
            <Query Id='0'>
                <Select Path='Application'>*</Select>
                <Select Path='Security'>*[System/Level&lt;4]</Select>
                <Select Path='System'>*</Select>
            </Query>
        </QueryList>
    </QueryXML>
    <Exec>
        $Message = to_json();
        to_syslog_bsd();
    </Exec>
</Input>

<Output to_chronicle>
    Module                 om_chronicle
    URL                    %BASE_URL%%ENDPOINT%?%API_KEY%
    HTTPSAllowUntrusted    TRUE
    LogType                WINEVTLOG
    ChronicleBatchSize     1024
</Output>
Output sample in JSON
{
  "log_type": "WINEVTLOG",
  "entries": [
    {
      "log_text": "<14>Jan 25 14:44:10 WIN2016-2 Service_Control_Manager[540]: {\"EventTime\":\"2022-01-25T14:44:10.864100Z\",\"Hostname\":\"WIN2016-2\",\"Keywords\":\"9259400833873739776\",\"LevelValue\":4,\"EventType\":\"INFO\",\"SeverityValue\":2,\"Severity\":\"INFO\",\"EventID\":7036,\"SourceName\":\"Service Control Manager\",\"ProviderGuid\":\"{555908D1-A6D7-4695-8E1E-26931D2012F4}\",\"Version\":0,\"TaskValue\":0,\"OpcodeValue\":0,\"RecordNumber\":15297,\"ExecutionProcessID\":540,\"ExecutionThreadID\":952,\"Channel\":\"System\",\"Message\":\"The nxlog service entered the running state.\",\"Level\":\"Information\",\"param1\":\"nxlog\",\"param2\":\"running\",\"EventData.Binary\":\"6E0078006C006F0067002F0034000000\",\"EventReceivedTime\":\"2022-01-25T14:44:11.958250Z\",\"SourceModuleName\":\"eventlog\",\"SourceModuleType\":\"im_msvistalog\"}",
      "ts_rfc3339": "2022-01-25T15:44:11.958250+01:00"
    }
  ]
}

Forwarding structured logs to Chronicle

For log sources not included in the list of unstructured log sources, the Unified Data Model (UDM) can be used to forward structured data to Chronicle. Examples of such log sources might be events associated with sending an email or creating a new user.

Before forwarding, the event fields need to be incorporated into the JSON structure that Chronicle expects for incoming structured data. The events can then be forwarded to Google Chronicle using the om_http output module.

Example 3. Forwarding UDM-formatted logs

This example shows how to collect a Windows Event Log event with Event ID 4688 (a new process has been created), format it to the PROCESS_LAUNCH type, and forward the results to Chronicle.

This XML sample event represents an event that matched the QueryXML and filter defined in the from_eventlog instance of the NXLog configuration below.

Input sample
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-Security-Auditing" Guid="{54849625-5478-4994-A5BA-3E3B0328C30D}" />
    <EventID>4688</EventID>
    <Version>2</Version>
    <Level>0</Level>
    <Task>13312</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8020000000000000</Keywords>
    <TimeCreated SystemTime="2021-01-16T05:50:13.788209900Z" />
    <EventRecordID>1683</EventRecordID>
    <Correlation />
    <Execution ProcessID="4" ThreadID="260" />
    <Channel>Security</Channel>
    <Computer>WIN-ET85AK2E1J1</Computer>
    <Security />
  </System>
  <EventData>
    <Data Name="SubjectUserSid">S-1-5-21-3213787892-1493673803-1430668809-500</Data>
    <Data Name="SubjectUserName">Administrator</Data>
    <Data Name="SubjectDomainName">WIN-ET85AK2E1J1</Data>
    <Data Name="SubjectLogonId">0x25369</Data>
    <Data Name="NewProcessId">0xcc8</Data>
    <Data Name="NewProcessName">C:\Windows\System32\ftp.exe</Data>
    <Data Name="TokenElevationType">%%1936</Data>
    <Data Name="ProcessId">0xb14</Data>
    <Data Name="CommandLine">ftp.exe</Data>
    <Data Name="TargetUserSid">S-1-0-0</Data>
    <Data Name="TargetUserName">-</Data>
    <Data Name="TargetDomainName">-</Data>
    <Data Name="TargetLogonId">0x0</Data>
    <Data Name="ParentProcessName">C:\Windows\System32\cmd.exe</Data>
    <Data Name="MandatoryLabel">S-1-16-12288</Data>
  </EventData>
</Event>

To parse Windows Event Log entries, the NXLog configuration uses the im_msvistalog module.

The BASE_URL, ENDPOINT, and API_KEY constants are used to construct the value needed for the URL directive in the to_chronicle instance of the om_http output module. The Exec block enables the UDM-formatted payload to be constructed, which is then forwarded to Chronicle.

nxlog.conf
define BASE_URL         https://malachiteingestion-pa.googleapis.com/v1/
define ENDPOINT         udmevents
define API_KEY          key=<YOURAPIKEY>

<Input from_eventlog>
    Module              im_msvistalog
    <QueryXML>
        <QueryList>
            <Query Id="0">
                <Select Path="Security">
                    *[System[Level=0 and (EventID=4688)]]
                </Select>
            </Query>
        </QueryList>
    </QueryXML>
    <Exec>
        if not ($NewProcessName =~ /.*ftp.exe/) drop();
        delete($Message);
    </Exec>
</Input>

<Output to_chronicle>
    Module              om_http
    URL                 %BASE_URL%%ENDPOINT%?%API_KEY%
    HTTPSAllowUntrusted TRUE
    ContentType         application/json
    <Exec>
        $timestamp = strftime($EventTime,'YYYY-MM-DDThh:mm:ss.sTZ');
        $raw_event = '{"events":[{"metadata":{"event_timestamp":"' \
            + $timestamp +'","event_type":"PROCESS_LAUNCH",' \
            + '"vendor_name":"Microsoft","product_name":"Windows"},' \
            + '"principal":{"hostname":"'+ $Hostname +'"},' \
            + '"target":{"process":{"pid":"'+ $NewProcessId +'",' \
            + '"file":{"full_path":"'+ $NewProcessName +'"}}}}]}';
        $raw_event = replace($raw_event,'\','\\');
    </Exec>
</Output>
Output sample in JSON
{
  "events": [
    {
      "metadata": {
        "event_timestamp": "2021-01-15T21:50:13.788209-08:00",
        "event_type": "PROCESS_LAUNCH",
        "vendor_name": "Microsoft",
        "product_name": "Windows"
      },
      "principal": {
        "hostname": "WIN-ET85AK2E1J1"
      },
      "target": {
        "process": {
          "pid": "0xcc8",
          "file": {
            "full_path": "C:\\Windows\\System32\\ftp.exe"
          }
        }
      }
    }
  ]
}

Forwarding logs using a central NXLog agent

Any NXLog agent can be configured to function as a central NXLog agent, which then relays events to one or more destinations for further processing and analysis. This is known as centralized log collection. In the case of Chronicle, a central NXLog agent receives logs from other NXLog agents installed locally on their respective log source hosts. The central NXLog agent can then interface directly with Chronicle to forward logs on behalf of the other NXLog agents.

The example below demonstrates the universal configuration of the central log collector for handling all types of unstructured logs.

Example 4. Using a central NXLog agent for forwarding logs

This configuration demonstrates how to read events produced by Osquery on a Linux host using the im_file input module. After each event is read and parsed by the from_file instance, the event is routed to the om_tcp module for further processing.

To send osquery events to Chronicle, further processing is required which can be done in the to_tcp instance of the om_tcp output module. After consulting the list of unstructured log sources, it is apparent that the unstructured Chronicle log type OSQUERY_EDR exists. The Exec block is used for constructing the JSON payload structure Chronicle requires. First, the value of $raw_event is assigned to the log_text key. This object is then defined as the sole element of the top-level JSON array entries. The top-level object is defined by two key-value pairs: entries and log_type which is assigned a value of OSQUERY_EDR. This JSON object is then sent over TCP to the central NXLog agent for relaying to Chronicle.

nxlog.conf (remote agent)
<Input from_file>
    Module              im_file
    File                '/var/log/osquery/osqueryd.snapshots.log'
    Exec                parse_json();
</Input>

<Output to_tcp>
    Module              om_tcp
    Host                192.168.31.157:10500
    <Exec>
        $raw_event = escape_json($raw_event);
        $raw_event = '{"log_type": "OSQUERY_EDR",' \
            + '"entries": [{"log_text":"'+ $raw_event+'"}]}';
    </Exec>
</Output>

The following configuration shows how the central NXLog agent is configured. The from_tcp instance of the im_tcp module is configured to listen for incoming events on TCP port 10500 using the network interface having an IP address of 192.168.31.157.

The to_chronicle instance of the om_http output module forwards the events to Chronicle. The BASE_URL, ENDPOINT, and API_KEY constants are used to construct the value needed for the URL directive. Since the events received over TCP were already processed to meet the JSON payload structure Chronicle requires, the events are forwarded directly to Chronicle without further processing.

nxlog.conf (central agent)
define BASE_URL         https://malachiteingestion-pa.googleapis.com/v1/
define ENDPOINT         unstructuredlogentries
define API_KEY          key=<YOURAPIKEY>

<Input from_tcp>
    Module              im_tcp
    ListenAddr          192.168.31.157:10500
</Input>

<Output to_chronicle>
    Module              om_http
    URL                 %BASE_URL%%ENDPOINT%?%API_KEY%
    HTTPSAllowUntrusted TRUE
    ContentType         application/json
</Output>

Forwarding log data via Chronicle Forwarder

Chronicle Forwarder, as introduced above, can only forward unstructured data. However, it is able to

  • run a Syslog server,

  • accept events from Splunk, and

  • provide passive network monitoring.

Chronicle Forwarder forwards events to Chronicle as soon as they are received.

Example 5. Forwarding messages to Chronicle Forwarder

The NXLog configuration below uses the im_etw module to collect log data from the Microsoft-Windows-DNS-Client provider. The to_tcp instance of the om_tcp output module is configured to send the DNS events to a Chronicle Forwarder agent listening on TCP port 10514 installed on a host with an IP address of 192.168.31.178.

nxlog.conf
<Input from_dns>
    Module              im_etw
    Provider            Microsoft-Windows-DNS-Client
</Input>

<Output to_tcp>
    Module              om_tcp
    Host                192.168.31.178:10514
</Output>
Output sample
{
  "SourceName": "Microsoft-Windows-DNS-Client",
  "ProviderGuid": "{1C95126E-7EEA-49A9-A3FE-A378B03DDB4D}",
  "EventID": 3008,
  "Version": 0,
  "ChannelID": 16,
  "OpcodeValue": 0,
  "TaskValue": 0,
  "Keywords": "9223372036854775808",
  "EventTime": "2021-01-20T10:34:40.130607-08:00",
  "ExecutionProcessID": 3324,
  "ExecutionThreadID": 2160,
  "EventType": "INFO",
  "SeverityValue": 2,
  "Severity": "INFO",
  "Hostname": "WIN-ET85AK2E1J1",
  "Domain": "WIN-ET85AK2E1J1",
  "AccountName": "Administrator",
  "UserID": "S-1-5-21-3213787892-1493673803-1430668809-500",
  "AccountType": "User",
  "Flags": "EXTENDED_INFO|IS_64_BIT_HEADER|PROCESSOR_INDEX (577)",
  "QueryName": "pipeline-incoming-prod-elb-149169523.us-west-2.elb.amazonaws.com",
  "QueryType": "28",
  "QueryOptions": "140737489406145",
  "QueryStatus": "9501",
  "EventReceivedTime": "2021-01-20T10:34:40.711997-08:00",
  "SourceModuleName": "from_dns",
  "SourceModuleType": "im_etw"
}

Below is the sample configuration of Chronicle Forwarder.

Chronicle Forwarder configuration sample
  - syslog:
      common:
        enabled: true
        data_type: WINDOWS_DNS
        data_hint:
        batch_n_seconds: 10
        batch_n_bytes: 1048576
      tcp_address: 192.168.31.178:10514
      udp_address: 192.168.31.178:10515
      connection_timeout_sec: 60

For more examples on forwarding logs to Chronicle Forwarder, see the Google Chronicle documentation on How to collect Microsoft Windows DNS data and Collect Zeek (Bro) logs.

Verifying data in Google Chronicle

Upon receipt, log data can be observed in Google Chronicle’s web interface:

  1. In a web browser, navigate to the Google Chronicle instance using the https://<customername>.backstory.chronicle.security address.

  2. To search for all entries, type a period (.) in the search field and click SEARCH.

  3. In the Raw Log Scan dialog, specify the search interval and click SEARCH.

    The search dialog
  4. On the Raw Log Scan page, click any event of interest in the Asset pane to see its details.

    The Asset pane
  5. The right-hand pane can be used to toggle (show/hide) both Raw Log and/or UDM Event details for the selected event.

    Details about the entry
Disclaimer

While we endeavor to keep the information in this topic up to date and correct, NXLog makes no representations or warranties of any kind, express or implied about the completeness, accuracy, reliability, suitability, or availability of the content represented here.

The accurateness of the content was tested and proved to be working in our lab environment at the time of the last revision with the following software versions:

NXLog version 5.4.7431

Last revision: 25 January 2022