Zeek (formerly Bro) Network Security Monitor

Zeek formerly known as the Bro Network Security Monitor, is a powerful open source Intrusion Detection System (IDS) and network traffic analysis framework. The Zeek engine captures traffic and converts it to a series of high-level events. These events are then analyzed according to customizable policies. Zeek supports real-time alerts, data logging for further investigation, and automatic program execution for detected anomalies. Zeek is able to analyze different protocols, including HTTP, FTP, SMTP, and DNS; as well as run host and port scans, detect signatures, and discover syn-floods.

NXLog can be configured to collect Zeek logs.

About Zeek logs

Zeek creates different log files in order to record network activities such as files transferred over the network, SSL sessions, and HTTP requests. By default, Zeek provides 60 different log files.

Table 1. A few of Zeek’s default log files
File Description

conn.log

TCP/UDP/ICMP connections

dhcp.log

DHCP leases

dns.log

DNS activity

files.log

Summaries of files transferred over the network

ftp.log

FTP activity

http.log

HTTP requests and replies

smtp.log

SMTP transactions

ssl.log

SSL/TLS handshake information

weird.log

Unexpected network-level activity

Zeek produces human-readable logs in a format similar to the W3C log format. Each log file uses a different set of fields.

dns.log sample
#separator \x09
#set_separator  ,
#empty_field    (empty)
#unset_field    -
#path   dns
#open   2020-05-27-22-00-01
#fields ts      uid     id.orig_h       id.orig_p       id.resp_h       id.resp_p       proto   trans_id        rtt     query   qclass  qclass_name     qtype   qtype_name      rcode   rcode_name      AA      TC      RD      RA      Z       answers TTLs    rejected
#types  time    string  addr    port    addr    port    enum    count   interval        string  count   string  count   string  count   string  bool    bool    bool    bool    count   vector[string]  vector[interval]        bool
1590634800.248362       C1ggH7liCnwAfLjw9       192.168.1.7     53743   192.168.1.1     53      udp     18876   -       250.255.255.239.in-addr.arpa    1       C_INTERNET      12      PTR     3       NXDOMAIN        F       F       T       F       0       -       -       F
1590634800.259227       C1ggH7liCnwAfLjw9       192.168.1.7     53743   192.168.1.1     53      udp     18876   -       250.255.255.239.in-addr.arpa    1       C_INTERNET      12      PTR     3       NXDOMAIN        F       F       T       F       0       -       -       F
1590634800.274483       CTQxOg2sSOuUO5AZy8      192.168.1.7     47182   192.168.1.1     53      udp     48442   -       7.1.168.192.in-addr.arpa        1       C_INTERNET      12      PTR     3       NXDOMAIN        F       F       T       F       0       -       -       F

For more information about Zeek logging, see the Zeek Manual.

Parsing Zeek logs

NXLog Enterprise Edition can parse Zeek logs with the xm_w3c module.

The following configurations have been tested with Zeek version 3.0.6 LTS.
Example 1. Using xm_w3c to parse Zeek logs

This configuration reads Zeek logs from a directory, parses with xm_w3c, and writes out events in JSON format.

nxlog.conf
<Extension _json>
    Module     xm_json
</Extension>

<Extension w3c_parser>
    Module     xm_w3c
</Extension>

<Input zeek>
    Module     im_file
    File       '/opt/zeek/logs/current/*.log'
    InputType  w3c_parser
</Input>

<Output zeek_json>
    Module     om_file
    File       '/tmp/zeek_logs.json'
    Exec       to_json();
</Output>

The following output from this configuration represents a sample event logged by Zeek after being parsed by NXLog and converted to JSON format. Spacing and line breaks have been added for readability.

Output sample
{
  "ts": "1590636144.680688",
  "uid": "C1InwK3K6fhY6YdvRe",
  "id.orig_h": "192.168.1.7",
  "id.orig_p": "45500",
  "id.resp_h": "35.222.85.5",
  "id.resp_p": "80",
  "version": "1",
  "cipher": "GET",
  "curve": "connectivity-check.ubuntu.com",
  "server_name": "/",
  "resumed": null,
  "last_alert": "1.1",
  "next_protocol": null,
  "established": null,
  "cert_chain_fuids": "0",
  "client_cert_chain_fuids": "0",
  "subject": "204",
  "issuer": "No Content",
  "client_subject": null,
  "client_issuer": null,
  "validation_status": "(empty)",
  "EventReceivedTime": "2020-05-27T22:22:26.917647-05:00",
  "SourceModuleName": "zeek",
  "SourceModuleType": "im_file"
}

The xm_w3c module is recommended because it supports reading the field list from the W3C-style log file header. For NXLog Community Edition, the xm_csv module could be used instead to parse Zeek logs. A separate instance of xm_csv must be configured for each log type.

Example 2. Using xm_csv to parse Zeek logs

This example has separate xm_csv module instances for the DNS and DHCP log types. Additional CSV parsers could be added for the remaining Zeek log types.

nxlog.conf
<Extension csv_parser_dns>
    Module      xm_csv
    Fields      ts, uid id.orig_h, id.orig_p, id.resp_h, id.resp_p, proto, \
                trans_id, rtt query, qclass, qclass_name, qtype, qtype_name, \
                rcode, rcode_name, AA, TC, RD, RA, Z, answers, TTLs, rejected
    Delimiter   \t
</Extension>

<Extension csv_parser_dhcp>
    Module      xm_csv
    Fields      ts, uid, id.orig_h, id.orig_p, id.resp_h, id.resp_p, mac, \
                assigned_ip, lease_time, trans_id
    Delimiter   \t
</Extension>

# xm_fileop provides the `file_basename()` function
<Extension _fileop>
    Module      xm_fileop
</Extension>

<Extension json>
    Module  xm_json
</Extension>

<Input zeek>
    Module      im_file
    File        '/opt/zeek/spool/zeek/*.log'
    <Exec>
        if file_basename(file_name()) == 'dhcp.log'
        {
            csv_parser_dhcp->parse_csv();
        }
        else if file_basename(file_name()) == 'dns.log'
        {
            csv_parser_dns->parse_csv();
        }
        else
        {
            log_warning('Zeek log type not supported, check configuration');
            drop();
        }
    </Exec>
</Input>

<Output zeek_json>
    Module      om_file
    File        '/tmp/ce_zeek_logs.json'
    Exec        to_json();
</Output>

The following output from this configuration represents a sample event logged by Zeek after being parsed by NXLog and converted to JSON format. Spacing and line breaks have been added for readability.

Output sample
{
  "EventReceivedTime": "2020-05-29 10:55:51",
  "SourceModuleName": "zeek",
  "SourceModuleType": "im_file",
  "ts": "1590767749.877652",
  "uid": "CAhAIX1Dl5KFfnhKbi",
  "id.orig_h": "192.168.1.7",
  "id.orig_p": "42157",
  "id.resp_h": "192.168.1.1",
  "id.resp_p": "53",
  "proto": "udp",
  "trans_id": "56765",
  "rtt": "0.051801",
  "query": "zeek.org",
  "qclass": "1",
  "qclass_name": "C_INTERNET",
  "qtype": "1",
  "qtype_name": "A",
  "rcode": "0",
  "rcode_name": "NOERROR",
  "AA": "F",
  "TC": "F",
  "RD": "T",
  "RA": "T",
  "Z": "0",
  "answers": "192.0.78.212,192.0.78.150",
  "TTLs": "60.000000,60.000000",
  "rejected": "F"
}
Disclaimer

While we endeavor to keep the information in this topic up to date and correct, NXLog makes no representations or warranties of any kind, express or implied about the completeness, accuracy, reliability, suitability, or availability of the content represented here.

Last revision: 12 June 2020