Zeek (formerly Bro) Network Security Monitor

Zeek, formerly known as the Bro Network Security Monitor, is a powerful open-source Intrusion Detection System (IDS) and network traffic analysis framework.

Zeek was designed for watching live network traffic, and even if it can process packet captures saved in PCAP format, most organizations deploy it to achieve near real-time insights into network usage patterns. Systems administrators utilize it to sniff packets on one or more network interfaces, capture traffic, and convert it to a series of high-level events. These events are then analyzed according to customizable policies.

Zeek supports real-time alerts, data logging for further investigation, and automatic program execution for detected anomalies. In addition, it can analyze various protocols, including HTTP, FTP, SMTP, and DNS, run host and port scans, detect signatures, and discover syn-floods.

About Zeek logs

By default, Zeek creates the following log files to record the most common types of network traffic as well as some internal messages.

Table 1. Default Zeek log files
conn.log	dns.log	http.log
files.log	ftp.log	ssl.log
x509.log	smtp.log	ssh.log
pe.log	dhcp.log	ntp.log
SMB Logs ¹	irc.log	rdp.log
traceroute.log	tunnel.log	dpd.log
known_*.log and software.log	weird.log and notice.log	capture_loss.log and reporter.log

¹ Plus DCE-RPC, Kerberos, and NTLM.

Zeek’s Script Reference covers seven broad categories of log files, which provides detailed descriptions of more than 60 different types of log files that Zeek can create.

Table 2. Zeek log file categories
Count	Log file category
5	Detection
4	Files
4	Miscellaneous
5	NetControl
5	Network Observations
28	Network Protocols
12	Zeek Diagnostics

The Zeek log format is similar to the W3C log format, which is human-readable. It is important to note that each log file contains different fields.

dns.log sample

#separator \x09
#set_separator  ,
#empty_field    (empty)
#unset_field    -
#path   conn
#open   2022-08-26-23-00-05
#fields ts      uid     id.orig_h       id.orig_p       id.resp_h       id.resp_p       proto   service duration        orig_bytes      resp_bytes      conn_state      local_orig      local_resp      missed_bytes    history orig_pkts       orig_ip_bytes   resp_pkts       resp_ip_bytes   tunnel_parents
#types  time    string  addr    port    addr    port    enum    string  interval        count   count   string  bool    bool    count   string  count   count   count   count   set[string]
1661543999.667181       CzgR8U3JBkjvE4m6mi      172.31.68.6     59270   35.224.170.84   80      tcp     -       0.285542        0       148     SHR     T       F       0       ^hCadf  0       0       5       416     -
1661543946.302917       CfMnzeFTQa32YYqbi       172.31.68.6     38706   185.125.190.57  123     udp     ntp     0.053387        0       48      SHR     T       F       0       Cd      0       0       1       76      -
1661543832.679298       C3irK84pkVkwL9vW99      172.31.68.6     41112   34.117.237.239  443     tcp     -       171.086161      0       1138    SHR     T       F       0       ^hCadCfA        1       52      16      1978    -
1661543958.797270       CXekPS349jF4RPQwl7      172.31.68.6     47753   172.31.64.1     53      udp     dns     37.595825       0       4784    SHR     T       T       0       CdC     0       0       24      5456    -

For more information about Zeek logging, refer to the Zeek Documentation.

Parsing Zeek logs

NXLog can parse Zeek logs with the xm_w3c module.

Example 1. Parsing Zeek logs with NXLog

NXLog Enterprise Edition exclusive feature

This configuration uses the im_file input module to read Zeek logs. It parses log records with the xm_w3c module and converts them to JSON format.

nxlog.conf

<Extension json>
    Module       xm_json
</Extension>

<Extension w3c_parser>
    Module       xm_w3c
</Extension>

<Input zeek>
    Module       im_file
    File         "/opt/zeek/logs/current/*.log" (1)
    InputType    w3c_parser (2)
    Exec         to_json(); (3)
</Input>

1	Reads all log files in the `/opt/zeek/logs/current` directory.
2	Sets the InputType directive to the name of the xm_w3c module instance.
3	Converts log records to JSON format using the to_json() procedure of the xm_json module.

Output sample

{
  "ts": "1661679527.064733",
  "uid": "Cvt5L64Ewv1OJcyl1d",
  "id.orig_h": "192.168.1.5",
  "id.orig_p": "50458",
  "id.resp_h": "54.161.241.46",
  "id.resp_p": "443",
  "fuid": "TLSv13",
  "file_mime_type": "TLS_AES_128_GCM_SHA256",
  "file_desc": "x25519",
  "proto": null,
  "note": "F",
  "msg": null,
  "sub": null,
  "src": "F",
  "dst": "si",
  "p": null,
  "n": null,
  "peer_descr": null,
  "actions": null,
  "EventReceivedTime": "2022-08-28T12:38:53.363478+03:00",
  "SourceModuleName": "zeek",
  "SourceModuleType": "im_file"
}

The xm_w3c module is recommended because it supports parsing the W3C-style log file header to determine the fields automatically. However, since this module is not available in NXLog Community Edition, xm_csv can be used to parse Zeek logs instead. You must configure separate xm_csv instances for each log type you want to collect.

Example 2. Parsing Zeek logs with NXLog

This configuration uses the im_file input module to read logs from file. It defines two instances of the xm_csv module to parse Zeek DNS and DHCP logs. Additional instances can be added to parse other Zeek log types.

nxlog.conf

<Extension dns_parser>
    Module       xm_csv
    Fields       ts, uid id.orig_h, id.orig_p, id.resp_h, id.resp_p, proto, \
                 trans_id, rtt query, qclass, qclass_name, qtype, qtype_name, \
                 rcode, rcode_name, AA, TC, RD, RA, Z, answers, TTLs, rejected
    Delimiter    \t
</Extension>

<Extension dhcp_parser>
    Module       xm_csv
    Fields       ts, uid, id.orig_h, id.orig_p, id.resp_h, id.resp_p, mac, \
                 assigned_ip, lease_time, trans_id
    Delimiter    \t
</Extension>

<Extension fileop>
    Module       xm_fileop (1)
</Extension>

<Extension json>
    Module       xm_json
</Extension>

<Input zeek>
    Module       im_file
    File         '/opt/zeek/logs/current/dhcp.log'
    File         '/opt/zeek/logs/current/dns.log'
    <Exec>
        if file_basename(file_name()) == 'dhcp.log'
        {
            dhcp_parser->parse_csv(); (2)
        }
        else if file_basename(file_name()) == 'dns.log'
        {
            dns_parser->parse_csv(); (3)
        }
        to_json(); (4)
    </Exec>
</Input>

1	The xm_fileop module provides the file_basename() function.
2	Parses the log record with the dhcp_parser instance.
3	Parses the log record with the dns_parser instance.
4	Converts log records to JSON format using the to_json() procedure of the xm_json module.

Output sample

{
  "EventReceivedTime": "2022-08-28T13:16:12.198761+03:00",
  "SourceModuleName": "zeek",
  "SourceModuleType": "im_file",
  "ts": "1661681771.517333",
  "uid": "CoAKAy5hqF65WakL9",
  "id.orig_h": "192.168.1.4",
  "id.orig_p": "5353",
  "id.resp_h": "224.0.0.251",
  "id.resp_p": "5353",
  "proto": "udp",
  "trans_id": "0",
  "rtt": "0.002248",
  "query": "desktop.local",
  "qclass": "1",
  "qclass_name": "C_INTERNET",
  "qtype": "255",
  "qtype_name": "*",
  "rcode": "0",
  "rcode_name": "NOERROR",
  "AA": "T",
  "TC": "F",
  "RD": "F",
  "RA": "F",
  "Z": "0",
  "answers": "2a02:587:4958:c2e1:406b:7ec8:80e4:5adc,2a02:587:4958:c2e1:9088:c550:3f68:f979,fe80::406b:7ec8:80e4:5adc,192.168.1.4",
  "TTLs": "60.000000,60.000000,60.000000,60.000000",
  "rejected": "F"
}

Zeek can also be configured to output logs in JSON format as follows:

# echo "@load policy/tuning/json-logs.zeek" | tee -a /opt/zeek/share/zeek/site/local.zeek

The following command instructs Zeek to remove the old configuration and reload all scripts included in /opt/zeek/share/zeek/site/local.zeek.

# /opt/zeek/bin/zeekctl deploy

Example 3. Parsing Zeek JSON logs with NXLog

This configuration uses the im_file input module to read Zeek logs. It expects JSON-formatted log records.

nxlog.conf

<Extension json>
    Module    xm_json
</Extension>

<Input zeek>
    Module    im_file
    File      "/opt/zeek/logs/current/*.log" (1)
    <Exec>
        parse_json(); (2)
        to_json(); (3)
    </Exec>
</Input>

1	Reads all log files in the `/opt/zeek/logs/current` directory.
2	Parses log records with the parse_json() procedure of the xm_json module.
3	Converts records back to JSON format with the to_json() procedure, which adds the NXLog core fields to the output.

Output sample

{
    "EventReceivedTime": "2022-08-29T23:31:18.517661+03:00",
    "SourceModuleName": "zeek",
    "SourceModuleType": "im_file",
    "ts": "1661805077.887304",
    "uid": "CVIHjvXZkFnM6wWae",
    "id.orig_h": "192.168.1.4",
    "id.orig_p": 5353,
    "id.resp_h": "224.0.0.251",
    "id.resp_p": 5353,
    "proto": "udp",
    "trans_id": 0,
    "rtt": "0.0019359588623046875",
    "query": "desktop.local",
    "qclass": 1,
    "qclass_name": "C_INTERNET",
    "qtype": 255,
    "qtype_name": "*",
    "rcode": 0,
    "rcode_name": "NOERROR",
    "AA": true,
    "TC": false,
    "RD": false,
    "RA": false,
    "Z": 0,
    "answers": ["2a02:587:4978:47c7:406b:7ec8:80e4:5adc","2a02:587:4978:47c7:14e1:5283:e01e:80e8","fe80::406b:7ec8:80e4:5adc","192.168.1.4"],
    "TTLs": [60.0,60.0,60.0,60.0],
    "rejected": false
}

Disclaimer

While we endeavor to keep the information in this topic up to date and correct, NXLog makes no representations or warranties of any kind, express or implied about the completeness, accuracy, reliability, suitability, or availability of the content represented here. We update our screenshots and instructions on a best-effort basis.

The accurateness of the content was tested and proved to be working in our lab environment at the time of the last revision with the following software versions:

NXLog version 5.6.7727
Ubuntu version 20.04.4 LTS
Zeek version 5.0.2
ZeekControl version 2.4.1

Last revision: 24 October 2022