NXLog Legacy Documentation

Apple macOS

NXLog can collect various types of system logs on the macOS platform. For deployment details, see the supported macOS platforms and macOS installation sections. How logs are captured on macOS is largely dependent on the macOS release, with one exception being the macOS Basic Security Module. Since it reads directly from the kernel, you can use it with any supported macOS platform.

Unified logging since macOS Sierra

Apple’s release of macOS Sierra 10.12 in September 2016 introduced a fundamentally new, unified logging system (ULS), which effectively replaced the traditional UNIX logging system that had been maintained since the very first release of OS X. One of Apple’s goals was to combine event messages from legacy APIs like NSLog, asl_log_message, and syslog into a unified logging system. To learn more about this major change in logging architecture see macOS Unified log: 1 why, what and how.

Regaining visibility of redacted, private data

One of the original goals of ULS was to introduce added security features, such as privacy for sensitive data that might be used for malicious purposes. This has resulted in dynamic strings, collections, and objects being redacted with the string <private>, while the static, generic message text — typically devoid of any useful details — is passed through unscathed, much to the dismay of developers and security analysts alike, as can be seen here in these Console messages:

Redacted, private data in the macOS onsole

Prior to the release of macOS 10.15.3 it was relatively simple to set the additional diagnostic flags required to expose private data from applications lacking the com.apple.private.set-atm-diagnostic-flag entitlement.

Fortunately an Apple developer has documented the challenges and a solution to acquire Catalina logs in Show private log messages in Catalina’s Console.app. The mobile configuration profile provided in this solution, enable-unified-log-private-data.mobileconfig (download the 4.6 kB signed XML profile), can be easily installed which will restore visibility of these private message details for macOS Catalina 10.15.3 and later as seen here:

Capturing ULS logs using im_maculs

NXLog Enterprise Edition exclusive feature

The im_maculs module reads ULS logs natively and can be used to collect any logs generated by macOS. This centralized logging facility captures all events and stores them internally as structured data. Each ULS logged event is stored using one of two schemas: either the standard schema or the signpost schema.

The standard schema is used for logging information, warnings, errors, and debugging information. This is the standard mode of logging for most macOS logs. For performance-related monitoring and debugging, the signpost type of events are stored using the signpost schema. Signpost events mark a point of interest in application code to assist developers in creating highly responsive applications. Apple encourages developers to "add signposts to record interesting time-based events." (Apple Developer: Recording Performance Data)

True to its name, the macOS unified logging system handles hundreds of log sources. In short, ULS is to macOS what Windows Event Log is to Windows platforms. Given the large number of logs stored in this global repository of system and application events, it is crucial to be selective when configuring each im_maculs module instance. Ideally, each instance should have a filter that captures only one or two log sources.

Example 1. Capturing macOS ULS logs

The following configuration reads all ULS logs being generated from hundreds of Mac log sources and filters out only two types of events:

Events containing an $eventType field with a value of signpostEvent
Events with the $subsystem set to com.apple.runningboard

Because the ULS schemas do not include any field for storing the log source’s host name, any event that matches the filter will have its schema enriched with a new $Hostname field that is assigned the value returned by the core function hostname(). Any event that fails to match the conditional statement is discarded by calling the drop() procedure.

Finally, the to_json() procedure is called, not only to convert the record to JSON format, but also to add this new field, along with the NXLog core fields common to all structured logs that have been processed by NXLog, to each event record.

nxlog.conf

<Extension _json>
    Module xm_json
</Extension>

<Input m1_uls>
    Module              im_maculs
    UUIDTextPath        "/var/db/uuidtext"
    TimeSyncPath        "/var/db/diagnostics/timesync"
    TraceV3Path         "/var/db/diagnostics"
    <Exec>
#       Filter only the events needed
        if $eventType == 'signpostEvent' or
           $subsystem == 'com.apple.runningboard'
        {
            $Hostname = hostname();
        }
        else
        {
            drop();
        }
        to_json();
    </Exec>
</Input>

<Output ndjson>
    Module              om_file
    File                "signpost-and-runningboard.ndjson"
</Output>

The following JSON sample exhibits the standard schema which is used for an event that matched the conditional statement $subsystem == 'com.apple.runningboard'.

Standard schema sample

{
  "eventMessage": "Acquired process power assertion with ID 41468 for pid 650",
  "formatString": "Acquired process power assertion with ID %{public}d for pid %{public}d",
  "activityIdentifier": 6872151,
  "subsystem": "com.apple.runningboard",
  "category": "power",
  "threadID": 11163152,
  "senderImageUUID": "cb3dc3fb-6454-3510-a1c2-0cc65d53b22d",
  "bootUUID": "80243e89-2bee-44e5-b841-736299e7c5e3",
  "processImagePath": "/usr/libexec/runningboardd",
  "senderImagePath": "/System/Library/PrivateFrameworks/RunningBoard.framework/Versions/A/RunningBoard",
  "EventTime": "2021-04-04T23:46:35.207740-05:00",
  "machTimestamp": 43611630764906,
  "messageType": "Default",
  "processImageUUID": "901f428b-2676-3774-b480-9b1604a32363",
  "processID": 371,
  "senderProgramCounter": 39036,
  "parentActivityIdentifier": 0,
  "TTL": 0,
  "EventReceivedTime": "2021-04-04T23:46:44.565064-05:00",
  "SourceModuleName": "m1_uls",
  "SourceModuleType": "im_maculs"
}

The following JSON sample exhibits the signpost schema which is used for an event that matched the conditional statement $eventType == 'signpostEvent'.

Signpost schema sample

{
  "eventType": "signpostEvent",
  "signpostID": -1229851353791926500,
  "signpostScope": "process",
  "formatString": "%{public, signpost.description:begin_time}llu",
  "activityIdentifier": 0,
  "subsystem": "com.apple.SkyLight",
  "category": "performance_instrumentation",
  "threadID": 3328,
  "senderImageUUID": "e50df124-c72e-3349-9b99-788bf3160420",
  "signpostType": "event",
  "bootUUID": "80243e89-2bee-44e5-b841-736299e7c5e3",
  "processImagePath": "/System/Library/PrivateFrameworks/SkyLight.framework/Versions/A/Resources/WindowServer",
  "senderImagePath": "/System/Library/PrivateFrameworks/SkyLight.framework/Versions/A/SkyLight",
  "signpostName": "CompositeLoop",
  "EventTime": "2021-04-04T22:57:19.476038-05:00",
  "machTimestamp": 43540691469555,
  "eventMessage": "43540691207161",
  "processImageUUID": "ddc145f0-07cc-3b36-9e61-742786349f3f",
  "processID": 353,
  "senderProgramCounter": 2753288,
  "parentActivityIdentifier": 0,
  "TTL": 0,
  "EventReceivedTime": "2021-04-04T22:59:17.683574-05:00",
  "SourceModuleName": "m1_uls",
  "SourceModuleType": "im_maculs"
}

Capturing ULS events using the built-in log stream command

It might be helpful when performing forensics to view the events emitted by the macOS built-in log command. It takes a single argument command, in this case stream, which can be followed by a number of options specific to the command chosen. This can be achieved by using the im_exec module.

Example 2. Using im_exec to run log stream

This example shows how to configure im_exec to invoke the built-in log command for generating a stream of ULS events. For a complete list of command line options, you can invoke man log within a terminal.

In this configuration, the --level option has been commented out, but could be used to explicitly choose one of three log level options: default, info, or debug. This live ULS event stream is already in JSON format, but after enriching it with a new $Hostname field that is assigned the value returned by the core function hostname(), the to_json() procedure is called which also adds the NXLog core fields to each event record.

nxlog.conf

<Extension _json>
    Module xm_json
</Extension>

<Input macos_log>
    Module   im_exec
    Command  /usr/bin/log
    Arg      stream
    Arg      --style=ndjson
    Arg      --type=log
#    Arg      --level=debug
    <Exec>
        parse_json();
        $Hostname = hostname();
        to_json();
    </Exec>
</Input>

This configuration yields line-delimited JSON (NDJSON) records which are suitable for routing directly to various NXLog output modules or most SIEMs. This single-line ULS event has been reformatted for display purposes.

Sample log stream event

{
  "EventReceivedTime": "2021-04-08T22:28:22.002528-05:00",
  "SourceModuleName": "macos_log",
  "SourceModuleType": "im_exec",
  "traceID": 1824493551157252,
  "eventMessage": "ocsp responder: (null) did not include status of requested cert",
  "eventType": "logEvent",
  "source": null,
  "formatString": "ocsp responder: %@ did not include status of requested cert",
  "activityIdentifier": 8049628,
  "subsystem": "com.apple.securityd",
  "category": "ocsp",
  "threadID": 13337131,
  "senderImageUUID": "AD8A1343-96A6-3E87-8CFA-14FE13754B06",
  "backtrace": {
    "frames": [
      {
        "imageOffset": 256208,
        "imageUUID": "AD8A1343-96A6-3E87-8CFA-14FE13754B06"
      }
    ]
  },
  "bootUUID": "",
  "processImagePath": "/usr/libexec/trustd",
  "timestamp": "2021-04-08T22:28:22.002306-05:00",
  "senderImagePath": "/usr/libexec/trustd",
  "machTimestamp": 51793593259945,
  "messageType": "Default",
  "processImageUUID": "AD8A1343-96A6-3E87-8CFA-14FE13754B06",
  "processID": 565,
  "senderProgramCounter": 256208,
  "parentActivityIdentifier": 0,
  "timezoneName": "",
  "Hostname": "mm"
}

Collecting logs from Apple’s Endpoint Security using im_maces

NXLog Enterprise Edition exclusive feature

The im_maces module collects logs from Apple’s Endpoint Security auditing system on MacOS 10.15 and later. Endpoint Security is an audit subsystem for monitoring system events for potentially malicious activity.

Example 3. Collecting Apple endpoint security events

With this configuration, NXLog will collect the events listed in the NotifyEvents directive.

nxlog.conf

<Input in>
    Module          im_maces
    NotifyEvents    get_task, proc_check, write
</Input>

Logging prior to macOS Sierra

Since the very first release of OS X, which traces its roots back to the Mach kernel and BSD, logging has been done in the traditional UNIX style: multiple files, each with containing a specific log source, typically stored as flat files. With the release of Sierra 10.12, a new unified logging system (ULS) was introduced. The following logs are produced by pre-Sierra releases of macOS.

macOS System Log files

The im_file and xm_asl modules can be used to collect and parse Apple Log (*.asl) files.

Example 4. Reading and parsing Mac System Logs

NXLog Enterprise Edition exclusive feature

This example reads macOS logs from input.asl and parses them with the xm_asl parser.

nxlog.conf

<Extension asl_parser>
    Module  xm_asl
</Extension>

<Input in>
    Module      im_file
    # Example: "/var/log/asl/*"
    File        "foo/input.asl"
    InputType   asl_parser
    Exec        delete($EventReceivedTime);
</Input>

Basic Security Mode (BSM) auditing

The im_bsm module collects logs directly from the BSM auditing system.

The following examples also apply to macOS releases after macOS Sierra.

Example 5. Collecting BSM audit logs from the kernel

NXLog Enterprise Edition exclusive feature

This configuration reads BSM audit logs directly from the kernel with the im_bsm module.

nxlog.conf

Group   wheel

<Input bsm>
    Module        im_bsm
    DeviceFile    /dev/auditpipe
</Input>

Alternatively, BSM logs can be read from the log files.

Example 6. Reading BSM audit logs from file

NXLog Enterprise Edition exclusive feature

This configuration reads from the BSM audit log files with im_file and parses the events with xm_bsm.

nxlog.conf

Group   wheel

<Extension bsm_parser>
    Module      xm_bsm
</Extension>

<Input bsm>
    Module      im_file
    File        '/var/audit/*'
    InputType   bsm_parser
</Input>

Custom programs

The im_exec module allows Mac log data to be collected from custom external programs.

Example 7. Using an external command

This example uses the tail command to read from a file.

The im_file module should be used to read log messages from files. This example only demonstrates the use of the im_exec module.

nxlog.conf

<Input systemlog>
    Module  im_exec
    Command /usr/bin/tail
    Arg     -f
    Arg     /var/log/system.log
</Input>

File integrity monitoring

File and directory changes can be detected and logged for auditing with the im_fim module. See File Integrity Monitoring.

Example 8. Monitoring file integrity

NXLog Enterprise Edition exclusive feature

This configuration watches for changes to files and directories under /bin and /usr/bin/.

nxlog.conf

<Input fim>
    Module          im_fim
    File            "/bin/*"
    File            "/usr/bin/*"
    ScanInterval    3600
    Recursive       TRUE
</Input>

Kernel

Logs from the kernel can be collected directly with the im_kernel module or via the local log file with im_file.

Local syslog

Events written to file in syslog format can be collected with im_file. The xm_syslog module can be used to parse the events. See the Collecting, parsing, and forwarding syslog logs section for more information.

Example 9. Reading syslog messages from file

This configuration file collects system logs from /var/log/system.log. This method does not read from /dev/klog directly, so it is not necessary to disable syslogd.

nxlog.conf

<Extension _syslog>
    Module  xm_syslog
</Extension>

<Input in>
    Module  im_file
    File    "/var/log/system.log"
    Exec    parse_syslog();
</Input>

Log files

The im_file module can be used to collect events from log files.

Example 10. Reading from log files

This configuration uses the im_file module to read events from the specified log file.

nxlog.conf

<Input in>
    Module  im_file
    File    "/foo/in.log"
</Input>

Process accounting

The im_acct module can be used to gather details about which owner (user and group) runs what processes.

Example 11. Reading process accounting logs

NXLog Enterprise Edition exclusive feature

With this configuration file, NXLog will enable process accounting to the specified file and reads events from it.

nxlog.conf

Group   wheel

<Input acct>
    Module  im_acct
    File    '/var/log/acct'
    AcctOn  TRUE
</Input>

Disclaimer

While we endeavor to keep the information in this topic up to date and correct, NXLog makes no representations or warranties of any kind, express or implied about the completeness, accuracy, reliability, suitability, or availability of the content represented here. We update our screenshots and instructions on a best-effort basis.

The accurateness of the content was tested and proved to be working in our lab environment at the time of the last revision with the following software versions:

NXLog 5.2.638 & NXLog 5.2.663
Apple macOS 10.15.3 (Catalina)
Apple macOS 11.2.3 (Big Sur)

Last revision: 11 April 2021