Grok (xm_grok)

This module supports parsing events with Grok patterns. A field is added to the event record for each pattern semantic. For more information about Grok, see the Logstash Grok filter plugin documentation.

To examine the supported platforms, see the list of installer packages in the Available Modules chapter.

Configuration

The xm_grok module accepts the following directives in addition to the common module directives.

Required directives

The following directives are required for the module to start.

Pattern

This mandatory directive specifies a directory or file containing Grok patterns. Wildcards may be used to specify multiple directories or files. This directive may be used more than once.

Functions

The following functions are exported by xm_grok.

boolean match_grok(string pattern): Execute the match_grok() procedure with the specified pattern on the $raw_event field. If the event is successfully matched, return TRUE, otherwise FALSE.

boolean match_grok(string field, string pattern): Execute the match_grok() procedure with the specified pattern on the specified field. If the event is successfully matched, return TRUE, otherwise FALSE.

Procedures

The following procedures are exported by xm_grok.

match_grok(string pattern);: Attempt to match and parse the $raw_event field of the current event with the specified pattern.

match_grok(string field, string pattern);: Attempt to match and parse the field of the current event with the specified pattern.

Example

Example 1. Using Grok patterns for parsing

This configuration reads Syslog events from a file and parses them with the parse_syslog() procedure (this sets the $Message field). Then the match_grok() function is used to attempt a series of matches on the $Message field until one is successful. If no patterns match, an internal message is logged.

nxlog.conf

<Extension _syslog>
    Module  xm_syslog
</Extension>

<Extension grok>
    Module  xm_grok
    Pattern modules/extension/grok/patterns2.txt
</Extension>

<Input in>
    Module  im_file
    File    'test2.log'
    <Exec>
        parse_syslog();
        if match_grok($Message, "%{SSH_AUTHFAIL_WRONGUSER}") {}
        else if match_grok($Message, "%{SSH_AUTHFAIL_WRONGCREDS}") {}
        else if match_grok($Message, "%{SSH_AUTH_SUCCESS}") {}
        else if match_grok($Message, "%{SSH_DISCONNECT}") {}
        else
        {
            log_info('Event did not match any pattern');
        }
    </Exec>
</Input>

patterns2.txt

USERNAME [a-zA-Z0-9_-]+
INT (?:[+-]?(?:[0-9]+))
BASE10NUM (?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))
NUMBER (?:%{BASE10NUM})
WORD \b\w+\b
GREEDYDATA .*
IP (?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])

SSH_AUTHFAIL_WRONGUSER Failed %{WORD:ssh_authmethod} for invalid user %{USERNAME:ssh_user} from %{IP:ssh_client_ip} port %{NUMBER:ssh_client_port} (?<ssh_protocol>\w+\d+)
SSH_AUTHFAIL_WRONGCREDS Failed %{WORD:ssh_authmethod} for %{USERNAME:ssh_user} from %{IP:ssh_client_ip} port %{NUMBER:ssh_client_port} (?<ssh_protocol>\w+\d+)
SSH_AUTH_SUCCESS Accepted %{WORD:ssh_authmethod} for %{USERNAME:ssh_user} from %{IP:ssh_client_ip} port %{NUMBER:ssh_client_port} (?<ssh_protocol>\w+\d+)(?:: %{WORD:ssh_pubkey_type} %{GREEDYDATA:ssh_pubkey_fingerprint})?
SSH_DISCONNECT Received disconnect from %{IP:ssh_client_ip} port %{INT:ssh_client_port}.*?:\s+%{GREEDYDATA:ssh_disconnect_reason}