Kubernetes

Kubernetes (also known as K8s or "kube" for short) is a container orchestration platform for automating the deployment, scaling, and maintenance of application containers in a cluster. NXLog Agent can integrate with Kubernetes to collect logs from containerized applications, as well as collect Kubernetes system and audit logs. This guide demonstrates how to collect logs from a Kubernetes cluster using an NXLog Agent Docker image.

Deploying NXLog Agent in Kubernetes

NXLog provides a Docker package that can be used to easily deploy NXLog Agent in a container. Log in to the NXLog website with your account to download the Docker archive, then follow these steps to build a Docker image. More detailed instructions are available in the NXLog Agent installation guide for Docker.

Extract the NXLog Agent Docker archive:
```
$ tar -xvf nxlog-6.4.9634-docker.tar.gz
```
Create an NXLog Agent configuration file containing your input, output, and routes, as well as any required extensions. By default, the Dockerfile copies all .conf files present in the build folder to /opt/nxlog/etc/nxlog.d in the container. The default NXLog Agent configuration file includes all .conf files found in this directory.
Build the NXLog Agent Docker image:
```
$ docker build -t nxlog .
```

Once built, the nxlog Docker image can be used in a Kubernetes manifest to deploy an NXLog Agent application container.

By default, NXLog Agent is configured to run using the nxlog account. In situations where you need NXLog Agent to run as root, replace USER nxlog with USER root in the Dockerfile before building the image.

Cluster-level logging

Logging from a Kubernetes cluster helps to monitor cluster activity and to troubleshoot problems when they arise. There are two types of Kubernetes logs that can be collected:

Application logs: Containerized applications commonly write their logs to the standard output and standard error streams. Application logs share the lifetime of the pod and are removed once a pod is deleted. Due to the transient nature of containers, it is important that application logs are centralized and have a separate life cycle from the containers, pods, and nodes. More information on application logs can be found in the Kubernetes documentation on Logging Architecture.
System logs: Kubernetes components create their own logging to record events that occur within the cluster. These logs help in troubleshooting cluster-level problems such as pods failing to start or networking issues. See the Kubernetes documentation on System Logs for more information.

The Kubernetes platform does not include a native, centralized storage solution for cluster logs; however, NXLog Agent can be deployed as a DaemonSet, collecting logs from each node without requiring any changes to the application containers. See the Kubernetes documentation for more information on DaemonSet.

For applications that do not write logs to the standard output and standard error streams, NXLog Agent can be deployed as a Kubernetes sidecar container, running in the same pod alongside the application. Logs collected from the application can either be streamed to the standard output to be written to the container log file or forwarded directly to a centralized log repository or SIEM.

NXLog Agent as a DaemonSet

Deploying NXLog Agent as a DaemonSet requires a ServiceAccount with permission to access all pods and namespaces. The DaemonSet needs to be deployed in the kube-system namespace and the NXLog Agent container must have the path of the logs on the node mounted as a hostPath. The steps below guide you to create the Kubernetes ServiceAccount and deploy the DaemonSet. NXLog Agent configuration examples are available in the Configuration examples section.

NXLog Agent needs to run as root to be able to read the logs generated on all pods. See Deploying NXLog Agent in Kubernetes for more information.

Set up a ServiceAccount for NXLog Agent

Create a ServiceAccount for the NXLog Agent DaemonSet.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nxlog

NXLog Agent needs to be granted permission to read, list, and watch pods and namespaces. Create a ClusterRole to define these permissions:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: nxlog
  namespace: kube-system
rules:
- apiGroups: [""]
  resources: ["pods", "namespaces"]
  verbs: ["get", "list", "watch"]

Bind the NXLog Agent ServiceAccount to the permissions:

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nxlog
roleRef:
  kind: ClusterRole
  name: nxlog
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: nxlog
  namespace: kube-system

Save the manifests above in a YAML file separated by --- and execute the following command to create the resources:
```
$ kubectl create -f my_svc_file.yml
```

The output should show that the resources were created successfully:

serviceaccount/nxlog created
clusterrole.rbac.authorization.k8s.io/nxlog created
clusterrolebinding.rbac.authorization.k8s.io/nxlog created

More information on role-based access control can be found in the Kubernetes documentation on Using RBAC Authorization.

Deploy the DaemonSet

Create a YAML file with the following manifest:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nxlog
  namespace: kube-system
  labels:
    k8s-app: nxlog-logging
spec:
  selector:
    matchLabels:
      name: nxlog
  template:
    metadata:
      labels:
        name: nxlog
    spec:
      tolerations:
      # this toleration is to have the daemonset runnable on master nodes
      # remove it if your masters can't run pods
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      volumes:
      - name: dockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: varlog
        hostPath:
          path: /var/log
      containers:
      - name: nxlog
        image: nxlog:latest
        imagePullPolicy: Never
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: dockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: varlog
          mountPath: /var/log
          readOnly: true
        env:
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
      terminationGracePeriodSeconds: 30

Execute the following command to deploy the DaemonSet:
```
$ kubectl apply -f my_deployment_file.yml
```
The output should show that the DaemonSet was created successfully:
```
daemonset.apps/nxlog created
```

Configuration examples

Example 1. Collecting container logs

When using Docker, logs from the standard output and standard error streams are written to file in the node’s /var/lib/docker/containers directory. To make it easier to access container logs, Kubernetes creates a directory structure with symbolic links to these logs in /var/log/pods and /var/log/containers.

The configuration below uses the im_file input module to read container logs from /var/log/containers. It parses logs using the xm_json extension and enriches the log records with metadata using the NODE_NAME environment variable and data extracted from the filename. It then converts the records back to JSON before forwarding them to Elasticsearch using the om_elasticsearch output module.

nxlog.conf

envvar NODE_NAME

<Extension json>
    Module    xm_json
</Extension>

<Input k8s_containers>
    Module    im_file
    File      '/var/log/containers/*.log'
    <Exec>
        parse_json();

        $log_type = "k8s_container";
        $log_file = file_name();
        $k8s_node = '%NODE_NAME%';

        # Log filenames are in the format
        # <pod-name>_<namespace>_<container-name-container-id>.log
        if $log_file =~ /\/.*\/(.+)_(.+)_(.+)-(.+).log$/
        {
            $k8s_pod = $1;
            $k8s_namespace = $2;
            $k8s_container = $3;
            $k8s_container_id = $4;
        }
    </Exec>
</Input>

<Output elasticsearch>
    Module    om_elasticsearch
    URL       http://192.168.1.123:9200/_bulk
    Exec      to_json();
</Output>

Input sample

The following is a container log record in JSON format. This input sample depicts the kind of data and format NXLog Agent will collect.

{
  "log": "2021-06-10 08:56:09 ERROR [xm_admin|agent_management] couldn't connect to 192.168.1.1:4041;The timeout specified has expired\n",
  "stream": "stderr",
  "time": "2021-06-10T08:56:09.216488+00:00"
}

Output sample

The following JSON shows the same event after it was processed by NXLog Agent.

{
  "EventReceivedTime": "2021-06-10T08:56:09.241401+00:00",
  "SourceModuleName": "k8s_containers",
  "SourceModuleType": "im_file",
  "log": "2021-06-10 08:56:09 ERROR [xm_admin|agent_management] couldn't connect to 192.168.1.1:4041;The timeout specified has expired\n",
  "stream": "stderr",
  "time": "2021-06-10T08:56:09.216488+00:00",
  "log_type": "k8s_container",
  "log_file": "/var/log/containers/nxlog-q5nh4_kube-system_nxlog-bdc9da43d94800e213477bbddbc9f96fa7a32b680509580cfce97fa4186d63e5.log",
  "k8s_node": "node001",
  "k8s_pod": "nxlog-q5nh4",
  "k8s_namespace": "kube-system",
  "k8s_container": "nxlog",
  "k8s_container_id": "bdc9da43d94800e213477bbddbc9f96fa7a32b680509580cfce97fa4186d63e5"
}

Example 2. Collecting cluster system logs

Kubernetes system logs are normally found in the /var/log directory on the node where the component is running. The following services create system logs:

kube-apiserver
kube-scheduler
kube-controller-manager
kube-proxy
kubelet
etcd

The configuration below uses the im_file input module to read Kubernetes system logs from /var/log. It enriches log records with metadata using the NODE_NAME environment variable and data extracted from the filename. It then converts the records to JSON before forwarding them to Elasticsearch using the om_elasticsearch output module.

nxlog.conf

envvar NODE_NAME

<Extension json>
    Module    xm_json
</Extension>

<Input k8s_cluster>
    Module    im_file
    File      '/var/log/kube*.log'
    File      '/var/log/etcd.log'
    <Exec>
        $log = $raw_event;
        $log_type = "k8s_cluster";
        $filename = file_name();
        $k8s_node = '%NODE_NAME%';

        if $filename =~ /.*\/(.+).log/
        {
            $log_source = $1;
        }
    </Exec>
</Input>

<Output elasticsearch>
    Module    om_elasticsearch
    URL       http://192.168.1.123:9200/_bulk
    Exec      to_json();
</Output>

Input sample

The following is a Kubernetes API server log record. This input sample depicts the kind of data and format NXLog Agent will collect.

I0610 09:48:25.796242       1 clientconn.go:948] ClientConn switching balancer to "pick_first"

Output sample

The following JSON shows the same event after it was processed by NXLog Agent.

{
  "EventReceivedTime": "2021-06-10T12:35:14.946171+02:00",
  "SourceModuleName": "k8s_cluster",
  "SourceModuleType": "im_file",
  "log": "I0610 09:48:25.796242       1 clientconn.go:948] ClientConn switching balancer to \"pick_first\"",
  "log_type": "k8s_cluster",
  "log_source": "kube-apiserver",
  "filename": "/opt/nxlog/etc/logs/k8s_logs/kube-apiserver.log",
  "k8s_node": "node001"
}

NXLog Agent as a sidecar

To deploy NXLog Agent as a sidecar, the Deployment needs to contain both the application container and the NXLog Agent container in the same pod. For collecting file-based logs, the path of the application log files must be mounted to the NXLog Agent container.

Kubernetes environment variables can be injected to the NXLog Agent container. These variables can then be used from the NXLog Agent configuration to enrich log records with metadata identifying the Kubernetes node, pod, and namespace.

The following is an example Deployment of NXLog Agent running alongside and collecting file-based logs from Tomcat. NXLog Agent configuration examples are available in the Configuration examples section.

Create a YAML file with the following manifest:

apiVersion: v1
kind: Service
metadata:
  name: tomcat-svc
  labels:
    k8s-app: tomcat-nxlog
spec:
  selector:
    k8s-app: tomcat-nxlog
  type: NodePort
  ports:
  - port: 80
    targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-nxlog-deployment
  labels:
    app: tomcat-nxlog
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tomcat-nxlog
  template:
    metadata:
      labels:
        k8s-app: tomcat-nxlog
    spec:
      securityContext:
        fsGroup: 2000
      volumes:
      - name: tomcatlogs
        emptyDir: {}
      containers:

      #Tomcat container
      - name: tomcat
        image: tomcat:9.0
        volumeMounts:
          - name: tomcatlogs
            mountPath: /usr/local/tomcat/logs
        ports:
        - containerPort: 8080
        lifecycle:
          postStart:
            exec:
              command: ["/bin/sh", "-c", "cp -R webapps.dist/. webapps"]

      #NXLog container
      - name: nxlog
        image: nxlog:latest
        imagePullPolicy: Never
        volumeMounts:
          - name: tomcatlogs
            mountPath: /usr/local/tomcat/logs
        env:
          - name:  NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name:  POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name:  POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name:  POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name:  POD_SERVICE_ACCOUNT
            valueFrom:
              fieldRef:
                fieldPath: spec.serviceAccountName

Execute the following command to create the Deployment:
```
$ kubectl apply -f my_deployment_file.yml
```
The output should show that the application service and deployment were created successfully:
```
service/tomcat-svc created
deployment.apps/tomcat-nxlog-deployment created
```

Configuration examples

Example 3. Writing logs to standard output

The configuration below uses the im_file input module to read Tomcat access logs. Logs are written to standard output using the om_exec module, where they will be picked up by the container engine and written to the container log file.

nxlog.conf

<Input tomcat_access_log>
    Module     im_file
    File       '/usr/local/tomcat/logs/localhost_access*'
</Input>

<Output stdout>
    Module     om_exec
    Command    /bin/sh
    Arg        -c
    Arg        cat $1
</Output>

Input sample

The following is a Tomcat access log record. This input sample depicts the kind of data and format NXLog Agent will collect.

172.17.0.1 - - [10/Jun/2021:12:24:24 +0000] "GET /tomcat.css HTTP/1.1" 200 5542

Output sample

The following JSON shows the same event found in the NXLog Agent container log file. The output format depends on the logging driver being used. In this example, the Docker container engine was configured to use the json-file logging driver.

{
  "log": "172.17.0.1 - - [10/Jun/2021:12:24:24 +0000] \"GET /tomcat.css HTTP/1.1\" 200 5542",
  "stream": "stdout",
  "time": "2021-06-10T12:24:34.592725312Z"
}

Example 4. Forwarding logs in JSON format

The configuration below uses the im_file input module to read Tomcat access logs. It parses log records into structured data and enriches them with metadata from environment variables. It then converts the records to JSON while forwarding them to Elasticsearch using the om_elasticsearch output module.

nxlog.conf

envvar NODE_NAME
envvar POD_NAMESPACE
envvar POD_NAME
envvar POD_SERVICE_ACCOUNT
envvar POD_IP

<Extension json>
    Module     xm_json
</Extension>

<Input tomcat_access_log>
    Module     im_file
    File       '/usr/local/tomcat/logs/localhost_access*'
    <Exec>
        if $raw_event =~ /(?x)^(\S+)\ \S+\ (\S+)\ \[([^\]]+)\]\ \"(\S+)\ (.+)
                          \ HTTP\/\d\.\d\"\ (\S+)\ (\S+)/
        {
            $Hostname = $1;
            if $2 != '-' $AccountName = $2;
            $EventTime = parsedate($3);
            $HTTPMethod = $4;
            $HTTPURL = $5;
            $HTTPResponseStatus = $6;
            if $7 != '-' $FileSize = $7;
        }

        $k8s_node = '%NODE_NAME%';
        $k8s_pod = '%POD_NAME%';
        $k8s_pod_namespace = '%POD_NAMESPACE%';
        $k8s_pod_svc_account = '%POD_SERVICE_ACCOUNT%';
        $k8s_pod_ip = '%POD_IP%';
    </Exec>
</Input>

<Output elasticsearch>
    Module    om_elasticsearch
    URL       http://192.168.1.123:9200/_bulk
    Exec      to_json();
</Output>

Input sample

The following is a Tomcat access log record. This input sample depicts the kind of data and format NXLog Agent will collect.

172.17.0.1 - - [10/Jun/2021:12:24:24 +0000] "GET /tomcat.css HTTP/1.1" 200 5542

Output sample

The following JSON shows the same event after it was processed by NXLog Agent.

{
  "EventReceivedTime": "2021-06-10T12:24:38.589128+00:00",
  "SourceModuleName": "tomcat_access_log",
  "SourceModuleType": "im_file",
  "Hostname": "172.17.0.1",
  "EventTime": "2021-06-10T12:24:24.000000+00:00",
  "HTTPMethod": "GET",
  "HTTPURL": "/tomcat.css",
  "HTTPResponseStatus": "200",
  "FileSize": "5542",
  "k8s_node": "node001",
  "k8s_pod": "tomcat-nxlog-deployment-5675bd78bd-ll5xf",
  "k8s_pod_namespace": "default",
  "k8s_pod_svc_account": "default",
  "k8s_pod_ip": "172.17.0.2"
}

Collecting Kubernetes audit logs

Kubernetes can be configured to audit requests made to the Kubernetes API server. Auditing provides visibility to who or what issued the request (users, applications, or the control plane itself), the resource that was requested, and the response returned by the API. These logs can be useful both for troubleshooting and detecting unusual behavior. For example, repeated failed attempts to access resources could indicate a user trying to breach the system.

One of the audit backends provided by Kubernetes is the Log backend, which writes audit events to file in JSON format, one record per line (see JSON Lines). NXLog Agent can be deployed in one of two ways to collect these events:

Installing NXLog Agent on the master node and reading the audit log file from the local filesystem. See the Configuration example below.
Deploying NXLog Agent as a DaemonSet and reading the audit log file from a mounted volume. See NXLog Agent as a DaemonSet for an example.

Applying an audit policy

To enable Kubernetes auditing, you first need to create an audit policy to define which events should be recorded and what information should be logged. For more information, see the Kubernetes documentation on Auditing and the Policy reference. The following is an example audit policy YAML file available in the Kubernetes documentation:

audit-policy.yml

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
# do not log requests to the following
- level: None
  nonResourceURLs:
  - "/healthz*"
  - "/logs"
  - "/metrics"
  - "/swagger*"
  - "/version"

# limit level to Metadata so token is not included in the spec/status
- level: Metadata
  omitStages:
  - RequestReceived
  resources:
  - group: authentication.k8s.io
    resources:
    - tokenreviews

# extended audit of auth delegation
- level: RequestResponse
  omitStages:
  - RequestReceived
  resources:
  - group: authorization.k8s.io
    resources:
    - subjectaccessreviews

# log changes to pods at RequestResponse level
- level: RequestResponse
  omitStages:
  - RequestReceived
  resources:
  - group: "" # core API group; add third-party API services and your API services if needed
    resources: ["pods"]
    verbs: ["create", "patch", "update", "delete"]

# log everything else at Metadata level
- level: Metadata
  omitStages:
  - RequestReceived

Once you have created your audit policy, apply it to the Kubernetes API server by following these steps:

SSH into the master node.
Create a directory where the audit log file will be stored, e.g., /var/log/kubernetes.
Configure kube-apiserver to use this audit policy and log events to file:
1. Open the manifest file of kube-apiserver located at /etc/kubernetes/manifests/kube-apiserver.yaml.
2. Add the following arguments to the commands section, replacing the paths with the ones corresponding to your master node:
  - --audit-policy-file=/etc/kubernetes/audit-policy.yml - --audit-log-path=/var/log/kubernetes/audit.log
3. Save the file and restart kube-apiserver.

If kube-apiserver is running as a pod, you will also need to add mounted volumes for the location of the policy and log files. See Log backend for details.

Configuration example

Example 5. Collecting audit logs on the master node

The configuration below uses the im_file input module to read Kubernetes audit logs. It parses logs using the xm_json extension and enriches the log records with the hostname and log type felds. It then converts the records back to JSON and forwards them to a SIEM using the om_http output module.

nxlog.conf

<Extension json>
    Module     xm_json
</Extension>

<Input k8s_audit>
    Module     im_file
    File       '/var/log/kubernetes/audit.log'
    <Exec>
        parse_json();
        $hostname = hostname();
        $log_type = "k8s_audit";
        to_json();
    </Exec>
</Input>

<Output siem>
    Module        om_http
    URL           http://<my_siem_url>/
    BufferSize    100000
</Output>

Kubernetes audit log records can be quite large. NXLog Agent input and output modules have a default buffer size of 65000 bytes. Depending on the module being used, the BufferSize may need to be increased, otherwise log records will be truncated. When this happens, the NXLog Agent log file will contain errors similar to the following: data size (69015) is over the limit (65000), will be truncated

Output sample

The following JSON object shows a Kubernetes audit log record after it was processed by NXLog Agent.

{
  "EventReceivedTime": "2021-06-10T20:09:19.677908+02:00",
  "SourceModuleName": "k8s_audit",
  "SourceModuleType": "im_file",
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "auditID": "319aac31-4143-4c7b-ac31-3c85189f4206",
  "stage": "ResponseStarted",
  "requestURI": "/api/v1/services?allowWatchBookmarks=true&resourceVersion=40051&timeout=6m34s&timeoutSeconds=394&watch=true",
  "verb": "watch",
  "user": {
    "username": "system:kube-scheduler",
    "groups": [
      "system:authenticated"
    ]
  },
  "sourceIPs": [
    "192.168.49.2"
  ],
  "userAgent": "kube-scheduler/v1.20.2 (linux/amd64) kubernetes/faecb19/scheduler",
  "objectRef": {
    "resource": "services",
    "apiVersion": "v1"
  },
  "responseStatus": {
    "metadata": {},
    "status": "Failure",
    "reason": "Forbidden",
    "code": 403
  },
  "requestReceivedTimestamp": "2021-06-07T17:42:55.723470+02:00",
  "stageTimestamp": "2021-06-07T17:42:55.723692+02:00",
  "annotations": {
    "authorization.k8s.io/decision": "forbid",
    "authorization.k8s.io/reason": ""
  },
  "hostname": "kubernetes-node001",
  "log_type": "k8s_audit"
}

Disclaimer

While we endeavor to keep the information in our guides up to date and correct, NXLog makes no representations or warranties of any kind, express or implied about the completeness, accuracy, reliability, suitability, or availability of the content represented here. We update our screenshots and instructions on a best-effort basis.

The accurateness of the content was tested and proved to be working in our lab environment at the time of the last revision with the following software versions:

Kubernetes version 1.20.2
NXLog Agent version 5.3.6720
Ubuntu 20.04

Last revision: 15 June 2021