Amazon S3 (om_amazons3)

Amazon Simple Storage Service (S3) is an object storage service offering industry-leading scalability, data availability, security, and performance.

This module can be used to send logs to Amazon S3 and compatible services.

Amazon S3 buckets, objects, keys, and structure

Amazon S3 stores objects inside containers called buckets. A finite number of buckets that can store an infinite number of objects are available to the user. See Getting Started with Amazon S3 in the Amazon S3 User Guide for more information.

Both the input and output modules interact with a single bucket on Amazon S3. The module will not create, delete, or alter the bucket or any of its properties, permissions, or management options. Instead, you must create the bucket, provide the appropriate permissions (ACL), and further configure any lifecycle, replication, encryption, or other options. Similarly, the module does not alter the storage class of the objects stored or any other properties or permissions.

We selected a schema where we store events in a single bucket. Each object has a key that references the server or service name, the date, and the time NXLog Agent received the event. Although Amazon S3 uses a flat structure to store objects, it groups objects with similar key prefixes resembling a filesystem structure. The following is a visual representation of our naming scheme. Note that the key name at the fourth level represents the time in UTC. However, Amazon S3 uses the colon (:) as a special character; therefore, we replace it with the dot (.) character to simplify matters.

  • MYBUCKET/

    • SERVER01/

      • 2018-05-17/

        • 12.36.34.1

        • 12.36.35.1

      • 2018-05-18/

        • 10.46.34.1

        • 10.46.35.1

        • 10.46.35.2

        • 10.46.36.1

    • SERVER02/

      • 2018-05-16/

        • 14.23.12.1

      • 2018-05-17/

        • 17.03.52.1

        • 17.03.52.2

        • 17.03.52.3

Configuration

The om_amazons3 module accepts the following directives in addition to the common module directives. The Bucket, Region, and Server directives are required.

The AccessKey and SecretKey directives are required if NXLog Agent is not running in the same tenant as the S3 bucket.

Required directives

The following directives are required for the module to start.

Bucket

This mandatory directive specifies the Amazon S3 bucket name.

Region

This mandatory directive specifies the service region code. It accepts any value when used in conjunction with the URL directive. Otherwise, the following codes are supported:

Provider Region Code

Amazon

US East (N. Virginia)

us-east-1

Amazon

US East (Ohio)

us-east-2

Amazon

US West (N. California)

us-west-1

Amazon

US West (Oregon)

us-west-2

Amazon

Canada (Central)

ca-central-1

Amazon

Africa (Cape Town)

af-south-1

Amazon

Asia Pacific (Hong Kong)

ap-east-1

Amazon

Asia Pacific (Mumbai)

ap-south-1

Amazon

Asia Pacific (Tokyo)

ap-northeast-1

Amazon

Asia Pacific (Seoul)

ap-northeast-2

Amazon

Asia Pacific (Osaka)

ap-northeast-3

Amazon

Asia Pacific (Singapore)

ap-southeast-1

Amazon

Asia Pacific (Sydney)

ap-southeast-2

Amazon

China (Beijing)

cn-north-1

Amazon

China (Ningxia)

cn-northwest-1

Amazon

Europe (Stockholm)

eu-north-1

Amazon

Europe (Frankfurt)

eu-central-1

Amazon

Europe (Ireland)

eu-west-1

Amazon

Europe (London)

eu-west-2

Amazon

Europe (Paris)

eu-west-3

Amazon

South America (São Paulo)

sa-east-1

Amazon

Middle East (Bahrain)

me-south-1

Digital Ocean

US East (New York City)

nyc3

Digital Ocean

Europe (Amsterdam)

ams3

Digital Ocean

Asia Pacific (Singapore)

sgp1

Digital Ocean

Europe (Frankfurt)

fra1

Yandex (Object Storage)

Russia

ru-central1

Wasabi

US East (N. Virginia)

wa-us-east-1

Wasabi

US East (N. Virginia)

wa-us-east-2

Wasabi

US West (Oregon)

wa-us-west-1

Wasabi

Europe (Amsterdam)

wa-eu-central-1

Server

This mandatory directive sets the object path prefix. The module will write object names starting with the specified value. See Amazon S3 buckets, objects, keys, and structure.

Optional directives

AccessKey

This optional directive specifies the AWS public access key ID. If AccessKey and SecretKey are missing, the module will try to read the credentials from the environment, STS, profile, or instance metadata. If none are available, the module will try to log in anonymously.

Connections

Number of allowed parallel HTTP connections. More connections mean bigger throughput. The default is 1.

OutputRequestSize

Maximum size of the request in bytes. The default is 64 MB.

OutputType

See the OutputType directive in the list of common module directives. If this directive is not specified, the default is LineBased, i.e., the module will use CRLF as the record terminator on Windows or LF on Unix.

This directive also supports data converters, see the description in the OutputType section.

PathStyle

This boolean directive changes how the module constructs the URL to cater to providers like MinIO, which accepts the bucket name in the path instead of a subdomain. The default is TRUE if the URL directive is specified. Otherwise, this directive is not used. If FALSE, the module will prefix the URL with the Bucket as a subdomain. So, for example, if the URL is https://s3-us-east-2.amazonaws.com and the bucket is mybucket, the resulting URL will be https://mybucket.s3-us-east-2.amazonaws.com.

Reconnect

This optional directive sets the reconnect interval in seconds. If it is set, the module attempts to reconnect in every defined second. If it is not set, the reconnect interval will start at 1 second and doubles on every attempt. If the duration of the successful connection is greater than the current reconnect interval, then the reconnect interval will be reset to 1 sec.

SecretKey

This optional directive specifies the AWS secret access key.

URL

Specify the URL for a custom endpoint. If the protocol is not specified, the module will use HTTPS.

Examples

Example 1. Sending logs to an Amazon S3 bucket

This configuration uses the om_amazons3 output module to forward logs to an Amazon S3 Bucket named MYBUCKET. Object names will be prefixed with SERVER01 specified by the Server directive.

<Output amazon_s3>
    Module       om_amazons3
    Region       us-east-1
    Bucket       MYBUCKET
    Server       SERVER01

    AccessKey    <YOUR_ACCESS_KEY> (1)
    SecretKey    <YOUR_SECRET_KEY> (2)
</Output>
1 The AccessKey directive specifies the AWS public access key ID.
2 The SecretKey directive specifies the AWS secret access key.
Example 2. Sending logs to other Amazon S3-compatible services

This configuration uses the om_amazons3 output module to forward logs to a self-hosted MinIO S3 instance.

<Output amazon_s3>
    Module        om_amazons3
    URL           https://example.net (1)
    Region        myminio
    Bucket        MYBUCKET
    Server        SERVER01

    AccessKey    <YOUR_ACCESS_KEY> (2)
    SecretKey    <YOUR_SECRET_KEY> (3)
</Output>
1 The URL directive is specified to use a custom endpoint.
2 The AccessKey directive specifies the AWS public access key ID.
3 The SecretKey directive specifies the AWS secret access key.