Amazon S3 (im_amazons3)
Amazon Simple Storage Service (S3) is an object storage service offering industry-leading scalability, data availability, security, and performance.
This module can be used to collect logs from Amazon S3 and compatible services.
Amazon S3 buckets, objects, keys, and structure
Amazon S3 stores objects inside containers called buckets. See Getting Started with Amazon S3 in the Amazon S3 User Guide for more information.
Every im_amazons3 instance interacts with a single bucket on Amazon S3. The module will not create, delete, or alter the bucket or any of its properties, permissions, or management options. Instead, you must create the bucket, provide the appropriate permissions (ACL), and further configure any lifecycle, replication, encryption, or other options. Similarly, the module does not alter the storage class of the objects or any other properties or permissions.
By default, the module uses a schema where each object has a key that references the server or service name, the date, and the time of the event. Although Amazon S3 uses a flat structure to store objects, it groups objects with similar key prefixes resembling a filesystem structure. The following is a visual representation of our default naming scheme.
MYBUCKET/
└── SERVER01/
└── 2025-06-18/
├── 12.30.45.0
├── 12.30.45.1
└── 12.30.45.2
└── SERVER02/
└── 2025-06-17/
├── 14.23.12.0
├── 14.23.12.1
└── 14.23.12.2You can change the schema by specifying the KeyTemplate directive.
Configuration
The im_amazons3 module accepts the following directives in addition to the common module directives. The Bucket, Region, and Server directives are required.
The AccessKey and SecretKey directives are required if NXLog Agent is not running in the same tenant as the S3 bucket.
Required directives
The following directives are required for the module to start.
This mandatory directive specifies the Amazon S3 bucket name. |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
This boolean directive changes how the module constructs the URL to cater to providers like MinIO, which accepts the bucket name in the path instead of a subdomain.
The default is |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
This mandatory directive specifies the service region code. It accepts any value when used in conjunction with the URL directive. Otherwise, the following codes are supported:
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
This mandatory directive sets the object path prefix.
The module will read object names starting with the specified value only.
Use |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Specify the URL for a custom endpoint. If the protocol is not specified, the module will use HTTPS. |
HTTPS directives
The following directives configure secure data transfer via HTTPS.
Set this directive to |
|
Set this directive to |
|
Set this directive to the path of a directory containing Certificate Authority (CA) certificates.
The module uses these certificates to verify the certificate presented by the remote host.
Name the certificate files using the OpenSSL hashed format: the hash of the certificate followed by For example, if the certificate hash is To trust a remote host’s self-signed certificate, include a copy of it in this directory. If this directive isn’t specified, NXLog Agent falls back to the default operating system root certificate store.
Unix-like operating systems commonly store root certificates in In addition, Microsoft’s PKI repository contains root certificates for Microsoft services. |
|
Set this directive to the path of the Certificate Authority (CA) certificate used to verify the certificate presented by the remote host. To trust a remote host’s self-signed certificate, specify the remote host’s certificate itself. For certificates signed by an intermediate CA, the specified certificate must contain the complete certificate chain (certificate bundle). |
|
Set this directive to the path of the certificate file to present to the remote host during the HTTPS handshake. |
|
Set this directive to the path of the private key file corresponding to the certificate specified by HTTPSCertFile. |
Optional directives
This optional directive specifies the AWS public access key ID. If AccessKey and SecretKey are missing, the module will try to read the credentials from the environment, STS, profile, or instance metadata. If none are available, the module will try to log in anonymously. |
|||||||||||||||||||||||||||||||||||||
Set this directive to a custom header to add to each HTTP request. Specify this directive multiple times to add more than one header. |
|||||||||||||||||||||||||||||||||||||
Set this directive to enable HTTP compression for outgoing HTTP messages.
The accepted values are Some HTTP servers may not accept compressed HTTP requests.
If the server doesn’t support the specified compression method, it may return |
|||||||||||||||||||||||||||||||||||||
Set this directive to the HTTP basic authentication password. This directive requires you to also set the HTTPBasicAuthUser directive to use HTTP authentication. |
|||||||||||||||||||||||||||||||||||||
Set this directive to the HTTP basic authentication username. If the HTTPBasicAuthPassword directive is not set, the module attempts HTTP Basic authentication using an empty password. |
|||||||||||||||||||||||||||||||||||||
This optional directive sets the reconnect interval in seconds. If it is set, the module attempts to reconnect in every defined second. If it is not set, the reconnect interval will start at 1 second and double with every attempt. In the latter case, when the system decides that the reconnection is successful, the reconnect interval is immediately reset to 1 sec.
|
|||||||||||||||||||||||||||||||||||||
This optional directive defines the behavior when the connection with the remote host is lost.
When set to |
|||||||||||||||||||||||||||||||||||||
This optional boolean directive is used to activate or deactivate the polling interval adjustment mechanism.
The default value is |
|||||||||||||||||||||||||||||||||||||
Use this directive to change the naming scheme of objects in the bucket. The key template must match the naming scheme used when writing the data to the bucket. You can use the following placeholders:
The value must adhere to the following requirements:
Refer to the Amazon S3 General purpose bucket naming rules. The default value is See Amazon S3 buckets, objects, keys, and structure for more information. |
|||||||||||||||||||||||||||||||||||||
This optional directive specifies how frequently the module will check for new events in seconds. If this directive is not specified, it defaults to 60 seconds. |
|||||||||||||||||||||||||||||||||||||
This boolean directive instructs the module on where to start reading events from the log source when NXLog Agent starts. When When The default is The following matrix shows the outcome of this directive in conjunction with the SavePos directive:
If the NoCache directive is |
|||||||||||||||||||||||||||||||||||||
Set this directive to The default is If the NoCache directive is |
|||||||||||||||||||||||||||||||||||||
This optional directive specifies the AWS secret access key. |
|||||||||||||||||||||||||||||||||||||
This optional directive specifies the time in RFC 3339 format of the first event to pull, e.g., |
Fields
The following fields are created by im_amazons3.
-
$raw_event(type: string) -
A record from the stored object
Examples
This configuration uses the im_amazons3 input module to collect logs from an Amazon S3 Bucket named MYBUCKET.
The Server directive specifies that only object names starting with SERVER01 should be collected.
<Input amazon_s3>
Module im_amazons3
Region us-east-1
Bucket MYBUCKET
Server SERVER01
StartFrom 2022-09-15T10:23:00-06:00 (1)
AccessKey <YOUR_ACCESS_KEY> (2)
SecretKey <YOUR_SECRET_KEY> (3)
</Input>This configuration uses the im_amazons3 input module to collect logs from a self-hosted MinIO S3 instance.
<Input amazon_s3>
Module im_amazons3
URL https://example.net (1)
Region myminio
Bucket MYBUCKET
Server SERVER01
AccessKey <YOUR_ACCESS_KEY> (2)
SecretKey <YOUR_SECRET_KEY> (3)
</Input>