key Must Be a Buffer S3 Upload

The Amazon S3 output plugin allows you to ingest your records into the S3 cloud object store.

The plugin can upload data to S3 using the multipart upload API or using S3 PutObject . Multipart is the default and is recommended; Fluent Bit will stream data in a series of 'parts'. This limits the corporeality of data it has to buffer on deejay at any point in time. By default, every time v MiB of data take been received, a new 'part' will be uploaded. The plugin tin can create files up to gigabytes in size from many small chunks/parts using the multipart API. All aspects of the upload process are configurable using the configuration options.

The plugin allows you to specify a maximum file size, and a timeout for uploads. A file volition be created in S3 when the max size is reached, or the timeout is reached- whichever comes starting time.

Records are stored in files in S3 equally newline delimited JSON.

Run across here for details on how AWS credentials are fetched.

Configuration Parameters

The AWS region of you S3 bucket

Specify the proper noun of the fourth dimension key in the output record. To disable the time key just set the value to false .

Specify the format of the engagement. Supported formats are double , epoch , iso8601 (eg: 2018-05-30T09:39:52.000681Z ) and java_sql_timestamp (eg: 2018-05-30 09:39:52.000681 )

Specifies the size of files in S3. Maximum size is 50G, minimim is 1M.

The size of each 'function' for multipart uploads. Max: 50M

Whenever this amount of fourth dimension has elapsed, Fluent Bit volition complete an upload and create a new file in S3. For example, set this value to 60m and y'all volition become a new file every hr.

Directory to locally buffer data earlier sending. When multipart uploads are used, data volition only be buffered until the upload_chunk_size is reached.

Format string for keys in S3. This option supports a UUID, strftime time formatters, a syntax for selecting parts of the Fluent log tag using a syntax inspired by the rewrite_tag filter. Add $UUID in the format string to insert a random string. Add $INDEX in the format string to insert an integer that increments each upload. Add $TAG in the format string to insert the full log tag; add together $TAG[0] to insert the commencement role of the tag in the s3 key. The tag is divide into "parts" using the characters specified with the s3_key_format_tag_delimiters option. Add extension directly after the last piece of the format string to insert a fundamental suffix. If you want to specify a primal suffix and you are in use_put_object mode, you must specify $UUID too. More explanations can exist found in use_put_object selection. See the in depth examples and tutorial in the documentation.

/fluent-bit-logs/$TAG/%Y/%k/%d/%H/%M/%S

s3_key_format_tag_delimiters

A series of characters which will be used to split the tag into 'parts' for use with the s3_key_format selection. Encounter the in depth examples and tutorial in the documentation.

Disables behavior where UUID string is automatically appended to end of S3 primal proper noun when $UUID is not provided in s3_key_format. $UUID, time formatters, $TAG, and other dynamic central formatters all work equally expected while this feature is set to truthful.

Employ the S3 PutObject API, instead of the multipart upload API. When this option is on, key extension is only available when $UUID is specified in s3_key_format . If $UUID is not included, a random string volition exist appended at the terminate of the format string and the primal extension cannot be customized in this case.

ARN of an IAM role to assume (ex. for cantankerous account access).

Custom endpoint for the S3 API. An endpoint can contain scheme and port.

Custom endpoint for the STS API.

Compression type for S3 objects. 'gzip' is currently the only supported value. The Content-Encoding HTTP Header will exist prepare to 'gzip'. Compression tin be enabled when use_put_object is on. If Apache Arrow back up was enabled at compile time, you lot can ready 'arrow' to this option.

A standard MIME blazon for the S3 object; this will exist set as the Content-Blazon HTTP header.

Ship the Content-MD5 header with PutObject and UploadPart requests, as is required when Object Lock is enabled.

Immediately retry failed requests to AWS services once. This option does non affect the normal Fluent Bit retry machinery with backoff. Instead, information technology enables an immediate retry with no delay for networking errors, which may assistance ameliorate throughput when there are transient/random networking bug.

TLS / SSL

To skip TLS verification, gear up tls.verify as false . For more details almost the properties available and general configuration, delight refer to the TLS/SSL section.

Permissions

The plugin requires s3:PutObject permission.

S3 Key Format and Tag Delimiters

In Fluent Flake, all logs take an associated tag. The s3_key_format option lets you inject the tag into the s3 fundamental using the following syntax:

$TAG[n] => the nth part of the tag (alphabetize starting at zero). This syntax is copied from the rewrite tag filter. Past default, "parts" of the tag are separated with dots, only you lot can change this with s3_key_format_tag_delimiters .

In the example below, assume the engagement is January 1st, 2020 00:00:00 and the tag associated with the logs in question is my_app_name-logs.prod .

s3_key_format /$TAG[2]/$TAG[0]/%Y/%m/%d/%H/%M/%S/$UUID.gz

s3_key_format_tag_delimiters .-

With the delimiters as . and -, the tag will be separate into parts as follows:

So the fundamental in S3 will exist /prod/my_app_name/2020/01/01/00/00/00/bgdHN1NM.gz .

Reliability

The store_dir is used to temporarily shop data before information technology is uploaded. If Fluent Chip is stopped of a sudden it will try to send all data and complete all uploads before information technology shuts down. If it tin non send some data, on restart it volition look in the store_dir for existing data and volition effort to send information technology.

Multipart uploads are ideal for most utilize cases considering they let the plugin to upload data in small chunks over fourth dimension. For example, 1 GB file can be created from 200 5MB chunks. While the file size in S3 will be 1 GB, only 5 MB volition be buffered on disk at any one point in fourth dimension.

There is ane minor drawback to multipart uploads- the file and data will non be visible in S3 until the upload is completed with a CompleteMultipartUpload call. The plugin volition effort to make this call whenever Fluent Bit is shut down to ensure your data is available in s3. Information technology will likewise store metadata about each upload in the store_dir , ensuring that uploads can be completed when Fluent Bit restarts (bold it has access to persistent disk and the store_dir files will still exist present on restart).

Using S3 without persisted disk

If you run Fluent Bit in an surroundings without persistent disk, or without the ability to restart Fluent Bit and requite it admission to the data stored in the store_dir from previous executions- some considerations utilise. This might occur if you run Fluent Bit on AWS Fargate .

In these situations, nosotros recommend using the PutObject API, and sending information oft, to avert local buffering every bit much as possible. This volition limit data loss in the result Fluent Fleck is killed unexpectedly.

The following settings are recommended for this employ example:

Worker back up

Fluent Bit ane.7 adds a new characteristic chosen workers which enables outputs to have dedicated threads. This s3 plugin has partial support for workers. The plugin can just support a single worker; enabling multiple workers will lead to errors/indeterminate beliefs.

If y'all enable a single worker, you are enabling a defended thread for your S3 output. We recommend starting without workers, evaluating the operation, and and so enabling a worker if needed. For about users, the plugin can provide sufficient throughput without workers.

Usage with MinIO

MinIO is a high-performance, S3 compatible object storage and yous can build your app with S3 functionality without S3.

endpoint http://localhost:9000

And so, the records volition be stored into the MinIO server.

Getting Started

In order to send records into Amazon S3, you can run the plugin from the command line or through the configuration file.

Control Line

The s3 plugin, can read the parameters from the command line through the -p argument (property), e.g:

$ fluent-flake -i cpu -o s3 -p bucket=my-bucket -p region=us-west-2 -p -m '*' -f 1

Configuration File

In your primary configuration file suspend the following Output section:

store_dir /home/ec2-user/buffer

An case that using PutObject instead of multipart:

store_dir /dwelling/ec2-user/buffer

AWS for Fluent Bit

Amazon distributes a container image with Fluent Bit and this plugins.

GitHub

Amazon ECR Public Gallery

Our images are available in Amazon ECR Public Gallery. You tin download images with unlike tags by following control:

docker pull public.ecr.aws/aws-observability/aws-for-fluent-flake:<tag>

For case, you tin pull the image with latest version by:

docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:latest

If you meet errors for epitome pull limits, effort log into public ECR with your AWS credentials:

aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws

Docker Hub

Amazon ECR

You tin utilize our SSM Public Parameters to discover the Amazon ECR image URI in your region:

aws ssm get-parameters-by-path --path /aws/service/aws-for-fluent-bit/

Advanced usage

Use Apache Arrow for in-memory information processing

Starting from Fluent Chip v1.8, the Amazon S3 plugin includes the back up for Apache Pointer . The back up is currently not enabled past default, as it depends on a shared version of libarrow as the prerequisite.

To use this feature, FLB_ARROW must be turned on at compile fourth dimension:

$ cmake -DFLB_ARROW=On ..

Once compiled, Fluent Bit tin upload incoming data to S3 in Apache Arrow format. For instance:

As shown in this example, setting Compression to arrow makes Fluent Bit to convert payload into Apache Arrow format.

The stored data is very like shooting fish in a barrel to load, analyze and process using popular information processing tools (such as Python pandas, Apache Spark and Tensorflow). The following lawmaking uses pyarrow to analyze the uploaded data:

>>> import pyarrow.plumage as feather

>>> import pyarrow.fs as fs

>>> s3 = fs.S3FileSystem()

>>> file = s3.open_input_file("my-saucepan/fluent-scrap-logs/cpu.0/2021/04/27/09/36/xv-object969o67ZF")

>>> df = plume.read_feather(file)

date cpu_p user_p system_p cpu0.p_cpu cpu0.p_user cpu0.p_system

0 2021-04-27T09:33:53.539346Z ane.0 one.0 0.0 1.0 1.0 0.0

1 2021-04-27T09:33:54.539330Z 0.0 0.0 0.0 0.0 0.0 0.0

ii 2021-04-27T09:33:55.539305Z one.0 0.0 i.0 1.0 0.0 1.0

3 2021-04-27T09:33:56.539430Z 0.0 0.0 0.0 0.0 0.0 0.0

four 2021-04-27T09:33:57.539803Z 0.0 0.0 0.0 0.0 0.0 0.0

jacksongoved1943.blogspot.com

Source: https://docs.fluentbit.io/manual/pipeline/outputs/s3/