Fluent-Bit

Fluent Bit is a lightweight and extensible log processor and forwarder. It is designed to be fast and efficient, and it is optimized for low resource consumption. Fluent Bit is part of the Fluentd project ecosystem, and it is licensed under the terms of the Apache License v2.0.

Ingest Methods

Setup the ingestion of this source using one of the following guides.

If using an AWS S3 bucket use the following SNS topic ARN to send your bucket notifications.

arn:aws:sns:<REGION>:253602268883:runreveal_generic

⚠️

SNS topic & Custom SQS. Use the ARN above in your event notification tied to your S3 bucket—the topic name must match (runreveal_…; hyphens in the source id become underscores). For Custom SQS, set the queue URL and region in RunReveal; see AWS S3 Bucket with Custom SQS.

Note: BYOC, On-Prem, and BYODB customers must use their AWS account ID in the ARN instead of 253602268883.

Setup

1. Stream logs to S3

To ingest fluent-bit logs into RunReveal, you will need to set up an S3 bucket to stream your fluent logs to, if you don’t have one already.

Below you’ll find an example configuration for fluent-bit to stream logs to an S3 bucket.

[OUTPUT]
    Name                         s3
    Match                        *
    bucket                       my-fluent-bit-bucket
    region                       us-east-2
    use_put_object               Off
    compression                  None
    s3_key_format                /$TAG/%Y/%m/%d/%H_%M_%S-$UUID.ndjson
    total_file_size              10M
    upload_timeout               60s

2. Source setup

From there, take the bucket you’ve configured fluent-bit to stream logs to, and follow our guide on setting up a Generic S3 source, paying close attention to the event transformation step to normalize your logs for optimal query performance.

FireHydrant Formal Security