VPC Flow Logs

AWS VPC Flow Logs capture information about the IP traffic going to and from network interfaces in a Virtual Private Cloud (VPC). These logs provide details such as source and destination IP addresses, ports, protocols, traffic direction, and the acceptance or rejection of the traffic. They are useful for network monitoring, troubleshooting connectivity issues, and auditing traffic for security analysis.

Ingest Methods

Setup the ingestion of this source using one of the following guides.

• AWS S3 Bucket
• AWS S3 Bucket with Custom SQS

arn:aws:sns:<REGION>:253602268883:runreveal_flow

Setup

Step 1: Create an S3 Bucket

1. Sign in to the AWS Management Console and open the Amazon S3 console.
2. Click on "Create bucket".
3. Enter a unique name for your bucket (e.g., "my-vpc-flow-logs-bucket").
4. Select the region where your VPC is located.
5. Configure the bucket settings as needed (e.g., versioning, encryption).
6. Click "Create bucket" to finish.

Step 2: Configure Bucket Policy

1. In the S3 console, select the bucket you just created.
2. Go to the "Permissions" tab.
3. Under "Bucket policy", click "Edit".
4. Paste the following policy, replacing {your-bucket-name} with your actual bucket name:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AWSLogDeliveryWrite",
            "Effect": "Allow",
            "Principal": {"Service": "delivery.logs.amazonaws.com"},
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::{your-bucket-name}/AWSLogs/*",
            "Condition": {
                "StringEquals": {
                    "s3:x-amz-acl": "bucket-owner-full-control"
                }
            }
        },
        {
            "Sid": "AWSLogDeliveryAclCheck",
            "Effect": "Allow",
            "Principal": {"Service": "delivery.logs.amazonaws.com"},
            "Action": "s3:GetBucketAcl",
            "Resource": "arn:aws:s3:::{your-bucket-name}"
        }
    ]
}

5. Click "Save changes".

Step 3: Create a Flow Log

1. Open the Amazon VPC console.
2. In the navigation pane, choose "Your VPCs".
3. Select the VPC for which you want to create a flow log.
4. Choose the "Flow logs" tab.
5. Choose "Create flow log".
6. Configure the flow log:
   • For "Filter", choose "All" to capture all traffic.
   • For "Maximum aggregation interval", choose your preferred interval.
   • For "Destination", select "Send to an S3 bucket".
   • For "S3 bucket ARN", enter the ARN of the bucket you created (format: arn:aws:s3:::{your-bucket-name}).
   • Optionally, specify a "Log file format" if you want to customize the fields.
   • For "Log record format", choose your preferred format (e.g., AWS default format).
7. Choose "Create flow log".

Step 4: Verify Log Delivery

1. Wait for some time to allow traffic to generate logs (this may take up to 15 minutes).
2. Go to the S3 console and open your bucket.
3. You should see folders created with the structure: AWSLogs/{aws-account-id}/vpcflowlogs/{region}/{year}/{month}/{day}/
4. Inside these folders, you'll find your VPC Flow Log files.

For more information, refer to the official AWS documentation on VPC Flow Logs.

Reducing Volume

VPC Flow Logs can be high volume, and some of those logs have little value. You can reduce ingest volume and cost by routing them to a dedicated pipeline and adding Drop steps to discard the low-value traffic.

Step 1: Create a topic targeting AWS VPC Flow Logs

Create a topic so only flow logs are routed to this pipeline, leaving every other source on its existing processing.

1. Go to the Topics page and click Create Topic.
2. Give it a name (e.g. aws-vpc-flow-logs).
3. Add a precondition that exact matches the flow-logs source type:
   • Field: sourceType
   • Type: exact
   • Value: flow (this is the source type for AWS VPC Flow Logs)
4. For the pipeline, choose Create a new pipeline or Copy from an existing pipeline so you start from your standard Transform → Enrich → Destination flow.

Step 2: Add the drop steps to the pipeline

In the pipeline editor, add the two Drop steps below after your Transform and Enrich steps — so records are already normalized/enriched and the conditions evaluate against the normalized fields — and before the Destination step:

1. Drop internal-to-internal traffic — discards east-west flows where both endpoints are in private (RFC 1918) space, typically the largest chunk of volume. Add one Drop step with two cidrMatch preconditions combined with AND:
   • Field: normalized.src.ip · Type: cidrMatch · Value: 172.16.0.0/12
   • Field: normalized.dst.ip · Type: cidrMatch · Value: 172.16.0.0/12

Drop · normalized.src.ip in 172.16.0.0/12 AND normalized.dst.ip in 172.16.0.0/12

2. Drop empty/no-event records — discards rows with no event (the - placeholder), which carry no analytical value:
   • Field: normalized.eventName · Type: exact · Value: -

Drop · normalized.eventName = -

So the final pipeline order is: Transform → Enrich → Drop (internal-to-internal) → Drop (empty events) → Destination.

Schema

The following columns are exposed for this source. RunReveal applies schema normalization across all sources, ensuring uniform field names and data types for cross-source queries and reusable detection logic.