Dropping Logs

Drop steps permanently remove events from your pipeline before they reach storage. This is useful for eliminating noise, reducing costs, and ensuring sensitive data never enters your logs.

⚠️

Permanent Deletion: Dropped events are permanently discarded and cannot be recovered. If you might need events for compliance or forensics, use Filtering to archive them instead.

When to Use Drop

Drop is ideal for:

Eliminating noise from health checks, heartbeats, and routine system operations
Reducing costs by removing high-volume, zero-value events
Preventing PII storage by dropping events containing sensitive data patterns
Simple, pipeline-specific rules that don’t need to be shared across pipelines

How Dropping Works

When you add a Drop step to a pipeline:

Events matching the step’s preconditions are identified
Matching events are permanently discarded
Non-matching events continue through the pipeline
Dropped events never reach storage or detections

Configuration

Drop steps use preconditions to determine which events to remove:

Precondition Type	Description	Example
Exact Match	Drop events with specific field values	`actor.email = "[email protected]"`
Not Equal	Drop events that don’t match a value	`environment != "production"`
Regex	Drop events matching a pattern	`eventName regex: ^Describe.*`
Is Empty	Drop events with missing fields	`actor.email isEmpty`
CIDR Match	Drop events from IP ranges	`src.ip cidrMatch: 10.0.0.0/8`

Use Cases

Drop AWS CloudTrail Read-Only API Calls

AWS generates high volumes of read-only API calls (Describe*, List*, Get*) that often have limited security value. Drop these to focus on write operations and configuration changes.

{
  "eventName": "DescribeInstances",
  "eventSource": "ec2.amazonaws.com",
  "userIdentity": {
    "arn": "arn:aws:iam::123456789012:role/monitoring-role",
    "invokedBy": "monitoring.amazonaws.com"
  },
  "readOnly": true
}

Alternative: Drop by Event Name Pattern

Drop Describe/List/Get callseventName regex: ^(Describe|List|Get)

Drop AWS KMS Decrypt Noise

AWS services automatically call KMS Decrypt for encrypted resources, generating thousands of routine events. Drop these internal operations while keeping user-initiated KMS activity.

{
  "eventName": "Decrypt",
  "eventSource": "kms.amazonaws.com",
  "userIdentity": {
    "invokedBy": "AWS Internal"
  },
  "requestParameters": {
    "encryptionContext": {
      "aws:lambda:FunctionArn": "arn:aws:lambda:us-east-1:123456789012:function:myFunction"
    }
  }
}

Adding multiple preconditions to the same step creates an AND condition—both must match for the event to be dropped.

Drop Okta System Events

Okta generates system-level events from the SystemPrincipal actor that are typically not relevant for security monitoring.

{
  "eventType": "system.org.rate_limit.warning",
  "actor": {
    "id": "00u0000000000000000",
    "type": "SystemPrincipal",
    "alternateId": "[email protected]",
    "displayName": "Okta System"
  },
  "outcome": { "result": "SUCCESS" }
}

Drop Health Check Traffic

Load balancer and Kubernetes health checks generate constant traffic with no security value. Drop these events to reduce noise.

{
  "type": "h2",
  "elb": "app/my-alb/50dc6c495c0c9188",
  "request_url": "https://10.0.1.50:443/health",
  "user_agent": "ELB-HealthChecker/2.0",
  "target_status_code": 200
}

Drop Events with Incomplete Data

Events missing critical fields (like actor information) may not be useful for security analysis. Drop events with empty required fields.

{
  "eventName": "anonymous_request",
  "actor": {},
  "src": { "ip": "10.0.0.1" }
}

Drop vs Filter vs Sample

Action	Data Retained	Use Case
Drop	❌ Permanently deleted	Zero-value events you never need
Filter	✅ Archived (searchable)	Low-value events you might need for compliance
Sample	✅ Percentage kept	High-volume events where statistical patterns suffice

Best Practices

Start with Filter, not Drop: Archive events first to validate your rules don’t catch important data
Use specific preconditions: Avoid overly broad patterns that might drop security-relevant events
Combine with enrichments: Tag events for dropping using enrichments to centralize your drop logic
Document your drops: Keep a record of what you’re dropping and why
Monitor for gaps: Periodically review if dropped event types become relevant

Using Enrichments with Drop Steps

For complex drop logic or centralized management, use enrichments to tag events, then drop based on the enrichment flag:

Enrichment-Based Dropping

Step 1: Create Enrichment

Create an enrichment named drop-candidates that tags events with {“should_drop”: true}

Step 2: Add Enrich Step

Add an Enrich step to your pipeline that applies the drop-candidates enrichment

Step 3: Add Drop Step

Add a Drop step with precondition: enrichments contains “should_drop”: true

This pattern centralizes your drop rules in enrichments, making them easier to manage and audit.

Sample Destinations