Filtering

Filters are reusable rules that match events based on regex patterns and either drop or archive them. Unlike the simpler Drop step, filters can be shared across pipelines, applied globally, and offer the option to archive events instead of permanently deleting them.

Filter Actions

Filters support two actions when an event matches:

Filter Types

Regex Filtering

Regex filters match a pattern against the entire raw log JSON. This is powerful for matching any content in the event, regardless of field structure.

{
  "eventName": "GetObject",
  "eventSource": "s3.amazonaws.com",
  "userIdentity": {
    "arn": "arn:aws:iam::123456789012:role/monitoring-role"
  },
  "userAgent": "ELB-HealthChecker/2.0"
}

Pattern: /ELB-HealthChecker/i
Matches: ✅ Yes - found in userAgent field

Pattern: /monitoring-role/
Matches: ✅ Yes - found in userIdentity.arn

Pattern: /DeleteObject/
Matches: ❌ No - not found anywhere in raw log

Pattern Format

Filters use the format /pattern/flags where:

• The pattern is enclosed in forward slashes
• Flags are appended after the closing slash

Supported Flags:

Source Filtering

Source filters match events from specific log sources. When you select sources for a filter, only events from those sources are evaluated against the filter pattern.

Where Do Filtered Events Go?

Global vs Source-Specific Filters

Filters can be configured to apply globally or only to specific sources:

Creating a Filter

Using Filters in Pipelines

Filters can be attached to pipelines as a Filter step. This allows you to apply reusable filter rules at specific points in your pipeline.

Use Cases

Name: health-check-archive
Type: Regex
Pattern: /ELB-HealthChecker|kube-probe|GoogleHC/i
Action: Archive
Scope: Global

// ✅ Matches - will be archived
{"userAgent": "ELB-HealthChecker/2.0", ...}
{"userAgent": "kube-probe/1.28", ...}
{"userAgent": "GoogleHC/1.0", ...}
 
// ❌ Does not match - goes to main logs
{"userAgent": "Mozilla/5.0...", ...}

Name: aws-readonly-drop
Type: Regex
Pattern: /"eventName":"(Describe|List|Get)[A-Za-z]+"/
Action: Drop
Scope: Source-specific (aws-cloudtrail)

// ✅ Matches - will be dropped
{"eventName": "DescribeInstances", ...}
{"eventName": "ListBuckets", ...}
{"eventName": "GetObject", ...}
 
// ❌ Does not match - kept in logs
{"eventName": "CreateUser", ...}
{"eventName": "DeleteBucket", ...}

Name: okta-system-archive
Type: Regex
Pattern: /"actor":\{[^}]*"type":"SystemPrincipal"/
Action: Archive
Scope: Source-specific (okta)

// ✅ Matches - will be archived
{
  "eventType": "system.org.rate_limit.warning",
  "actor": {
    "type": "SystemPrincipal",
    "alternateId": "[email protected]"
  }
}
 
// ❌ Does not match - kept in logs
{
  "eventType": "user.session.start",
  "actor": {
    "type": "User",
    "alternateId": "[email protected]"
  }
}

Filter vs Drop Step

When to use Filter:

• You need to archive events instead of dropping
• You want reusable rules across multiple pipelines
• You want to match patterns anywhere in the raw log

When to use Drop:

• Simple, one-off dropping within a single pipeline
• You want to match on specific normalized fields using preconditions
• You don’t need the archive option

Common Patterns