Filtering

Filters are reusable rules that match events based on regex patterns and drop them when they match. Unlike the simpler Drop step, filters can be shared across pipelines and applied globally.

Filter action

When an event matches a filter, it is dropped: it is permanently discarded and not indexed or forwarded.

Filter Types

Regex Filtering

Regex filters match a pattern against the entire raw log JSON. This is powerful for matching any content in the event, regardless of field structure.

{
  "eventName": "GetObject",
  "eventSource": "s3.amazonaws.com",
  "userIdentity": {
    "arn": "arn:aws:iam::123456789012:role/monitoring-role"
  },
  "userAgent": "ELB-HealthChecker/2.0"
}

Pattern: /ELB-HealthChecker/i
Matches: ✅ Yes - found in userAgent field

Pattern: /monitoring-role/
Matches: ✅ Yes - found in userIdentity.arn

Pattern: /DeleteObject/
Matches: ❌ No - not found anywhere in raw log

Pattern Format

Filters use the format /pattern/flags where:

• The pattern is enclosed in forward slashes
• Flags are appended after the closing slash

Supported Flags:

Source Filtering

Source filters match events from specific log sources. When you select sources for a filter, only events from those sources are evaluated against the filter pattern.

Where do matched events go?

Dropped events are not stored in ClickHouse (they will not appear in Explore) and are not sent to destinations.

Global vs Source-Specific Filters

Filters can be configured to apply globally or only to specific sources:

Creating a Filter

Using Filters in Pipelines

Filters can be attached to pipelines as a Filter step. This allows you to apply reusable filter rules at specific points in your pipeline.

Use Cases

Name: health-check-drop
Type: Regex
Pattern: /ELB-HealthChecker|kube-probe|GoogleHC/i
Action: Drop
Scope: Global

// ✅ Matches - will be dropped
{"userAgent": "ELB-HealthChecker/2.0", ...}
{"userAgent": "kube-probe/1.28", ...}
{"userAgent": "GoogleHC/1.0", ...}
 
// ❌ Does not match - goes to main logs
{"userAgent": "Mozilla/5.0...", ...}

Name: aws-readonly-drop
Type: Regex
Pattern: /"eventName":"(Describe|List|Get)[A-Za-z]+"/
Action: Drop
Scope: Source-specific (aws-cloudtrail)

// ✅ Matches - will be dropped
{"eventName": "DescribeInstances", ...}
{"eventName": "ListBuckets", ...}
{"eventName": "GetObject", ...}
 
// ❌ Does not match - kept in logs
{"eventName": "CreateUser", ...}
{"eventName": "DeleteBucket", ...}

Name: okta-system-drop
Type: Regex
Pattern: /"actor":\{[^}]*"type":"SystemPrincipal"/
Action: Drop
Scope: Source-specific (okta)

// ✅ Matches - will be dropped
{
  "eventType": "system.org.rate_limit.warning",
  "actor": {
    "type": "SystemPrincipal",
    "alternateId": "[email protected]"
  }
}
 
// ❌ Does not match - kept in logs
{
  "eventType": "user.session.start",
  "actor": {
    "type": "User",
    "alternateId": "[email protected]"
  }
}

Filter vs Drop Step

When to use Filter:

• You want reusable drop rules across multiple pipelines
• You want to match patterns anywhere in the raw log

When to use Drop:

• Simple, one-off dropping within a single pipeline
• You want to match on specific normalized fields using preconditions

Common Patterns