Federated Search

Federated Search lets you query data that lives in an external S3-compatible bucket directly, without ingesting it into RunReveal first. It is a kind of Custom View — a federated view — that reads from your bucket via ClickHouse's s3() table function at query time instead of from the runreveal.logs table.

Use it to search cold archives, query large datasets you already store in object storage, or explore public datasets, all from the same Search Explorer and SQL detections you already use.

Internal vs. Federated Views

How It Works

External bucket → Federated view → Search and detections

1. Files stay in your bucket — NDJSON, CSV, TSV, Parquet, or ORC, with optional compression and path partitioning.
2. RunReveal builds a federated view — ClickHouse's s3() reads matching objects at query time. NDJSON views use JSON path columns; structured formats use the file's native schema.
3. You query the view — Use Search Explorer or SQL detections against the workspace-prefixed table name. No data is copied into RunReveal.

When you query a federated view, ClickHouse reads the matching objects from your bucket on demand, applies your column definitions, and returns rows — nothing is copied into RunReveal. Credentials are encrypted at rest and are never returned in API responses.

Availability

Supported Providers

Supported Formats

Supported compression options: auto-detect (default), none, gzip, zstd, lz4, brotli, and xz.

Creating a Federated View

Connection Settings

Authentication

IAM Role Assumption (AWS S3)

For AWS S3 buckets we recommend authenticating with an IAM Role ARN instead of access keys. RunReveal runs Federated Search queries on ClickHouse Cloud, which uses role chaining to assume a role you create in your AWS account. You grant that role read access to your bucket and add a trust relationship that allows ClickHouse Cloud's service principal to assume it. No long-lived credentials ever leave your account.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::426924874929:role/CH-S3-indigo-xo-42-ue2-29-Role"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject"
      ],
      "Resource": "arn:aws:s3:::YOUR_BUCKET/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Resource": "arn:aws:s3:::YOUR_BUCKET"
    }
  ]
}

Partitioning

If your bucket is organized into key=value/ path segments (for example date=2024-01-01/hour=12/), you can prune the files ClickHouse scans so queries only touch the objects they need. There are two mutually exclusive approaches:

s3://my-bucket/logs/date=2024-01-01/hour=12/events.parquet

Querying a Federated View

Federated views follow the same naming convention as internal views: {workspace_name}_{your_view_name}. Query them like any other table.

-- Search archived events in your bucket
SELECT
  receivedAt,
  actor_email,
  src_ip,
  event_name
FROM workspace_archived_auth_logs
WHERE event_name = 'login_failed'
ORDER BY receivedAt DESC
LIMIT 100;

For partitioned data, filter on a partition column to reduce the number of files scanned:

SELECT *
FROM workspace_archived_auth_logs
WHERE date = '2024-01-01'
LIMIT 100;

You can also use federated views in SQL detections, just like internal custom views. See Using Custom Views in Detections.

Use Cases

• Cold archive search: keep older logs in cheap object storage and query them on demand without re-ingesting.
• Bring your own data lake: query data you already export to S3/R2/GCS from other systems.
• Public datasets: point an anonymous (NOSIGN) view at a public bucket to explore open data.
• Cost control: avoid storing rarely queried data in the workspace database while keeping it searchable.

Limitations

• Available on Pro and Enterprise plans only. Like all custom views, it is not available for BYODB workspaces.
• Query performance depends on bucket size, file format, compression, and partitioning. Large unpartitioned scans can be slow and costly — prefer columnar formats (Parquet/ORC) and partition pruning for big datasets.
• Endpoints must use https (except MinIO, which may use http for local development).
• Federated views cannot be used in Sigma streaming detections (the same limitation as internal custom views).

Next Steps

• Custom Views: column configuration, JSON paths, and querying basics.
• Search: explore your views in the Search Explorer.
• Detections: alert on federated view data with SQL detections.
• Destinations: route ingested events to object storage you can later search with federated views.