RRQ (RunReveal Queue) Configuration Guide

RRQ ingests data from your configured sources, runs it through the processing pipeline (parsing, enrichment, detections), and writes it to ClickHouse and any other configured destinations. This guide covers how RRQ is configured, and how to scale and tune it as your data volume grows.

Table of Contents

• Deploying and Configuring RRQ
• Configuration Reference
• Scaling and Performance Tuning
• Running RRQ Standalone
• Source Types

Deploying and Configuring RRQ

If you're running RunReveal on Kubernetes, RRQ is deployed via our Helm chart. The chart renders one Deployment (with its own ConfigMap) per entry under rrq.groups in your values file. Splitting sources across groups is how you isolate and independently scale different workloads — see Scaling and Performance Tuning below.

rrq:
  # Defaults merged into every group's config.json
  configJson:
    stream:
      parallelism: 8
    common:
      clickhouse:
        flushLength: 300000
        flushFrequency: 30
        flushParallelism: 3
 
  groups:
    main:
      replicas: 2
      resources:
        requests:
          cpu: 500m
          memory: 4Gi
        limits:
          cpu: '1'
          memory: 8Gi
      configJson:
        stream:
          sources:
            - type: cloudtrail
              sqsQueue: your_cloudtrail_queue
              parallelism: 15

Each group's configJson is deep-merged on top of rrq.configJson, so a group only needs to specify what's different for it — everything else falls back to the shared defaults.

If you're not deploying via the Helm chart, the same configuration keys apply — you write them directly into a config.json file and point the rrq binary at it (see Running RRQ Standalone).

Configuration Reference

The keys below are what actually end up in a group's config.json, whether it got there through Helm values or a hand-written file.

{
  "common": {
    "chdsn": "clickhouse://user:password@localhost:9000/dbname",
    "baseurl": "https://your-domain.com",
    "pgdsn": "postgres://user:password@localhost/dbname?sslmode=disable",
    "memoryLimitMB": 6500,
    "memoryLimitHeadroomPercentage": 15,
    "clickhouse": {
      "flushLength": 300000,
      "flushFrequency": 30,
      "flushParallelism": 3
    }
  },
  "stream": {
    "pprof": "0.0.0.0:6060",
    "parallelism": 4,
    "watchdogTimeout": 300,
    "sources": [
      {
        "type": "cloudtrail",
        "sqsQueue": "your_cloudtrail_queue",
        "parallelism": 15
      }
    ]
  }
}

Common Configuration Options

common Object

stream Object

Per-source options

Each entry in stream.sources accepts a parallelism field alongside its type-specific settings (queue name, credentials, etc.) — see Source parallelism for tuning guidance. Backfills are configured the same way, as a sources entry with "type": "backfill-process", rather than as a separate top-level flag.

Scaling and Performance Tuning

RRQ's pipeline looks like this:

Sources (queue readers / pollers / webhooks)
  → Processor pool (configurable parallelism)
    → Pipeline (transforms, enrichments, detections)
      → Destinations (batched writes to ClickHouse, S3, etc.)

There are three independent places to scale: how many sources you run and how fast each reads, how many events the processor pool handles concurrently, and how batches are flushed to destinations. Tuning them together — rather than just turning up one number — is what keeps a growing deployment stable.

Splitting sources into groups

Every group defined under rrq.groups becomes its own Deployment, with its own resource allocation and its own subset of sources. This is the main lever for scaling out: put high-volume or high-latency sources in their own group so they don't compete with everything else for processor time.

A source that reads slowly (a high-latency object storage bucket, a rate-limited API) can end up dominating a shared processor pool even with a low event rate, because RRQ's internal fan-in gives read-ready sources priority. Isolating it into its own group with its own parallelism and resource budget keeps a slow source from starving fast ones.

Processor parallelism

stream.parallelism controls how many goroutines process events through the pipeline for a given group. It defaults to NumCPU - 1 if unset.

• Increase it when queue depth is growing but CPU usage is low — the processor pool is idle waiting on something else (usually a slow destination or slow source).
• Decrease it when CPU is consistently above 80% and destabilizing other services on the same node.

Source parallelism

Each source's parallelism field controls how many goroutines that source uses to read messages/files concurrently. More is not always better — most sources process one message start-to-finish per goroutine, including any blocking network I/O, and don't acknowledge it until processing completes.

If a source's per-message processing time is close to (or exceeds) its queue's visibility timeout, running it with high parallelism makes things worse, not better: more in-flight messages means more of them time out and get redelivered, which increases load further. As a rule of thumb:

• Fast, low-latency sources (same-region object storage, low-latency APIs): parallelism of 10-20 is fine.
• Slow or high-latency sources (cross-region buckets, rate-limited APIs, very large files): keep parallelism low (1-5) and prefer adding processor parallelism or a dedicated group over raising it.

Memory limits and backpressure

When running under Kubernetes, set resources.limits.memory on the group and set memoryLimitHeadroomPercentage. The chart wires the container's actual memory limit into RRQ's config automatically (via the Kubernetes downward API), so RRQ knows its own ceiling without you having to keep two numbers in sync:

groups:
  main:
    memoryLimitHeadroomPercentage: 15
    resources:
      requests:
        memory: 4Gi
      limits:
        memory: 8Gi

With this set, RRQ tracks its own memory usage and applies backpressure — pausing new event intake, forcing GC, and retrying with backoff — before it hits the container's hard limit. Without it, the only thing standing between a burst of traffic and an OOM kill is the container runtime, which doesn't give RRQ a chance to shed load gracefully first. memoryLimitHeadroomPercentage defaults to 0 (no headroom) if unset, so it's worth setting explicitly rather than relying on the default.

The container's requests.memory is also wired in automatically as RRQ's GOMEMLIMIT (Go's own soft heap target), which is a second, independent lever on the same problem: it makes the Go runtime garbage-collect more proactively as usage approaches that number, rather than only the hard limits.memory ceiling.

There's no universally right headroom percentage — it depends on how bursty a group's sources are and how large its flush batches are. Start with something conservative, watch RRQ's logs for how often the limiter actually engages, and adjust from there rather than guessing a number up front.

Destination flush tuning

common.clickhouse.flushLength, flushFrequency, and flushParallelism control how events are batched before being written to ClickHouse:

• Larger flushLength: better write throughput, more memory held per in-flight batch (flushLength × flushParallelism × average event size).
• Shorter flushFrequency: lower latency before events are queryable, less efficient writes.
• Higher flushParallelism: more throughput for high event rates, more concurrent ClickHouse connections.

If you're memory-constrained, reduce flushLength and flushParallelism together rather than one in isolation — the memory cost scales with their product.

Autoscaling (HPA/VPA)

The Helm chart supports both horizontal (HPA) and vertical (VPA) autoscaling per group:

rrq:
  hpa:
    enabled: true
    minReplicas: 2
    maxReplicas: 8

HPA scales on CPU (75% utilization target) and memory (80% utilization target) by default. This works well for CPU-bound processing groups, but be aware that a group dominated by I/O-bound sources (waiting on slow network calls, not burning CPU) may not trigger scale-up even while its queue backs up — watch queue depth as well as CPU/memory when deciding whether a group needs more replicas or is just under-provisioned on parallelism.

VPA (vpa.enabled: true) is useful when you don't yet know a group's steady-state resource needs and want Kubernetes to right-size requests/limits for you over time, rather than guessing up front.

Running RRQ Standalone

If you're not deploying via the Helm chart, RRQ can be run directly from a config.json file:

rrq run --config config.json

Command Line Arguments

Environment Variables

Configuration values can reference environment variables by prefixing the value with $. For example:

{
  "common": {
    "pgdsn": "$POSTGRES_DSN"
  }
}

Next Steps

• See Deployment for the full Helm chart install process
• See Source Types for detailed configuration of each source type
• Refer to the monitoring documentation for operational best practices