RunReveal
Bring Your Own CloudRunReveal Scheduler

RRSCH Scheduler Types

RRSCH runs background jobs configured in the schedule.schedulers array of config.json. Each entry selects a worker via its type field.

Schedule Section Options

OptionTypeDescriptionDefault
schedulersarrayList of scheduler configurations to runRequired
pprofstringListen address for the /metrics and /debug/pprof/ HTTP server (for example, 0.0.0.0:6065)Disabled

Common Queue Options

Queue-backed schedulers share four polling and concurrency fields:

OptionTypeDescriptionDefault
timerintegerPolling interval in milliseconds (minimum 500)500
fetchCountintegerMaximum jobs claimed per polling tick10
visibilityTimeoutintegerJob lock timeout in minutes before an unfinished job can be claimed again (minimum 5)5
parallelismintegerNumber of concurrent worker goroutines (minimum 3 unless noted)3

Available Schedulers

Detections and Queries

query

Executes workspace scheduled queries and SQL detections. Query runtime is capped at visibilityTimeout (up to a maximum of 10 minutes), and the queue visibility lock is set to the query timeout plus 30 seconds.

{
  "type": "query",
  "timer": 500,
  "fetchCount": 30,
  "visibilityTimeout": 10,
  "parallelism": 8
}

managedquery

Executes managed detection queries across subscribed workspaces. Uses the same timeout and common queue settings as query.

{
  "type": "managedquery",
  "timer": 500,
  "fetchCount": 30,
  "visibilityTimeout": 10,
  "parallelism": 8
}

async-query

Executes asynchronous ClickHouse queries and streams results to S3. While a query runs, the worker heartbeats every 5 minutes to extend its lock and polls for user-initiated cancellation.

{
  "type": "async-query",
  "resultBucket": "runreveal-async-query-results",
  "s3RoleARN": "arn:aws:iam::123456789012:role/ClickHouseAsyncQueryS3Role",
  "timer": 5000,
  "fetchCount": 10,
  "visibilityTimeout": 10,
  "parallelism": 5
}
FieldTypeDescriptionDefault
resultBucketstringS3 bucket where query results are written (required)-
s3RoleARNstringIAM role ARN assumed by ClickHouse for S3 access; falls back to ambient AWS credentials when omitted""

managed-detection-sync

Syncs managed detections from a .tar/.tar.gz archive, .zip archive, or local filesystem directory into managed_detection_configs. Uses a PostgreSQL advisory lock so only one RRSCH replica runs a sync at a time. Does not use the common queue options.

{
  "type": "managed-detection-sync",
  "syncConfig": {
    "type": "tar",
    "url": "https://api.github.com/repos/runreveal/public-detection-packs/tarball/main",
    "path": "detections/",
    "stripComponents": 1,
    "authHeader": {
      "key": "Authorization",
      "value": "$GH_SYNC_TOKEN"
    }
  },
  "pollIntervalSeconds": 3600,
  "strictMode": false
}
FieldTypeDescriptionDefault
syncConfig.typestringSource type: tar, zip, or fs (required)-
syncConfig.urlstringArchive URL (required for tar and zip)-
syncConfig.pathstringSubdirectory inside the archive, or local path on disk when type is fs"."
syncConfig.stripComponentsintegerNumber of leading path segments to strip when extracting tar or zip archives0
syncConfig.authHeaderobjectOptional { "key": "...", "value": "..." } HTTP header sent when downloading the archive-
pollIntervalSecondsintegerSeconds between sync runs; 0 runs once at startup and blocks until shutdown0
strictModebooleanAborts the sync without writing database changes if any detection file fails validationfalse

Threat Intelligence and Enrichment

threatfeed

Polls configured threat intelligence feeds and writes indicators to ClickHouse, fanning out public feeds to workspaces with custom destinations. Uses the common queue options.

{
  "type": "threatfeed",
  "timer": 30000,
  "fetchCount": 2,
  "visibilityTimeout": 3
}

enrichment

Runs scheduled managed enrichment tasks. Uses the common queue options.

{
  "type": "enrichment",
  "timer": 60000,
  "fetchCount": 1,
  "visibilityTimeout": 10
}

Notifications

notification-send

Starts a delivery consumer for each notification channel type, reaps delivered notification events older than 24 hours once per hour, and reports per-channel queue depth metrics. Does not use the common queue options.

{
  "type": "notification-send"
}

notification-retry (Deprecated)

Accepted as a no-op so existing configuration files continue to load, and logs a warning at startup. Notification retries are now handled directly by notification-send. Remove this entry from new configurations.

AI Agents and MCP

ai-agent

Evaluates scheduled AI agent triggers and enqueues asynchronous agent tasks. Uses the common queue options.

{
  "type": "ai-agent",
  "timer": 500,
  "fetchCount": 10,
  "visibilityTimeout": 10,
  "parallelism": 5
}
FieldTypeDescriptionDefault
apiBaseURLstringOverrides the API base URL; falls back to common.internalBaseURL or common.baseurl""

async-agent

Executes queued asynchronous AI agent runs and writes results to ClickHouse. Uses the common queue options.

{
  "type": "async-agent",
  "timer": 500,
  "fetchCount": 10,
  "visibilityTimeout": 10,
  "parallelism": 5,
  "maxConcurrentPerWorkspace": 3
}
FieldTypeDescriptionDefault
apiBaseURLstringOverrides the API base URL; falls back to common.internalBaseURL or common.baseurl""
maxConcurrentPerWorkspaceintegerMaximum concurrent async agent tasks per workspace (0 uses the store default)0

mcp-oauth-flow-sweep

Deletes expired, unfinished MCP OAuth consent flows from PostgreSQL.

{
  "type": "mcp-oauth-flow-sweep",
  "timer": 900000
}
FieldTypeDescriptionDefault
timerintegerSweep interval in milliseconds (minimum 60000)900000 (15m)

mcp-session-refresh

Refreshes MCP OAuth grants whose access tokens expire before the next tick.

{
  "type": "mcp-session-refresh",
  "timer": 3600000,
  "fetchCount": 200,
  "visibilityTimeout": 5,
  "parallelism": 2
}
FieldTypeDescriptionDefault
timerintegerRefresh check interval in milliseconds3600000 (1h)
fetchCountintegerMaximum sessions claimed per tick200
visibilityTimeoutintegerLock timeout in minutes5
parallelismintegerNumber of concurrent refresh workers (minimum 1)2

Health Checks, Destinations, and Maintenance

health-check

Evaluates scheduled source health checks against ClickHouse, writes results to health_check_results, and enqueues notifications. Uses the common queue options.

{
  "type": "health-check",
  "timer": 1000,
  "fetchCount": 20,
  "visibilityTimeout": 5,
  "parallelism": 5
}

destination-health-check

Evaluates scheduled destination health checks against PostgreSQL destination_errors and enqueues notifications. Uses the common queue options.

{
  "type": "destination-health-check",
  "timer": 1000,
  "fetchCount": 20,
  "visibilityTimeout": 5,
  "parallelism": 5
}

destination-migration

Keeps ClickHouse destination schemas and access policies aligned with the running binary version. Keep enabled set to false while the API server applies destination migrations at startup (common.migration.clickhouse.destinationAutoApply), so the two processes do not race.

{
  "type": "destination-migration",
  "enabled": false,
  "timer": 30000,
  "fetchCount": 10,
  "visibilityTimeout": 15,
  "recheckIntervalMinutes": 15,
  "retryBaseSeconds": 60,
  "retryMaxMinutes": 360,
  "parallelism": 4
}
FieldTypeDescriptionDefault
enabledbooleanEnables the destination migration sweepfalse
recheckIntervalMinutesintegerMinutes a destination waits after a completed pass before being rechecked15
retryBaseSecondsintegerInitial backoff in seconds for a failed destination, doubling per consecutive failure60
retryMaxMinutesintegerMaximum retry backoff in minutes360 (6h)
parallelismintegerConcurrent destination migration workers (minimum 1)4

custom-view-sync

Applies queued custom-view ClickHouse DDL tasks (create, update, delete) asynchronously with exponential backoff. Uses the common queue options.

{
  "type": "custom-view-sync",
  "timer": 2000,
  "fetchCount": 10,
  "visibilityTimeout": 5,
  "parallelism": 3,
  "maxAttempts": 5
}
FieldTypeDescriptionDefault
maxAttemptsintegerMaximum attempts before a task is marked as a terminal failure5

scim-directory-sync

Polls WorkOS directory sync events to provision and deprovision users and groups. Requires the WORKOS_API_KEY environment variable; if unset, the worker logs a warning and stays idle.

{
  "type": "scim-directory-sync",
  "timer": 600000
}
FieldTypeDescriptionDefault
timerintegerPolling interval in milliseconds (minimum 10000)10000 (10s)

batch-queue-reaper

Purges orphaned rows from batch_notification_queue that belong to deleted/inactive destinations or have exceeded the batch TTL. Does not use the common queue options.

{
  "type": "batch-queue-reaper",
  "pollIntervalSeconds": 300,
  "destinationGraceMinutes": 60,
  "batchTTLHours": 96
}
FieldTypeDescriptionDefault
pollIntervalSecondsintegerInterval in seconds between reaper runs300 (5m)
destinationGraceMinutesintegerGrace period in minutes before reaping rows for an inactive or deleted destination60 (1h)
batchTTLHoursintegerMaximum age in hours before queued batch notifications are reaped96 (4d)

source-delete

Permanently deletes sources that have passed their scheduled delete_at timestamp, including their logs in ClickHouse. Does not use the common queue options.

{
  "type": "source-delete",
  "pollIntervalSeconds": 300
}
FieldTypeDescriptionDefault
pollIntervalSecondsintegerInterval in seconds between deletion checks300 (5m)

Complete Configuration Example

{
  "common": {
    "chdsn": "clickhouse://user:password@localhost:9000/dbname",
    "pgdsn": "postgres://user:password@localhost:5432/dbname?sslmode=verify-full",
    "baseurl": "https://api.example.com",
    "appBaseURL": "https://app.example.com",
    "secretsConfig": {
      "type": "none"
    }
  },
  "schedule": {
    "pprof": "0.0.0.0:6065",
    "schedulers": [
      {
        "type": "query",
        "timer": 500,
        "fetchCount": 30,
        "visibilityTimeout": 10,
        "parallelism": 8
      },
      {
        "type": "managedquery",
        "timer": 500,
        "fetchCount": 30,
        "visibilityTimeout": 10,
        "parallelism": 8
      },
      {
        "type": "notification-send"
      },
      {
        "type": "threatfeed",
        "timer": 30000,
        "fetchCount": 2,
        "visibilityTimeout": 3
      },
      {
        "type": "enrichment",
        "timer": 60000,
        "fetchCount": 1,
        "visibilityTimeout": 10
      },
      {
        "type": "ai-agent",
        "timer": 500,
        "fetchCount": 10,
        "visibilityTimeout": 10,
        "parallelism": 5
      },
      {
        "type": "async-agent",
        "timer": 500,
        "fetchCount": 10,
        "visibilityTimeout": 10,
        "parallelism": 5
      },
      {
        "type": "health-check",
        "timer": 1000,
        "fetchCount": 20,
        "visibilityTimeout": 5,
        "parallelism": 5
      },
      {
        "type": "destination-health-check",
        "timer": 1000,
        "fetchCount": 20,
        "visibilityTimeout": 5,
        "parallelism": 5
      },
      {
        "type": "destination-migration",
        "enabled": false,
        "timer": 30000,
        "fetchCount": 10,
        "visibilityTimeout": 15,
        "recheckIntervalMinutes": 15,
        "retryBaseSeconds": 60,
        "retryMaxMinutes": 360,
        "parallelism": 4
      },
      {
        "type": "custom-view-sync",
        "timer": 2000,
        "fetchCount": 10,
        "visibilityTimeout": 5,
        "parallelism": 3
      },
      {
        "type": "async-query",
        "resultBucket": "runreveal-async-query-results",
        "timer": 5000,
        "fetchCount": 10,
        "visibilityTimeout": 10,
        "parallelism": 5
      },
      {
        "type": "managed-detection-sync",
        "syncConfig": {
          "type": "tar",
          "url": "https://api.github.com/repos/runreveal/public-detection-packs/tarball/main",
          "path": "detections/",
          "stripComponents": 1,
          "authHeader": {
            "key": "Authorization",
            "value": "$GH_SYNC_TOKEN"
          }
        },
        "pollIntervalSeconds": 3600
      },
      {
        "type": "scim-directory-sync",
        "timer": 600000
      },
      {
        "type": "mcp-oauth-flow-sweep",
        "timer": 900000
      },
      {
        "type": "mcp-session-refresh",
        "timer": 3600000,
        "fetchCount": 200,
        "parallelism": 2
      },
      {
        "type": "batch-queue-reaper",
        "pollIntervalSeconds": 300,
        "destinationGraceMinutes": 60,
        "batchTTLHours": 96
      },
      {
        "type": "source-delete",
        "pollIntervalSeconds": 300
      }
    ]
  }
}