Google Cloud Pub/Sub Queue Ingestion

Use this guide when your GCP logs are delivered to a Pub/Sub topic and you want RunReveal to pull messages directly from a subscription. This ingest method is a good fit when log payloads are already on Pub/Sub and you want near-real-time delivery without routing through Cloud Storage.

GCP Logs and Generic GCP PubSub sources both support this ingest method. GCP Logs applies Google Cloud audit-log normalization; Generic GCP PubSub stores the raw message payload.

Overview

The direct Pub/Sub flow works like this:

A GCP log router sink forwards logs to a Pub/Sub topic.
A pull subscription holds messages until RunReveal reads them.
RunReveal polls the subscription, normalizes each log entry, and sends events to the pipeline.

Unlike Google Cloud Storage ingestion, there is no bucket and RunReveal does not relay full message bodies through an internal object-notification queue.

Use a Pull subscription. Push subscriptions that target a RunReveal webhook are covered in the GCP Logs webhook setup.

Resource Setup

Service account

Create or choose a service account in the GCP project that owns the Pub/Sub subscription. Grant it the Pub/Sub Subscriber role on the subscription RunReveal will poll.

See Google Cloud Storage ingestion — Service Account Creation for the same service-account workflow if you are new to GCP IAM setup.

Pub/Sub topic and pull subscription

In the GCP console, open Pub/Sub and create a topic (for example, runreveal-logs).
Create a subscription on that topic.
Set Delivery type to Pull.
Save the full subscription resource path (for example, projects/my-project/subscriptions/runreveal-logs).

Grant the service account Pub/Sub Subscriber on the subscription.

Log router sink

Route Cloud Logging output to the topic:

Open Log Router.
Create a sink with Cloud Pub/Sub topic as the destination.
Select the topic created above.
Add inclusion or exclusion filters as needed.

For screenshots and sink options, see the log-router steps in GCP Logs — Webhook Ingestion. The sink configuration is the same; only the subscription type differs (pull here vs. push for webhook).

Authentication Setup

RunReveal supports the same GCP authentication options as GCS ingestion:

Service account key — JSON key for the service account above.
Workload identity federation — AWS IAM role that impersonates the GCP service account.

Follow Google Cloud Storage ingestion — Authentication Setup for step-by-step instructions.

RunReveal source setup

Open Connect a source.
Choose GCP Logs (or Generic GCP PubSub for unstructured payloads).
Select Google Cloud Pub/Sub Queue as the ingest method.
Enter the subscription path and authentication details.
Run Test to confirm RunReveal can pull from the subscription.

On-prem and BYOC stream configuration

SaaS deployments enable this path automatically. For on-prem, BYOC, and other self-hosted stream workers, add a dedicated source entry to your stream config in addition to the existing GCP object-storage processor:

{
  "type": "gcp-pubsub-logs"
}

Keep { "type": "gcp" } for sources that ingest via Cloud Storage. Keep { "type": "gcp-queue" } under queueReaders for GCS bucket-notification forwarding.

Example:

"queueReaders": [
  { "type": "gcp-queue" }
],
"sources": [
  { "type": "gcp" },
  { "type": "gcp-pubsub-logs" }
]

{ "type": "gcp-pubsub-logs" } is required for GCP Logs sources that use the Pub/Sub Queue ingest method. It is separate from { "type": "gcp-pubsub" }, which serves Generic GCP PubSub sources.

GCP Logs - Source overview and comparison of ingest methods
Google Cloud Storage ingestion - Batch logs through a GCS bucket instead of direct Pub/Sub pull

On this page