Sources
Object Storage
Google Cloud Storage

Google Cloud Storage Ingestion

This guide explains how to set up log ingestion from Google Cloud Storage (GCS) using either service account keys or workload identity federation with AWS.

Overview

The process for ingesting logs from Google Cloud Storage (GCS) involves 3 steps:

  1. Send your logs/events to a storage bucket.
  2. An Object Created event is triggered and adds a message to a pub/sub topic.
  3. Every minute, RunReveal will poll a pub/sub subscription on the created topic, pull any new notifications, and process the objects that are returned.

Resource Setup

Setting up a GCS ingestion source requires the following steps:

  1. Create a service account that will have access to the resources.
  2. Create a storage bucket giving read access to the created service account.
  3. Create a GCS bucket notification that writes to a new pub/sub topic.
  4. Create a pub/sub subscription that RunReveal can use to poll for new events giving subscribe access to the service account.
  5. Give RunReveal authentication access to your service account using either service account keys or workload identity federation.
  6. Create a RunReveal source and provide us with the details of your setup.

Service accounts require the following permissions for RunReveal to process logs:

  1. Storage Object Viewer on the storage bucket where logs are stored.
  2. Pub/Sub Subscriber on the pub/sub subscription where event notifications are sent.

Service Account Creation

Choose a project for the storage bucket, pub/sub topic, and service account. Create a new service account (or use an existing one) at https://console.cloud.google.com/iam-admin/serviceaccounts/create (opens in a new tab)

Give the service account a name, id, and a description. Copy the generated service email address for future steps.

gcp service account creation

Storage Bucket Creation

Create a new bucket in the GCP cloud storage (opens in a new tab) page.

If adding a retention policy to the bucket, make sure it's no less than 4 hours to ensure RunReveal has sufficient time to import all logs before object deletion.

Add the service account email to the bucket's permissions with the Storage Object Viewer role.

bucket permissions

GCS Bucket Notification Creation

Create a pub/sub notification for new objects added to the bucket using the GCP cloud shell:

gcloud storage buckets notifications create gs://BUCKET_NAME --topic=TOPIC_NAME -e OBJECT_FINALIZE

Replace BUCKET_NAME with your bucket name and TOPIC_NAME with your desired topic name.

Pub/Sub Subscription Creation

Create a new pull subscription for the topic:

  1. Give the subscription an ID and select the topic created in the previous step.
  2. Choose Pull as the delivery type.
  3. Set other settings according to your preferences.

topic creation

Grant the service account the Pub/Sub Subscriber role on the subscription.

topic creation

Save the subscription name for setting up your RunReveal source.

Authentication Setup

RunReveal offers two authentication methods for GCS access, service account keys where you will provide RunReveal with a JSON config file containing a private key, or using workload identity federation where you give an AWS IAM role access to authenticate as your chosen service account.

Service Account Keys

  1. Open the service account and navigate to the keys menu.
  2. Create a new JSON private key.
  3. Download the new key to provide to RunReveal when setting up your source.

gcp service account key

RunReveal Source Setup

Navigate to the Connect a source (opens in a new tab) page and find the source you are adding. Select the Google Cloud Storage Bucket ingest method and fill in the fields. Regardless of the authentication method you will need to provide the subscription path created in the Pub/Sub Subscription Creation step.

Service Account Keys

If authenticating with service account keys select the Service account key method and paste or select the private key file that was downloaded from GCP for the service account.

gcs-setup

Workload Identity Federation

If authenticating with identity federation you will need to provide the AWS role arn that RunReveal will assume. Leave this field blank if you plan to use the RunReveal provided role.

If your role uses an external ID for authentication make sure the same value is pasted in the External ID field.

Copy or select the config file that was downloaded in the Grant Service Account Access steps.

gcs-setup2

Final Steps

At this point, all pieces are in place for RunReveal to access logs stored in the bucket. Continue to the source docs for specific instructions on how to get your logs into the GCS bucket.

Azure Blob StorageR2