Google Cloud Platform logs

GCP logs are ingested by routing your projects or organization logs to a storage bucket and allowing RunReveal access to read them (using a service account).

We currently support two methods for getting logs into a storage bucket and notifying RunReveal about the new logs.

Method 1, routes the logs, using a logging sink, directly into a storage bucket. Then you create an event notification for new objects that will forward notifications to a pub/sub topic. RunReveal will poll this topic and process the objects based on the notifications.

Method 2, routes the logs, using a logging sink, to a pub/sub topic. This topic has a subscription created that writes the logs into a storage bucket. RunReveal will then poll the storage bucket for new logs every minute and process the objects that are returned.

You should use method 1 if you are worried about pub/sub costs or if you are ok with logs being delayed by about 1 hour.

You should use method 2 if you want logs ingested in as little as 1-2 minutes and are ok with a slightly higher pub/sub cost.

Setup

Setting up this GCP source requires a few steps that will be detailed below. Regardless of which method you choose, steps 1 & 2 will need to be performed for both.

1. Create a service account with a JSON key.
2. Create a storage bucket giving read access to the created service account.

Method 1

3. Create a GCS bucket notification that writes to a new pub/sub topic.
4. Create a pub/sub subscription that RunReveal can use to poll for new events.
5. Setup log router to send logs to the created storage bucket.
6. Create a RunReveal GCP source and provide us with the details of your setup.

Method 2

3. Create a pub/sub topic that forwards events to the storage bucket.
4. Setup log router to send logs to the created pub/sub topic.
5. Create a RunReveal GCP source and provide us with the details of your setup.

1. Service Account Creation

Choose a project that you would like the storage bucket, pub/sub topic, and service account to reside and switch to that project in the GCP Console.

Create a new service account (or use an existing one) by navigating to the service account creation page, https://console.cloud.google.com/iam-admin/serviceaccounts/create (opens in a new tab)

Give the service account a name, id, and a description. Make sure to copy the generated service email address as it will be needed in a future steps.

Once created, open the service account and navigate to the keys menu. Create a new JSON private key, this will be uploaded to RunReveal to allow access to your GCP resources.

2. Storage Bucket Creation

Navigate to the GCP cloud storage (opens in a new tab) page and create a new bucket.

Give the bucket a name, and fill in the remaining settings based on your needs.

Once the bucket is created, go to the permission settings for it and click on Grant Access. Add the email address for the service account that was created in step 1, and assign the Storage Object Viewer role to it.

Method 1: GCS bucket notifications

3. GCS Bucket Notification Creation

After the bucket is created you will need to create a pub/sub notification when new objects are added to the bucket. Currently GCP doesn't offer a way to do this in the UI, but following along with their guide you can accomplish this with the gcloud cli tool. https://cloud.google.com/storage/docs/pubsub-notifications (opens in a new tab)

4. Pub/Sub Subscription Creation

Once the notification has been added a new topic should have been created. Create a new pull subscription for the topic so RunReveal will be able to access the notifications.

Give the new subscription an id and select the topic that was created when setting up the bucket notification. Choose Pull as the delivery type and set the rest of the settings according to your preferences.

Now that you have a topic and subscription created you will need to grant the service account, that you created in step 1, the Pub/Sub Subscriber role. Click on the Subscriptions menu option and select the subscription that was created. In the permissions info panel add a new principal and enter the service account email and select Pub/Sub Subscriber as the role. This will allow RunReveal to subscribe to this topic and process the new object notifications.

5. GCP Log Router Sink Setup

All of the pieces are now in place to send GCP logs to your storage bucket. Navigate to the GCP Log router (opens in a new tab) setup page to create a new sink. You can setup a logging sink for a single project or your entire organization and forward logs to the bucket created in step 2. Create a new logging sink giving it a name, choose Cloud Storage bucket as the sink service. Enter the bucket information that was created in step 2 as the destination bucket. Add an optional inclusion or exclusion filter to limit the logs that are forwarded by this logging sink.

6. RunReveal GCP source

Now that you have all of the GCP pieces created you can add a GCP source (opens in a new tab) in RunReveal.

Provide a name for the source and select which notification type you are using. For method 1 select GCS Object Notification. Fill in the name of the subscription that was created in step 4. Upload the credential file that was downloaded from step 1 or paste the contents from it into the credential field.

Verify your settings to make sure that RunReveal has access to subscribe to the subscription and read the objects in your bucket.

Method 2: GCS Bucket Polling

3. Pub/Sub Topic Setup

Navigate to the GCP Pub/Sub (opens in a new tab) setup page and create a new topic.

When creating a new topic, give it a name and you can remove the default subscription. Instead select Backup message data to Cloud Storage. Select the bucket that was created in step 2. If GCP warns about missing permissions follow their instructions to grant them. For the remaining settings, make sure to select Text as the file format. Prefix, suffix, and file batching can be set based on your needs.

Now that you have a topic and subscription created you will need to grant the service account, that you created in step 1, the Pub/Sub Viewer role. Click on the Subscriptions menu option and select the subscription that was created with the topic. In the permissions info panel add a new principal and enter the service account email and select Pub/Sub Viewer as the role. This will allow RunReveal to read the settings you have chosen when creating this subscription.

4. GCP Log Router Sink Setup

All of the pieces are now in place to send GCP logs to your storage bucket. Navigate to the GCP Log router (opens in a new tab) setup page to create a new sink. You can setup a logging sink for a single project or your entire organization and forward logs to the bucket created in step 2. Create a new logging sink giving it a name. Choose Cloud Pub/Sub topic as your sink destination and select the option to use a topic in a project. It will have you supply the project ID that the topic that was created under, and the topic ID that were created in step 3. Add an optional inclusion or exclusion filter to limit the logs that are sent to the pub/sub topic.

5. RunReveal GCP source

Now that you have all of the GCP pieces created you can add a GCP source (opens in a new tab) in RunReveal.

Provide a name for the source and select which notification type you are using. For method 2 select GCS Bucket Polling. Fill in the name of the subscription that was created in step 3. Upload the credential file that was downloaded from step 1 or paste the contents from it into the credential field.

Verify your settings to make sure that RunReveal has access to view the subscription details and list/read the objects in your bucket.

Ingest

Your GCP source is now setup and logs should start showing up in RunReveal once the logging sink has written them to the storage bucket. For additional questions, reach out to the RunReveal team for assistance.