Azure Blob Storage
The current process for ingesting logs from Azure blob storage involves 3 steps.
- Send your logs/events to a storage account to be saved as blob storage objects.
- A Blob Created event is triggered and adds a message to a Storage Queue.
- Every minute, RunReveal will subscribe to your storage queue, pull any new notifications, and process the objects that are listed.
When setting up the Azure resources, each unique source will require its own storage queue in order to process events.
RunReveal recommends using a separate storage account for each source but is not required.
In order to setup this format you will need to create the following pieces in Azure:
- Add a new App Registration to your Entra account. (This registraion can be reused between different sources)
- Create a new Client Secret for this new app.
- Create a new Storage Account.
- Create an Event to notify on Blob Created event types and forward the notification to a Storage Queue.
- Provide access to the created app to read blob objects and process storage queue messages.
For steps 3-5 RunReveal provides an Azure Deploy Template that can be used to simplify setup.
App Registration
The first step is to create a new App Registration. Navigate to the Microsoft Entra ID (opens in a new tab) service and choose Add -> App Registration.
Give the app a name and fill in the remaining settings for your organization. Once done click Register.
You should now be presented with the details of the new app. Copy and save the Application (client) ID and the Directory (tenant) ID as these will be needed when setting up a RunReveal source.
If you plan to use the provided deploy template click on the link for the Managed application in local directory, this will open a properties page where you will need to copy the listed Object ID.
App Secret
Once the app has been registered you will need to create a client secret to provide to RunReveal for access.
Back on the main app registration page under the Manage category click on 'Certificates & secrets'.
From this screen click on the 'Client secrets' option and select New client secret. Give the secret a name and choose an expiration date.
Once created copy the secret value and save it for later, this is needed when creating the source in RunReveal.
Set a reminder for the expiration date of this secret. You will need to generate a new secret and update your RunReveal source before the expiration date to make sure RunReveal continues to have access.
Storage Account Setup
For each logging type (Activity Logs, Entra, etc) a new storage account must be created with the same setup. The easiest way to set up your storage account is to use our deployment template. This will automatically setup the storage account, the object notifications, and provide the correct permissions for the created app.
Deployment Template
Click on the to Azure button to load the template in the Azure portal. From there choose your Subscription and Resource group where this storage account should reside.
Give the storage account a name and paste the App's Object ID that was copied earlier. Wait for the deployment to complete, and click on the Outputs menu item.
This will list the storageAccountName and storageQueueName that were created by the template. These values will be needed when setting up the RunReveal source.
Azure displays multiple App object IDs, if the wrong one is used you may get a deployment error with the template. All of the resources will have been created but the IAM permissions will not have been set. You can manually apply the needed permissions to the storage account if this happens.
Manual Setup
If you already have logs being stored in a storage account or want to change other setup settings follow these steps.
Storage Account
Navigate to the Storage account (opens in a new tab) creation page and create a new storage account to store your log files.
The only settings that we recommend is to select the Standard performance option and to select the Hot access tier.
Event Subscription
In your new storage account resource, go to the Events menu and create a new Event Subscription.
Fill in the name fields for the subscription and system topic name and select Event Grid Schema as the event schema. Change the event types to only filter for Blob Created events.
Select Storage Queue as your endpoint type and configure a new queue. For the easiest setup, we recommend putting the storage queue in the same storage account.
Make sure to take note of the name of the storage queue that was created as this will be needed in the RunReveal source.
IAM
Go to the access control (IAM) screen for your storage account and add a new Role assignment.
You will need to add two separate roles the first being Storage Blob Data Reader and the second Storage Queue Data Message Processor.
For both of these the member should be the App that was registered earlier.
RunReveal Source Setup
Sources that allow you to use Azure storage accounts to ingest logs will all have the same setup.
Navigate to the Connect a source (opens in a new tab) page and find the source you are adding.
Once open, select the Azure Storage Account ingest method and fill in the fields.
You will need the tenant id, client id, and the client secret for the app that was created. You will also need the storage account name and storage queue name where the notifications are being sent.
The storage account name that is provided should be the location of the storage queue, not necessarily the location of the blob objects. The storage account of the blob object is listed in the notification that is created.
Final Steps
At this point all of the pieces are in place for RunReveal to access logs stored in the bucket. Continue to the source docs for specific instructions on how to get your logs into the storage account.