Custom Source

Connect any JSON or NDJSON log source to RunReveal — even if there isn’t a dedicated source type for it. The Custom Source wizard walks you through selecting an ingest method, configuring the connection, previewing sample data, and optionally mapping fields to a normalized schema.

When to use Custom Source

You have logs in JSON or NDJSON format that don’t match an existing RunReveal source type.
You want to bring your own schema and optionally map fields to RunReveal’s normalized columns.
You need flexibility to ingest from a webhook, S3, GCS, Azure Blob, R2, or a self-hosted MinIO instance.

Wizard walkthrough

The Custom Source wizard in the RunReveal dashboard has four steps. Steps 1–3 are required; Step 4 is optional.

Choose ingest method

Navigate to Sources → Add Source and select the Custom Source tile. Pick how your data will reach RunReveal:

Method	Best for
Webhook	Applications that can POST JSON events in real time.
AWS S3 Bucket	Log files already landing in S3 with SNS notifications.
AWS S3 (Custom SQS)	S3 ingestion using your own SQS queue instead of RunReveal’s SNS topic.
Google Cloud Storage	Log files in GCS with Pub/Sub notifications.
Azure Blob Storage	Log files in Azure Blob with storage queue notifications.
Cloudflare R2	Log files in R2 with Workers Queue notifications.
MinIO	Self-hosted or S3-compatible storage with PostgreSQL-backed event notifications.

Click a card to select it, then proceed to the next step.

Configure connection

Enter a Source Name (and optional display name).
Fill in the connection settings for your chosen ingest method — see the Ingest method setup tabs below.
Click Verify Settings. RunReveal tests the connection and fetches sample records.

⚠️

If you need to switch ingest methods, click Change next to the method badge. This resets your connection settings for that step.

Preview data

After verification succeeds, review the sample records that RunReveal pulled from your source. Confirm the JSON structure and field names look correct.

You can click Preview Data to inspect the records, or Skip Preview to move on.

Field mapping (optional)

Map JSON paths from your data (e.g. $.user.id, $.timestamp) to RunReveal’s normalized fields (e.g. actor.user.id, eventTime). This creates a custom view and, when normalized fields are mapped, a transform.

Click Suggest mappings to auto-detect fields from the sample. AI-assisted suggestions are used when available.
Click Skip to create the source without field mappings — data is still ingested as raw JSON.

After clicking Connect Source, a progress dialog shows each step being created (source, custom view, transform). When everything succeeds, you can open the new view in Explorer or return to the Sources list.

Ingest method setup

Each ingest method has its own connection settings. Select the tab that matches the method you chose in Step 1.

Webhook

After you save the source, RunReveal generates a unique Webhook URL (and an optional bearer token). Configure your application or log shipper to send events to that URL.

Required configuration on your side:

Setting	Value
URL	The HTTPS webhook URL from RunReveal.
Method	`POST`
Content-Type	`application/json`
Body	A JSON object, JSON array, or NDJSON (one object per line).
Authorization (optional)	`Bearer YOUR_TOKEN` — set this header if you enabled token auth.

Example curl request:

curl -X POST https://ingest.runreveal.com/sources/s/YOUR_SOURCE_SLUG \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -d '{"event": "user.login", "user": "[email protected]", "ts": "2025-06-01T12:00:00Z"}'

AWS S3 Bucket

For AWS S3 Bucket (RunReveal-managed SNS), point your S3 bucket’s event notifications to RunReveal’s SNS topic:

arn:aws:sns:<REGION>:253602268883:runreveal_generic

Replace <REGION> with your bucket’s AWS region (e.g. us-east-1, eu-west-1).

For the full walkthrough — including bucket policy, IAM permissions, and event notification setup — follow the AWS S3 Bucket guide.

AWS S3 with Custom SQS

If you prefer to use your own SQS queue for S3 event notifications (instead of RunReveal’s SNS topic), follow the AWS S3 Bucket with Custom SQS guide.

In the wizard, enter the SQS queue URL and any required credentials.

MinIO

Ingest from a self-hosted MinIO (or any S3-compatible) server. MinIO uses a PostgreSQL notification table to alert RunReveal about new objects, instead of SNS or Pub/Sub.

Connection details

Enter the following in the wizard:

Field	Description	Example
Endpoint	MinIO server address including port.	`minio.example.com:9000`
Bucket Name	The bucket containing your log files.	`my-logs`
Region (optional)	MinIO region if configured. Defaults to `us-east-1`.	`us-east-1`
Compression (optional)	File compression format. Defaults to `auto` (auto-detect).	`auto`, `gzip`, `none`
Access Key ID	MinIO access key with read access to the bucket.	`minioadmin`
Secret Access Key	The corresponding secret key.	—

Configure bucket notifications

MinIO must write s3:ObjectCreated:* event notifications to a PostgreSQL table that RunReveal can poll. Set this up using the MinIO client (mc):

Add a PostgreSQL notification target

mc admin config set myminio notify_postgres:runreveal \
  connection_string="host=db.example.com port=5432 dbname=minio_events user=minio password=secret sslmode=require" \
  table="runreveal_notifications" \
  format="namespace"

Restart MinIO

mc admin service restart myminio

Add a bucket event notification

mc event add myminio/my-logs arn:minio:sqs::runreveal:postgresql --event put

This writes a row to runreveal_notifications every time an object is created in the bucket. RunReveal polls this table to discover and download new files.

The PostgreSQL notification table must be accessible to RunReveal’s backend. If you’re connecting an external MinIO instance, coordinate with the RunReveal team to ensure connectivity.

Verify it’s working

Once your source is created, confirm data is flowing:

Go to Sources and check your source’s health status.
Open Explorer and query your data. If you created a custom view, use the view table name. Otherwise, query the raw logs:

SELECT * FROM logs WHERE sourceType = 'custom-source' LIMIT 10

If using a webhook, send a test event and confirm it appears within a few seconds.

Troubleshooting

Symptom	What to check
Verify Settings fails	Confirm credentials, bucket name, and endpoint are correct. For S3, verify the bucket policy allows RunReveal access.
No data appearing	For object storage: confirm event notifications are configured and new files are landing in the bucket. For webhook: send a test POST and check the response status code.
Field mapping errors	Retry from the progress dialog — RunReveal allows retrying failed view/transform steps without recreating the source.
MinIO notifications not working	Verify the PostgreSQL target is reachable from MinIO (`mc admin config get myminio notify_postgres`). Check that `mc event ls myminio/my-logs` shows the event rule.

Cursor Audit Logs Cyberhaven