SPIFFE/mTLS Authentication

SPIFFE (Secure Production Identity Framework for Everyone) provides a standardized way to issue and manage service identities using X.509 certificates. RunReveal supports SPIFFE-based mutual TLS (mTLS) authentication for ClickHouse connections, enabling secure service-to-service communication without passwords.

Why Use SPIFFE/mTLS?

  • Enhanced Security: Certificates are more secure than static passwords and can be automatically rotated
  • Zero Trust Architecture: Every connection is authenticated with cryptographic proof of identity
  • Kubernetes Native: Works seamlessly with SPIFFE implementations like SPIRE in Kubernetes environments
  • Automatic Rotation: Certificates can be rotated without service restarts

SPIFFE/mTLS is recommended for production BYOC deployments where ClickHouse runs in the same network as RunReveal services.

Supported Destinations

SPIFFE authentication works with both:

  • Default ClickHouse - The global/shared ClickHouse cluster configured via environment
  • Custom Destinations (BYODB) - Per-workspace ClickHouse instances configured in destination settings

For BYODB destinations, enable SPIFFE by setting useSPIFFE: true in the destination configuration.

Configuration

Enable SPIFFE authentication by setting the following configuration options:

{
  "clickhouse": {
    "spiffe": {
      "defaultDestEnabled": true,
      "certPath": "/etc/identity/tls.crt",
      "keyPath": "/etc/identity/tls.key",
      "caCertPath": "/etc/identity/ca.crt",
      "refreshInterval": "10m"
    }
  }
}

Configuration Options

OptionDescriptionDefault
defaultDestEnabledEnable SPIFFE for the default/multi-tenant ClickHouse destinationfalse
certPathPath to the client certificate file/etc/identity/tls.crt
keyPathPath to the client private key file/etc/identity/tls.key
caCertPathPath to the CA certificate for server verification(none)
refreshIntervalHow often to check for certificate updates10m
⚠️

When SPIFFE is enabled, the certificate Common Name (CN) or the X-ClickHouse-User header is used for ClickHouse authentication. Ensure your ClickHouse user configuration matches.

Certificate Requirements

Your SPIFFE certificates must meet these requirements:

  • Format: PEM-encoded X.509 certificates
  • Key Usage: Client authentication (clientAuth extended key usage)
  • Common Name: Must match the ClickHouse username you want to authenticate as
  • Validity: Certificate must be valid (not expired)

Example Certificate Structure

/etc/identity/
├── tls.crt    # Client certificate (PEM format)
└── tls.key    # Private key (PEM format)

ClickHouse Server Configuration

Configure your ClickHouse server to accept certificate-based authentication:

Enable HTTPS

Add SSL configuration to your ClickHouse config.xml:

<clickhouse>
    <https_port>8443</https_port>
    <openSSL>
        <server>
            <certificateFile>/etc/clickhouse-server/certs/server.crt</certificateFile>
            <privateKeyFile>/etc/clickhouse-server/certs/server.key</privateKeyFile>
            <caConfig>/etc/clickhouse-server/certs/ca.crt</caConfig>
            <verificationMode>relaxed</verificationMode>
            <loadDefaultCAFile>false</loadDefaultCAFile>
        </server>
    </openSSL>
</clickhouse>

Create Certificate-Authenticated User

Add a user that authenticates via certificate CN in users.xml:

<clickhouse>
    <users>
        <runreveal_service>
            <ssl_certificates>
                <common_name>runreveal-service</common_name>
            </ssl_certificates>
            <networks>
                <ip>::/0</ip>
            </networks>
            <profile>default</profile>
            <quota>default</quota>
            <access_management>0</access_management>
        </runreveal_service>
    </users>
</clickhouse>

Verify Configuration

Test the connection using curl:

curl --cert /etc/identity/tls.crt \
     --key /etc/identity/tls.key \
     --cacert /etc/clickhouse-server/certs/ca.crt \
     -H "X-ClickHouse-SSL-Certificate-Auth: on" \
     -H "X-ClickHouse-User: runreveal_service" \
     "https://clickhouse:8443/?query=SELECT%201"

Certificate Rotation

RunReveal automatically watches for certificate changes and reloads them without requiring a restart. The refreshInterval setting controls how often the certificate files are checked for updates.

When certificates are rotated:

  1. Your SPIFFE provider (e.g., SPIRE) writes new certificates to the configured paths
  2. RunReveal detects the file change within the refreshInterval period
  3. New connections use the updated certificates automatically
  4. Existing connections continue until they’re recycled by the connection pool

Set refreshInterval to match your certificate rotation frequency. For short-lived certificates (common in SPIRE), use a shorter interval like 1m.

Kubernetes with SPIRE

If you’re using SPIRE in Kubernetes, configure the SPIRE agent to project certificates to your RunReveal pods:

apiVersion: v1
kind: Pod
metadata:
  name: runreveal-api
spec:
  containers:
  - name: api
    volumeMounts:
    - name: spiffe-workload-api
      mountPath: /etc/identity
      readOnly: true
  volumes:
  - name: spiffe-workload-api
    csi:
      driver: "csi.spiffe.io"
      readOnly: true

Troubleshooting

Connection refused on HTTPS port

  • Verify ClickHouse is listening on the HTTPS port: netstat -tlnp | grep 8443
  • Check ClickHouse logs for SSL initialization errors
  • Ensure the server certificate and key are readable by ClickHouse

Debug Logging

Enable debug logging to troubleshoot certificate issues:

{
  "logging": {
    "level": "debug"
  }
}

Look for log messages containing “SPIFFE” or “watching SPIFFE certificates” to verify the certificate watcher is running.