SourcesForwardersReveald

Reveald Log Forwarder

Reveald is RunReveal’s official log collection agent, designed specifically for sending data to RunReveal. It’s optimized for performance, reliability, and ease of use with RunReveal’s platform.

Quick Start

This setup demonstrates reading logs from a local file and forwarding them to your RunReveal webhook. The example creates test JSON logs, configures Reveald to monitor the log file, and sends events to RunReveal in real-time.

Step 1: Create a Reveald Source in RunReveal

This step creates a Reveald source in RunReveal and generates the webhook URL you’ll need for Reveald configuration.

  1. Navigate to RunReveal: Go to your RunReveal dashboard
  2. Create Source: Click on “Sources” in the left sidebar
  3. Add Reveald Source: Click on “Reveald” from the source options
  4. Configure: Give your source a name and description
  5. Copy URL: Copy the generated webhook URL to use in your configuration (It’s always available after you save the source)

Note: The webhook URL will look like: https://api.runreveal.com/sources/reveald/webhook/[YOUR_WEBHOOK_ID]

Step 2: Install Reveald

This step installs Reveald on your system. Reveald is a lightweight daemon for collecting and forwarding logs.

# Download latest Reveald for Linux
curl -L https://github.com/runreveal/reveald/releases/latest/download/reveald-linux-amd64.tar.gz | sudo tar --directory /usr/local/bin -xz
 
# Make executable
sudo chmod +x /usr/local/bin/reveald
 
# Verify installation
reveald --help

Step 3: Create Test Logs

This step creates sample log files with security events that Reveald will monitor and forward to RunReveal.

# Create test-logs directory
mkdir -p test-logs
 
# Create sample JSON logs (each line is a separate JSON event)
cat > test-logs/test.log << 'EOF'
{"eventName":"UserLogin","eventTime":"2025-01-15T10:30:00Z","actor":{"email":"[email protected]","id":"USR001"},"service":{"name":"webapp"},"src":{"ip":"192.168.1.100"}}
{"eventName":"FileAccess","eventTime":"2025-01-15T10:31:00Z","actor":{"email":"[email protected]","id":"USR002"},"service":{"name":"webapp"},"src":{"ip":"192.168.1.101"}}
{"eventName":"DataExport","eventTime":"2025-01-15T10:32:00Z","actor":{"email":"[email protected]","id":"USR003"},"service":{"name":"webapp"},"src":{"ip":"192.168.1.102"}}
EOF
 
# Verify logs were created
echo "Test logs created:"
cat test-logs/test.log | jq -c '.'

Step 4: Configure Reveald

This step creates the Reveald configuration file. Note that Reveald uses a map-based configuration where sources and destinations are objects with named keys, not arrays.

# Create config.json with your webhook URL
cat > config.json << 'EOF'
{
  "sources": {
    "test": {
      "type": "file",
      "path": "./test-logs/",
      "extension": ".log"
    }
  },
  "destinations": {
    "runreveal": {
      "type": "runreveal",
      "webhookURL": "YOUR_WEBHOOK_URL_HERE"
    }
  }
}
EOF
 
# Replace YOUR_WEBHOOK_URL_HERE with your actual webhook URL
# Example: sed -i 's|YOUR_WEBHOOK_URL_HERE|https://api.runreveal.com/sources/reveald/webhook/YOUR_ID|' config.json
⚠️

Important: Replace YOUR_WEBHOOK_URL_HERE with your actual RunReveal webhook URL from Step 1.

Step 5: Start Reveald

This step starts Reveald with your configuration, which will begin monitoring the log files and forwarding events to RunReveal.

# Run Reveald with your configuration
reveald run --config config.json
 
# You should see output like:
# time=2025-01-15T10:35:00.000-00:00 level=INFO source=main.go:53 msg="starting reveald" version=0.1.5
# time=2025-01-15T10:35:00.100-00:00 level=INFO msg="processing file" path="test-logs/test.log"

To run Reveald in the background:

# Run as background process
nohup reveald run --config config.json > reveald.log 2>&1 &
 
# Check logs
tail -f reveald.log
 
# Stop background process
pkill reveald

Step 6: Validate Delivery

This step verifies that logs are successfully being delivered to RunReveal.

  1. Check RunReveal Dashboard:

    • Go to RunReveal → Sources → Your Reveald source
    • Look for “Last Event Received” timestamp
    • Check event count is increasing

  2. Query for Events:

    -- Find your test events
SELECT
  *
FROM
  events
WHERE
  (eventTime >= now() - INTERVAL 1 HOUR)
  AND (eventTime < now())
  AND (sourceType = 'reveald')
ORDER BY
  eventTime DESC
LIMIT
  10

  3. Add More Test Events (optional):

    # Append new events to trigger real-time forwarding
echo '{"eventName":"NewEvent","eventTime":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","actor":{"email":"[email protected]"}}' >> test-logs/test.log

Configuration Reference

Sources Configuration

Reveald sources are defined as a map (object) where each source has a unique key name:

{
  "sources": {
    "source-name": {
      "type": "file",
      "path": "/path/to/logs/",
      "extension": ".log"
    }
  }
}

File Source Options:

  • type: Must be “file”
  • path: Directory path containing log files (absolute or relative)
  • extension: File extension to monitor (e.g., “.log”, “.json”)
  • codec: (optional) “json” for JSON parsing
  • startAt: (optional) “beginning” or “end” (default: “end”)

Syslog Source Example:

{
  "sources": {
    "syslog-nginx": {
      "type": "syslog",
      "addr": "0.0.0.0:5514",
      "contentType": "application/json; rrtype=nginx-json"
    }
  }
}

Destinations Configuration

Destinations are also defined as a map where each destination has a unique key:

{
  "destinations": {
    "destination-name": {
      "type": "runreveal",
      "webhookURL": "https://api.runreveal.com/sources/reveald/webhook/YOUR_ID"
    }
  }
}

RunReveal Destination Options:

  • type: Must be “runreveal”
  • webhookURL: Your RunReveal webhook URL
  • batchSize: (optional) Number of events per batch
  • flushInterval: (optional) Time between batches (e.g., “5s”)

Complete Production Example

{
  "sources": {
    "app-logs": {
      "type": "file",
      "path": "/var/log/application/",
      "extension": ".log",
      "codec": "json",
      "startAt": "end"
    },
    "nginx-syslog": {
      "type": "syslog",
      "addr": "0.0.0.0:5514",
      "contentType": "application/json"
    }
  },
  "destinations": {
    "runreveal-prod": {
      "type": "runreveal",
      "webhookURL": "${RUNREVEAL_WEBHOOK_URL}",
      "batchSize": 100,
      "flushInterval": "10s"
    }
  }
}

Advanced Configurations

Systemd Service Configuration

For production Linux deployments, create a systemd service:

# /etc/systemd/system/reveald.service
[Unit]
Description=Reveald Log Forwarder
After=network.target
 
[Service]
Type=simple
User=reveald
Group=reveald
ExecStart=/usr/local/bin/reveald run --config /etc/reveald/config.json
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal
SyslogIdentifier=reveald
 
# Security hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/log
 
[Install]
WantedBy=multi-user.target

Enable and start the service:

# Create user and directories
sudo useradd -r -s /bin/false reveald
sudo mkdir -p /etc/reveald
sudo chown reveald:reveald /etc/reveald
 
# Copy configuration
sudo cp config.json /etc/reveald/
sudo chown reveald:reveald /etc/reveald/config.json
 
# Enable and start service
sudo systemctl daemon-reload
sudo systemctl enable reveald
sudo systemctl start reveald
 
# Check status
sudo systemctl status reveald
sudo journalctl -u reveald -f

Docker Compose Example

# docker-compose.yml
version: '3.8'
 
services:
  reveald:
    image: runreveal/reveald:latest
    container_name: reveald
    restart: unless-stopped
    volumes:
      - ./config.json:/config.json:ro
      - /var/log:/logs:ro
    environment:
      - RUNREVEAL_WEBHOOK_URL=${RUNREVEAL_WEBHOOK_URL}
    command: run --config /config.json
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"

Environment Variable Configuration

Use environment variables for sensitive data:

{
  "sources": {
    "logs": {
      "type": "file",
      "path": "${LOG_PATH:-/var/log/app/}",
      "extension": ".log"
    }
  },
  "destinations": {
    "runreveal": {
      "type": "runreveal",
      "webhookURL": "${RUNREVEAL_WEBHOOK_URL}",
      "batchSize": ${BATCH_SIZE:-100},
      "flushInterval": "${FLUSH_INTERVAL:-10s}"
    }
  }
}

Set environment variables:

export RUNREVEAL_WEBHOOK_URL="https://api.runreveal.com/sources/reveald/webhook/YOUR_ID"
export LOG_PATH="/var/log/application/"
export BATCH_SIZE=500
export FLUSH_INTERVAL="30s"
 
reveald run --config config.json

Best Practices

Security Recommendations

  1. Webhook URL Protection:

    • Never commit webhook URLs to version control
    • Use environment variables or secrets management
    • Rotate webhook URLs periodically

  2. File Permissions:

    # Restrict config file access
chmod 600 /etc/reveald/config.json
chown reveald:reveald /etc/reveald/config.json

  3. Network Security:

    • Use TLS for all webhook connections
    • Implement egress filtering for Reveald hosts
    • Monitor for unusual outbound traffic patterns

Performance and Recommendations

For production deployments, consider these optimizations:

{
  "sources": {
    "production": {
      "type": "file",
      "path": "/var/log/app/",
      "extension": ".log",
      "startAt": "end",  // Only new events
      "maxFileSize": "100MB"  // Skip huge files
    }
  },
  "destinations": {
    "runreveal": {
      "type": "runreveal",
      "webhookURL": "${RUNREVEAL_WEBHOOK_URL}",
      "batchSize": 500,  // Larger batches for throughput
      "flushInterval": "30s",  // Less frequent sends
      "maxRetries": 3,
      "timeout": "60s"
    }
  }
}

  1. Batch Configuration:

    • Start with batch size of 100-500 events
    • Adjust based on event size and network latency
    • Monitor memory usage with larger batches

  2. File Monitoring:

    • Use startAt: "end" for production to avoid replaying old logs
    • Set maxFileSize to prevent memory issues with large files
    • Use specific paths rather than wildcards when possible

  3. Resource Limits:

    # Set memory limits for systemd service
# Add to [Service] section
MemoryLimit=512M
CPUQuota=50%

FAQ

Q: Can Reveald handle multiple log formats? A: Yes, configure multiple sources with different codecs for each format.

Q: How do I handle log rotation? A: Reveald automatically detects rotated files and continues processing new files matching the pattern.

Q: What’s the maximum batch size? A: While there’s no hard limit, optimal performance is typically achieved with batches of 100-1000 events.

Q: Can I filter events before sending? A: Currently, filtering should be done at the destination. Consider using RunReveal’s filtering features.

Q: How do I debug connection issues? A: Run with -v debug flag and check network connectivity to api.runreveal.com on port 443.

Q: Is there a Windows version? A: Windows binaries are available in GitHub releases. Use Windows paths in configuration.

Q: Can I send to multiple RunReveal workspaces? A: Yes, configure multiple destinations with different webhook URLs.

Troubleshooting

Common Issues and Solutions

Error: json: cannot unmarshal "[...]" into Go struct

Solution: Ensure your config uses objects (maps) not arrays:

// ❌ Wrong - arrays
{
  "sources": [
    { "type": "file", ... }
  ]
}
 
// ✅ Correct - objects with keys
{
  "sources": {
    "my-source": { "type": "file", ... }
  }
}

Resources

ForwardersVector