SourcesForwardersVector

Vector Log Forwarder

Vector is a high-performance observability data pipeline that can collect, transform, and route logs to RunReveal.

Quick Start

This setup demonstrates reading logs from a local file and forwarding them to your RunReveal webhook. The example creates test JSON logs, configures Vector to monitor the log file, and sends events to RunReveal in real-time.

Step 1: Create a Webhook Source in RunReveal

This step creates a webhook source in RunReveal and generates the webhook URL you’ll need for Vector configuration.

  1. Navigate to RunReveal: Go to your RunReveal dashboard
  2. Create Source: Click on “Sources” in the left sidebar
  3. Add Webhook: Create a source for “Structured Webhook or Generic Webhook” based on your data format.
  4. Configure: Give your webhook a name and description
  5. Copy URL: Copy the generated webhook URL to use in your configuration. (It’s always available after you save the source)

Step 2: Install Vector

This step installs Vector on your system using your preferred method. Vector will be used to collect, process, and forward logs to RunReveal.

# Install via Homebrew
brew install vector
 
# Verify installation
vector --version

Step 3: Create Test Logs

This step creates sample log files with security events that Vector will monitor and forward to RunReveal. These logs simulate real application events for testing purposes.

# Create vector directory for testing
mkdir -p vector
cd vector
 
# Create sample application logs
cat > application.log << EOF
{"eventName":"UserLogin","eventTime":"2024-01-15T10:30:00Z","readOnly":false,"actor":{"email":"[email protected]","id":"MFA777777","username":"mfa_user"},"src":{"ip":"192.168.1.114","port":443},"service":{"name":"vector-application"},"tags":{"environment":"production","source":"vector"}}
{"eventName":"FileAccess","eventTime":"2024-01-15T10:31:00Z","readOnly":true,"actor":{"email":"[email protected]","id":"DB888888","username":"database_service"},"src":{"ip":"192.168.1.115","port":443},"service":{"name":"vector-application"},"tags":{"environment":"production","source":"vector"}}
{"eventName":"DataExport","eventTime":"2024-01-15T10:32:00Z","readOnly":false,"actor":{"email":"[email protected]","id":"MET999999","username":"metrics_exporter"},"src":{"ip":"192.168.1.116","port":443},"service":{"name":"vector-application"},"tags":{"environment":"production","source":"vector"}}
EOF
 
# Verify logs were created
ls -la application.log
cat application.log

Step 4: Configure Vector

This step creates the Vector configuration file that defines how to collect logs and send them to RunReveal.

Basic Configuration - No Authentication Required

Use this configuration when your RunReveal webhook doesn’t require a bearer token for authentication.

# vector.toml
data_dir = "./vector-data"
 
# Source: Read log files
[sources.my_logs]
type = "file"
include = ["application.log"]
read_from = "beginning"
 
# Sink: Send to RunReveal
[sinks.runreveal]
type = "http"
inputs = ["my_logs"]
uri = "YOUR_WEBHOOK_URL"
method = "post"
 
[sinks.runreveal.encoding]
codec = "json"
 
[sinks.runreveal.request]
timeout_secs = 30

Replace YOUR_WEBHOOK_URL with your actual webhook URL from Step 1.

Authentication Notes:

  • Export YOUR_WEBHOOK_URL as an environment variable instead of adding the url directly to the config.
  • Use bearer token authentication only if your RunReveal webhook requires it
  • Always use environment variables for sensitive data like tokens
  • Never commit authentication tokens to version control
  • Bearer auth support requires Vector v0.9.0 or later (see documentation)

Step 5: Start Vector

This step starts Vector with your configuration, which will begin monitoring the log files and forwarding events to RunReveal in real-time.

# Make sure you're in the vector directory where the config file is located
cd vector
 
# Validate configuration
vector validate vector.toml
 
# Run Vector in foreground
vector --config vector.toml
 
# Or run in background
vector --config vector.toml --quiet &

Step 6: Verify Delivery in RunReveal

  • Return to Sources in RunReveal and open your Vector Webhook source card.
  • Confirm the Last Event timestamp and recent volume.

Click Query to explore incoming logs. Start with the sourceID which corresponds to the source id found on each source card on the sources page to only view logs from this source:

SELECT *
FROM {table:Identifier}
WHERE ({tsColumn:Identifier} >= now() - INTERVAL 1 HOUR) 
 AND ({tsColumn:Identifier} < now()) 
 AND ((sourceID = 'WEBHOOK_SOURCE_ID'))
LIMIT 10

Vector Configuration

Sources

Sources collect data from various inputs. For RunReveal, you’ll typically use file sources or system sources. See the Vector Sources Reference for comprehensive documentation.

File Source Example

[sources.application_logs]
type = "file"
include = ["/var/log/app/*.log", "/var/log/api/*.json"]
read_from = "beginning"

Collects application logs in JSON format, perfect for security event analysis and compliance reporting.

System Source Example

[sources.system_logs]
type = "journald"
include_units = ["ssh", "sudo", "audit"]

Captures system authentication and authorization events, essential for security monitoring and threat detection.

Docker Source Example

[sources.container_logs]
type = "docker_logs"
include_images = ["nginx", "postgres", "redis"]

Monitors containerized applications for security incidents, performance issues, and compliance violations.

Next Steps

Now that you have Vector configured and logs flowing to RunReveal, explore the detailed configuration guides:

  • Detections - Create and manage security detection rules
  • Sigma Streaming - Use Sigma rules for standardized threat detection
  • Detection as Code - Manage detections through code and version control
  • Notifications - Set up alerting and notification channels
  • AI Chat - Use AI-powered analysis for threat hunting and investigation
  • Enrichments - Add context and metadata to your security events
  • Filters - Create custom filters to focus on specific security events
  • Pipelines - Build data processing pipelines for complex workflows

Troubleshooting

Vector Won’t Start

  1. Check Configuration Syntax

    vector validate vector.toml

  2. Verify File Permissions

    • Ensure Vector has read access to log files
    • Check that Vector can write to its working directory

  3. Review Vector Logs

    vector --config vector.toml --log-level debug

Logs Not Reaching RunReveal

  1. Verify Webhook URL

    • Check that the webhook URL is correct and active
    • Ensure the URL includes the full path from RunReveal

  2. Test Network Connectivity

    curl -X POST https://api.runreveal.com/sources/hook/YOUR_WEBHOOK_URL \
  -H "Content-Type: application/json" \
  -d '{"test": "message"}'

  3. Check Vector Sink Configuration

    • Verify the HTTP sink is properly configured
    • Check for any authentication requirements

High Resource Usage

  1. Optimize Buffer Settings
    • Adjust buffer size and flush intervals
    • Consider using disk buffers for high-volume data

Data Format Issues

  1. Check Log Format

    • Ensure logs are in the expected format (JSON, syslog, etc.)

  2. Review Transform Rules

    • Check that remap transforms are correctly parsing data
    • Verify field mappings match your log structure

  3. Test with Sample Data

    • Use Vector’s vector test command to validate transforms
    • Check output format before sending to RunReveal

Resources

RevealdFluent Bit