reveald log collection

Reveald (opens in a new tab) is an open source log collection agent that is efficient, performant and simple.

Reveald supports a variety of host-level log sources and is appropriate to use in any situation where the logs aren't already being collected to an S3 bucket or accessible via another first-party API.

To get your logs into RunReveal via reveald, we'll first create a reveald source in the RunReveal UI, which will give us a webhookURL to use as the target to send our logs to from reveald.

Next, download a reveald release for the operating system and architecture of the host you'll be running the daemon on from the releases page on github (opens in a new tab). Extract the package to a directory of your choosing.

As of this writing, reveald supports the following sources:

journald
file
command
syslog
command
mqtt
nginx access logs (via syslog)

And the following destinations:

runreveal
s3
printer
mqtt

You can see examples of how to configure reveald for each of these sources in the example config file provided in the repository's examples directory, or an abridged example config below.

Replace {{YOUR-REVEALD-WEBHOOKURL}} with the webhook given in the UI.

{
  // this is an example config file for kawa
  // it is parsed using hujson so you can use comments and trailing commas, but
  // is otherwise identical to JSON
  "sources": {
    // The keys here are the identifiers for the sources, and are used to
    // refernece them when logging or in metrics.
    "kubernetes": {
      "type": "file",
      "path": "/var/log/pods/",
      "extension": ".log",
    },
    "hostlogs": {
      "type": "file",
      "path": "/var/log/mylogs/",
      "extension": ".log",
    },
    "myjournald": {
      "type": "journald",
    },
    "cmdexample": {
      // The comand source type allows you to run a command and collect its
      // output as logs. This is useful for running custom scripts or commands
      // that output logs to stdout.
      "type": "command",
      "cmd": "/bin/date",
      "args": ["+%Y-%m-%dT%H:%M:%S%z"],
      // inheritEnv allows you to pass the environment of the reveald process
      // to the command being run.  Defaults to false.
      "inheritEnv": false,
      // env allows you to set environment variables for the command being run.
      // if inheritEnv is set, any keys this json object override the
      // inherited environment of the reveald process.
      "env": {
        "TZ": "Asia/Taipei",
      },
      // interval is the time between runs of the command, the polling
      // frequency. Defaults to 5s. Whatever this is set to, the command
      // being run will also have a deadline to complete before the
      // next run, or be killled with an error returned to reveald.
      "interval": "5s",
    },
    // "mynginx": {
    //   "type": "nginx_syslog",
    //   "addr": "0.0.0.0:5514",
    // },
  },
  "destinations": {
    "webhook": {
      "type": "runreveal",
      // Replace this webhook URL with your own, created on https://www.runreveal.com
      // as a "Reveald" type source
      "webhookURL": "https://example.runreveal.com/sources/kawa/webhook/0123456789",
      // You can also use environment variables by referencing them with a
      // dollar sign. The value must be quoted, start with a dollar sign and be
      // a valid environment variable name
      // "webhookURL": "$WEBHOOK_URL",
    },
    "blobstore": {
      "type": "s3",
      // bucketName is required
      "bucketName": "our-s3-archive",
      // bucketRegion is required
      "bucketRegion": "us-west-2",
      // pathPrefix is optional and default unset
      "pathPrefix": "logs/",
      // accessKeyID is required if not present in the environment as
      // AWS_ACCESS_KEY_ID. If running on AWS, reveald can also be authenticated
      // at the ec2 host/pod level using IAM roles.
      "accessKeyID": "AKIAIOS...",
      // secretAccessKey is required if not present in the environment as
      // AWS_SECRET_ACCESS_KEY
      "secretAccessKey": "SECRET",
      // batchSize is optional
      "batchSize": 100000,
    },
  },
}

This is a valid config file if you're running on Linux. We can run reveald pointing at the configuration like so (assuming you download it to the same directory as the binary).

./reveald run --config=./config.json

If configured correctly, you should start seeing the journald logs and kubelet pod logs (if the host is running a kubelet) in your RunReveal workspace.

If running inside kubernetes, you can configure this binary to run as a daemonset with the given config file as a configmap mounted somewhere inside the pod, and the given log directories shared from the host to the pod.

Palo Alto Panorama Traffic SentinelOne