No description
Find a file
2018-10-06 00:51:26 +02:00
src refactor: Replace reqwest library with ureq 2018-10-05 23:58:01 +02:00
.gitignore feat(journald): Implement initial libsystemd journal calls 2018-05-27 20:09:37 +02:00
.travis.yml feat(build): Enable Nix builds in Travis.CI 2018-06-15 17:00:48 +02:00
build.rs feat(build): Configure linking to libsystemd 2018-05-27 20:09:20 +02:00
Cargo.lock refactor: Replace reqwest library with ureq 2018-10-05 23:58:01 +02:00
Cargo.toml refactor: Replace reqwest library with ureq 2018-10-05 23:58:01 +02:00
CODE_OF_CONDUCT.md docs: Add code of conduct 2018-06-15 17:02:04 +02:00
CONTRIBUTING.md docs(README): Add notes about error reporting & minor improvements 2018-10-06 00:51:26 +02:00
default.nix refactor: Replace reqwest library with ureq 2018-10-05 23:58:01 +02:00
LICENSE chore: License under GPL 3.0 2018-06-15 17:00:26 +02:00
README.md docs(README): Add notes about error reporting & minor improvements 2018-10-06 00:51:26 +02:00

journaldriver

This is a small daemon used to forward logs from journald (systemd's logging service) to Stackdriver Logging.

Many existing log services are written in inefficient dynamic languages with error-prone "cover every possible use-case" configuration. journaldrive instead aims to fit a specific use-case very well, instead of covering every possible logging setup.

journaldriver can be run on GCP-instances with no additional configuration as authentication tokens are retrieved from the metadata server.

Table of Contents

Features

  • journaldriver persists the last forwarded position in the journal and will resume forwarding at the same position after a restart
  • journaldriver will recognise log entries in JSON format and forward them appropriately to make structured log entries available in Stackdriver
  • journaldriver can be used outside of GCP by configuring static credentials
  • journaldriver will recognise journald's log priority levels and convert them into equivalent Stackdriver log severity levels

Usage on Google Cloud Platform

journaldriver does not require any configuration when running on GCP instances.

  1. Install journaldriver on the instance from which you wish to forward logs.

  2. Ensure that the instance has the appropriate permissions to write to Stackdriver. Google continously changes how IAM is implemented on GCP, so you will have to refer to Google's documentation.

    By default instances have the required permissions if Stackdriver Logging support is enabled in the project.

  3. Start journaldriver, for example via systemd.

Usage outside of Google Cloud Platform

When running outside of GCP, the following extra steps need to be performed:

  1. Create a Google Cloud Platform service account with the "Log Writer" role and download its private key in JSON-format.

  2. When starting journaldriver, configure the following environment variables:

    • GOOGLE_CLOUD_PROJECT: Name of the GCP project to which logs should be written.
    • GOOGLE_APPLICATION_CREDENTIALS: Filesystem path to the JSON-file containing the service account's private key.
    • LOG_STREAM: Name of the target log stream in Stackdriver Logging. This will be automatically created if it does not yet exist.
    • LOG_NAME: Name of the target log to write to. This defaults to journaldriver if unset, but it is recommended to - for example - set it to the machine hostname.

Log levels / severities / priorities

journaldriver recognises journald's priorities and converts them into equivalent severities in Stackdriver. Both sets of values correspond to standard syslog priorities.

The easiest way to emit log messages with priorites from an application is to use priority prefixes, which are compatible with structured log messages.

For example, to emit a simple warning message (structured and unstructured):

$ echo '<4>{"fnord":true, "msg":"structured log (warning)"}' | systemd-cat
$ echo '<4>unstructured log (warning)' | systemd-cat

NixOS module

The NixOS package repository contains a module for setting up journaldriver on NixOS machines. NixOS by default uses systemd for service management and journald for logging, which means that log output from most services will be captured automatically.

On a GCP instance the only required option is this:

services.journaldriver.enable = true;

When running outside of GCP, the configuration looks as follows:

services.journaldriver = {
  enable                 = true;
  logStream              = "prod-environment";
  logName                = "hostname";
  googleCloudProject     = "gcp-project-name";
  applicationCredentials = keyFile;
};

Note: The journaldriver-module is not yet included in a stable release of NixOS, but it is available on the unstable-channel.

Stackdriver Error Reporting

The Stackdriver Error Reporting service of Google's monitoring toolbox supports automatically detecting and correlating errors from log entries.

To use this functionality log messages must be logged in the expected log format.

Note: Currently errors logged from non-GCP instances are not ingested into Error Reporting. Please see issue #4 for more information about this.