Commit graph

21 commits

Author SHA1 Message Date
Thomas ten Cate
a89be8d715 feat(main): Pass log levels along to Stackdriver
If a priority is present, it is passed as-is into the Stackdriver API.
This allows filtering by severity in the logs UI. Conveniently, the
levels are the same between journald and Stackdriver.

Fixes #11.
2018-09-24 16:03:58 +02:00
Vincent Ambo
7557d8ff5b refactor(main): Move cursor file to /var/lib/journaldriver
This directory is the default for state directories managed by
systemd, which will be the main use-case of journaldriver when running
it on NixOS.

Relates to nixos/nixpkgs#42134
2018-06-18 20:12:43 +02:00
Vincent Ambo
9f29992d03 refactor(main): Split LOG_NAME & LOG_STREAM variables
Splitting these in two lets users better configure how the logs show
up in Stackdriver Logging.
2018-06-17 22:36:01 +02:00
Vincent Ambo
79afc5f474 feat(main): Support manually configured resource descriptors
Lets users specify environment variables to configure the monitored
resource descriptor. In combination with service account support this
can be used to configure journaldriver on non-GCP machines.

The supported environment variables are:

* `GOOGLE_CLOUD_PROJECT`: Project ID of the project to which to report
  logs
* `LOG_NAME`: Name of the Stackdriver Logging log into which to write
  entries
2018-06-17 17:43:33 +02:00
Vincent Ambo
62745db1a0 feat(main): Support static service account credentials
Adds support for authenticating towards Google's APIs by using static
service account credentials which can be retrieved from the GCP
console.

Users can toggle the behaviour by specifying the
`GOOGLE_APPLICATION_CREDENTIALS` environment variable.
2018-06-17 17:43:33 +02:00
Vincent Ambo
c2308b5ba1 chore(main): Bump chunk size to 750
Still slightly below the Stackdriver limit. However, previous errors
seem to have mostly been related to JSON payloads - which has been
fixed in the previous commit.
2018-06-17 15:46:05 +02:00
Vincent Ambo
05fa476929 fix(main): Correct name of JSON payload field 2018-06-17 15:46:05 +02:00
Vincent Ambo
68189d4872 refactor(main): Log error responses from Stackdriver
In some cases Stackdriver seems to return error responses for batched
inserts.

This change will log the error response body and status from
Stackdriver for all insertion errors.
2018-06-17 15:46:05 +02:00
Vincent Ambo
87ab3c806c feat(main): Parse timestamps out of journald entries
Instead of relying on Stackdriver's ingestion timestamps, parse
timestamps out of journal entries and provide those to Stackdriver.

If a timestamp could not be parsed out of a log entry, the ingestion
time is used as the fallback.
2018-06-17 15:46:05 +02:00
Vincent Ambo
bd19132eff chore(main): Send logs in smaller chunks of 250 entries
In some cases sending 1000 entries seemingly results in a vague "Bad
request" error.

This reduces the chunk size to something that should be more
manageable.
2018-06-16 21:38:53 +02:00
Vincent Ambo
54b03a8dad fix(main): Attempt to read with next_record before waiting
Without this fix new records are only "pushed out" when something
appends to the journal.
2018-06-16 21:38:53 +02:00
Vincent Ambo
e4b4830a04 feat(main): Persist & load persisted cursor positions
Adds support for persisting the cursor position in a file (by default
`/var/journaldriver/cursor.pos`, overridable via the environment
variable `CURSOR_POSITION_FILE`).

This lets journaldriver resume from a known journal position after
service restarts.

This closes #3
2018-06-16 21:38:53 +02:00
Vincent Ambo
10f23a9dfb feat(main): Implement parsing of JSON payloads
Stackdriver supports structured JSON payloads in addition to simple
plain-text payloads.

This commit introduces a new feature in which journaldriver will
attempt to parse incoming log messages into JSON vaues and forward
them as structured payloads if they are JSON objects.

Messages that can not be parsed into JSON objects will continue to be
forwarded as plain text messages.
2018-06-16 17:57:11 +02:00
Vincent Ambo
a47bd5675f chore: License under GPL 3.0 2018-06-15 17:00:26 +02:00
Vincent Ambo
4ef98fc2ba feat(main): Implement record conversion & flushing to API
This implements the "meat" of the initial version of journaldriver.

Records from journald are converted into the representation required
by Stackdriver and forwarded to the API.

In this initial version journaldriver is only supported on instances
running on GCP.
2018-06-15 16:45:17 +02:00
Vincent Ambo
2d8e057118 feat(main): Add fetching of tokens from metadata server 2018-06-14 16:48:43 +02:00
Vincent Ambo
2541d25fba feat(main): Emit output in chunks of max. 1000 records
Required by the Stackdriver API.
2018-06-05 15:24:03 +02:00
Vincent Ambo
ef638bfa20 feat(stackdriver): Add initial Stackdriver API type definitions 2018-05-28 22:24:32 +02:00
Vincent Ambo
0db4512df4 refactor(main): Simplify receiver
Departing from the initial approach. There's no reason to multithread
this for now.
2018-05-28 09:45:07 +02:00
Vincent Ambo
6793b25a67 feat(main): Implement receiver & flushing logic
The only thing missing for a 0.1 test run is the actual gRPC call to
Stackdriver.
2018-05-27 23:57:24 +02:00
Vincent Ambo
c5cd12b81f feat(journald): Implement initial libsystemd journal calls 2018-05-27 20:09:37 +02:00