Commit graph

35 commits

Author SHA1 Message Date
Vincent Ambo
7557d8ff5b refactor(main): Move cursor file to /var/lib/journaldriver
This directory is the default for state directories managed by
systemd, which will be the main use-case of journaldriver when running
it on NixOS.

Relates to nixos/nixpkgs#42134
2018-06-18 20:12:43 +02:00
Vincent Ambo
9f29992d03 refactor(main): Split LOG_NAME & LOG_STREAM variables
Splitting these in two lets users better configure how the logs show
up in Stackdriver Logging.
2018-06-17 22:36:01 +02:00
Vincent Ambo
208b5dec2c chore: Bump to 1.0-rc1 2018-06-17 18:22:13 +02:00
Vincent Ambo
7b6f4cb89b docs(README): Add non-GCP instructions to README 2018-06-17 17:43:33 +02:00
Vincent Ambo
79afc5f474 feat(main): Support manually configured resource descriptors
Lets users specify environment variables to configure the monitored
resource descriptor. In combination with service account support this
can be used to configure journaldriver on non-GCP machines.

The supported environment variables are:

* `GOOGLE_CLOUD_PROJECT`: Project ID of the project to which to report
  logs
* `LOG_NAME`: Name of the Stackdriver Logging log into which to write
  entries
2018-06-17 17:43:33 +02:00
Vincent Ambo
62745db1a0 feat(main): Support static service account credentials
Adds support for authenticating towards Google's APIs by using static
service account credentials which can be retrieved from the GCP
console.

Users can toggle the behaviour by specifying the
`GOOGLE_APPLICATION_CREDENTIALS` environment variable.
2018-06-17 17:43:33 +02:00
Vincent Ambo
d56fedbe8d chore(cargo): Add dependency on medallion JWT library 2018-06-17 17:43:33 +02:00
Vincent Ambo
c2308b5ba1 chore(main): Bump chunk size to 750
Still slightly below the Stackdriver limit. However, previous errors
seem to have mostly been related to JSON payloads - which has been
fixed in the previous commit.
2018-06-17 15:46:05 +02:00
Vincent Ambo
05fa476929 fix(main): Correct name of JSON payload field 2018-06-17 15:46:05 +02:00
Vincent Ambo
68189d4872 refactor(main): Log error responses from Stackdriver
In some cases Stackdriver seems to return error responses for batched
inserts.

This change will log the error response body and status from
Stackdriver for all insertion errors.
2018-06-17 15:46:05 +02:00
Vincent Ambo
87ab3c806c feat(main): Parse timestamps out of journald entries
Instead of relying on Stackdriver's ingestion timestamps, parse
timestamps out of journal entries and provide those to Stackdriver.

If a timestamp could not be parsed out of a log entry, the ingestion
time is used as the fallback.
2018-06-17 15:46:05 +02:00
Vincent Ambo
f626d88438 chore(cargo): Enable chrono with serde as a dependency 2018-06-17 15:46:05 +02:00
Vincent Ambo
bd19132eff chore(main): Send logs in smaller chunks of 250 entries
In some cases sending 1000 entries seemingly results in a vague "Bad
request" error.

This reduces the chunk size to something that should be more
manageable.
2018-06-16 21:38:53 +02:00
Vincent Ambo
54b03a8dad fix(main): Attempt to read with next_record before waiting
Without this fix new records are only "pushed out" when something
appends to the journal.
2018-06-16 21:38:53 +02:00
Vincent Ambo
b6c0610278 feat(build): Add toggle for tests to Nix derivation 2018-06-16 21:38:53 +02:00
Vincent Ambo
e4b4830a04 feat(main): Persist & load persisted cursor positions
Adds support for persisting the cursor position in a file (by default
`/var/journaldriver/cursor.pos`, overridable via the environment
variable `CURSOR_POSITION_FILE`).

This lets journaldriver resume from a known journal position after
service restarts.

This closes #3
2018-06-16 21:38:53 +02:00
Vincent Ambo
03a6ea742c
Merge pull request #5 from aprilabank/feat/json-payloads
feat(main): Implement parsing of JSON payloads
2018-06-16 18:28:49 +02:00
Vincent Ambo
10f23a9dfb feat(main): Implement parsing of JSON payloads
Stackdriver supports structured JSON payloads in addition to simple
plain-text payloads.

This commit introduces a new feature in which journaldriver will
attempt to parse incoming log messages into JSON vaues and forward
them as structured payloads if they are JSON objects.

Messages that can not be parsed into JSON objects will continue to be
forwarded as plain text messages.
2018-06-16 17:57:11 +02:00
Vincent Ambo
71f0afe4b5 fix(build): Update cargo dependency hash after deps change 2018-06-15 17:08:06 +02:00
Vincent Ambo
23286d13b9 docs: Add contribution guidelines
Adds contribution guidelines. These are a slightly modified version of
the guidelines I use for my personal projects.
2018-06-15 17:03:38 +02:00
Vincent Ambo
bb968ed8ed docs: Add code of conduct 2018-06-15 17:02:04 +02:00
Vincent Ambo
01ef180cff feat(build): Enable Nix builds in Travis.CI 2018-06-15 17:00:48 +02:00
Vincent Ambo
a47bd5675f chore: License under GPL 3.0 2018-06-15 17:00:26 +02:00
Vincent Ambo
e2504a2da9 docs(README): Add usage instructions & other information to README 2018-06-15 16:58:35 +02:00
Vincent Ambo
7258f31d29 feat(build): Add initial Nix build derivation 2018-06-15 16:47:48 +02:00
Vincent Ambo
4ef98fc2ba feat(main): Implement record conversion & flushing to API
This implements the "meat" of the initial version of journaldriver.

Records from journald are converted into the representation required
by Stackdriver and forwarded to the API.

In this initial version journaldriver is only supported on instances
running on GCP.
2018-06-15 16:45:17 +02:00
Vincent Ambo
2d8e057118 feat(main): Add fetching of tokens from metadata server 2018-06-14 16:48:43 +02:00
Vincent Ambo
2541d25fba feat(main): Emit output in chunks of max. 1000 records
Required by the Stackdriver API.
2018-06-05 15:24:03 +02:00
Vincent Ambo
ef638bfa20 feat(stackdriver): Add initial Stackdriver API type definitions 2018-05-28 22:24:32 +02:00
Vincent Ambo
0db4512df4 refactor(main): Simplify receiver
Departing from the initial approach. There's no reason to multithread
this for now.
2018-05-28 09:45:07 +02:00
Vincent Ambo
6793b25a67 feat(main): Implement receiver & flushing logic
The only thing missing for a 0.1 test run is the actual gRPC call to
Stackdriver.
2018-05-27 23:57:24 +02:00
Vincent Ambo
c5cd12b81f feat(journald): Implement initial libsystemd journal calls 2018-05-27 20:09:37 +02:00
Vincent Ambo
1ed238b449 feat(build): Configure linking to libsystemd 2018-05-27 20:09:20 +02:00
Vincent Ambo
869d74723e chore: Add project bootstrap 2018-05-27 20:09:13 +02:00
Vincent Ambo
44904c619e chore: Initial commit 2018-05-27 20:08:56 +02:00