Commit graph

49 commits

Author SHA1 Message Date
Vincent Ambo
1942ec574f docs: Mention global log stream value in README 2019-02-04 16:31:12 +01:00
Vincent Ambo
6d2a8adacc feat: Add best-effort support for non-GCP Error Reporting
This adds support for a special log stream name called "global", which
instead of setting that as a log stream changes the resource
descriptor to be of the `global` monitored resource descriptor.

According to the information we have in #4 logs submitted to that
resource descriptor should be compatible with Stackdriver Error
Reporting. This is relevant when running journaldriver outside of GCP
as there is no way to utilise Error Reporting with the log stream
resource.

This relates to #4. I'm unsure about saying whether or not it "fixes"
it.
2019-02-04 16:31:12 +01:00
Vincent Ambo
9fed0bdb45 docs(README): Add note about inclusion in NixOS 18.09 2018-10-10 15:16:37 +02:00
Vincent Ambo
623a84f6f8 refactor: Require directory instead of file path for cursors
The previous change, which makes journaldriver write the cursor
position in two steps, requires that journaldriver can write files
adjacent to the cursor position file itself.

Instead of simply guessing that this is possible (e.g. by changing the
file suffix), expect the user to provide a directory that
journaldriver can work with.
2018-10-09 11:38:41 +02:00
Vincent Ambo
61b2577a19 fix: Refuse to write empty journald cursors and inform users
Exits from `persist_cursor` early if the cursor received from journald
is an empty string.

We don't currently know if this actually happens (please see #2 for
more details), so an error message has been added that asks users to
report this if it ever occurs.
2018-10-09 11:38:41 +02:00
Vincent Ambo
c1ab78c05a fix: Write cursor into temporary file and move it
This deals with a potential issue where creating a new file in place
of an existing cursor position file may cause position files to be
empty.

The cause for this is that the newly created file will truncate the
previous content, if journaldriver is then terminated before it
completes the cursor write to this file, it will not have written a
valid cursor (or anything at all).

Potentially relates to #2
2018-10-09 11:38:41 +02:00
Vincent Ambo
7ec507f8cb chore: Bump version to 1.1.0
... in preparation of a new release!
2018-10-06 01:04:56 +02:00
Vincent Ambo
55c4943df1 docs(README): Add notes about error reporting & minor improvements 2018-10-06 00:51:26 +02:00
Vincent Ambo
86c25cc226 refactor: Replace reqwest library with ureq
This replaces reqwest with the more simplistic ureq library for
performing required HTTP requests.

Reqwest comes with a lot of (tokio-based) machinery for
high-performance requesting that is a bit out of scope for
journaldriver's needs.

This clocks in at 62 (!) fewer dependencies after the change, with
equivalent functionality. Wew.
2018-10-05 23:58:01 +02:00
Vincent Ambo
a4084bf1e0 docs(README): Add note about log levels / priorities / severities
(kjære barn har mange navn :P)

Adds a small piece of documentation about the conversion between
journald priorities and Stackdriver severities to the README, as well
as information about how to easily emit messages at different
priorities from applications logging to journald.
2018-09-24 16:03:58 +02:00
Vincent Ambo
3b20abe0f8 docs(main): Expand information on priority->severity conversion
Adds a few additional notes about where to find more information and
how to log information correctly, as was discussed on issue #11.
2018-09-24 16:03:58 +02:00
Thomas ten Cate
a89be8d715 feat(main): Pass log levels along to Stackdriver
If a priority is present, it is passed as-is into the Stackdriver API.
This allows filtering by severity in the logs UI. Conveniently, the
levels are the same between journald and Stackdriver.

Fixes #11.
2018-09-24 16:03:58 +02:00
Vincent Ambo
d9569ad80a docs(README): Add section about new NixOS module 2018-06-24 23:43:08 +02:00
Vincent Ambo
375cd8d93f chore(nix): Bump version to 1.0.0 2018-06-18 20:14:54 +02:00
Vincent Ambo
7557d8ff5b refactor(main): Move cursor file to /var/lib/journaldriver
This directory is the default for state directories managed by
systemd, which will be the main use-case of journaldriver when running
it on NixOS.

Relates to nixos/nixpkgs#42134
2018-06-18 20:12:43 +02:00
Vincent Ambo
9f29992d03 refactor(main): Split LOG_NAME & LOG_STREAM variables
Splitting these in two lets users better configure how the logs show
up in Stackdriver Logging.
2018-06-17 22:36:01 +02:00
Vincent Ambo
208b5dec2c chore: Bump to 1.0-rc1 2018-06-17 18:22:13 +02:00
Vincent Ambo
7b6f4cb89b docs(README): Add non-GCP instructions to README 2018-06-17 17:43:33 +02:00
Vincent Ambo
79afc5f474 feat(main): Support manually configured resource descriptors
Lets users specify environment variables to configure the monitored
resource descriptor. In combination with service account support this
can be used to configure journaldriver on non-GCP machines.

The supported environment variables are:

* `GOOGLE_CLOUD_PROJECT`: Project ID of the project to which to report
  logs
* `LOG_NAME`: Name of the Stackdriver Logging log into which to write
  entries
2018-06-17 17:43:33 +02:00
Vincent Ambo
62745db1a0 feat(main): Support static service account credentials
Adds support for authenticating towards Google's APIs by using static
service account credentials which can be retrieved from the GCP
console.

Users can toggle the behaviour by specifying the
`GOOGLE_APPLICATION_CREDENTIALS` environment variable.
2018-06-17 17:43:33 +02:00
Vincent Ambo
d56fedbe8d chore(cargo): Add dependency on medallion JWT library 2018-06-17 17:43:33 +02:00
Vincent Ambo
c2308b5ba1 chore(main): Bump chunk size to 750
Still slightly below the Stackdriver limit. However, previous errors
seem to have mostly been related to JSON payloads - which has been
fixed in the previous commit.
2018-06-17 15:46:05 +02:00
Vincent Ambo
05fa476929 fix(main): Correct name of JSON payload field 2018-06-17 15:46:05 +02:00
Vincent Ambo
68189d4872 refactor(main): Log error responses from Stackdriver
In some cases Stackdriver seems to return error responses for batched
inserts.

This change will log the error response body and status from
Stackdriver for all insertion errors.
2018-06-17 15:46:05 +02:00
Vincent Ambo
87ab3c806c feat(main): Parse timestamps out of journald entries
Instead of relying on Stackdriver's ingestion timestamps, parse
timestamps out of journal entries and provide those to Stackdriver.

If a timestamp could not be parsed out of a log entry, the ingestion
time is used as the fallback.
2018-06-17 15:46:05 +02:00
Vincent Ambo
f626d88438 chore(cargo): Enable chrono with serde as a dependency 2018-06-17 15:46:05 +02:00
Vincent Ambo
bd19132eff chore(main): Send logs in smaller chunks of 250 entries
In some cases sending 1000 entries seemingly results in a vague "Bad
request" error.

This reduces the chunk size to something that should be more
manageable.
2018-06-16 21:38:53 +02:00
Vincent Ambo
54b03a8dad fix(main): Attempt to read with next_record before waiting
Without this fix new records are only "pushed out" when something
appends to the journal.
2018-06-16 21:38:53 +02:00
Vincent Ambo
b6c0610278 feat(build): Add toggle for tests to Nix derivation 2018-06-16 21:38:53 +02:00
Vincent Ambo
e4b4830a04 feat(main): Persist & load persisted cursor positions
Adds support for persisting the cursor position in a file (by default
`/var/journaldriver/cursor.pos`, overridable via the environment
variable `CURSOR_POSITION_FILE`).

This lets journaldriver resume from a known journal position after
service restarts.

This closes #3
2018-06-16 21:38:53 +02:00
Vincent Ambo
03a6ea742c
Merge pull request #5 from aprilabank/feat/json-payloads
feat(main): Implement parsing of JSON payloads
2018-06-16 18:28:49 +02:00
Vincent Ambo
10f23a9dfb feat(main): Implement parsing of JSON payloads
Stackdriver supports structured JSON payloads in addition to simple
plain-text payloads.

This commit introduces a new feature in which journaldriver will
attempt to parse incoming log messages into JSON vaues and forward
them as structured payloads if they are JSON objects.

Messages that can not be parsed into JSON objects will continue to be
forwarded as plain text messages.
2018-06-16 17:57:11 +02:00
Vincent Ambo
71f0afe4b5 fix(build): Update cargo dependency hash after deps change 2018-06-15 17:08:06 +02:00
Vincent Ambo
23286d13b9 docs: Add contribution guidelines
Adds contribution guidelines. These are a slightly modified version of
the guidelines I use for my personal projects.
2018-06-15 17:03:38 +02:00
Vincent Ambo
bb968ed8ed docs: Add code of conduct 2018-06-15 17:02:04 +02:00
Vincent Ambo
01ef180cff feat(build): Enable Nix builds in Travis.CI 2018-06-15 17:00:48 +02:00
Vincent Ambo
a47bd5675f chore: License under GPL 3.0 2018-06-15 17:00:26 +02:00
Vincent Ambo
e2504a2da9 docs(README): Add usage instructions & other information to README 2018-06-15 16:58:35 +02:00
Vincent Ambo
7258f31d29 feat(build): Add initial Nix build derivation 2018-06-15 16:47:48 +02:00
Vincent Ambo
4ef98fc2ba feat(main): Implement record conversion & flushing to API
This implements the "meat" of the initial version of journaldriver.

Records from journald are converted into the representation required
by Stackdriver and forwarded to the API.

In this initial version journaldriver is only supported on instances
running on GCP.
2018-06-15 16:45:17 +02:00
Vincent Ambo
2d8e057118 feat(main): Add fetching of tokens from metadata server 2018-06-14 16:48:43 +02:00
Vincent Ambo
2541d25fba feat(main): Emit output in chunks of max. 1000 records
Required by the Stackdriver API.
2018-06-05 15:24:03 +02:00
Vincent Ambo
ef638bfa20 feat(stackdriver): Add initial Stackdriver API type definitions 2018-05-28 22:24:32 +02:00
Vincent Ambo
0db4512df4 refactor(main): Simplify receiver
Departing from the initial approach. There's no reason to multithread
this for now.
2018-05-28 09:45:07 +02:00
Vincent Ambo
6793b25a67 feat(main): Implement receiver & flushing logic
The only thing missing for a 0.1 test run is the actual gRPC call to
Stackdriver.
2018-05-27 23:57:24 +02:00
Vincent Ambo
c5cd12b81f feat(journald): Implement initial libsystemd journal calls 2018-05-27 20:09:37 +02:00
Vincent Ambo
1ed238b449 feat(build): Configure linking to libsystemd 2018-05-27 20:09:20 +02:00
Vincent Ambo
869d74723e chore: Add project bootstrap 2018-05-27 20:09:13 +02:00
Vincent Ambo
44904c619e chore: Initial commit 2018-05-27 20:08:56 +02:00