Commit graph

14 commits

Author SHA1 Message Date
Vincent Ambo
68189d4872 refactor(main): Log error responses from Stackdriver
In some cases Stackdriver seems to return error responses for batched
inserts.

This change will log the error response body and status from
Stackdriver for all insertion errors.
2018-06-17 15:46:05 +02:00
Vincent Ambo
87ab3c806c feat(main): Parse timestamps out of journald entries
Instead of relying on Stackdriver's ingestion timestamps, parse
timestamps out of journal entries and provide those to Stackdriver.

If a timestamp could not be parsed out of a log entry, the ingestion
time is used as the fallback.
2018-06-17 15:46:05 +02:00
Vincent Ambo
bd19132eff chore(main): Send logs in smaller chunks of 250 entries
In some cases sending 1000 entries seemingly results in a vague "Bad
request" error.

This reduces the chunk size to something that should be more
manageable.
2018-06-16 21:38:53 +02:00
Vincent Ambo
54b03a8dad fix(main): Attempt to read with next_record before waiting
Without this fix new records are only "pushed out" when something
appends to the journal.
2018-06-16 21:38:53 +02:00
Vincent Ambo
e4b4830a04 feat(main): Persist & load persisted cursor positions
Adds support for persisting the cursor position in a file (by default
`/var/journaldriver/cursor.pos`, overridable via the environment
variable `CURSOR_POSITION_FILE`).

This lets journaldriver resume from a known journal position after
service restarts.

This closes #3
2018-06-16 21:38:53 +02:00
Vincent Ambo
10f23a9dfb feat(main): Implement parsing of JSON payloads
Stackdriver supports structured JSON payloads in addition to simple
plain-text payloads.

This commit introduces a new feature in which journaldriver will
attempt to parse incoming log messages into JSON vaues and forward
them as structured payloads if they are JSON objects.

Messages that can not be parsed into JSON objects will continue to be
forwarded as plain text messages.
2018-06-16 17:57:11 +02:00
Vincent Ambo
a47bd5675f chore: License under GPL 3.0 2018-06-15 17:00:26 +02:00
Vincent Ambo
4ef98fc2ba feat(main): Implement record conversion & flushing to API
This implements the "meat" of the initial version of journaldriver.

Records from journald are converted into the representation required
by Stackdriver and forwarded to the API.

In this initial version journaldriver is only supported on instances
running on GCP.
2018-06-15 16:45:17 +02:00
Vincent Ambo
2d8e057118 feat(main): Add fetching of tokens from metadata server 2018-06-14 16:48:43 +02:00
Vincent Ambo
2541d25fba feat(main): Emit output in chunks of max. 1000 records
Required by the Stackdriver API.
2018-06-05 15:24:03 +02:00
Vincent Ambo
ef638bfa20 feat(stackdriver): Add initial Stackdriver API type definitions 2018-05-28 22:24:32 +02:00
Vincent Ambo
0db4512df4 refactor(main): Simplify receiver
Departing from the initial approach. There's no reason to multithread
this for now.
2018-05-28 09:45:07 +02:00
Vincent Ambo
6793b25a67 feat(main): Implement receiver & flushing logic
The only thing missing for a 0.1 test run is the actual gRPC call to
Stackdriver.
2018-05-27 23:57:24 +02:00
Vincent Ambo
c5cd12b81f feat(journald): Implement initial libsystemd journal calls 2018-05-27 20:09:37 +02:00