Commit graph

31 commits

Author SHA1 Message Date
Florian Klink
5161c3c072 feat(users/flokli/archivist): init
This introduces a new structure, as activities are moved to a separate
AWS Account for reasons (tm).

Change-Id: Ic310eca2dc0d4ee81bae8944238b26910067336a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10124
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-11-25 16:17:46 +00:00
Florian Klink
9c0586e3a7 feat(users/flokli/keyboard): align 3rd layer with thinkpad
Make these keys behave a bit more like the Fn+F* keys on the thinkpad
keyboards.

Seems there's currently no trivial way to get mic mute, keyboard mute
and wifi toggle sent out, but considering the thinkpad usb keyboard is
able to, this should be possible somehow here too - but not today, left
for a followup.

Change-Id: I529a958c78116dd9f7250c938e2e7989b296d6c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10076
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-11-19 12:24:32 +00:00
Florian Klink
abe099b6ba fix(users/flokli/archeology/parse_bucket_logs): fix regex and skip
It seems the regex is not perfect, it choked on a single log line:

```
Nov 13 03:10:19 archeology-ec2 59nkrwmih3ywaxrgxqj79pn395fs6m17-parse-bucket-logs-continuously[11105]: Code: 117. DB::Exception: Line "d57bd890fbd1ae16625bdb8168064125e013198099b7e1b3c24878a4d03c3ab8 nix-cache [12/Nov/2023:09:13:02 +0000] xxx.xx.xxx.xxx - VB7SJVZ108DSSN67 REST.POST.OBJECT index.html "POST /index.html HTTP/1.1" 405 MethodNotAllowed 348 - 4 - "-" "Mozilla/5.0 (Macintosh;                 Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML,                 like Gecko) Chrome/39.0.2171.95 Safari/537.36" - 0bFdGKbi0n9JHXU1a2hijcJwmYdc6lG2xgbdozc3wS6mlUkBE7ssrQCHIDdOLebo78o2cGbhivY= - ECDHE-RSA-AES128-GCM-SHA256 - nix-cache.s3.amazonaws.com TLSv1.2 - -" doesn't match the regexp.: (in file/uri log/2023-11-12-10-19-50-80805A702ECF65EB): (at row 5)
```

This was due to the user-agent field. The regex is now fixed.

The request itself is fun (someone trying to POST an index.html to the
bucket), and we should probably filter this on the Fastly side already,
not via IAM,

In any case, there's no point failing to parse if a single line doesn't
match the regex - we can just skip them.

For the sake of completeness, logs for that day have been reprocessed
and reuploaded.

Change-Id: Id98a7167a381cda06d150ad5118ee9e70ead277e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10034
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-11-14 21:46:09 +00:00
edef
1091b1e623 feat(users/flokli/archeology): install parquet-tools
Change-Id: I64cd83fbce920eeabace5b49ef623c033d98a8be
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10000
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-14 16:17:33 +00:00
edef
f467f0d801 feat(users/flokli/archeology): install DuckDB
Change-Id: I76bc20711c7e59d184659db134ba224cfcd7f6cb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9999
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-11-14 16:17:33 +00:00
edef
423ab20f43 feat(users/flokli/archeology): turn on task_delayacct
More ClickHouse perf stats ^_^

Change-Id: I4f6882b1a6c1ebfed9a430e62ca634a141cd1cf1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9998
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-13 14:53:41 +00:00
Florian Klink
160dac0a22 feat(users/flokli/keyboard): add missing KC_PSCR
Change-Id: I005defb868151ecec95e710523db3d23c859e489
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10021
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-13 09:38:20 +00:00
Florian Klink
a49d5d2279 feat(users/flokli/keyboard): init
This packages up my keyboard firmware used for the Keychron K6 Pro.

We add a custom keymap to the `keyboards/keychron/k6_pro/ansi/rgb/
keymaps` directory, a copy from the `default` one (with a modified
`keymap.c`), and then build that as a makefile target.

`via` is *disabled*, as their keybindings take priority over keymap.c.

Luckily, only `qmk` seems to be sufficient to build it.

A simple `:flash` target/script is provided as well, it relies on some
udev rules set in the global system
(`hardware.keyboard.qmk.enable = true`).

Change-Id: I9f7a7a992e13516c32033127f94e37aec62d6b67
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10020
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-13 08:18:33 +00:00
Florian Klink
c93086848f feat(users/flokli/archeology): add AWS config to shell
This allows using awscli inside a shell.

Clickhouse AWS SSO integration still seems broken unfortunately, even
with https://github.com/ClickHouse/ClickHouse/pull/54347 included in
our bump - it seems it's coming up with another token file path than the
AWS SDK:

> SSOCredentialsProvider: Unable to open token file on path: /home/flokli/.aws/sso/cache/da39a3ee5e6b4b0d3255bfef95601890afd80709.json

This is the sha1sum of the sso_start_url, not the sha1sum of the
session-name (nixos / f2f059b8b7298f1ad52636d67cef8b719aa83bf5).

Change-Id: Ia1bdec03c4f269a7415c42c90c1f4fd3d928f770
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10012
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-11-12 22:43:26 +00:00
Florian Klink
e4adca0880 feat(users/flokli/nixos/archeology-ec2): automate bucket log parsing
This adds a `parse-bucket-logs.{service,timer}`, running once every
night at 3AM UTC, figuring out the last time it was run and parsing
bucket logs for all previous days.

It invokes the `archeology-parse-bucket-logs` script to produce
a .parquet file with the bucket logs in `s3://nix-cache-log/log/` for
that day (inside a temporary directory), then on success uploads the
produced parquet file to
`s3://nix-archeologist/nix-cache-bucket-logs/yyyy-mm-dd.parquet`.

Change-Id: Ia75ca8c43f8074fbaa34537ffdba68350c504e52
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10011
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-11-12 16:46:06 +00:00
Florian Klink
46964f6d8f fix(users/flokli/archaeology): don't use file but column compression
Clickhouse also has column compression, configurable with the
output_format_parquet_compression_method setting.

It defaults to lz4, and the previous setting got a a zstd-compressed
parquet file with lz4 data.

Set output_format_parquet_compression_method to zstd instead, and sort
by timestamp before assembling the parquet file.

The existing files were updated to the same format with the following query:

```
SELECT * FROM file('bucket_logs_2023-11-11*.pq', 'Parquet', 'auto') ORDER BY timestamp ASC INTO OUTFILE 'bucket_logs_2023-11-11.parquet' SETTINGS output_format_parquet_compression_method = 'zstd'
```

Change-Id: Id63b14c82e7bf4b9907a500528b569a51e277751
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10008
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-11-11 19:49:13 +00:00
Florian Klink
281cb93ba8 feat(users/flokli/nixos/archeology-ec2): add parse-bucket-logs
This adds a `archeology-parse-bucket-logs` CLI tool to `$PATH`.

It can be invoked like this:

```
archeology-parse-bucket-logs http://nix-cache-log.s3.amazonaws.com/log/2023-11-10-00-* bucket_logs_2023-11-10-00.pq.zstd
````

… and will produce a zstd-compressed Parquet file for (roughly) that
time range.

As the EC2 instance credentials don't give access to the logs bucket
(yet), other AWS credentials need to be provided.

This can be accomplished by using "AWS_ACCESS_KEY_ID",
"AWS_SECRET_ACCESS_KEY", "AWS_SESSION_TOKEN" from
"Option 2: Manually add a profile to your AWS credentials file (Short-
term credentials)" in AWS IAM Identity Center.

Processing logs for a one-hour range takes a minute or two, the
resulting zstd-compressed Parquet file is around 40-80M in size.

Processing logs for a whole day takes some 25mins, due to the sheer
amount of data (12 GB of raw log data, distributed among 450k individual
files, 20Mio log lines), but at least clickhouse isn't able to parse the
resulting parquet file back in:

> Code: 36. DB::Exception: IOError: Couldn't deserialize thrift: MaxMessageSize reached

For future automation tasks, it's probably better to run this once an
hour, and further join the data later on.

Change-Id: I6c8108c0ec17dc8d4e2dbe923175553325210a5c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10007
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-11 12:24:23 +00:00
Florian Klink
aaf53614b3 fix(users/flokli/archeology): make clickhouse use ambient AWS creds
Rather than picking up from clickhouse-specific config files, this gets
it to pick up from the ambient environment, which is closer to (but not
the same as) the AWS default credentials chain.

Change-Id: I9c498c231974ed345c3e3d354ec230052b4d0ff2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10006
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-11 12:24:23 +00:00
Florian Klink
7075aacc9c feat(users/flokli/archeology): show clickhouse-local progress
This behaviour might change (or not), see https://github.com/ClickHouse/
ClickHouse/pull/42003, but as of now, a `--progress` will provide some
progress.

Change-Id: I4891b6e2f96f2656858e71f88a226d24f0d45dc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10005
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-11-11 12:24:23 +00:00
Florian Klink
314d1facd4 feat(users/flokli/archeology): add shell
Change-Id: Ic34fefdaac82fd1e23d248f2e5fec282384b8fc0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9984
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-11 12:24:23 +00:00
Florian Klink
5ae49c5ccf feat(users/flokli/archeology): init parse-bucket-logs
Change-Id: I096b6fed8c73ddd5a417f5183cc113356ffd98c9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9983
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-11 12:24:23 +00:00
Florian Klink
c37c9cc770 feat(users/flokli/nixos): add direnv support
Expose `deps` separately, add a direnv with PATH_add for it to bring
tooling into $PATH.

Change-Id: I432cd2b082cad89e08bef78dc4653e10e137cd6b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9842
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-30 14:32:18 +00:00
Florian Klink
5b03bebce8 feat(users/flokli/nixos): use lazydeps for shell
Avoid having to re-enter the shell whenever the config is changed.

Change-Id: Ib9f6bb4075e29acaeb4863d64c017695ca85b60b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9841
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-30 12:15:36 +00:00
Florian Klink
5d2789d1ad feat(users/flokli/archeology): add awscli, htop, kitty terminfo
Change-Id: Ib7ae1871a5d0b16a68c79b68e7e79fd302da79bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9840
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-30 12:15:36 +00:00
Florian Klink
f9323d5aa5 feat(users/flokli): use nix-copy-closure instead of nix copy
nix copy seems to stall on the EC2 box for unknown reasons.

Change-Id: I30639a52758814968d3b54d716522fb88db80cfe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9839
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-30 12:15:35 +00:00
Florian Klink
d83b574be7 feat(users/flokli): expose readTree targets to CI
Change-Id: I3ea801b47267f4c985c2ab5cb1b79b2659894307
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9838
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-30 12:15:35 +00:00
Florian Klink
9e2f1f4583 refactor(users/flokli): move common stuff to archeology profile
Change-Id: I8470c0a2416c0c397e009affb44f8c7a852cd526
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9837
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-30 12:15:35 +00:00
Florian Klink
71fa4110fa feat(users/flokli): add archeology-ec2
This add the EC2 box config to the repo.

Change-Id: Id7a888a2cfbf1454cd9f9465018df377e14b4e9f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9836
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-30 09:31:49 +00:00
Florian Klink
2513ddd2b7 feat(users/flokli/nixos): add deploy script
This adds a deploy-archeology script.

I tried getting morph to work first, but passing it a
depot.ops.nixos.nixosFor seems to be very hard - the NixOS module system
doesn't like the arguments it's called with.

Replace morph with a 3 line bash script, which assumes your ssh_config
contains config for an `archeology` host.

Change-Id: I2bf694c60ded39c201efbbb899f3b5512aa4d0f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9835
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-10-29 17:49:25 +00:00
Florian Klink
85cd7d64fe refactor(users/flokli): drop some optional args and complexity
Change-Id: Ifdcac829ede4ec469a7ce1b608e78bae11f2766b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9834
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-10-29 17:49:24 +00:00
Florian Klink
48b6242313 feat(users/flokli/nixos): init archeology
Change-Id: Ic31cb8030179ff37b1cc3d3d9241e2582cfe3e5e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9833
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
2023-10-29 16:31:35 +00:00
Florian Klink
8b09ae54b1 feat(users/flokli): add ASG Lightning talk presentation
This adds the materials for the lightning talk held at All Systems Go
Conference 2023, Berlin.

Talk lives at https://media.ccc.de/v/all-systems-go-2023-245-tvix-store

Change-Id: I114b1aec9f1953c148dd29ca88888c16b9fc655d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9751
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-16 11:49:06 +00:00
Florian Klink
b8ef3e5263 feat(users/flokli): add NixCon 2023 presentation
This adds the materials for the talk held at NixCon 2023, Darmstadt.

It seems it's not really possible to export a PDF out of this,
at least I couldn't get reveal-md to do it properly.

The video recordings live on
https://media.ccc.de/v/nixcon-2023-35254-tvix and
https://www.youtube.com/watch?v=j67prAPYScY .

Change-Id: Id27dc8c8637ffd455981141383c54d8d1484679e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9750
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-16 11:49:06 +00:00
Luke Granger-Brown
f190712b7f chore(gerrit): migrate OWNERS files to code-owners style
Change-Id: Iacc521dfdd4b4a2d5cef3920cf8189bcce35a488
2022-09-19 11:13:28 +00:00
Florian Klink
372b35dffa feat(ops/nixos/whitby): add flokli user
Change-Id: Ibdb5b498f8bbc837fffdb38cdf95499b279773aa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2683
Reviewed-by: lukegb <lukegb@tvl.fyi>
Tested-by: BuildkiteCI
2021-03-26 20:31:48 +00:00
Florian Klink
26eaae587b chore(flokli): init user folder
Change-Id: Id19eafb3f2cc7dfa1ec8c47cbe9c5766ac491516
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2682
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2021-03-26 20:31:48 +00:00