Commit graph

63 commits

Author SHA1 Message Date
Aspen Smith
756539a596 chore(3p/sources): Bump channels & overlays (2024-07-28)
* Treewide: re-run depotfmt

* //third_party/nixpkgs:html5validator: build with Python 3.11,
  dependency openstackdocstheme doesn't support 3.12

* //users/sterni/machines/ingeborg: adapt to poorly handled fcgiwrap
  module API change: https://github.com/NixOS/nixpkgs/pull/318599

* //tvix/*-go: regenerate protobuf files

* //third_party/nixpkgs:treefmt: Remove patch for merged pull request

* //users/flokli/ipu6-softisp: rebase, drop upstreamed kernel patches

Change-Id: Ie4e0df007c287e8cd6207683a9a25838aa5bd39a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11971
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-01 10:06:33 +00:00
sterni
71435d9546 chore(3p/sources): Bump channels & overlays
* agenix has not been updated (https://github.com/ryantm/agenix/pull/241).
* wasm-bindgen bumped to 0.2.92 in Rust WASM projects
* 3p/lisp: port lispPackages from stable channel

  The Lisp package set we are using (`pkgs.lispPackages`) is the "old
  old" package set, whereas we were supposed to have been using
  `pkgs.lispPackages_new` (which is the "old new" package set).

  Either way we missed that train, and now there's a "new new" package
  set, but with a twist: Lisp packages in nixpkgs are now tied to
  their compilers, so the most generic way to access them seems to be
  from `pkgs.sbclPackages`.

  Switching to the packages from the "new new" package set doesn't
  work: Lots of stuff stops building if we just switch the sources
  over, and not everything is trivially fixable.

  For now we stay on the lispPackages from the stable channel. We need
  to look into the migration later.

  Or rewrite panettone.
* tvix: update generated protobuf files
* 3p/nixpkgs: pick trunk from stable channel; newer versions try to
  read files and do network I/O during build, but don't print enough
  details in error messages to figure out why.
* 3p/overlays: remove tdlib override (nixpkgs is currently new enough)
* 3p/overlays: override telega.el sources while updates are lagging in
  nixpkgs
* users/flokli/ipu6-softisp: update firmware paths, which NixOS now
  stores zstd-compressed.

Change-Id: I5a7a6c8b5d0688461bca92b9e6d654356d3a1cf1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11711
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2024-06-08 15:20:48 +00:00
edef
ce1aa10b69 fix(users/flokli/archeology): clickhose -> clickhouse
Change-Id: I9709aa2b663df5d3fd95d48e463e9b087e4d4ce1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11629
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-14 17:58:37 +00:00
Florian Klink
8324913045 chore(users/flokli/kb/dilemma): bump qmk to latest releasewt
Change-Id: Ia0b4784384b0f1dda0f849f14e5619274dec1dd3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11631
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-12 11:36:59 +00:00
Florian Klink
2cdf8346a9 fix(users/flokli/keyboards/*): fix flash command
`qmk flash` disappeared from the `qmk` command itself.

See `qmk_cli/script_qmk.py`:

```
This program can be run from anywhere, with or without a qmk_firmware
repository. If a qmk_firmware repository can be located we will use that
to augment our available subcommands.
```

By pointing QMK_HOME to our firmware sources, we can make it re-appear.

Change-Id: I5348fff546e864773063fb1f712bf2abd198972e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11627
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
2024-05-11 12:36:01 +00:00
sterni
6e2c143756 chore(3p/sources): Bump channels & overlays
- agenix has not been updated
  (https://github.com/ryantm/agenix/pull/241).

- Re-enable now fixed dependency of flokli/archeology-ec2.

Change-Id: I4e0399e5b5dbaf5e504076e029013f165dd4d191
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11363
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-06 18:04:14 +00:00
Florian Klink
f7cdbbef45 chore(users/flokli/nixos): restore archeology-ec2 build
Temporarily remove parquet-tools, so this builds again.

Can be re-added once https://github.com/NixOS/nixpkgs/pull/301032 landed
upstream.

Change-Id: Ie74f014eb8158d5f529a5f1c55788a4edc5c805d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11347
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2024-04-03 11:19:34 +00:00
Florian Klink
c641fce62a feat(users/flokli/ipu6-softisp): expose qcam to $PATH
This allows testing the camera connection without pipewire in between:

```
LIBCAMERA_LOG_LEVELS=*:DEBUG qcam
Warning: Ignoring XDG_SESSION_TYPE=wayland on Gnome. Use QT_QPA_PLATFORM=wayland to run on Wayland anyway.
[0:26:31.326112005] [51496]  INFO IPAManager ipa_manager.cpp:143 libcamera is not installed. Adding '/nix/store/src/ipa' to the IPA search path
[0:26:31.326354338] [51496] DEBUG IPAModule ipa_module.cpp:334 ipa_ipu3.so: IPA module /nix/store/da6s46p3dk9x7fhh0qj0wrxrqrq7y8av-libcamera-0.2.0/lib/libcamera/ipa_ipu3.so is signed
[0:26:31.326417399] [51496] DEBUG IPAManager ipa_manager.cpp:245 Loaded IPA module '/nix/store/da6s46p3dk9x7fhh0qj0wrxrqrq7y8av-libcamera-0.2.0/lib/libcamera/ipa_ipu3.so'
[0:26:31.326484670] [51496] DEBUG IPAModule ipa_module.cpp:334 ipa_soft_simple.so: IPA module /nix/store/da6s46p3dk9x7fhh0qj0wrxrqrq7y8av-libcamera-0.2.0/lib/libcamera/ipa_soft_simple.so is signed
[0:26:31.326518633] [51496] DEBUG IPAManager ipa_manager.cpp:245 Loaded IPA module '/nix/store/da6s46p3dk9x7fhh0qj0wrxrqrq7y8av-libcamera-0.2.0/lib/libcamera/ipa_soft_simple.so'
[0:26:31.326559430] [51496]  INFO Camera camera_manager.cpp:284 libcamera v0.2.0
[0:26:31.326728389] [51505] DEBUG Camera camera_manager.cpp:69 Starting camera manager
[0:26:31.345213320] [51505] DEBUG DeviceEnumerator device_enumerator.cpp:230 New media device "intel-ipu6" created from /dev/media1
[0:26:31.345249850] [51505] DEBUG DeviceEnumerator device_enumerator_udev.cpp:96 Defer media device /dev/media1 due to 36 missing dependencies
[0:26:31.350335759] [51505] DEBUG DeviceEnumerator device_enumerator_udev.cpp:322 All dependencies for media device /dev/media1 found
[0:26:31.350365700] [51505] DEBUG DeviceEnumerator device_enumerator.cpp:258 Added device /dev/media1: intel-ipu6
[0:26:31.350738881] [51505] DEBUG Camera camera_manager.cpp:113 Found registered pipeline handler 'SimplePipelineHandler'
[0:26:31.350765332] [51505] DEBUG Camera camera_manager.cpp:113 Found registered pipeline handler 'PipelineHandlerIPU3'
[0:26:31.350779457] [51505] DEBUG Camera camera_manager.cpp:113 Found registered pipeline handler 'PipelineHandlerUVC'
```

Change-Id: I30b4ede89f90e2455bf1313b3c7c4d638089e5db
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11180
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-03-19 06:59:57 +00:00
Florian Klink
9948eb64d1 chore(users/flokli/ipu6-softisp): refresh libcamera patches
Refresh them with the patches from
https://patchwork.libcamera.org/cover/19663/.

This is still based off v0.2.0.

Change-Id: I875fd64e3bb71a95c92af1108a23d27c0f3494e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11179
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
2024-03-19 06:59:57 +00:00
Florian Klink
622efa86fa feat(users/flokli/ipu6-softisp): test pipewire & wireplumber
This ensures we also test building pipewire.

Change-Id: I7527ca2e4259f26fecd453f5da5d91bbd3862bb3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11178
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-03-19 06:59:52 +00:00
Vincent Ambo
3633d846f8 chore(3p/sources): bump channels & overlays (2024-03-17)
In hope that iwlwifi works again on this commit, and I don't actually
have to debug it.

Includes following changes:

* users/aspen: home-manager is shuffling around pinentry options again
* users/flokli: rebase ipu6-softisp patches to Linux 6.8
                make cl/11097 a separate patch
* ops/modules: remove unused (and now broken) v4l2loopback module

Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: I763f1f075778f2ed8db7803f87248c9dabde4213
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11174
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-03-18 16:28:40 +00:00
Florian Klink
c499e424d8 feat(users/flokli/ipu6-softisp): provide example closure
This introduces a NixOS configuration using config.nix as a NixOS
module, and also checks the firmware is present in the location it's
expected to.

Change-Id: I3ec4333d73fe1b28e10c589c1cf351c59372bb99
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11099
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-03-07 14:43:02 +00:00
Florian Klink
525388fde7 fix(users/flokli/ipu6-softisp): skip subdirs from readTree
There's no readTree targets in there.

Change-Id: I267e3c363ccf1084adb40738361ea1ce702e2a8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11098
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-03-07 14:29:17 +00:00
Florian Klink
4749e2e703 chore(users/flokli/ipu6-softisp): align with linux-firmware
The firmware files ended up in linux-firmware, and
https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/commit/?id=dbc294d3e49aa63a61bb4026b5784bae5b75aa2b
moved them to a ipu/ subpath, so update our patches to find them there,
and stop using the pkgs.ipu6-camera-bins package.

https://github.com/NixOS/nixpkgs/pull/290216 bumped our linux-firmware
image past this, so this should work for people tracking nixos-unstable.

Change-Id: Ic3eb9fb364c93ee0d10951451a59f7e98888b5d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11097
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-03-07 13:56:33 +00:00
Florian Klink
46249cc587 feat(users/flokli/kb/dilemma): fix grave key
Change-Id: I5037c95d1a69cc55e766466821a9406695641c0f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11036
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-02-27 12:22:46 +00:00
Florian Klink
7ecf43a1bd feat(users/flokli/kb/dilemma): fix clipboard
It seems linux, or my desktop environment don't support the
clipboard-related keycodes.

Replace them with Ctrl-C and Ctrl-V, which is good enough.

Change-Id: Ibac27b6e022157341ba6eb9f8f6ff62f9e97137b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11034
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-02-26 12:52:35 +00:00
Florian Klink
1f3fe7d41f feat(users/flokli/kb/dilemma): add RALT
äöüß.

Change-Id: I2e1df9dc504f892e9176cabf4a1479cd25a9c59f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11030
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-02-26 11:46:59 +00:00
Florian Klink
d5ec9a0c6f chore: update links to tvixbolt
Use bolt.tvix.dev everywhere.

Change-Id: Ifebbea60058418c12cde20a2a3879a8a7f5f830c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11022
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-02-24 10:17:56 +00:00
Florian Klink
24a089f87d chore(users/flokli/ipu6-softisp): simplify expression
nixos-unstable got the bump to libcamera 0.2, no need to also do this
here.
We only need to use the patched libcamera for pipewire and wireplumber
running on the system.

Change-Id: I4ba5d30f68a5ded56cba07a5ad4aaa84d2711b8f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10825
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-02-13 15:37:59 +00:00
Florian Klink
41cd3c44d2 chore(users/flokli/ipu6-softisp): inline kernel patches
"media: ov2740: Fix hts value" got backported into linux-stable:

```
commit c5883a8a3676b94fe7cefde97c8eec50bb879eb2
Author: Hans de Goede <hdegoede@redhat.com>
Date:   Mon Dec 4 13:39:42 2023 +0100

    media: ov2740: Fix hts value

    [ Upstream commit 3735228bbe3511f844e03dfcc4003fadb59dde23 ]

    HTS must be more then width, so the 1080 value clearly is wrong,
    this is then corrected with some weird math dividing clocks in
    to_pixels_per_line() which results in the hts getting multiplied by 2,
    resulting in 2160.

    Instead just directly set hts to the correct value of 2160 and
    drop to_pixels_per_line().

    Signed-off-by: Hans de Goede <hdegoede@redhat.com>
    Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
    Signed-off-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
    Signed-off-by: Sasha Levin <sashal@kernel.org>
```

So drop that patch from the list of kernel patches to apply (and vendor
in the patches).

Change-Id: I314ccb524d156c0f445ed923ce0600f27b8cc699
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10813
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: flokli <flokli@flokli.de>
2024-02-13 15:37:59 +00:00
Florian Klink
8699a2b945 fix(users/flokli/kb/dillemma): update comments
Some of the comments didn't yet account for this layout now being
colemak, and the function keys swapped to match stock Miryoku more
closely.

Change-Id: I38ae92153f80855ac21cc62cd603b7020b5be4ac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10787
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
2024-02-10 12:01:27 +00:00
Florian Klink
1dad7a144d chore(users/flokli/nixos): drop archeology nixos config
The machine at Hetzner is gone, only the one in EC2 remains.

Change-Id: Ia7266d56ef1174267b95086c51e6d80015c2f905
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10711
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: flokli <flokli@flokli.de>
2024-01-30 15:04:40 +00:00
Florian Klink
af9a8d372b feat(users/flokli/ipu6-softisp): init
This code adds support for the ipu6 webcams via libcamera, based on the work in
https://copr.fedorainfracloud.org/coprs/jwrdegoede/ipu6-softisp/.

It's supposed to be included in your NixOS configuration imports.

Change-Id: Ifb71999ad61161fa23506b97cb449f73fb1270e3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10709
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
2024-01-30 09:54:24 +00:00
Florian Klink
0b3245211e feat(users/flokli/kb/dilemma): switch to qmk upstream
Looks like the develop branch at least has everything that's needed.

Change-Id: Ic25d571f4e6fe2a45f8f11f917622fe50a39ce07
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10690
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-01-25 11:40:38 +00:00
Florian Klink
e81c02311c feat(users/flokli/kb/dilemma): enable taps
Change-Id: I08e4718879da3e832f38265b5741f3e0bfc586cb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10682
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-23 12:14:43 +00:00
Florian Klink
0ae751a89f feat(users/flokli/kb/dilemma): enable kinetic mouse mode
Change-Id: I959edf0996ecf0993131b554b2732cd3a265d4d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10680
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-22 13:37:34 +00:00
Florian Klink
a27edf6405 feat(users/flokli/kb/dilemma): configure miryoku
The original layout is pretty far from Miryoku. It's not using Colemak
DH, but QWERTY, and the rest also looks quite different.

Align with the official Miryoku reference, from
https://github.com/manna-harbour/miryoku/tree/master/docs/reference.

Notable differences:

 - Button layer left out (normally activated by holding a bottom row
   pinkie key)
 - Media layer is missing external power and bluetooth-related keys
   (understandably).

Change-Id: Icd8b2222c9a68650d9d4ee81aa96df0f4bc3bc99
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10673
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-20 17:00:57 +00:00
Florian Klink
da99d6db46 feat(users/flokli/kb/dilemma): disable VIA
Change-Id: I922e79233b4c9f7ee68352d155be830f4ca66644
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10671
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-20 10:28:01 +00:00
Florian Klink
8ab7489418 feat(users/flokli/kb/dilemma): init custom layout
So far, this is still a copy from the "via" target, but makes
customization possible.

Change-Id: I9e23808b72b874901a5f8c074763346181b05c1f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10670
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-20 10:28:01 +00:00
Florian Klink
93dfc65f10 feat(users/flokli/keyboards): add dilemma 3x5_3
This builds the stock "via" layout, which, in addition to via support,
contains some other usefulness from miryoku.

Change-Id: I967a9bfcc87e108d4c0a2cb37b512d0b39912f7a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10666
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-19 23:01:17 +00:00
Florian Klink
3af72dad39 chore(users/flokli): move keyboard to keyboards/k6_pro
Change-Id: I8039053ee1fb7ed4beb564e20f98adcfc9e79fc6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10665
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-01-19 23:00:16 +00:00
Florian Klink
5cfa5e26fb feat(users/flokli/keyboard): enable kinetic mode
Change-Id: Ieae46ff2fa4617c113852763980a5c55d3fb0b22
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10459
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-29 22:14:09 +00:00
Florian Klink
5161c3c072 feat(users/flokli/archivist): init
This introduces a new structure, as activities are moved to a separate
AWS Account for reasons (tm).

Change-Id: Ic310eca2dc0d4ee81bae8944238b26910067336a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10124
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-11-25 16:17:46 +00:00
Florian Klink
9c0586e3a7 feat(users/flokli/keyboard): align 3rd layer with thinkpad
Make these keys behave a bit more like the Fn+F* keys on the thinkpad
keyboards.

Seems there's currently no trivial way to get mic mute, keyboard mute
and wifi toggle sent out, but considering the thinkpad usb keyboard is
able to, this should be possible somehow here too - but not today, left
for a followup.

Change-Id: I529a958c78116dd9f7250c938e2e7989b296d6c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10076
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-11-19 12:24:32 +00:00
Florian Klink
abe099b6ba fix(users/flokli/archeology/parse_bucket_logs): fix regex and skip
It seems the regex is not perfect, it choked on a single log line:

```
Nov 13 03:10:19 archeology-ec2 59nkrwmih3ywaxrgxqj79pn395fs6m17-parse-bucket-logs-continuously[11105]: Code: 117. DB::Exception: Line "d57bd890fbd1ae16625bdb8168064125e013198099b7e1b3c24878a4d03c3ab8 nix-cache [12/Nov/2023:09:13:02 +0000] xxx.xx.xxx.xxx - VB7SJVZ108DSSN67 REST.POST.OBJECT index.html "POST /index.html HTTP/1.1" 405 MethodNotAllowed 348 - 4 - "-" "Mozilla/5.0 (Macintosh;                 Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML,                 like Gecko) Chrome/39.0.2171.95 Safari/537.36" - 0bFdGKbi0n9JHXU1a2hijcJwmYdc6lG2xgbdozc3wS6mlUkBE7ssrQCHIDdOLebo78o2cGbhivY= - ECDHE-RSA-AES128-GCM-SHA256 - nix-cache.s3.amazonaws.com TLSv1.2 - -" doesn't match the regexp.: (in file/uri log/2023-11-12-10-19-50-80805A702ECF65EB): (at row 5)
```

This was due to the user-agent field. The regex is now fixed.

The request itself is fun (someone trying to POST an index.html to the
bucket), and we should probably filter this on the Fastly side already,
not via IAM,

In any case, there's no point failing to parse if a single line doesn't
match the regex - we can just skip them.

For the sake of completeness, logs for that day have been reprocessed
and reuploaded.

Change-Id: Id98a7167a381cda06d150ad5118ee9e70ead277e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10034
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-11-14 21:46:09 +00:00
edef
1091b1e623 feat(users/flokli/archeology): install parquet-tools
Change-Id: I64cd83fbce920eeabace5b49ef623c033d98a8be
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10000
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-14 16:17:33 +00:00
edef
f467f0d801 feat(users/flokli/archeology): install DuckDB
Change-Id: I76bc20711c7e59d184659db134ba224cfcd7f6cb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9999
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-11-14 16:17:33 +00:00
edef
423ab20f43 feat(users/flokli/archeology): turn on task_delayacct
More ClickHouse perf stats ^_^

Change-Id: I4f6882b1a6c1ebfed9a430e62ca634a141cd1cf1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9998
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-13 14:53:41 +00:00
Florian Klink
160dac0a22 feat(users/flokli/keyboard): add missing KC_PSCR
Change-Id: I005defb868151ecec95e710523db3d23c859e489
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10021
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-13 09:38:20 +00:00
Florian Klink
a49d5d2279 feat(users/flokli/keyboard): init
This packages up my keyboard firmware used for the Keychron K6 Pro.

We add a custom keymap to the `keyboards/keychron/k6_pro/ansi/rgb/
keymaps` directory, a copy from the `default` one (with a modified
`keymap.c`), and then build that as a makefile target.

`via` is *disabled*, as their keybindings take priority over keymap.c.

Luckily, only `qmk` seems to be sufficient to build it.

A simple `:flash` target/script is provided as well, it relies on some
udev rules set in the global system
(`hardware.keyboard.qmk.enable = true`).

Change-Id: I9f7a7a992e13516c32033127f94e37aec62d6b67
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10020
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-13 08:18:33 +00:00
Florian Klink
c93086848f feat(users/flokli/archeology): add AWS config to shell
This allows using awscli inside a shell.

Clickhouse AWS SSO integration still seems broken unfortunately, even
with https://github.com/ClickHouse/ClickHouse/pull/54347 included in
our bump - it seems it's coming up with another token file path than the
AWS SDK:

> SSOCredentialsProvider: Unable to open token file on path: /home/flokli/.aws/sso/cache/da39a3ee5e6b4b0d3255bfef95601890afd80709.json

This is the sha1sum of the sso_start_url, not the sha1sum of the
session-name (nixos / f2f059b8b7298f1ad52636d67cef8b719aa83bf5).

Change-Id: Ia1bdec03c4f269a7415c42c90c1f4fd3d928f770
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10012
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-11-12 22:43:26 +00:00
Florian Klink
e4adca0880 feat(users/flokli/nixos/archeology-ec2): automate bucket log parsing
This adds a `parse-bucket-logs.{service,timer}`, running once every
night at 3AM UTC, figuring out the last time it was run and parsing
bucket logs for all previous days.

It invokes the `archeology-parse-bucket-logs` script to produce
a .parquet file with the bucket logs in `s3://nix-cache-log/log/` for
that day (inside a temporary directory), then on success uploads the
produced parquet file to
`s3://nix-archeologist/nix-cache-bucket-logs/yyyy-mm-dd.parquet`.

Change-Id: Ia75ca8c43f8074fbaa34537ffdba68350c504e52
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10011
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-11-12 16:46:06 +00:00
Florian Klink
46964f6d8f fix(users/flokli/archaeology): don't use file but column compression
Clickhouse also has column compression, configurable with the
output_format_parquet_compression_method setting.

It defaults to lz4, and the previous setting got a a zstd-compressed
parquet file with lz4 data.

Set output_format_parquet_compression_method to zstd instead, and sort
by timestamp before assembling the parquet file.

The existing files were updated to the same format with the following query:

```
SELECT * FROM file('bucket_logs_2023-11-11*.pq', 'Parquet', 'auto') ORDER BY timestamp ASC INTO OUTFILE 'bucket_logs_2023-11-11.parquet' SETTINGS output_format_parquet_compression_method = 'zstd'
```

Change-Id: Id63b14c82e7bf4b9907a500528b569a51e277751
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10008
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-11-11 19:49:13 +00:00
Florian Klink
281cb93ba8 feat(users/flokli/nixos/archeology-ec2): add parse-bucket-logs
This adds a `archeology-parse-bucket-logs` CLI tool to `$PATH`.

It can be invoked like this:

```
archeology-parse-bucket-logs http://nix-cache-log.s3.amazonaws.com/log/2023-11-10-00-* bucket_logs_2023-11-10-00.pq.zstd
````

… and will produce a zstd-compressed Parquet file for (roughly) that
time range.

As the EC2 instance credentials don't give access to the logs bucket
(yet), other AWS credentials need to be provided.

This can be accomplished by using "AWS_ACCESS_KEY_ID",
"AWS_SECRET_ACCESS_KEY", "AWS_SESSION_TOKEN" from
"Option 2: Manually add a profile to your AWS credentials file (Short-
term credentials)" in AWS IAM Identity Center.

Processing logs for a one-hour range takes a minute or two, the
resulting zstd-compressed Parquet file is around 40-80M in size.

Processing logs for a whole day takes some 25mins, due to the sheer
amount of data (12 GB of raw log data, distributed among 450k individual
files, 20Mio log lines), but at least clickhouse isn't able to parse the
resulting parquet file back in:

> Code: 36. DB::Exception: IOError: Couldn't deserialize thrift: MaxMessageSize reached

For future automation tasks, it's probably better to run this once an
hour, and further join the data later on.

Change-Id: I6c8108c0ec17dc8d4e2dbe923175553325210a5c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10007
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-11 12:24:23 +00:00
Florian Klink
aaf53614b3 fix(users/flokli/archeology): make clickhouse use ambient AWS creds
Rather than picking up from clickhouse-specific config files, this gets
it to pick up from the ambient environment, which is closer to (but not
the same as) the AWS default credentials chain.

Change-Id: I9c498c231974ed345c3e3d354ec230052b4d0ff2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10006
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-11 12:24:23 +00:00
Florian Klink
7075aacc9c feat(users/flokli/archeology): show clickhouse-local progress
This behaviour might change (or not), see https://github.com/ClickHouse/
ClickHouse/pull/42003, but as of now, a `--progress` will provide some
progress.

Change-Id: I4891b6e2f96f2656858e71f88a226d24f0d45dc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10005
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-11-11 12:24:23 +00:00
Florian Klink
314d1facd4 feat(users/flokli/archeology): add shell
Change-Id: Ic34fefdaac82fd1e23d248f2e5fec282384b8fc0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9984
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-11 12:24:23 +00:00
Florian Klink
5ae49c5ccf feat(users/flokli/archeology): init parse-bucket-logs
Change-Id: I096b6fed8c73ddd5a417f5183cc113356ffd98c9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9983
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-11 12:24:23 +00:00
Florian Klink
c37c9cc770 feat(users/flokli/nixos): add direnv support
Expose `deps` separately, add a direnv with PATH_add for it to bring
tooling into $PATH.

Change-Id: I432cd2b082cad89e08bef78dc4653e10e137cd6b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9842
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-30 14:32:18 +00:00
Florian Klink
5b03bebce8 feat(users/flokli/nixos): use lazydeps for shell
Avoid having to re-enter the shell whenever the config is changed.

Change-Id: Ib9f6bb4075e29acaeb4863d64c017695ca85b60b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9841
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-30 12:15:36 +00:00