This operation is particularly used when invoking the following
nix commands:
```
nix-store --add-fixed some-path
nix-store --add-fixed --recursive some-path
```
Change-Id: I0f9b129c838c00e10415881f1e6e0d7bc1d7a3a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12800
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Validation is done inside ingest_nar_and_hash and
is used by Fetch::NAR and the nar-bridge.
Change-Id: I7e2be4cc13d2447035f1e5a444f44b62339988bf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12836
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
With nar-bridge supporting zstd content-encoding, we don't need the
nginx zstd module and can re-enable http2.
We also need to propagate the Accept-Encoding sent by the client to
nar-bridge, so it actually knows it can send zstd.
This reduces the time measured in the microbenchmark from ~13s to this:
```
hyperfine 'rm -rf /tmp/cache; nix copy --from https://nixos.tvix.store/ --to "file:///tmp/cache?compression=none" /nix/store/jlkypcf54nrh4n6r0l62ryx93z752hb2-firefox-132.0'
Benchmark 1: rm -rf /tmp/cache; nix copy --from https://nixos.tvix.store/ --to "file:///tmp/cache?compression=none" /nix/store/jlkypcf54nrh4n6r0l62ryx93z752hb2-firefox-132.0
Time (mean ± σ): 4.880 s ± 0.207 s [User: 4.661 s, System: 2.377 s]
Range (min … max): 4.700 s … 5.274 s 10 runs
```
Change-Id: Id092307423636163ae95ef87ec8fa558b83ce0bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12835
Reviewed-by: Jörg Thalheim <joerg@thalheim.io>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
We previously didn't handle compression in nar-bridge, and left it up to
a fronting reverse proxy.
However, at least nginx with http2 enabled pins each connection to a
single core, causing compression to be limited by the throughput of a
single CPU.
Change-Id: Ia11c2ff5c012192b25eb8ad05dae5542a2d2f777
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12834
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: Jörg Thalheim <joerg@thalheim.io>
We don't need to calculate dependencies between and on targets that are
part of the parent target map since they will be skipped by buildkite
anyways. This speeds up 🦙 considerably for pipeline runs that have a
limited number of changed targets and a parent target map passed
in (i.e. pipeline runs of most CLs, but not canon runs). In my testing
it was about a minute faster (1/6 of the time 🦙 takes currently) for a
pipeline where under five drv targets changed. For the full
pipeline (i.e. no parentTargetMap) 🦙 takes about the same time as
before (it's a few seconds slower as is to be expected, but nothing
significant).
Change-Id: Ia5a80e142da8f40bc591e2c6cfaf48c325b2f577
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12818
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
With more implementing NixDeserialize, this error message changed.
Fixtures were regenerated by running:
```
TRYBUILD=overwrite cargo test -p nix-compat-derive-tests --all-features
```
Unfortunately, it's not possible to loop this into CI, as trybuild
invokes cargo during the build.
Change-Id: Ia0ab07d0907d21366845fe06e01df9fb1fe3e7cb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12831
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
We don't need a separate instance of opentelemetry-collector, alloy can
also do this job for us.
Change-Id: I1b671ba57d70b080f7db112e1afcfe2e0cbdd13e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12829
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Jonas Chevalier <zimbatm@zimbatm.com>
Tested-by: BuildkiteCI
These are quite bursty, and I've seen messages about getting rate
limited.
Change-Id: I73058140957cb5718971fa432c003c2d1b0305e3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12828
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Use grafana-alloy to collect system metrics.
Change-Id: I592e64ca722701d4f12e69a531a434b54954955a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12827
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We got some double-quoted strings at narinfo_str, and it didn't align
well with our other field names.
Change-Id: I5c08786d2c4435542bf39ff44b9d4ada5400550d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12826
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
We should be able to distinguish tvix-store and nar-bridge.
Change-Id: I616c8e0c1ce2dbacab92975582dd36141b673aa2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12824
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
This provides some global HTTP statistics.
Change-Id: I8bd3e034123154a49d94720b0c8d0c3babde5ae3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12557
Reviewed-by: Jonas Chevalier <zimbatm@zimbatm.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This creates and registers a global meter provider, which uses the same
mechanism to get notified of flushes.
Change-Id: I856a67f0b282d494de3b2c2a1b79c06ae8ffe252
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12556
Reviewed-by: Jonas Chevalier <zimbatm@zimbatm.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This enables routing of metrics to an instance of VictoriaMetrics, and
configures opentelemetry-collector to route metrics there.
Change-Id: If765191a4cc70ddcaad821d45132b96a10a12148
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12812
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Jonas Chevalier <zimbatm@zimbatm.com>
This is a fetch-through mirror of cache.nixos.org, hosted by NumTide.
The current machine is a SX65 Hetzner dedicated server with 4x22TB SATA disks,
and 2x1TB NVMe disks.
The goals of this machine:
- Exercise tvix-store and nar-bridge code
- Collect usage metrics (see https://nixos.tvix.store/grafana)
- Identify bottlenecks
- Replace cache.nixos.org?
Be however aware that there's zero availability guarantees. Since Tvix doesn't
support garbage collection yet, we either will delete data or order a bigger
box.
Change-Id: Id24baa18cae1629a06caaa059c0c75d4a01659d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12811
Tested-by: BuildkiteCI
Reviewed-by: Jonas Chevalier <zimbatm@zimbatm.com>
Reviewed-by: flokli <flokli@flokli.de>
This can be used to transitively enable the `xp-composition-cli` feature
flag in the `tvix-store` crate, which is unnecessarily hard to do with
crate2nix.
Change-Id: I3f7c505a3fd505561c9a7b2f063b6540532cdfd4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12809
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
This got lost somehow, but is necessary to keep `mg` in `$PATH`.
Change-Id: I2100d68225284bfe825bcc5ab01628891ebd09a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12810
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: flokli <flokli@flokli.de>
When sending nars over the wire to the nix-daemon, nix protocol versions
>= 1.23 use this framing protocol.
This change implements an AsyncRead for this protocol, to be used in
AddToStoreNar and any other operations when necessary.
Change-Id: I5f7972fe1c9ea145780bf449321bd3efeb833d18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12814
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
When sending nars over the wire to the nix-daemon, nix protocol versions
1.21 to 1.23 use this framing protocol.
This change implements an AsyncRead for this protocol, to be used in
AddToStoreNar and any other operations when necessary.
Change-Id: I571f1adbb2343c14c98503d1a2c12eea4c783ec9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12813
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
The TracingHandle::flush function allowed a user to pass in their
own (optional) oneshot::Sender<()> to get notified once the flush is
completed, but that's making things unnecessary complicated.
By simply having the flush() function await the flush, we make its
interface more intuitive, and callsites (only inside tvix-tracing itself
so far) simpler.
We can also remove the Option around the oneshot::Sender entirely, as we
now always call it with that.
For some more clarity, we can remove the channel from the struct fields
entirely if otlp support isn't compiled in.
Change-Id: I0870b9e8e88c6be6494a9c201c1c70b87e0f0810
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12801
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Marijan Petričević <marijan.petricevic94@gmail.com>
Tested-by: BuildkiteCI
This should get rid of some deprecation warnings in 🦙.
Change-Id: I8588169553a312a4df01d9bf71de7d673f28c431
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12808
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
* //3p/overlays/tvl: build nixos-option with latest Nix version (2.24)
as is required now. It would be nice to avoid this somehow to prevent
NixOS machines in depot having to carry around two versions of Nix.
Maybe we can at least use a statically linked nixos-option?
* //3p/{gerrit,gerrit_plugins}: update deps hash
* //tvix/eval: adjust our nixVersion “user agent” so that it'll pass the
new 2.3.17 minimum version nixpkgs prescribes (to check for zstd
support when substituting from the binary cache).
Change-Id: I4eb715afdc3dbb857340839f08ce86612aa7f117
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12805
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This machine hasn't existed for a while. Seems like I forgot to remove
the expression after its final month ran out.
Change-Id: I0e4abbd9af75eabfab0db106f851a1e43aa8c90f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12807
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: sterni <sternenseemann@systemli.org>
All repositories under code.sterni.lv are mirrors, so there's no value
in AI startups endlessly crawling the nixpkgs git history on
code.sterni.lv…
Change-Id: Iaac296315f325ced3cfd0852ae1d8d3f3815ea5b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12803
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
I want to add a warpper script to the mdmonitor program anyways, so
there's not really a point in this.
Change-Id: I92166bd44b54507b782a8d19b9676d91d8fa0f99
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12802
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: sterni <sternenseemann@systemli.org>
1.21.1 fixes an exploit in 1.21 without any other changes, so we can
safely skip it.
Change-Id: I72503c9f3869d7bafdfc78842b61804627a1d452
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12798
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This switches to a ZMK branch with support for mouse movement, and sets
MIRYOKU_KLUDGE_MOUSEKEYSPR so miryoku makes use of the functionality.
Change-Id: I3d4f48f10d50c202f909bec15189106a1bbcc1b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12796
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
`requires` is not recommended in this context. I think I tried it out of
desperation trying to combat the issue that these units would be started
before they were able to resolve names in switch-to-configuration.
Unfortunately, network access during switch-to-configuration can't be
detected using network-online.target, it seems.
Change-Id: Ia98a0a3b505ffa56eb37fb58a5375a1215d6cb1b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12794
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Mostly it's important (which I did this time at least) to remember that
with how things work at the moment, the initrd sshd is only reachable
via IPv4.
Change-Id: Ie9a87b6a38b2e128a8a2141d2221bbe7cfe24cdb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12792
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
I've discovered that it is possible for irccat to fail enough times to
run into the restart limit before network is online after booting.
Change-Id: Ia54a46d56bdc765a825fee50e7bdc8206718edc0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12790
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
* update wasm-bindgen in all wasm projects
* //users/wpcarro/website: declare missing dependency on
string-conversions. Presumably this was propagated
before from some other dependency which got updated now.
Change-Id: Ib93de576408974441d532196601e6e53d22cdafe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12770
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
I don't have this keyboard anymore, no need to build the firmware
anymore.
Change-Id: I1ab25dfaa71d03dc7006312ecab86453d6f90333
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12788
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
`buildSplitKeyboard`, as well as all the patching of `miryoku_zmk` is
independent of the specific keyboard used, so it can be moved one layer
up.
`config-flat` is now provided through a helper function, accepting the
name of the keymap to use when rendering the config.
This all makes the amount of code added for the new keyboard itself
pleasantly small.
Change-Id: I2216aa246502eddaf9bc4f4d126b0639d574ad87
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12787
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
These are required to support certain nix's local-overlay store
operations, it's safer to return empty results for these
operations than failing with "operation not implemented" errors.
Change-Id: Ic9b69d75dd52af5a826bfb6a8b283b082a0f6bcf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12766
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This change contains 2 improvements to the tvix-store copy command:
1. Allows reading the reference graph from stdin, using `-` argument
2. Supports json representation produced by `nix path-info --json`
command.
In general it makes is easier and faster to import arbitrary closures
from an existing nix store with e.g the following command:
```
nix path-info ./result --json --closure-size --recursive | \
jq -s '{closure: add}' | \
tvix-store copy -
```
Change-Id: Id6eea2993da233ecfbdc186f1a8c37735b686264
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12765
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
By manually writing out the async function, and moving the owned String
we can avoid for S to be Sync in the NixSerialize for StorePath<S> impl.
Co-Authored-By: edef@edef.eu
Change-Id: I8427b38d2bd61de2818088728cbad79cda69d17d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12764
Reviewed-by: Vladimir Kryachko <v.kryachko@gmail.com>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
