Commit graph

12 commits

Author SHA1 Message Date
Florian Klink
0715163779 chore(tvix): bump opentelemetry to 0.27
Change-Id: I3afbd8c94e0bc2cdf30da1886ae05b922f8a718f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12822
Reviewed-by: Jonas Chevalier <zimbatm@zimbatm.com>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2024-11-23 09:40:21 +00:00
Vova Kryachko
fa9c067dc9 feat(nix-daemon): Implement more nix daemon operations.
In particular QueryPathFromHashPart, QueryValidPaths, QueryValidDerivers

Change-Id: Ie6ad83cec5ce9580044b85e201e4e23394f87075
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12762
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: flokli <flokli@flokli.de>
2024-11-13 21:21:57 +00:00
Vova Kryachko
b564ed9d43 feat(nix-daemon): Implement client handler.
This change includes only the basic nix handshake protocol handling and
sets up a client session. The only supported operation at this point is
SetOptions.

Additional operations will be implemented in subsequent cls.

Change-Id: I3eccd9e0ceb270c3865929543c702f1491768852
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12743
Autosubmit: Vladimir Kryachko <v.kryachko@gmail.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: Brian Olsen <me@griff.name>
2024-11-12 02:15:04 +00:00
edef
84a82f6f41 feat(users/edef/weave): use tracing_indicatif for progress
Progress bars :3

Change-Id: I770d0f8381521b6efc8b38c0db4d59c771887fee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12673
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-10-20 21:22:23 +00:00
edef
201d8f0cf2 feat(users/edef/weave): use FxHashSet and dedupe early
Deduping early saves a fair bit of memory, but the extra hashing is
costly.

We switch to FxHash, since we don't need a DoS-proof hash, but we do
need it to be *fast*.

Change-Id: Ic6b7010874c417862baa9b882593208c8dd1d5e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12648
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-10-19 17:01:38 +00:00
edef
06d2536eec feat(users/edef/weave): ingest roots in Parquet format
Parsing of store-paths.xz is now handled by //users/edef/fetchroots.

Change-Id: I78be5aada0c0a321ed79d80c9b615e5f997ac3e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12670
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-10-19 17:01:38 +00:00
edef
313899c291 refactor(users/edef/weave/swizzle): use polars streaming
This vastly reduces the memory requirements, so we can run in ~40G RAM.

Change-Id: I4952a780df294bd852a8b4682ba2fd59b9bae675
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12667
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-10-19 17:01:38 +00:00
edef
bdc2891053 refactor(users/edef/weave): simplify OwningRef handling
We switch to working with the underlying Arc<Bytes<u8>> type, since
Buffer<u8> is already essentially an OwningRef-esque subslice.

Because we're now working with an exposed Arc directly, we don't need
to have our own `unsafe impl StableAddress` any more.

Change-Id: I9ce2edc6899177145e15b72aa5380f708a62173c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12668
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-10-19 13:49:02 +00:00
edef
549c51f200 refactor(users/edef/weave): use swap rather than CAS
We only care about ordering on the same variable, and we rely on the
release barrier provided by rayon's thread joining.

The comparison failing is always an error path, and we're indifferent
about which thread it fails on.

Change-Id: I592a7eaae5b6935cf424c6576a49786f39909fb6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12666
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-10-19 13:49:02 +00:00
edef
25671c284f refactor(users/edef/weave): simplify DoubleEndedIterator bound
Thanks, Clippy!

Change-Id: I116b478124b72e070e94150ee850532752f64d60
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12669
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-10-19 13:49:02 +00:00
edef
dfff592784 fix(users/edef/weave): use safer_owning_ref
owning_ref has serious unsoundness.

Change-Id: Ie760697cd6399e6bc75f1ad17c9bb74adc077a35
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12656
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Autosubmit: edef <edef@edef.eu>
Reviewed-by: flokli <flokli@flokli.de>
2024-10-17 22:47:41 +00:00
Vincent Ambo
bb34210a64 chore(tvix/tools): move weave to //users/edef
This is not a core Tvix tool, it's a tool that uses a Tvix component.

Change-Id: I705f2c4ab87f1512e005007c933e16b84ed4279f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12605
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2024-10-12 23:11:25 +00:00