Commit graph

20252 commits

Author SHA1 Message Date
Florian Klink
de68d8321f feat(tvix/buildkite): apply a 10 minutes timeout
We should cap the time of a `cargo build && cargo test` to a reasonable
amount of time.

Change-Id: If618198459ac267f9f6d7eb60466534a95a95c2c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11404
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-13 11:44:37 +00:00
Florian Klink
cc42cac39c refactor(tvix/castore/import): rename ingest_entries function arg
This is a stream of DirEntry, so let's call it direntry_stream.

Change-Id: I5b3cb4efba899d746393f75f6ece7eaa79424717
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11401
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-13 10:51:58 +00:00
Florian Klink
820e98fa7f refactor(tvix/boot): use tvix-store copy
Instead of using an additional Nix build producing a directory structure
containing xz-compressed NAR files and NARInfos, and uploading these via
`curl` to NAR-bridge, which then decomposes them and uploads them to
tvix-store, have tvix-store interpret the exported reference graph
directly, and deal with the uploading on its own.

Change-Id: I44ee0b132944c53d11abb688d2d017f0cc9c3d97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11398
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-13 10:44:56 +00:00
Florian Klink
a4f65ddba0 feat(tvix/store/bin): add "copy" command
This allows copying individual store paths from the local /nix/store
into tvix-store.

As we don't support getting this information from Nix yet, we currently
expect metadata to be provided externally:

Nix' `exportReferencesGraph` feature contains pretty much all data we
need for this. Expect a list of this information at a well-known key
(`closure`, similar to nixpkgs' `pkgs/build-support/binary-cache/
default.nix`).

We currently simply upload all store paths sequentially, without any
parallelism or awareness in how the reference graph looks like.

As long as the connected stores don't enforce this, this is fine, at
least for now.

Change-Id: Ib83c998465adddfdb110db994843c44e26b3d3d8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11397
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-04-13 10:44:56 +00:00
Ryan Lahfa
863c4207cc feat(tvix/eval): contextful JSON operations
`toJSON` transform a Nix structure into a JSON string.

For each context in that Nix structure, the JSON string must possess it.

Thus, it is necessary to take the union of all contexts and attach it to
the final structure.

Unfortunately, the return type of `into_json` is a serde's JSON object,
not a string. Therefore, it is not possible to reuse `NixString`
machinery.

Context tests are reinforced as Nix does not test those behaviors.

Fixes b/393.

Change-Id: I5afdbc4e18dd70469192c1aa657d1049ba330149
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11266
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-13 10:11:18 +00:00
Florian Klink
45cf7ae657 refactor(tvix/nix-compat): move nar writer to tokio
There's little reason to keep the nar writer using Async{Read,Write}
traits from futures, while everything else async in tvix (and
nix-compat) uses tokio.

Change-Id: I8cd1efcd0dd5bb76471de997603c7b701a5095de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11391
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Brian Olsen <me@griff.name>
2024-04-13 10:09:35 +00:00
Florian Klink
742937d55c refactor(tvix/nix-compat): move worker_protocol into nix_daemon mod
This doesn't have much to do with the plain "wire" format, it's merely
one user of it.

Also, use the more "public" `wire::` API to read/write bytes, strings,
bools and u64s.

Change-Id: I98dddcc3004dfde7a0c009958fe84a840f77b188
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11390
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Brian Olsen <me@griff.name>
2024-04-13 10:09:35 +00:00
Florian Klink
36b296609b refactor(tvix/nix-compat): reorganize wire and bytes
Move everything bytes-related into its own module, and re-export
both bytes and primitive in a flat space from wire/mod.rs.

Expose this if a `wire` feature flag is set. We only have `async` stuff
in here.

Change-Id: Ia4ce4791f13a5759901cc9d6ce6bd6bbcca587c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11389
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-04-13 10:09:35 +00:00
Florian Klink
839c971a0f docs(tvix/eval): remove link to cargo manifest reference
We don't have it in all Cargo.toml files either.

Change-Id: Ifcb333a39603a7e402ab0068e0f0f1b5af769e13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11388
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
2024-04-13 10:09:35 +00:00
Florian Klink
eeca2d92e2 feat(tvix/nix-compat): add BytesReader
This adds AsyncRead counterpart for read_bytes.

Change-Id: I751da9944984c7a523abee305f8f8a050e705f04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11385
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
2024-04-13 10:09:35 +00:00
Florian Klink
82cf89ee6e feat(tvix/nix-compat): introduce ExportedPathInfo struct
This is another representation about information Nix has on a Store
Path. It's encountered in the exportReferencesGraph feature.

Change-Id: Ia0c08c369f4af9e50ebe3a5053bd98fadaa0003d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11396
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-13 10:03:50 +00:00
Florian Klink
329c4a1ace docs(tvix/nix-compat): rename as_narinfo() to to_narinfo()
This actually does a bit of allocation. There's two Vecs, one for
references and one for signatures.

We can get rid of the Vec at nar_hash.

Change-Id: Ie025309b6678f83f5b961d49ff75dcfc7da145a1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11395
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-13 10:03:50 +00:00
Florian Klink
3d7f80c0e3 feat(tvix/nix-compat): impl Deserialize, Serialize for Signature
Change-Id: I30294079129b0e5b4faa3272e09df982d3ef2178
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11394
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-13 10:03:50 +00:00
Florian Klink
a2322d7c14 feat(tvix/nix-compat): implement Serialize, Deserialize for NixHash
We use the (slightly more tolerant) from_str to deserialize, and
serialize out as SRI.

Change-Id: If76b0ed2d4e243904f02df34f6c90b976c0bab8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11393
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-13 10:03:50 +00:00
Florian Klink
57fba1f167 feat(nix-compat/store_path): impl [Partial]Ord for StorePathRef
Move the code implementing it from StorePath to StorePathRef, and have
the StorePath impls use that too.

Drop the debug_assert in every comparison - we have tests for this to
ensure it keeps working, and built up some confidence by piping a lot of
other store paths through it in the meantime.

Change-Id: I288bad3dfa597f68d63c4bcda7791f722b7a8ced
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11392
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-13 10:03:50 +00:00
Florian Klink
fd4c7c10a8 chore(3p/overlays/patches): vendor cbtemulator UDS patch
I updated https://github.com/googleapis/google-cloud-go/pull/9665 in the
meantime, and GH decided to GC the patches.

Vendor the patch in for now (manually stripping the `bigtable/` prefix
in the path). Hopefully the PR itself gets merged soon.

Change-Id: I5b7ba78ccaf5c792c1445818b23b52d6f17155a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11402
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-04-13 10:02:13 +00:00
Florian Klink
dcd68155f0 feat(tvix/store/pathinfo/grpc): instrument functions
Change-Id: Idb7d7144be1917fbaf83e9fd76c5b2ebb3df98d2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11400
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-12 22:32:37 +00:00
Florian Klink
7bcb896e48 feat(tvix/castore/directory/grpc): instrument functions
Change-Id: I9cc0a6a32184773597556ab5f9250257aa18ca4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11399
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-12 22:32:37 +00:00
Florian Klink
f8800ba189 chore(tvix): bump rstest to 0.19.0
Change-Id: Ib2f5e84fdb8be1210b3507da67d4fe84f061651e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11387
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-12 22:16:56 +00:00
Florian Klink
fbf31f45ef feat(tvix/store): add bigtable pathinfoservice backend
Put behind the "cloud" backend, like in the `tvix-castore` crate.

Change-Id: Ib38d198baf11ab2a4b6dc405121676147c424611
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11362
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-04-09 19:40:14 +00:00
Florian Klink
d6cadee941 fix(tvix/nix-compat): fix writing padding partially
We forgot to only write the part of the padding that has not been
written already.

Unfortunately it seems the tokio mocking thing does buffer small writes,
so the tests succeeds even with the bug :-/

Change-Id: I5a6cf04212d559728639427c57207094d507ec75
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11384
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
2024-04-09 19:18:09 +00:00
Connor Brewster
63116d8c21 fix(tvix): Avoid buffering file into memory in builtins.hashFile
Right now `builtins.hashFile` always reads the entire file into memory
before hashing, which is not ideal for large files. This replaces
`read_to_string` with `open_file` which allows calculating the hash of
the file without buffering it entirely into memory. Other callers can
continue to buffer into memory if they choose, but they still use the
`open_file` VM request and then call `read_to_string` or `read_to_end`
on the `std::io::Reader`.

Fixes b/380

Change-Id: Ifa1c8324bcee8f751604b0b449feab875c632fda
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11236
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-09 17:31:58 +00:00
Florian Klink
17849c5c00 feat(tvix/castore/directory): add bigtable backend
This adds a Directory service using
https://cloud.google.com/bigtable/docs/ as a K/V store.

Directory (closures) are put in individual keys.

We don't do any bucketed upload of directory closures (yet), as castore/
fs does query individually, does not request recursively (and buffers).
This will be addressed by store composition at some point.

Change-Id: I7fada45bf386a78b7ec93be38c5f03879a2a6e22
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11212
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-04-09 15:50:34 +00:00
Florian Klink
84ad8a0bbd fix(tvix/glue/derivation): fix __structuredAttrs "outputs" handling
In case structured attrs is enabled, rather than adding a
space-separated list to the "outputs" environment variable, a "proper"
list is added to the JSON itself, at the "outputs" key.

Fixes b/395.

Reported-By: Alyssa Ross <hi@alyssa.is>
Change-Id: I2834ede9cfcf49d5e81e1056bf8f9bb9b65ddad8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11386
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-08 21:43:08 +00:00
Florian Klink
f1496f5722 refactor(tvix/nix-compat/wire): rename padding_len to total_padding_len
Make it a bit more clear that this is the total padding length, not the
padding length we still need to write.

Change-Id: I9ff4aa16f256fda367b4b9295abf82ed01b1f989
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11383
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-08 16:51:50 +00:00
Florian Klink
24fd4e963a refactor(tvix/nix-compat/wire): mv Bytes{WriterState,PacketPosition}
This is perfectly fine to track the position inside a reader too, so
rename it to reflect that.
Also make the docstring a bit less write-specific.

Change-Id: I831b0a8fe44a2477d4af96fefc692b9aabc378f1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11382
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-04-08 16:27:13 +00:00
Florian Klink
35d70f94b7 fix(tvix/nix-compat/wire): fix BytesWriter test payloads
We actually did construct other payload sizes than the test function was
intended to, because I'm too stupid to count. Fix that, and also update
a leftover _debug in a test name.

Change-Id: I1bff8551f2078afe82f789646b851f2fcff3e82c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11381
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-08 16:26:12 +00:00
Picnoir
9825404388 refactor(tvix/nix-compat): move handshake function to nix-compat
Moving out the handshake route out of tvix-daemon to nix-compat.

We're bumping the protocol to version 37. It does not change anything
for us. Version 37 introduced a change in the build result
serialization. We do not implement that for now.

Tested tvix-daemon against Nix via the integration test to make sure
we did not break anything in the process.

Taking advantage of this refactor to remove the unused anyhow
dependency in tvix-daemon.

Change-Id: I8a2aad0b6dbb0aaa997446a612a365fd7d70336c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11379
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-08 12:32:19 +00:00
Florian Klink
c05f90e611 refactor(tvix/nix-compat/wire): express magics as u64
This allows using read_u64, write_u64, which is a bit easier to juggle
with.

Also, update names to align with the nix codebase, which makes it easier
to spot both the constant name as well as the value.
Leave the ASCII interpretation as a comment afterwards.

Change-Id: I0b9ab187acd22807e2785b0722aa4300dab37c51
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11378
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Autosubmit: flokli <flokli@flokli.de>
2024-04-08 08:40:57 +00:00
Florian Klink
fd749070e2 refactor(tvix/nix-compat/wire): move magic bytes to worker_protocol
`primitive.rs` implements reading and writing primitive (fixed-length)
types in the wire format, used in the the nix daemon protocol and NAR
format.

Move worker-protocol specific magic bytes to worker_protocol.rs (and
possibly further split there once needed)

Change-Id: If681c01e9460294619f1d000229b81f0ac745810
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11377
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-08 07:02:10 +00:00
Florian Klink
71a3855f09 feat(tvix/nix-compat/wire): have write_bytes accept AsRef<[u8]> payloads
This includes String, &str etc. An example testcase with &str is
provided.

Change-Id: I900186d6ceb52f52bd41ef4596524c1f5b52470b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11376
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-08 07:02:10 +00:00
Florian Klink
acee489866 refactor(tvix/nix-compat/wire): simplify write_bytes a bit
Use the same EMPTY_BYTES trick from BytesWriter to write out the
padding, rather than allocating a Vec.

Change-Id: Ifb4ba1b45b7388adbc135fc8e46fd3d3cedd30aa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11375
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-08 06:18:22 +00:00
Florian Klink
eb910dfa3a docs(tvix/nix-compat/wire): update docstrings
Only describe the format once (in `read_bytes`, and simplify the other
docstrings a bit.

Change-Id: Iff898f3c4173d506a357bc14bdffbf69c4c6e0e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11374
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Autosubmit: flokli <flokli@flokli.de>
2024-04-08 06:18:22 +00:00
Florian Klink
f6359bf91f refactor(tvix/nix-compat): drop read_bytes_unchecked
This isn't used anywhere so far, and if someone really wants to, they
can simply make the range open-ended.

Change-Id: Iae9bcaa1f7ea032dd3ee76c8c142a38b6b72894d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11373
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Tested-by: BuildkiteCI
2024-04-07 21:57:11 +00:00
Florian Klink
d00c75ea9c fix(nix-compat/store_path): fix Deserialize with borrow
We were wrongly using `'de` as a lifetime for both `Deserializer` and
`StorePathRef`.

This prevented Deserializing into a struct containing `StorePathRef`.

See https://serde.rs/lifetimes.html#the-deserializede-lifetime, the last
part of the paragraph:

The 'de lifetime should not appear in the type to which the Deserialize
impl applies.

- // Do not do this. Sooner or later you will be sad.
- impl<'de> Deserialize<'de> for Q<'de> {
+ // Do this instead.
+ impl<'de: 'a, 'a> Deserialize<'de> for Q<'a> {

This fixes it, and adds a test, deserializing into a `Container` struct.

It also fixes the existing test cases, deserialize_ref was actually
deserialize_owned, and deserialize_owned didn't exist yet - but they
alone are not enough to provoke the lifetime issues.

Change-Id: Iaed2832998cae5f192eafe7fd5243e82ff6e051e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11372
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-07 21:21:09 +00:00
Picnoir
efd8ab5e0b feat(users/picnoir/tvix-daemon): implement set_options operation
The protocol is more stateful than I initially thought. We need to
keep track to a bunch of things, including but not limited to: the
client settings, the client version. I moved things around a bit to
keep this state along with the client socket.

Change-Id: Ibd34fbe7821c20a460934ea1af0719f5de46e491
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11359
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-07 20:38:33 +00:00
Picnoir
199f9b0a79 feat(tvix/nix-compat): read client setting from wire
Add the primitives necessary to read the client settings from the Nix
daemon wire protocol.

Introducing the read_string primitive. This trivial primitive parses a
read_bytes call, check the bytes are valid utf-8 bytes and wraps the
result in a String.

Change-Id: Ie1253523a6bd4e31e7924e9898a0898109da2fa0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11358
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-07 20:30:03 +00:00
Florian Klink
289b3126db feat(tvix/castore): drop test-case crate dep
Change-Id: I5049a3682a58ce848d80f413b2964331025a90a8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11370
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-07 14:51:47 +00:00
Florian Klink
936a175b2f refactor(tvix/castore/directoryservice/from_addr): migrate to rstest
Change-Id: I52b14652822766421d66e95bc646ed7baecc705f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11369
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Tested-by: BuildkiteCI
2024-04-07 14:51:47 +00:00
Florian Klink
d94ff54d42 refactor(tvix/castore): migrate closure_validator to rstest
Change-Id: I6c594d2e670a681484b858c3e04bc25b9e5a2077
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11368
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-07 14:51:47 +00:00
Florian Klink
c715b6d448 refactor(tvix/castore/tonic): migrate to rstest
Change-Id: Ie088bf03c739bf64abf3432175362a8f92f501c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11367
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-07 14:51:47 +00:00
Florian Klink
71fb99a265 refactor(tvix/castore/hashing_reader): migrate to rstest
Change-Id: I99ae0e27b4db4799db8af7cd6b9cc8d7f09227de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11366
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-07 14:51:47 +00:00
Florian Klink
c7c66abd85 refactor(tvix/castore/blobservice/object_store): drop individual tests
This (and more) should now be covered by the generic testsuite
(in crate::blobservice::tests).

Change-Id: Ib3afc4f19f7e37a561b7398d43663dc941971f5c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11365
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-07 14:51:47 +00:00
Florian Klink
32ac9bd110 refactor(tvix/blobservice/from_addr): move from test_case to rstest
This allows conditionalizing test cases on feature flags.

Change-Id: Ic9ed9ef52f703c973fda2ca4aae5f425e33b67de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11364
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Tested-by: BuildkiteCI
2024-04-07 14:51:47 +00:00
Aspen Smith
bab3db2fa4 fix(aspen/system): Fix and reenable system build
Change-Id: I36f7a772e70be8711c03150b4b6dfcd6d4e299f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11371
Reviewed-by: aspen <root@gws.fyi>
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
2024-04-07 14:19:01 +00:00
sterni
6e2c143756 chore(3p/sources): Bump channels & overlays
- agenix has not been updated
  (https://github.com/ryantm/agenix/pull/241).

- Re-enable now fixed dependency of flokli/archeology-ec2.

Change-Id: I4e0399e5b5dbaf5e504076e029013f165dd4d191
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11363
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-06 18:04:14 +00:00
Florian Klink
f3d8c633f2 feat(tvix/nix-compat/wire): introduce BytesWriter
This deals with writing byte packets of larger sizes to an underlying
AsyncWrite.

Its constructor receives the expected size. It also deals with writing
padding if flush/shutdown is called after writing all the payload.

Change-Id: I8acbf992467f3862ffb8c7d669e8c0c8eced14c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11355
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
2024-04-06 14:12:05 +00:00
Florian Klink
39276dc5b4 feat(tvix/store/proto): avoid clone in PathInfo::validate()
Have this return a StorePathRef<'_>, rather than a StorePath, and leave
it up to the caller to possibly convert it to a owned StorePath.

This avoids some allocations, if we only want to validate.

Change-Id: I5cf8e246fe02bd4e631f46a5cb86d3f77a728a0d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11361
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-05 23:28:11 +00:00
Vincent Ambo
9e81063050 feat(tazjin/khamovnik): enable docker
Change-Id: I105aad32cfcebd7a1b9108e4fd847073fb30da15
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11360
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-04-05 15:53:56 +00:00
sterni
5262e5bf6c refactor(3p/mime4cl): use SB-POSIX for FILE-LENGTH
This is slightly better than the (mostly untested) mess we had before:
Just implement the one thing we need using the tools the one
implementation we support (SBCL) gives us.

Eventually, we'll want to make this portable, probably using osicat.
Unfortunately, packaging this requires support for cffi-grovel (b/383)
which buildLisp lacks at the moment.

Change-Id: I6960015f80e6a5dfde67baf55537c5274a19e4e2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11356
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
2024-04-05 11:01:39 +00:00