Commit graph

20221 commits

Author SHA1 Message Date
Florian Klink
c19bc95b8a fix(tvix/castore/blobservice): update bytes_read only on successful read
We previously updated this.pos also in case the underlying read returned
an error.

Also, use the ready! macro to remove the match block, and instrument
errors returned during start_seek.

Change-Id: Ic32e26579d964a76b45687134acc48d72d67c36f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11421
Reviewed-by: Brian Olsen <me@griff.name>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-04-15 14:04:47 +00:00
Florian Klink
bccb4c92c3 refactor(tvix/castore/fs): add InodeData::as_fuse_{entry,file_attr}
Remove the now unused gen_file_attr (which had a wrong docstring).

Change-Id: Ie86b14d1ad798e6233bc44c43ace3f8b95c67ea9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11430
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 13:59:46 +00:00
Florian Klink
50065afd36 refactor(tvix/castore/fs): remove "add … handle" debug messages
Change-Id: Iac22bbef96a2afa0416f011d073934b52b19975d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11433
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-15 13:54:14 +00:00
Florian Klink
d3d88431f3 feat(tvix/castore/fs): assign read[dir*]/release[dir] ops to parent span
When a directory or file is open()'ed, we already put some data into
a lookup table, and subsequent operations then use the returned handle
id.

By also adding the span that's been created during these calls into the
lookup table, we can properly set the span parent for these requests,
nicely connecting the individual operations to the bigger picture.

Change-Id: Ia354842fccdbc7f45c2d3efda3acf058b2dbc48e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11429
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
2024-04-15 13:54:14 +00:00
Florian Klink
34d9d54aae fix(tvix/castore/fs): use record to add fields to current span
Instead of creating another child span, we can use
`tracing::Span::current().record(k,v)` to add an additional field to the
current span.

Change-Id: I337faac0e73a0da6eb0a52cb75c2e8c026eff774
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11428
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-15 13:23:33 +00:00
Florian Klink
ff5835008f feat(tvix/castore/fs): implement readdirplus
This currently shares some code with readdir, except it's also providing
a second `fuse_backend_rs::api::filesystem::Entry` argument to the
`add_entry` function call.

Refactoring this to reduce some duplication is left for a future CL.

Change-Id: I282c8dfc6a711d00a4482c87cbb84d4950c0aee9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11426
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-15 13:22:08 +00:00
Florian Klink
456191e69e refactor(tvix/castore/fs): use consistent span field name for handle
Use rq.handle in `release` too, and remove interpolating it into the
log message itself.

Also update the comment, we don't get ownership, just simply drop, and
change the level to warn!, as suggested in cl/11425.

Change-Id: If4e6cff6d8b580671b1548ae3862851db4af6694
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11427
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-04-15 13:20:07 +00:00
Florian Klink
2a47ad1d90 feat(tvix/castore/fs): implement opendir/releasedir
Similar to how we already handle opening files, implement opendir/
releasedir, and keep a map of dir_handles. They point to the rx side of
a channel.

This greatly improves performance listing the root of the filesystem
when used inside tvix-store, as we don't need to re-request the listing
(and skip to the desired position) all the time.

Change-Id: I0d3ec4cb70a8792c5a1343439cf47d78d9cbb1d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11425
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-15 13:18:36 +00:00
Florian Klink
3ed7eda79b refactor(tvix/castore/fs): use std::sync::Mutex
This allows us acquiring the lock in sync code still. Also, simplify
some of the error handling a bit.

Change-Id: I29e83b715f92808e95ecb0ae9de787339d1a371d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11424
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-15 09:27:04 +00:00
Florian Klink
1bf6b9f5a0 refactor(tvix/castore/fs): simplify some separate spawn and blocks
We can just pass an async move closure to `self.tokio_handle.block_on`
and make this a bit shorter.

Change-Id: Iba674f34f22ba7a7de7c5bae59d64584884cb17c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11423
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-15 09:27:04 +00:00
Florian Klink
515bfa18fb feat(tvix/castore/fs): support extended attributes
This exposes `user.tvix.castore.{blob,directory}.digest` xattr keys for
files and directories:

```
❯ getfattr -d /tmp/tvix/06jrrv6wwp0nc1m7fr5bgdw012rfzfx2-nano-7.2-info
getfattr: Removing leading '/' from absolute path names
user.tvix.castore.directory.digest="b3:SuYDcUM9RpWcnA40tYB1BtYpR0xw72v3ymhKDQbBfe4="

❯ getfattr -d /tmp/tvix/156a89x10c3kaby9rgf3fi4k0p6r9wl1-etc-shells
getfattr: Removing leading '/' from absolute path names
user.tvix.castore.blob.digest="b3:pZkwZoHN+/VQ8wkaX0wYVXZ0tV/HhtKlSqiaWDK7uRs="
```

It's currently mostly used for debugging, though it might be useful for
tvix-castore-aware syncing programs using the filesystem too.

Change-Id: I26ac3cb9fe51ffbf7f880519f26741549cb5ab6a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11422
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-04-15 09:27:04 +00:00
Ilan Joselevich
23871649bb feat(tvix/nix-compat/path_info): derive Hash for ExportedPathInfo
Change-Id: Icd2431f27043f120eee1d156cfcdc9bc85519ead
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11412
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-13 21:41:47 +00:00
Florian Klink
b70e01a4db feat(tvix/castore/import): remove copying in find_ancestor
We don't need to copy if we explicitly say that the returned
Option<Path> may hold onto bytes from the passed in &DirEntry.

Change-Id: Ib46b6fd2f8f19a45f8bef79c4c1d2fa6b490cad7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11410
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-13 13:20:29 +00:00
Ryan Lahfa
a919efbe11 fix(tvix/eval): fix the unsafeGetAttrPos stub return value
It was returning `{ line, col, file }` instead of `{ line, column, file
}`.

Note this is not a regression in Nix, Nix always had `column` since
2013.

Change-Id: I4871a201b32324dff0432526d0449806a535828f
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11409
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-13 13:16:02 +00:00
Florian Klink
01f8ef1560 fix(tvix): apply cbtemulator patch in tvix view
This tells josh to include the patch from
third_party/overlays/patches/cbtemulator-uds.patch at
nixpkgs/cbtemulator-uds.patch.

We then use the `additionalOverlays` feature introduced earlier, and
pass it an overlay adding the patch to `cbtemulator`.

This only gets applied if tvix/shell.nix is not called with a pkgs
argument, which is the case in the tvix view.

Change-Id: I7bc5543460ddd30325ab0998e891c15fcaf8bcfe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11408
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2024-04-13 12:09:47 +00:00
Florian Klink
e5b87d5496 feat(3p/nixpkgs): add additionalOverlays optional arg
This allows applying additional overlays to nixpkgs before
instantiating it.

Change-Id: Iee486086e13bb73e6bd20a817b1106c3cd99c935
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11407
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-13 12:01:05 +00:00
Florian Klink
31e3382129 feat(tvix/*store/bigtable): limit retries connecting to cbtemulator
This kept retrying indefinitely if the socket didn't appear.

Change-Id: I4d4ef61df73cef6abda698501432f370abc8a82c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11406
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-13 12:01:00 +00:00
Florian Klink
0dd421414e feat(tvix/nix-compat/wire/bytes): add #[inline] for ensure_* funcs
While the compiler might realize it's a good idea to inline these,
especially considering this is in the same crate [1], it doesn't hurt to
add that annotation too.

Suggested in cl/11385.

[1]: https://nnethercote.github.io/perf-book/inlining.html

Change-Id: If679fce0e1aab9ec681ba60f689d386d6fd92d61
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11403
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2024-04-13 11:45:38 +00:00
Florian Klink
f6bb8dbe1c chore(3p/nixpkgs): drop most of cbtemulator expression
We bumped nixpkgs past that bump, so we only need to carry our
downstream UDS support patch.

Change-Id: I56504d71a8327cba2e43aa36b7b657f93cf35e51
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11405
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-13 11:44:37 +00:00
Florian Klink
de68d8321f feat(tvix/buildkite): apply a 10 minutes timeout
We should cap the time of a `cargo build && cargo test` to a reasonable
amount of time.

Change-Id: If618198459ac267f9f6d7eb60466534a95a95c2c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11404
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-13 11:44:37 +00:00
Florian Klink
cc42cac39c refactor(tvix/castore/import): rename ingest_entries function arg
This is a stream of DirEntry, so let's call it direntry_stream.

Change-Id: I5b3cb4efba899d746393f75f6ece7eaa79424717
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11401
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-13 10:51:58 +00:00
Florian Klink
820e98fa7f refactor(tvix/boot): use tvix-store copy
Instead of using an additional Nix build producing a directory structure
containing xz-compressed NAR files and NARInfos, and uploading these via
`curl` to NAR-bridge, which then decomposes them and uploads them to
tvix-store, have tvix-store interpret the exported reference graph
directly, and deal with the uploading on its own.

Change-Id: I44ee0b132944c53d11abb688d2d017f0cc9c3d97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11398
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-13 10:44:56 +00:00
Florian Klink
a4f65ddba0 feat(tvix/store/bin): add "copy" command
This allows copying individual store paths from the local /nix/store
into tvix-store.

As we don't support getting this information from Nix yet, we currently
expect metadata to be provided externally:

Nix' `exportReferencesGraph` feature contains pretty much all data we
need for this. Expect a list of this information at a well-known key
(`closure`, similar to nixpkgs' `pkgs/build-support/binary-cache/
default.nix`).

We currently simply upload all store paths sequentially, without any
parallelism or awareness in how the reference graph looks like.

As long as the connected stores don't enforce this, this is fine, at
least for now.

Change-Id: Ib83c998465adddfdb110db994843c44e26b3d3d8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11397
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-04-13 10:44:56 +00:00
Ryan Lahfa
863c4207cc feat(tvix/eval): contextful JSON operations
`toJSON` transform a Nix structure into a JSON string.

For each context in that Nix structure, the JSON string must possess it.

Thus, it is necessary to take the union of all contexts and attach it to
the final structure.

Unfortunately, the return type of `into_json` is a serde's JSON object,
not a string. Therefore, it is not possible to reuse `NixString`
machinery.

Context tests are reinforced as Nix does not test those behaviors.

Fixes b/393.

Change-Id: I5afdbc4e18dd70469192c1aa657d1049ba330149
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11266
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-13 10:11:18 +00:00
Florian Klink
45cf7ae657 refactor(tvix/nix-compat): move nar writer to tokio
There's little reason to keep the nar writer using Async{Read,Write}
traits from futures, while everything else async in tvix (and
nix-compat) uses tokio.

Change-Id: I8cd1efcd0dd5bb76471de997603c7b701a5095de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11391
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Brian Olsen <me@griff.name>
2024-04-13 10:09:35 +00:00
Florian Klink
742937d55c refactor(tvix/nix-compat): move worker_protocol into nix_daemon mod
This doesn't have much to do with the plain "wire" format, it's merely
one user of it.

Also, use the more "public" `wire::` API to read/write bytes, strings,
bools and u64s.

Change-Id: I98dddcc3004dfde7a0c009958fe84a840f77b188
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11390
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Brian Olsen <me@griff.name>
2024-04-13 10:09:35 +00:00
Florian Klink
36b296609b refactor(tvix/nix-compat): reorganize wire and bytes
Move everything bytes-related into its own module, and re-export
both bytes and primitive in a flat space from wire/mod.rs.

Expose this if a `wire` feature flag is set. We only have `async` stuff
in here.

Change-Id: Ia4ce4791f13a5759901cc9d6ce6bd6bbcca587c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11389
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-04-13 10:09:35 +00:00
Florian Klink
839c971a0f docs(tvix/eval): remove link to cargo manifest reference
We don't have it in all Cargo.toml files either.

Change-Id: Ifcb333a39603a7e402ab0068e0f0f1b5af769e13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11388
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
2024-04-13 10:09:35 +00:00
Florian Klink
eeca2d92e2 feat(tvix/nix-compat): add BytesReader
This adds AsyncRead counterpart for read_bytes.

Change-Id: I751da9944984c7a523abee305f8f8a050e705f04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11385
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
2024-04-13 10:09:35 +00:00
Florian Klink
82cf89ee6e feat(tvix/nix-compat): introduce ExportedPathInfo struct
This is another representation about information Nix has on a Store
Path. It's encountered in the exportReferencesGraph feature.

Change-Id: Ia0c08c369f4af9e50ebe3a5053bd98fadaa0003d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11396
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-13 10:03:50 +00:00
Florian Klink
329c4a1ace docs(tvix/nix-compat): rename as_narinfo() to to_narinfo()
This actually does a bit of allocation. There's two Vecs, one for
references and one for signatures.

We can get rid of the Vec at nar_hash.

Change-Id: Ie025309b6678f83f5b961d49ff75dcfc7da145a1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11395
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-13 10:03:50 +00:00
Florian Klink
3d7f80c0e3 feat(tvix/nix-compat): impl Deserialize, Serialize for Signature
Change-Id: I30294079129b0e5b4faa3272e09df982d3ef2178
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11394
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-13 10:03:50 +00:00
Florian Klink
a2322d7c14 feat(tvix/nix-compat): implement Serialize, Deserialize for NixHash
We use the (slightly more tolerant) from_str to deserialize, and
serialize out as SRI.

Change-Id: If76b0ed2d4e243904f02df34f6c90b976c0bab8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11393
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-13 10:03:50 +00:00
Florian Klink
57fba1f167 feat(nix-compat/store_path): impl [Partial]Ord for StorePathRef
Move the code implementing it from StorePath to StorePathRef, and have
the StorePath impls use that too.

Drop the debug_assert in every comparison - we have tests for this to
ensure it keeps working, and built up some confidence by piping a lot of
other store paths through it in the meantime.

Change-Id: I288bad3dfa597f68d63c4bcda7791f722b7a8ced
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11392
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-13 10:03:50 +00:00
Florian Klink
fd4c7c10a8 chore(3p/overlays/patches): vendor cbtemulator UDS patch
I updated https://github.com/googleapis/google-cloud-go/pull/9665 in the
meantime, and GH decided to GC the patches.

Vendor the patch in for now (manually stripping the `bigtable/` prefix
in the path). Hopefully the PR itself gets merged soon.

Change-Id: I5b7ba78ccaf5c792c1445818b23b52d6f17155a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11402
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-04-13 10:02:13 +00:00
Florian Klink
dcd68155f0 feat(tvix/store/pathinfo/grpc): instrument functions
Change-Id: Idb7d7144be1917fbaf83e9fd76c5b2ebb3df98d2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11400
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-12 22:32:37 +00:00
Florian Klink
7bcb896e48 feat(tvix/castore/directory/grpc): instrument functions
Change-Id: I9cc0a6a32184773597556ab5f9250257aa18ca4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11399
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-12 22:32:37 +00:00
Florian Klink
f8800ba189 chore(tvix): bump rstest to 0.19.0
Change-Id: Ib2f5e84fdb8be1210b3507da67d4fe84f061651e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11387
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-12 22:16:56 +00:00
Florian Klink
fbf31f45ef feat(tvix/store): add bigtable pathinfoservice backend
Put behind the "cloud" backend, like in the `tvix-castore` crate.

Change-Id: Ib38d198baf11ab2a4b6dc405121676147c424611
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11362
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-04-09 19:40:14 +00:00
Florian Klink
d6cadee941 fix(tvix/nix-compat): fix writing padding partially
We forgot to only write the part of the padding that has not been
written already.

Unfortunately it seems the tokio mocking thing does buffer small writes,
so the tests succeeds even with the bug :-/

Change-Id: I5a6cf04212d559728639427c57207094d507ec75
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11384
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
2024-04-09 19:18:09 +00:00
Connor Brewster
63116d8c21 fix(tvix): Avoid buffering file into memory in builtins.hashFile
Right now `builtins.hashFile` always reads the entire file into memory
before hashing, which is not ideal for large files. This replaces
`read_to_string` with `open_file` which allows calculating the hash of
the file without buffering it entirely into memory. Other callers can
continue to buffer into memory if they choose, but they still use the
`open_file` VM request and then call `read_to_string` or `read_to_end`
on the `std::io::Reader`.

Fixes b/380

Change-Id: Ifa1c8324bcee8f751604b0b449feab875c632fda
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11236
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-09 17:31:58 +00:00
Florian Klink
17849c5c00 feat(tvix/castore/directory): add bigtable backend
This adds a Directory service using
https://cloud.google.com/bigtable/docs/ as a K/V store.

Directory (closures) are put in individual keys.

We don't do any bucketed upload of directory closures (yet), as castore/
fs does query individually, does not request recursively (and buffers).
This will be addressed by store composition at some point.

Change-Id: I7fada45bf386a78b7ec93be38c5f03879a2a6e22
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11212
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-04-09 15:50:34 +00:00
Florian Klink
84ad8a0bbd fix(tvix/glue/derivation): fix __structuredAttrs "outputs" handling
In case structured attrs is enabled, rather than adding a
space-separated list to the "outputs" environment variable, a "proper"
list is added to the JSON itself, at the "outputs" key.

Fixes b/395.

Reported-By: Alyssa Ross <hi@alyssa.is>
Change-Id: I2834ede9cfcf49d5e81e1056bf8f9bb9b65ddad8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11386
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-08 21:43:08 +00:00
Florian Klink
f1496f5722 refactor(tvix/nix-compat/wire): rename padding_len to total_padding_len
Make it a bit more clear that this is the total padding length, not the
padding length we still need to write.

Change-Id: I9ff4aa16f256fda367b4b9295abf82ed01b1f989
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11383
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-08 16:51:50 +00:00
Florian Klink
24fd4e963a refactor(tvix/nix-compat/wire): mv Bytes{WriterState,PacketPosition}
This is perfectly fine to track the position inside a reader too, so
rename it to reflect that.
Also make the docstring a bit less write-specific.

Change-Id: I831b0a8fe44a2477d4af96fefc692b9aabc378f1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11382
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-04-08 16:27:13 +00:00
Florian Klink
35d70f94b7 fix(tvix/nix-compat/wire): fix BytesWriter test payloads
We actually did construct other payload sizes than the test function was
intended to, because I'm too stupid to count. Fix that, and also update
a leftover _debug in a test name.

Change-Id: I1bff8551f2078afe82f789646b851f2fcff3e82c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11381
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-08 16:26:12 +00:00
Picnoir
9825404388 refactor(tvix/nix-compat): move handshake function to nix-compat
Moving out the handshake route out of tvix-daemon to nix-compat.

We're bumping the protocol to version 37. It does not change anything
for us. Version 37 introduced a change in the build result
serialization. We do not implement that for now.

Tested tvix-daemon against Nix via the integration test to make sure
we did not break anything in the process.

Taking advantage of this refactor to remove the unused anyhow
dependency in tvix-daemon.

Change-Id: I8a2aad0b6dbb0aaa997446a612a365fd7d70336c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11379
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-08 12:32:19 +00:00
Florian Klink
c05f90e611 refactor(tvix/nix-compat/wire): express magics as u64
This allows using read_u64, write_u64, which is a bit easier to juggle
with.

Also, update names to align with the nix codebase, which makes it easier
to spot both the constant name as well as the value.
Leave the ASCII interpretation as a comment afterwards.

Change-Id: I0b9ab187acd22807e2785b0722aa4300dab37c51
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11378
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Autosubmit: flokli <flokli@flokli.de>
2024-04-08 08:40:57 +00:00
Florian Klink
fd749070e2 refactor(tvix/nix-compat/wire): move magic bytes to worker_protocol
`primitive.rs` implements reading and writing primitive (fixed-length)
types in the wire format, used in the the nix daemon protocol and NAR
format.

Move worker-protocol specific magic bytes to worker_protocol.rs (and
possibly further split there once needed)

Change-Id: If681c01e9460294619f1d000229b81f0ac745810
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11377
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-08 07:02:10 +00:00
Florian Klink
71a3855f09 feat(tvix/nix-compat/wire): have write_bytes accept AsRef<[u8]> payloads
This includes String, &str etc. An example testcase with &str is
provided.

Change-Id: I900186d6ceb52f52bd41ef4596524c1f5b52470b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11376
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-08 07:02:10 +00:00