Commit graph

20289 commits

Author SHA1 Message Date
sterni
8fed8982b7 feat(nix/buildkite): reflect deps between derivations in pipelines
Most of the steps in our buildkite pipeline build derivations without
doing anything else. A lot of those derivations depend on each other.
Consequently, buildkite will schedule builds of derivations whose
dependencies are still in the process of being built. The result is many
buildkite agents doing nothing but blocking on other derivations being
built. We can easily prevent this by using the dependency information we
can get from the derivation (files) of the targets we want to build and
translating them into buildkite step dependencies.

The hard part of this has already been done for a while:
//nix/dependency-analyzer finds the dependencies between a list of
“known” derivations (even if they only depend on each other through
intermediate derivations) without depending on a specific derivation
builder convention, but rather relying on `.drv` files. It still has a
few rough edges, but has been working reliably for our purposes.

Since our steps are identified by derivation hashes, we can just
directly use the available dependency data. Luckily, buildkite seems to
just takes a step as if it was completed if it is skipped, so we don't
even have to check whether dependencies have been skipped or not.

On whitby it seems that the dependency analysis costs about a minute
additionally (which is how long it takes to run
//nix/dependency-analyzer in isolation just about).

Supersedes cl/5063, cl/5060, cl/5064 and cl/5065.

Change-Id: I91d2eb2b43d60811cac0d26fa94467298f622970
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11116
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
2024-04-16 16:16:05 +00:00
sterni
dce1a7480c feat(nix/buildkite): use keys based on drvPaths where possible
This will make it easier to emit intra pipeline dependencies based on
the dependencies between derivations contained therein later.

A consequence of this change is that it is no longer possible to have
the same derivation be exposed as multiple steps in the pipeline. I
doubt that having this is very useful, though.

Keys for extraSteps are not changed significantly, but are distinguished
from derivation based steps (with prefix `drv-*`) by their prefix
`extra-step-*`.

Change-Id: I4165900e512b3967fa3ca4cd5bffd44bc15915fc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11115
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: ezemtsov <eugene.zemtsov@gmail.com>
Tested-by: BuildkiteCI
2024-04-16 16:15:35 +00:00
Florian Klink
80d0b305a7 docs(tvix/castore/blobservice): explain open_read for small blobs more
State that this case applies if the blob is small enough to fit inside a
single chunk.

Change-Id: I0383514729e686799599b629cf1303b284147bb4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11440
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 19:33:37 +00:00
Florian Klink
e958cb0251 feat(tvix/castore/blobs/object_store): chunks() method for small blobs
We previously returned Ok(None) when being asked for more granular
chunking info, signalling the blob does not exist at all.

This is however incorrect, we should return an empty Vec instead, as
documented in the trait.

Change-Id: I83ecc2027e0767134c7598792c2ee6d964853c66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11439
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-04-15 19:33:06 +00:00
Florian Klink
c936c1c042 fix(tvix/castore/blob/object_store): tweak log levels
Don't log with info! here, bug debug!.

Change-Id: I57bd5f2a45276090b893a4051fd175e3948ddfa4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11438
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-04-15 19:32:35 +00:00
Florian Klink
9dc621cd95 fix(tix/castore/blobservice): don't warn if chunk list is empty
It's perfectly normal if we ask for more granular chunking info and the
backend responds it does not have it.

Change-Id: I593ab3e53b4f4e70c99f39b266546d2ac8eb10c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11437
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-04-15 19:32:04 +00:00
Florian Klink
7156697010 feat(tvix/castore/blob/grpc_wrapper): add blob.digest field
We're receiving bytes over the wire, and encode them the same way
B3Digest does internally, but don't use it for formatting, as we're
discarding that string.

In case the sent bytes don't have the right length, the string will be
short, but it's better to still have it as a field, even if it's not a
valid b3 digest.

Change-Id: I6ef08275d51c8a0d98f5e46844b15dfd05d17cd8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11436
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 19:32:04 +00:00
sterni
4a91197802 fix(nix/dependency-analyzer): ignore non-drv paths for Nix < 2.6
Looking for .drv file names in non .drv files doesn't make sense, as it
less reliably a reference in those cases. Matches behavior of the
function for Nix >= 2.6.

Change-Id: I79fc1da3e55df869f03702fa5137d030790bc4eb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11114
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: aspen <root@gws.fyi>
2024-04-15 19:21:02 +00:00
Florian Klink
d1da9f5c84 refactor(tvix/castore/fs): add parenthesis for readability
As suggested in cl/11426.

Change-Id: Ic2bb8cf2838bf0be09fb8bc62b8e598a3d153699
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11434
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 15:09:49 +00:00
Florian Klink
b025a30d27 refactor(tvix/castore/fs): remove From<Node> for InodeData
These were copying unnecessarily. Instead, have a
InodeData::from_node(), which *consumes* the Node entirely, returns
`InodeData` and the split-off name (which is not part of InodeData).

Callers can then use the result in various helper functions, like:

 - InodeData::as_fuse_type
 - InodeData::as_fuse_file_attr
 - InodeData::as_fuse_entry

… to prepare their replies to the kernel.

This removes not only a bunch of clones, but also a lot of copy-pasted
code.

Change-Id: Idbca5f25cc29e96c1f4c614b33dff2becb0a8738
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11435
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 14:49:44 +00:00
Florian Klink
fb852b0245 fix(tvix/castore/blobs): reply to has() for chunks
We allow reading individual chunks via open_read(), it's inconsistent if
a has() would return Ok(false).

Change-Id: Ie713d968172ccd2687d2e6e0dfef89ee152ef511
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11420
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-04-15 14:47:12 +00:00
Florian Klink
f1349caf3f refactor(tvix/castore): relax trait bounds on BlobService
We don't need to clone BlobService anymore.

Change-Id: I2f3b9a595f604ec0f1e081f6e90cd8b67cbb8961
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11419
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-04-15 14:47:12 +00:00
Florian Klink
9498ac936e fix(tvix/castore/directory): fix graph traversal
Use a proper graph library to ensure all nodes are reachable from the
root.

We had a bit of that handrolled during add(), as well as later, which
had an annoying bug:

Redundant nodes were omitted during insert, but when returning the list
during finalize, we did not properly account they need to be introduced
before their parents are sent.

We now simply populate a petgraph DiGraph during insert (skipping
inserting nodes we already saw), and use petgraph's DfsPostOrder to
traverse the graph during finalize.

If the number of returned indices equals the total number of nodes in
the graph, all nodes are reachable from the root, we can consume the
graph and return the nodes as a vec, in the same order as the traversal
(and insertion).

Providing a regression test for the initial bug is challenging, as the
current code uses a bunch of HashSets. I manually tested ingesting a
full NixOS closure using this mechanism (via gRPC, which exposes this
problem, as it validates twice), and it now works.

Change-Id: Ic1d5e3e981f2993cc08c5c6b60ad895e578326dc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11418
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-04-15 14:47:12 +00:00
Florian Klink
6ebaa7b88a refactor(tvix/castore/import): restructure directory uploader a bit
Have a Option<Box<dyn DirectoryPutter>>, which is lazily initialized
whenever we first want to upload a directory.

Have the loop explicitly break when it encounters the root_node, and
deal with the flushing after the loop.

Deal with the FUTUREWORK (assertion for root directory digest matching
what the DirectoryPutter returns).

Change-Id: Iefc4904d8b8387e868fb752d40e3e4e4218c7407
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11417
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 14:47:12 +00:00
Florian Klink
c088123d4e refactor(tvix/castore/import): put invariant checker into a .inspect()
Separate this a bit stronger from the main application flow.

Change-Id: I2e9bd3ec47cc6e37256ba6afc6e0586ddc9a051f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11416
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 14:37:35 +00:00
Florian Klink
b70744fda6 refactor(tvix/*/import): rename direntry_stream, entries_per_depths
Align these names and comments with the two users, to make it more
obvious we're doing the same thing here, just use a different method to
come up with entries_per_depths.

Change-Id: I42058e397588b6b57a6299e87183bef27588b228
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11415
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 14:06:50 +00:00
Florian Klink
bcc00fba8f refactor(tvix/castore/import): inline process_entry
This did very little, and especially the part of relying on the outside
caller to pass in a Directory if the type is a directory required having
per-entry-type specific logic anyways.

It's cleaner to just inline it.

Change-Id: I997a8513ee91c67b0a2443cb5cd9e8700f69211e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11414
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 14:05:49 +00:00
Florian Klink
d47bd4f4bc refactor(tvix/castore/import): move process_entry to the end of the file
This makes it easier to understand the code.

Change-Id: I0a9047433000551a6ba1f50a8c5c93527bc86216
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11413
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 14:05:18 +00:00
Florian Klink
c19bc95b8a fix(tvix/castore/blobservice): update bytes_read only on successful read
We previously updated this.pos also in case the underlying read returned
an error.

Also, use the ready! macro to remove the match block, and instrument
errors returned during start_seek.

Change-Id: Ic32e26579d964a76b45687134acc48d72d67c36f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11421
Reviewed-by: Brian Olsen <me@griff.name>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-04-15 14:04:47 +00:00
Florian Klink
bccb4c92c3 refactor(tvix/castore/fs): add InodeData::as_fuse_{entry,file_attr}
Remove the now unused gen_file_attr (which had a wrong docstring).

Change-Id: Ie86b14d1ad798e6233bc44c43ace3f8b95c67ea9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11430
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 13:59:46 +00:00
Florian Klink
50065afd36 refactor(tvix/castore/fs): remove "add … handle" debug messages
Change-Id: Iac22bbef96a2afa0416f011d073934b52b19975d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11433
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-15 13:54:14 +00:00
Florian Klink
d3d88431f3 feat(tvix/castore/fs): assign read[dir*]/release[dir] ops to parent span
When a directory or file is open()'ed, we already put some data into
a lookup table, and subsequent operations then use the returned handle
id.

By also adding the span that's been created during these calls into the
lookup table, we can properly set the span parent for these requests,
nicely connecting the individual operations to the bigger picture.

Change-Id: Ia354842fccdbc7f45c2d3efda3acf058b2dbc48e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11429
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
2024-04-15 13:54:14 +00:00
Florian Klink
34d9d54aae fix(tvix/castore/fs): use record to add fields to current span
Instead of creating another child span, we can use
`tracing::Span::current().record(k,v)` to add an additional field to the
current span.

Change-Id: I337faac0e73a0da6eb0a52cb75c2e8c026eff774
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11428
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-15 13:23:33 +00:00
Florian Klink
ff5835008f feat(tvix/castore/fs): implement readdirplus
This currently shares some code with readdir, except it's also providing
a second `fuse_backend_rs::api::filesystem::Entry` argument to the
`add_entry` function call.

Refactoring this to reduce some duplication is left for a future CL.

Change-Id: I282c8dfc6a711d00a4482c87cbb84d4950c0aee9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11426
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-15 13:22:08 +00:00
Florian Klink
456191e69e refactor(tvix/castore/fs): use consistent span field name for handle
Use rq.handle in `release` too, and remove interpolating it into the
log message itself.

Also update the comment, we don't get ownership, just simply drop, and
change the level to warn!, as suggested in cl/11425.

Change-Id: If4e6cff6d8b580671b1548ae3862851db4af6694
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11427
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-04-15 13:20:07 +00:00
Florian Klink
2a47ad1d90 feat(tvix/castore/fs): implement opendir/releasedir
Similar to how we already handle opening files, implement opendir/
releasedir, and keep a map of dir_handles. They point to the rx side of
a channel.

This greatly improves performance listing the root of the filesystem
when used inside tvix-store, as we don't need to re-request the listing
(and skip to the desired position) all the time.

Change-Id: I0d3ec4cb70a8792c5a1343439cf47d78d9cbb1d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11425
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-15 13:18:36 +00:00
Florian Klink
3ed7eda79b refactor(tvix/castore/fs): use std::sync::Mutex
This allows us acquiring the lock in sync code still. Also, simplify
some of the error handling a bit.

Change-Id: I29e83b715f92808e95ecb0ae9de787339d1a371d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11424
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-15 09:27:04 +00:00
Florian Klink
1bf6b9f5a0 refactor(tvix/castore/fs): simplify some separate spawn and blocks
We can just pass an async move closure to `self.tokio_handle.block_on`
and make this a bit shorter.

Change-Id: Iba674f34f22ba7a7de7c5bae59d64584884cb17c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11423
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-15 09:27:04 +00:00
Florian Klink
515bfa18fb feat(tvix/castore/fs): support extended attributes
This exposes `user.tvix.castore.{blob,directory}.digest` xattr keys for
files and directories:

```
❯ getfattr -d /tmp/tvix/06jrrv6wwp0nc1m7fr5bgdw012rfzfx2-nano-7.2-info
getfattr: Removing leading '/' from absolute path names
user.tvix.castore.directory.digest="b3:SuYDcUM9RpWcnA40tYB1BtYpR0xw72v3ymhKDQbBfe4="

❯ getfattr -d /tmp/tvix/156a89x10c3kaby9rgf3fi4k0p6r9wl1-etc-shells
getfattr: Removing leading '/' from absolute path names
user.tvix.castore.blob.digest="b3:pZkwZoHN+/VQ8wkaX0wYVXZ0tV/HhtKlSqiaWDK7uRs="
```

It's currently mostly used for debugging, though it might be useful for
tvix-castore-aware syncing programs using the filesystem too.

Change-Id: I26ac3cb9fe51ffbf7f880519f26741549cb5ab6a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11422
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-04-15 09:27:04 +00:00
Ilan Joselevich
23871649bb feat(tvix/nix-compat/path_info): derive Hash for ExportedPathInfo
Change-Id: Icd2431f27043f120eee1d156cfcdc9bc85519ead
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11412
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-13 21:41:47 +00:00
Florian Klink
b70e01a4db feat(tvix/castore/import): remove copying in find_ancestor
We don't need to copy if we explicitly say that the returned
Option<Path> may hold onto bytes from the passed in &DirEntry.

Change-Id: Ib46b6fd2f8f19a45f8bef79c4c1d2fa6b490cad7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11410
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-13 13:20:29 +00:00
Ryan Lahfa
a919efbe11 fix(tvix/eval): fix the unsafeGetAttrPos stub return value
It was returning `{ line, col, file }` instead of `{ line, column, file
}`.

Note this is not a regression in Nix, Nix always had `column` since
2013.

Change-Id: I4871a201b32324dff0432526d0449806a535828f
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11409
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-13 13:16:02 +00:00
Florian Klink
01f8ef1560 fix(tvix): apply cbtemulator patch in tvix view
This tells josh to include the patch from
third_party/overlays/patches/cbtemulator-uds.patch at
nixpkgs/cbtemulator-uds.patch.

We then use the `additionalOverlays` feature introduced earlier, and
pass it an overlay adding the patch to `cbtemulator`.

This only gets applied if tvix/shell.nix is not called with a pkgs
argument, which is the case in the tvix view.

Change-Id: I7bc5543460ddd30325ab0998e891c15fcaf8bcfe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11408
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2024-04-13 12:09:47 +00:00
Florian Klink
e5b87d5496 feat(3p/nixpkgs): add additionalOverlays optional arg
This allows applying additional overlays to nixpkgs before
instantiating it.

Change-Id: Iee486086e13bb73e6bd20a817b1106c3cd99c935
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11407
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-13 12:01:05 +00:00
Florian Klink
31e3382129 feat(tvix/*store/bigtable): limit retries connecting to cbtemulator
This kept retrying indefinitely if the socket didn't appear.

Change-Id: I4d4ef61df73cef6abda698501432f370abc8a82c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11406
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-13 12:01:00 +00:00
Florian Klink
0dd421414e feat(tvix/nix-compat/wire/bytes): add #[inline] for ensure_* funcs
While the compiler might realize it's a good idea to inline these,
especially considering this is in the same crate [1], it doesn't hurt to
add that annotation too.

Suggested in cl/11385.

[1]: https://nnethercote.github.io/perf-book/inlining.html

Change-Id: If679fce0e1aab9ec681ba60f689d386d6fd92d61
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11403
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2024-04-13 11:45:38 +00:00
Florian Klink
f6bb8dbe1c chore(3p/nixpkgs): drop most of cbtemulator expression
We bumped nixpkgs past that bump, so we only need to carry our
downstream UDS support patch.

Change-Id: I56504d71a8327cba2e43aa36b7b657f93cf35e51
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11405
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-13 11:44:37 +00:00
Florian Klink
de68d8321f feat(tvix/buildkite): apply a 10 minutes timeout
We should cap the time of a `cargo build && cargo test` to a reasonable
amount of time.

Change-Id: If618198459ac267f9f6d7eb60466534a95a95c2c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11404
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-13 11:44:37 +00:00
Florian Klink
cc42cac39c refactor(tvix/castore/import): rename ingest_entries function arg
This is a stream of DirEntry, so let's call it direntry_stream.

Change-Id: I5b3cb4efba899d746393f75f6ece7eaa79424717
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11401
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-13 10:51:58 +00:00
Florian Klink
820e98fa7f refactor(tvix/boot): use tvix-store copy
Instead of using an additional Nix build producing a directory structure
containing xz-compressed NAR files and NARInfos, and uploading these via
`curl` to NAR-bridge, which then decomposes them and uploads them to
tvix-store, have tvix-store interpret the exported reference graph
directly, and deal with the uploading on its own.

Change-Id: I44ee0b132944c53d11abb688d2d017f0cc9c3d97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11398
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-13 10:44:56 +00:00
Florian Klink
a4f65ddba0 feat(tvix/store/bin): add "copy" command
This allows copying individual store paths from the local /nix/store
into tvix-store.

As we don't support getting this information from Nix yet, we currently
expect metadata to be provided externally:

Nix' `exportReferencesGraph` feature contains pretty much all data we
need for this. Expect a list of this information at a well-known key
(`closure`, similar to nixpkgs' `pkgs/build-support/binary-cache/
default.nix`).

We currently simply upload all store paths sequentially, without any
parallelism or awareness in how the reference graph looks like.

As long as the connected stores don't enforce this, this is fine, at
least for now.

Change-Id: Ib83c998465adddfdb110db994843c44e26b3d3d8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11397
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-04-13 10:44:56 +00:00
Ryan Lahfa
863c4207cc feat(tvix/eval): contextful JSON operations
`toJSON` transform a Nix structure into a JSON string.

For each context in that Nix structure, the JSON string must possess it.

Thus, it is necessary to take the union of all contexts and attach it to
the final structure.

Unfortunately, the return type of `into_json` is a serde's JSON object,
not a string. Therefore, it is not possible to reuse `NixString`
machinery.

Context tests are reinforced as Nix does not test those behaviors.

Fixes b/393.

Change-Id: I5afdbc4e18dd70469192c1aa657d1049ba330149
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11266
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-13 10:11:18 +00:00
Florian Klink
45cf7ae657 refactor(tvix/nix-compat): move nar writer to tokio
There's little reason to keep the nar writer using Async{Read,Write}
traits from futures, while everything else async in tvix (and
nix-compat) uses tokio.

Change-Id: I8cd1efcd0dd5bb76471de997603c7b701a5095de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11391
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Brian Olsen <me@griff.name>
2024-04-13 10:09:35 +00:00
Florian Klink
742937d55c refactor(tvix/nix-compat): move worker_protocol into nix_daemon mod
This doesn't have much to do with the plain "wire" format, it's merely
one user of it.

Also, use the more "public" `wire::` API to read/write bytes, strings,
bools and u64s.

Change-Id: I98dddcc3004dfde7a0c009958fe84a840f77b188
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11390
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Brian Olsen <me@griff.name>
2024-04-13 10:09:35 +00:00
Florian Klink
36b296609b refactor(tvix/nix-compat): reorganize wire and bytes
Move everything bytes-related into its own module, and re-export
both bytes and primitive in a flat space from wire/mod.rs.

Expose this if a `wire` feature flag is set. We only have `async` stuff
in here.

Change-Id: Ia4ce4791f13a5759901cc9d6ce6bd6bbcca587c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11389
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-04-13 10:09:35 +00:00
Florian Klink
839c971a0f docs(tvix/eval): remove link to cargo manifest reference
We don't have it in all Cargo.toml files either.

Change-Id: Ifcb333a39603a7e402ab0068e0f0f1b5af769e13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11388
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
2024-04-13 10:09:35 +00:00
Florian Klink
eeca2d92e2 feat(tvix/nix-compat): add BytesReader
This adds AsyncRead counterpart for read_bytes.

Change-Id: I751da9944984c7a523abee305f8f8a050e705f04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11385
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
2024-04-13 10:09:35 +00:00
Florian Klink
82cf89ee6e feat(tvix/nix-compat): introduce ExportedPathInfo struct
This is another representation about information Nix has on a Store
Path. It's encountered in the exportReferencesGraph feature.

Change-Id: Ia0c08c369f4af9e50ebe3a5053bd98fadaa0003d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11396
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-13 10:03:50 +00:00
Florian Klink
329c4a1ace docs(tvix/nix-compat): rename as_narinfo() to to_narinfo()
This actually does a bit of allocation. There's two Vecs, one for
references and one for signatures.

We can get rid of the Vec at nar_hash.

Change-Id: Ie025309b6678f83f5b961d49ff75dcfc7da145a1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11395
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-13 10:03:50 +00:00
Florian Klink
3d7f80c0e3 feat(tvix/nix-compat): impl Deserialize, Serialize for Signature
Change-Id: I30294079129b0e5b4faa3272e09df982d3ef2178
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11394
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-13 10:03:50 +00:00