Commit graph

228 commits

Author SHA1 Message Date
edef
f222325339 fix(tvix/castore/path): join, push -> try_{join,push}
These are fallible methods, so they should be named accordingly.

Change-Id: I6dc271c42989dd6500173488190f65381835d6fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11572
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 13:40:24 +00:00
edef
2513120ff5 feat(tvix/castore/path): add PathBuf::{new, with_capacity, push}
Change-Id: Ia64f4bda80e91adbdb804f4f26cef5ace8f5406a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11571
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 13:40:24 +00:00
edef
687291cebc feat(tvix/castore/path): more conversions
Change-Id: I3ee510b444848316df520dc8ca445d0f3c7d607f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11567
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 13:40:24 +00:00
edef
1bb023df91 feat(tvix/castore/path): single-component paths are children of ROOT
The empty path (Path::ROOT) is explicitly a valid path, and "foo" is
simply a child of "". The root itself is the only path without a parent.

Change-Id: Iff00dc8aed89eaf98702b664c0df658bd5a1d88a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11569
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-01 13:40:24 +00:00
edef
aa53338ddb test(tvix/castore/path): test the null byte case
Change-Id: Ia4ceaf56f6cb4d8ed1ad935c77b3898d711da73a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11568
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 13:40:24 +00:00
edef
2d7f4135ec feat(tvix/castore/path): use proto::validate_node_name
Use the shared code for validating node names, since that is what path
components represent.

Change-Id: I12109c1306b224718faa66cf1f2874c78c1436a7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11566
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 13:40:24 +00:00
edef
4b3223a621 chore(tvix/castore/path): drop now-duplicate tests
Since PathBuf doesn't have inherent methods anymore, these just forward
to Path itself.

Change-Id: I30f44adc9994337c367bad985ada0e8fcb98dd6a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11570
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 13:40:24 +00:00
edef
c5b3187002 feat(tvix/castore/path): implement Debug + Display
We implement Debug explicitly, so that we don't just see raw integers.

Change-Id: I11213094728f3e0c674562ee71c092a950041632
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11565
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-01 11:27:59 +00:00
edef
ccb93a65a8 feat(tvix/castore/path): make Path unsized, repr-equivalent to [u8]
Change-Id: Ia1e6a10dba0bcda44e06e4cab7d4e52ca88b5859
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11564
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 11:03:12 +00:00
Florian Klink
959c50c4ac feat(tvix/castore/path): add Path{Buf}
This contains Path and PathBuf, representing platform-independent paths
representable by the castore model.

These are always relative, and platform-independent, which distinguishes
them from the ones provided in the standard library.

A subsequent CL will move IngestionEntry (and more) to use them.

Change-Id: Ib85857f4159ebc2f3c00192c95d4e5b54ffd4fcf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11558
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2024-05-01 11:01:30 +00:00
Florian Klink
c9d3946cb5 refactor(tvix/castore/import): restructure error types
Have ingest_entries return an Error type with only three kinds:

 - Error while uploading a specific Directory
 - Error while finalizing the directory upload
 - Error from the producer

Move all ingestion method-specific errors to the individual
implementations.

Change-Id: I2a015cb7ebc96d084cbe2b809f40d1b53a15daf3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11557
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-30 17:12:39 +00:00
Florian Klink
77546d734e refactor(tvix/castore): remove IngestionEntry::Unknown
We shouldn't try to represent non-representable things in the ingestion
entries (only to throw an error).

It's cleaner to throw the error directly in the part producing the
stream.

Change-Id: I6b6f6d8c2f677425210142a39f1829ddeefec812
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11556
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: firefly <firefly@firefly.nu>
2024-04-30 15:53:58 +00:00
Florian Klink
4c5c810c6f refactor(tvix/castore/import): move upload_blob_at_path into fs mod
This is only useful for when we have access to a filesystem, so it
shouldn't be in the root.

Change-Id: I9923aaed1aef9d3a1e8fad41f58821d51c2eb34b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11555
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: firefly <firefly@firefly.nu>
Tested-by: BuildkiteCI
2024-04-30 15:53:58 +00:00
Florian Klink
5e8cfcfcd6 fix(tvix/castore/import): symlink targets are Vec<u8>
These can be arbitrary bytes in theory. Some of our libraries might
be more strict, or inconsistent w.r.t. their representation of path
separators.

Change-Id: I7981b74fc7d3dd79f5589cf2ef52ced7b71dd003
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11551
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2024-04-30 13:18:03 +00:00
Florian Klink
ca64881cb3 docs(tvix/castore): fix tvix_castore::import sub-mod docstrings
The one for `fs` was wrong, and ended up being attached to ingest_path,
and the one for `archive` was missing entirely.

Change-Id: I8a4c32fb5293badb1ea0764c278a88e4ca33c018
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11552
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2024-04-30 10:06:17 +00:00
Alice Carroll
8d49ff3d64 test(tvix): Fix tvix tests on macOS
Prior to this, some tests would not build
or would fail in an obscure way.

Change-Id: I68587cc7592492ebfd71ca02fc7ccc9ff7c0196f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11544
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-30 00:55:34 +00:00
sterni
69e4a78818 chore(3p/sources): Bump channels & overlays
- //tvix: address new clippy lints

- //users/tazjin: Satisfy gonic module's new need for a playlist folder.

- //users/aspen/games: adjust for changed location of df's default
  init.txt and d_init.txt.

Change-Id: I00a2adb506ae866206fb6f88c39c9a6af320380f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11509
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
2024-04-28 16:39:26 +00:00
edef
d93633937c fix(tvix): typo
Change-Id: Ibe4741b8086e9da442232c14cdb337556704cef6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11514
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-25 23:47:49 +00:00
Connor Brewster
d2e67f021e refactor(tvix/castore): add separate Error enum for archives
The `Error` enum for the `imports` crate has both filesystem and archive
specific errors and was starting to get messy.

This adds a separate `Error` enum for archive-specific errors and then
keeps a single `Archive` variant in the top-level import `Error` for all
archive errors.

Change-Id: I4cd0746c864e5ec50b1aa68c0630ef9cd05176c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11498
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
Reviewed-by: flokli <flokli@flokli.de>
2024-04-24 15:41:38 +00:00
Connor Brewster
79698c470c feat(tvix/castore): upload blobs concurrently when ingesting archives
Ingesting tarballs with a lot of small files is very slow because of the
round trip time to the `BlobService`. To mitigate this, small blobs can
be buffered into memory and uploaded concurrently in the background.

Change-Id: I3376d11bb941ae35377a089b96849294c9c139e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11497
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
2024-04-23 17:02:07 +00:00
Connor Brewster
fa69becf4d refactor(tvix/castore): switch to ingest_entries for tarball ingestion
With `ingest_entries` being more generalized, we can now use it for
ingesting the directory entries generated from tarballs.

Change-Id: Ie1f7a915c456045762e05fcc9af45771f121eb43
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11489
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-04-23 15:31:22 +00:00
Florian Klink
5fc403587f refactor(tvix/castore): ingest filesystem entries in parallel
Rather than carrying around an Future in the IngestionEntry::Regular,
simply carry the plain B3Digest.

Code reading through a non-seekable data stream has no choice but to
read and upload blobs immediately, and code seeking through something
seekable (like a filesystem) probably knows better what concurrency to
pick when ingesting, rather than the consuming side.

(Our only) one of these seekable source implementations is now doing
exactly that. We produce a stream of futures, and then use
[StreamExt::buffered] to process more than one, concurrently.

We still keep the same order, to avoid shuffling things and violating
the stream order.

This also cleans up walk_path_for_ingestion in castore/import, as well
as ingest_dir_entries in glue/tvix_store_io.

Change-Id: I5eb70f3e1e372c74bcbfcf6b6e2653eba36e151d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11491
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-04-20 18:54:28 +00:00
Connor Brewster
b0bdeb2e89 feat(tvix/castore): Fix build warnings in release mode
Fixes some build warnings that only happen when building in release mode
which disables `debug_assertions`.

Change-Id: I554d5fce7c869c23cf4aa93179f0ee9f7f7c834e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11490
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
Reviewed-by: flokli <flokli@flokli.de>
2024-04-20 16:47:12 +00:00
Connor Brewster
18ab59ed70 fix(tvix/castore): ensure all directories are present during ingestion
`ingest_entries` requires that all directories referenced by entries in
the ingestion stream have an explicit entry in the stream.

For example, if the stream contains a file with path `foo/bar`, there
must be an entry that comes later in the stream for the directory `foo`.

Change-Id: I61b4fbbb73ea7278715e04271d8073b484e05e61
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11488
Autosubmit: Connor Brewster <cbrewster@hey.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-20 16:46:41 +00:00
Aspen Smith
3107961428 feat(tvix/eval): Implement builtins.fetchTarball
Implement a first pass at the fetchTarball builtin.

This uses much of the same machinery as fetchUrl, but has the extra
complexity that tarballs have to be extracted and imported as store
paths (into the directory- and blob-services) before hashing. That's
reasonably involved due to the structure of those two services.

This is (unfortunately) not easy to test in an automated way, but I've
tested it manually for now and it seems to work:

    tvix-repl> (import ../. {}).third_party.nixpkgs.hello.outPath
    => "/nix/store/dbghhbq1x39yxgkv3vkgfwbxrmw9nfzi-hello-2.12.1" :: string

Co-authored-by: Connor Brewster <cbrewster@hey.com>
Change-Id: I57afc6b91bad617a608a35bb357861e782a864c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11020
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-20 14:58:04 +00:00
Florian Klink
f34e0fa342 feat(tvix/castore/import): only allow normal components in entry paths
Explicitly document and add a debug assertion for that.

It's up to callers to ensure this doesn't happen.

Change-Id: Ib5d154809c2ad2920258e239993d0b790d846dc8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11487
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-20 14:14:19 +00:00
Florian Klink
e9db0449e7 refactor(tvix/castore/import): make module, split off fs and error
Move error types and filesystem-specific functions to a separate file,
and keep the fs:: namespace in public exports.

Change-Id: I5e9e83ad78d9aea38553fafc293d3e4f8c31a8c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11486
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-04-20 14:14:19 +00:00
Florian Klink
c4cb099823 refactor(tvix/castore/import): rename ingest_entries arg
This is not a stream of direntries anymore, but a stream of ingestion
entries.

Change-Id: I387f4497b6567066b24c58ca0262e710348180e9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11485
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-20 14:14:19 +00:00
Connor Brewster
259d7a3cfa refactor(tvix/castore): generalize store ingestion streams
Previously the store ingestion code was coupled to `walkdir::DirEntry`s
produced by the `walkdir` crate which made it impossible to reuse
ingesting from other sources like tarballs or NARs.

This introduces a `IngestionEntry` which carries enough information for
store ingestion and a future for computing the Blake3 digest of files.
This allows the producer to perform file uploads in a way that makes
sense for the source, ie. the filesystem upload could concurrently
upload multiple files at the same time, while the NAR ingestor will need
to ingest the entire blob before yielding the next blob in the stream.
In the future we can buffer small blobs and upload them concurrently,
but the full blob still needs to be read from the NAR before advancing.

Change-Id: I6d144063e2ba5b05e765bac1f27d41b3c8e7b283
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11462
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-19 20:37:05 +00:00
Connor Brewster
150106610e feat(tvix/castore): add convenience add method to Directory
This adds `Directory::add` which is a convenience helper for adding
nodes into a `Directory` while preserving sorted order.

This implements `Ord` and `PartialOrd` for `FileNode`, `SymlinkNode`,
and `DirectoryNode` so `binary_search` can be used.

Change-Id: I94b86bdef5d0da55aa352e098988b9704cafca19
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11481
Autosubmit: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-19 20:10:14 +00:00
Florian Klink
538d5fc8ee fix(tvix/castore/blobservice/grpc): don't use NaiveSeeker for now
Userland likes to seek backwards, and until we have store composition
and can serve chunks from a local cache, we need to buffer the
individual chunks in memory.

Change-Id: I66978a0722d5f55ed4a9a49d116cecb64a01995d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11448
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-16 18:45:52 +00:00
Florian Klink
8107678632 fix(tvix/castore/src): record rq.handle field in read()
This makes it easier to separate concurrent requests on the same inode.

Change-Id: I7637c1d889336beeb0d186182ce22fbf60fd16c3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11447
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-16 18:45:52 +00:00
Florian Klink
99bc926d1e fix(tvix/castore/fs): use io::copy to fill kernel-provided buffer
The docs state we must fill all of the buffer, except on EOF.

Change-Id: Id977ba99c0b15132422474ebbf82bb92b79d55ba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11446
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-16 18:45:52 +00:00
Florian Klink
bfd342873c feat(tvix/castore/blob/naive_seeker): add some more tracing
Change-Id: Iecf4a82a7d84008a8620825570b34e9094e6d590
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11445
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-16 18:45:52 +00:00
Florian Klink
9d9c731147 feat(tvix/castore/blob/chunked_reader): add some more traces
Change-Id: I2408707a7bc0e1c0cd8bd2933f8d68805b9e12c9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11444
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-16 18:45:52 +00:00
Florian Klink
28e98af9bc fix(tvix/castore/blobservice/chunk_rd): only skip *first* chunk bytes
When (re)initializing a chunked reader, we were erroneously skipping the
first n bytes from all chunks, not just the first one.

Fix this, by passing in an enumerated list of chunks, and only calling
SeekFrom::Start() on the first chunk in the stream.

With this, I'm able to invoke b3sum on bin/bash successfully.

Change-Id: I52ea480569267e093b0ac9d6bcd5c2d1b4db25f7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11443
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-16 18:45:52 +00:00
Florian Klink
9398bc46b6 refactor(tvix/castore/blob/naive_seeker): rework skipping for clarity
Increase the discard_buf to 4096 (as I've seen this size).
Use the ready! macro to propagate pendings.
Make it more clear what exactly should be skipped in total, and what
during the current iteration.

Also write down that poll_read call already takes care of updating
self.pos, as I ran into that trap earlier (and added it here).

Change-Id: I2d22e1c8a835c0f3dd0c648917009b2bad4fd57c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11442
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-16 18:45:52 +00:00
Florian Klink
4d802fa0ae feat(tvix/castore/blob/chunked_reader): only reassemble on real seek
If the resulting offset equals to our current position, there's no need
to recreate a reader.

Change-Id: I855f0c79c514c16ca48a78e12978af2835fbbd6a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11441
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-16 18:45:52 +00:00
Florian Klink
80d0b305a7 docs(tvix/castore/blobservice): explain open_read for small blobs more
State that this case applies if the blob is small enough to fit inside a
single chunk.

Change-Id: I0383514729e686799599b629cf1303b284147bb4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11440
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 19:33:37 +00:00
Florian Klink
e958cb0251 feat(tvix/castore/blobs/object_store): chunks() method for small blobs
We previously returned Ok(None) when being asked for more granular
chunking info, signalling the blob does not exist at all.

This is however incorrect, we should return an empty Vec instead, as
documented in the trait.

Change-Id: I83ecc2027e0767134c7598792c2ee6d964853c66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11439
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-04-15 19:33:06 +00:00
Florian Klink
c936c1c042 fix(tvix/castore/blob/object_store): tweak log levels
Don't log with info! here, bug debug!.

Change-Id: I57bd5f2a45276090b893a4051fd175e3948ddfa4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11438
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-04-15 19:32:35 +00:00
Florian Klink
9dc621cd95 fix(tix/castore/blobservice): don't warn if chunk list is empty
It's perfectly normal if we ask for more granular chunking info and the
backend responds it does not have it.

Change-Id: I593ab3e53b4f4e70c99f39b266546d2ac8eb10c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11437
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-04-15 19:32:04 +00:00
Florian Klink
7156697010 feat(tvix/castore/blob/grpc_wrapper): add blob.digest field
We're receiving bytes over the wire, and encode them the same way
B3Digest does internally, but don't use it for formatting, as we're
discarding that string.

In case the sent bytes don't have the right length, the string will be
short, but it's better to still have it as a field, even if it's not a
valid b3 digest.

Change-Id: I6ef08275d51c8a0d98f5e46844b15dfd05d17cd8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11436
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 19:32:04 +00:00
Florian Klink
d1da9f5c84 refactor(tvix/castore/fs): add parenthesis for readability
As suggested in cl/11426.

Change-Id: Ic2bb8cf2838bf0be09fb8bc62b8e598a3d153699
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11434
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 15:09:49 +00:00
Florian Klink
b025a30d27 refactor(tvix/castore/fs): remove From<Node> for InodeData
These were copying unnecessarily. Instead, have a
InodeData::from_node(), which *consumes* the Node entirely, returns
`InodeData` and the split-off name (which is not part of InodeData).

Callers can then use the result in various helper functions, like:

 - InodeData::as_fuse_type
 - InodeData::as_fuse_file_attr
 - InodeData::as_fuse_entry

… to prepare their replies to the kernel.

This removes not only a bunch of clones, but also a lot of copy-pasted
code.

Change-Id: Idbca5f25cc29e96c1f4c614b33dff2becb0a8738
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11435
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 14:49:44 +00:00
Florian Klink
fb852b0245 fix(tvix/castore/blobs): reply to has() for chunks
We allow reading individual chunks via open_read(), it's inconsistent if
a has() would return Ok(false).

Change-Id: Ie713d968172ccd2687d2e6e0dfef89ee152ef511
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11420
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-04-15 14:47:12 +00:00
Florian Klink
f1349caf3f refactor(tvix/castore): relax trait bounds on BlobService
We don't need to clone BlobService anymore.

Change-Id: I2f3b9a595f604ec0f1e081f6e90cd8b67cbb8961
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11419
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-04-15 14:47:12 +00:00
Florian Klink
9498ac936e fix(tvix/castore/directory): fix graph traversal
Use a proper graph library to ensure all nodes are reachable from the
root.

We had a bit of that handrolled during add(), as well as later, which
had an annoying bug:

Redundant nodes were omitted during insert, but when returning the list
during finalize, we did not properly account they need to be introduced
before their parents are sent.

We now simply populate a petgraph DiGraph during insert (skipping
inserting nodes we already saw), and use petgraph's DfsPostOrder to
traverse the graph during finalize.

If the number of returned indices equals the total number of nodes in
the graph, all nodes are reachable from the root, we can consume the
graph and return the nodes as a vec, in the same order as the traversal
(and insertion).

Providing a regression test for the initial bug is challenging, as the
current code uses a bunch of HashSets. I manually tested ingesting a
full NixOS closure using this mechanism (via gRPC, which exposes this
problem, as it validates twice), and it now works.

Change-Id: Ic1d5e3e981f2993cc08c5c6b60ad895e578326dc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11418
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-04-15 14:47:12 +00:00
Florian Klink
6ebaa7b88a refactor(tvix/castore/import): restructure directory uploader a bit
Have a Option<Box<dyn DirectoryPutter>>, which is lazily initialized
whenever we first want to upload a directory.

Have the loop explicitly break when it encounters the root_node, and
deal with the flushing after the loop.

Deal with the FUTUREWORK (assertion for root directory digest matching
what the DirectoryPutter returns).

Change-Id: Iefc4904d8b8387e868fb752d40e3e4e4218c7407
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11417
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 14:47:12 +00:00
Florian Klink
c088123d4e refactor(tvix/castore/import): put invariant checker into a .inspect()
Separate this a bit stronger from the main application flow.

Change-Id: I2e9bd3ec47cc6e37256ba6afc6e0586ddc9a051f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11416
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-15 14:37:35 +00:00