Commit graph

20279 commits

Author SHA1 Message Date
edef
fdecf52a52 refactor(nix-compat/wire/bytes): fold TrailerReader into BytesReader
The TrailerReader has no purpose separate from BytesReader, and the
code gets a fair bit simpler this way.

EOF handling is simplified, since we just rely on the implicit
behaviour of the existing case.

Change-Id: Id9b9f022c7c89fbc47968a96032fc43553af8290
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11539
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-29 15:33:30 +00:00
edef
44bd9543a6 feat(nix-compat/wire/bytes/reader): expose the remaining data length
The API is a bit odd here, because we don't have a distinct type for a
known-length reader.

Change-Id: I4a1dd07fbed0a400004dbe4aa2095c51898ad3bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11538
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
2024-04-29 15:33:30 +00:00
edef
5070f9eff7 refactor(nix-compat/nar/reader): always enable poisoning *API*
The poisoning API is now always available, whether debug_assertions is
enabled or not. When debug assertions are not enabled, it is equivalent
to a unit struct, and is always considered ready and unpoisoned.

Change-Id: I950237f4816d480330d9acab32661ed4f1663049
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11525
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-29 13:32:46 +00:00
Florian Klink
98b85b4580 refactor(tvix/glue): clone less
Prepare the NixAttr to return without an intermediate
Vec<(String, NixString)>, and without into_iter(), and send off the
unmoved Derivation struct to known_paths without having to clone it.

Change-Id: Icdb9f78938e998a27d0313c5d9ab15b93af5821d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11531
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
2024-04-29 13:19:41 +00:00
Florian Klink
26b77b2cf3 refactor(tvix/glue): move decompression into fetchers/ subdir
This is specifically used for the fetcher code (only).
Moving it to there for now.

Change-Id: I1e1d0541b85340ef4ff3a4c6b3fa99b51853f539
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11532
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-29 13:11:36 +00:00
sterni
69e4a78818 chore(3p/sources): Bump channels & overlays
- //tvix: address new clippy lints

- //users/tazjin: Satisfy gonic module's new need for a playlist folder.

- //users/aspen/games: adjust for changed location of df's default
  init.txt and d_init.txt.

Change-Id: I00a2adb506ae866206fb6f88c39c9a6af320380f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11509
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
2024-04-28 16:39:26 +00:00
sterni
eace431e53 feat(tools/depot-deps): include josh-filter
This is occasionally useful for testing //views or preparing new
subtrees or subtree updates.

Change-Id: I6841e5c3d105406fb960b59e16019f8444da238b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11458
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-04-28 15:20:16 +00:00
sterni
545eb05357 refactor(3p): use josh from nixpkgs
The change we need has been released and propagated to nixos channels.

Change-Id: Ib10a1d42d7ef6deaf5665a13b72ece345e83d7dc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11457
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2024-04-28 15:19:15 +00:00
Vincent Ambo
d816b18fa6 chore(tazj.in): add a convenience redirect
Change-Id: Ia41fbff390a2b1df0926ab33e9f4f66b1fd92512
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11533
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-04-28 14:40:24 +00:00
sterni
f169cc273d feat(sterni/machines/ingeborg): switch to quassel
Change-Id: I2970f47e22e6984db49d455b31903814fa94ec76
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11529
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2024-04-27 10:38:57 +00:00
sterni
61e512f294 chore(3p/rust-crates:libgit2-sys): 0.16.1+1.7.1 -> 0.16.2+1.7.2
Change-Id: I21ab23177cbbc903eae02ea95dc3fe51f731f777
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11528
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: sterni <sternenseemann@systemli.org>
2024-04-27 09:02:08 +00:00
Florian Klink
9401cccf84 feat(tvix/cli): change log format to compact, add RUST_LOG support
This brings feature parity with the tvix-store CLI command, at least as
far as logging (not tracing) is concerned.

Alternative to cl/11482, RUST_LOG seems to be the more canonical way to
influence this, and is consistent with how it's done in tvix-store.

Change-Id: I923a0b0ae55dc49af7efdacdcf5b1f24e561b3c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11527
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-26 15:10:58 +00:00
Florian Klink
88922bb207 feat(tvix/store/bin): disable --json option, set log to compact
This wasn't really used - to ingest logs in meachine-readable form, the
OTLP infrastructure is more suitable to provide structured logs than
parsing JSON from std{err}, as it also captures span information.

Also, the non-JSON output is a bit too spammy, as remarked in cl/11483 -
change it to `compact`.

Change-Id: I48007b84ba076ab566abbb6131a02868fe0eb397
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11526
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-26 15:08:15 +00:00
edef
cab9c774b7 chore(nix-compat/nar/wire): space Tag::Buf docs properly
Change-Id: If15b14b5328b9e7f85ea3fa7644d327dfb50e1a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11523
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-26 08:22:58 +00:00
edef
6e41e0917e feat(nix-compat/wire/bytes/reader): split out reading the trailer block
We separate ingesting the trailer block into a Future of its own,
parametrised on the specifics of the trailer pattern.

This is intended to be used for future work on an async NAR reader,
which needs to read a terminating parenthesis as well as the regular
padding.

Thanks to @griff for suggesting separating the ingestion into its own
Future.

Change-Id: I36c2503baa67937046a63e9bf0cfc38201394025
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11522
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-26 08:22:58 +00:00
edef
b2dc135d92 refactor(nix-compat/wire/bytes/reader): drop random whitespace
Change-Id: Ic683eab435576acc8f7e03f5684767ffa468851a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11521
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-26 08:22:58 +00:00
edef
a39894c158 refactor(nix-compat/wire/bytes/reader): drop Pin::get_mut
Pin<&mut T> is DerefMut when T: Unpin, so we don't actually need to
explicitly call get_mut.

Change-Id: Iaa312ec49c87100010e09c94f319e57e31da0cd5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11520
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-26 06:56:18 +00:00
edef
89316b22bc refactor(nix-compat/wire): move BytesPacketPosition into writer
We don't use it in the reader anymore.

Change-Id: I98fe204a747711464e9e7ca17df06fa9854eb344
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11519
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-25 23:47:49 +00:00
edef
ba46b1818a fix(nix-compat/bytes): make BytesReader less hazardous
We now *never* return the final bytes until we've read the padding
in full, so read_exact is safe to use.

This is implemented by TrailerReader, which splits the phases of
reading (and validating) the final 8-byte block, and providing
the contained payload bytes to the caller.

Change-Id: I0d05946a8af9c260a18d71d2b763ba7a68b3c27f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11518
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-25 23:47:49 +00:00
edef
70c679eac4 feat(nix-compat/wire/bytes): allow specifying a pre-read size
Change-Id: I9c94239c308cfbc2e6dae871ba77fb33507433c9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11517
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-25 23:47:49 +00:00
edef
859bfcb68b refactor(nix-compat/wire/bytes): drop pin_project, clean up
We already require R: Unpin in the constructor, so there's not much use
to pin projection.

Change-Id: Ia7bf734dc3aa86ffa6d1d5de778939baa9676bb9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11516
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-25 23:47:49 +00:00
edef
652413c97d refactor(nix-compat/wire/bytes): don't parametrise on RangeBounds
This is semantically a RangeInclusive, since we can only have
0..=u64::MAX at most, and monomorphising on the bounds doesn't
buy us anything.

Change-Id: Ib601d7fd77d703d6c8c5ec27ac9e67bb122ce1c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11515
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-25 23:47:49 +00:00
edef
d93633937c fix(tvix): typo
Change-Id: Ibe4741b8086e9da442232c14cdb337556704cef6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11514
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-25 23:47:49 +00:00
tcmal
671bdff5dc test(tvix/nix-compat): add debug assertions for nar reader
Adds debug assertions to ensure that the reader's variants are upheld.
If any of the following happens, then the currently in use reader must
be abandoned:
  * A directory or file reader encounters an error
  * A directory or file reader is dropped before being fully read from
Additionally, a directory reader must not be read from again after it
has returned None.
These checks are only used when debug_assertions are on, so vanish in
release mode.

Resolves two TODO items added by edef

Change-Id: I27bd9643a632798db5351957506c166b9bd5ca4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11508
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Autosubmit: Aria Shrimpton <me@aria.rip>
Tested-by: BuildkiteCI
2024-04-25 17:29:57 +00:00
sterni
f0e428db75 feat(ops/users): add caralice to users
Change-Id: I3d907589c75939c86faa3c1276e4023126ad3d17
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11513
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-04-25 15:15:19 +00:00
Florian Klink
f5bb9b8467 docs(tvix): add a list of TODOs/ideas
This contains a rough collection of ideas on the TODO list, trying to
keep track of it somewhere.

Change-Id: Ifc5b0cf9f7ac38f7a8e56515882bdf70e349544b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11512
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-25 15:07:45 +00:00
Florian Klink
7345986ff1 docs(tvix/nix-compat/wire/bytes/reader): fix typo
This is a ReadBuf, not a BufRead.

Change-Id: Ie80e894f4b24b77cdd60409ddfaa66dae0ffeec9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11511
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2024-04-24 20:06:17 +00:00
Florian Klink
2fd9dc11c2 docs(tvix/nix-compat): add missing reference to BytesReader
Change-Id: Ideed83d191b55e131720e598b7591e8375a26cfd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11510
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
2024-04-24 20:06:17 +00:00
Connor Brewster
d2e67f021e refactor(tvix/castore): add separate Error enum for archives
The `Error` enum for the `imports` crate has both filesystem and archive
specific errors and was starting to get messy.

This adds a separate `Error` enum for archive-specific errors and then
keeps a single `Archive` variant in the top-level import `Error` for all
archive errors.

Change-Id: I4cd0746c864e5ec50b1aa68c0630ef9cd05176c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11498
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
Reviewed-by: flokli <flokli@flokli.de>
2024-04-24 15:41:38 +00:00
Connor Brewster
79698c470c feat(tvix/castore): upload blobs concurrently when ingesting archives
Ingesting tarballs with a lot of small files is very slow because of the
round trip time to the `BlobService`. To mitigate this, small blobs can
be buffered into memory and uploaded concurrently in the background.

Change-Id: I3376d11bb941ae35377a089b96849294c9c139e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11497
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
2024-04-23 17:02:07 +00:00
Connor Brewster
fa69becf4d refactor(tvix/castore): switch to ingest_entries for tarball ingestion
With `ingest_entries` being more generalized, we can now use it for
ingesting the directory entries generated from tarballs.

Change-Id: Ie1f7a915c456045762e05fcc9af45771f121eb43
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11489
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-04-23 15:31:22 +00:00
Florian Klink
49b63fceee fix(users/picnoir/tvix-daemon): gitignore target dir, use cleaned src
This prevents the nix build copying the target/ dir into the store
whenever this is built through Nix.

Change-Id: I397228fd8e2e3265ed87d3400fe927bc505da090
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11496
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-04-23 15:14:42 +00:00
Florian Klink
e18bc33529 fix(tvix/glue/tvix_store_io): remove early return
Doing the fetch comes up with the root node, but we still need to
descend from there to the desired subpath.

Move things around to ensure the fetch case also only sets root_node.

This logic should probably be moved into smaller, easier to consume
functions.

Change-Id: I6ab9317df794f53d2504029bbc77859e89fef1ed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11507
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-23 14:53:31 +00:00
Florian Klink
72d3f9b914 fix(tvix/glue/fetchers): rename node name for all three types
We also need to rename the node in case it's a directory or symlink at
the root.

Change-Id: I6e9957200f65991645ae3e1755b943200453dfd5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11506
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-04-23 14:41:25 +00:00
Florian Klink
8181817e53 feat(tvix/glue/fetchers): support file:// URLs
Nix supports file:// - URLs for `fetchurl` and `fetchTarball`.

Convert the enums and function arguments to hold a URL type.
reqwest::Url is a re-export of the url crate, but they don't re-export
the parsing errors, and as we want to hold these in our Error types, add
it to Cargo.toml explicitly.

The Fetcher::download function now checks on the scheme, and either
opens the file locally, or does do a HTTP request as before.

Fetch gets its custom debug impl, removing potentially sensitive
username and password out of URLs.

Change-Id: I777db1fe487370e822cbfec4624034aca5e08045
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11504
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-23 12:41:17 +00:00
Florian Klink
dfef3d18d1 test(tvix/glue): add tests for fetchurl and fetchTarball
Change-Id: I53a0590ecf4e5fcb1bfd1d127824211338e28256
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11503
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-23 12:41:05 +00:00
Florian Klink
30950833c9 feat(tvix/glue/store_io): have KnownPaths track fetches too
Have fetcher builtins call queue_fetch() whenever they don't need to
fetch something immediately, and teach TvixStoreIO::store_path_to_node
on how to look up (and call ingest_and persist on our Fetcher).

Change-Id: Id4bd9d639fac9e4bee20c0b1c584148740b15c2f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11501
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-04-23 12:40:55 +00:00
Florian Klink
091de12a9a refactor(tvix/glue): move Fetch[er] into its own types, fetch lazily
We actually want to delay fetching until we actually need the file. A
simple evaluation asking for `.outPath` or `.drvPath` should work even
in a pure offline environment.

Before this CL, the fetching logic was quite distributed between
tvix_store_io, and builtins/fetchers.rs.

Rather than having various functions and conversions between structs,
describe a Fetch as an enum type, with the fields describing the fetch.

Define a store_path() function on top of `Fetch` which can be used to
ask for the calculated store path (if the digest has been provided
upfront).

Have a `Fetcher` struct, and give it a `fetch_and_persist` function,
taking a `Fetch` as well as a desired name, and have it deal with all
the logic of persisting the PathInfos. It also returns a StorePathRef,
similar to the `.store_path()` method on a `Fetch` struct.

In a followup CL, we can extend KnownPaths to track fetches AND
derivations, and then use `Fetcher` when we need to do IO into that
store path.

Change-Id: Ib39a96baeb661750a8706b461f8ba4abb342e777
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11500
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-23 12:30:48 +00:00
Florian Klink
dc444e55dc feat(tvix/boot): default CH_NUM_CPUS=2, and set num_queues= to it
See https://patchwork.kernel.org/project/qemu-devel/patch/20200706135650.438362-5-stefanha@redhat.com/

Change-Id: I3ba9a1c1f8b0be27f215da6b71cd3a0ded513a75
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11502
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-22 21:15:49 +00:00
Florian Klink
ff9e8743f6 fix(tvix/eval): don't impl From<NixString> for String
This caught me by accident in an earlier revision of cl/11500 -
I had a `NixString`, wanted to return it as a `String`, so I was naively
calling `s.into()`.

That unfortunately gave me the `Display` implementation of `NixString`,
which quotes strings, causing an annoying error further up the stack.

NixStrings are bytes, we can keep the impl From<NixString> for BString,
but having a `.into()` suddenly do quoting is more than unexpected.

Change-Id: I5434ba94bfe6c493d0a57e68225ecc22daa4b948
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11505
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-22 20:28:55 +00:00
Florian Klink
6d79cf39b9 docs(nix/buildkite): update error message
With custom phases, using `prompt` is not limited to the "release" phase
only, we only care about it not being in the "build" phase.

Change-Id: I8af27016865c728125dbe144d31006915b20b105
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11499
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2024-04-22 14:04:11 +00:00
Florian Klink
4124f0e679 refactor(tvix/cli/default.nix): make eval tests and benches derivations
This makes the eval tests and benchmarks standalone Nix derivations and
readTree targets:

```
nix-build -A tvix.cli.eval-nixpkgs-cross-hello-outpath
nix-build -A tvix.cli.benchmark-nixpkgs-attrnames
```

Even without doing any fetches, We need to set `SSL_CERT_FILE`, so
reqwest is able to load its CA roots.

Change-Id: Ib45282d01044165c7816391adbeeb26334f8e924
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11493
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-04-22 13:15:22 +00:00
Florian Klink
f3ec974895 feat(tvix/store/bin): upload paths concurrently in copy
We can speedup uploads further by not uploading all store paths
sequentially, but in parallel.

We still don't respect the reference graph, so nothing changed here.

Change-Id: I40edb5725fe7a15170f5b9f8f4c8bec2130ecca3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11492
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-04-20 18:54:28 +00:00
Florian Klink
5fc403587f refactor(tvix/castore): ingest filesystem entries in parallel
Rather than carrying around an Future in the IngestionEntry::Regular,
simply carry the plain B3Digest.

Code reading through a non-seekable data stream has no choice but to
read and upload blobs immediately, and code seeking through something
seekable (like a filesystem) probably knows better what concurrency to
pick when ingesting, rather than the consuming side.

(Our only) one of these seekable source implementations is now doing
exactly that. We produce a stream of futures, and then use
[StreamExt::buffered] to process more than one, concurrently.

We still keep the same order, to avoid shuffling things and violating
the stream order.

This also cleans up walk_path_for_ingestion in castore/import, as well
as ingest_dir_entries in glue/tvix_store_io.

Change-Id: I5eb70f3e1e372c74bcbfcf6b6e2653eba36e151d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11491
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-04-20 18:54:28 +00:00
Connor Brewster
01239a4f6f fix(tvix): fix outdated comment and error in TvixStoreIO::open
This function was originally called `read_to_string` but was changed to
`open` to make it so that file contents aren't always held in memory.
A comment and error message were not updated to reflect the new name of
this method.

Change-Id: I3d86e2f6d7006c2e1513121fc3c62efcb7e7b9bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11495
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-20 18:49:16 +00:00
Connor Brewster
b0bdeb2e89 feat(tvix/castore): Fix build warnings in release mode
Fixes some build warnings that only happen when building in release mode
which disables `debug_assertions`.

Change-Id: I554d5fce7c869c23cf4aa93179f0ee9f7f7c834e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11490
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
Reviewed-by: flokli <flokli@flokli.de>
2024-04-20 16:47:12 +00:00
Connor Brewster
18ab59ed70 fix(tvix/castore): ensure all directories are present during ingestion
`ingest_entries` requires that all directories referenced by entries in
the ingestion stream have an explicit entry in the stream.

For example, if the stream contains a file with path `foo/bar`, there
must be an entry that comes later in the stream for the directory `foo`.

Change-Id: I61b4fbbb73ea7278715e04271d8073b484e05e61
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11488
Autosubmit: Connor Brewster <cbrewster@hey.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-20 16:46:41 +00:00
Aspen Smith
3107961428 feat(tvix/eval): Implement builtins.fetchTarball
Implement a first pass at the fetchTarball builtin.

This uses much of the same machinery as fetchUrl, but has the extra
complexity that tarballs have to be extracted and imported as store
paths (into the directory- and blob-services) before hashing. That's
reasonably involved due to the structure of those two services.

This is (unfortunately) not easy to test in an automated way, but I've
tested it manually for now and it seems to work:

    tvix-repl> (import ../. {}).third_party.nixpkgs.hello.outPath
    => "/nix/store/dbghhbq1x39yxgkv3vkgfwbxrmw9nfzi-hello-2.12.1" :: string

Co-authored-by: Connor Brewster <cbrewster@hey.com>
Change-Id: I57afc6b91bad617a608a35bb357861e782a864c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11020
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-20 14:58:04 +00:00
Florian Klink
f34e0fa342 feat(tvix/castore/import): only allow normal components in entry paths
Explicitly document and add a debug assertion for that.

It's up to callers to ensure this doesn't happen.

Change-Id: Ib5d154809c2ad2920258e239993d0b790d846dc8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11487
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-20 14:14:19 +00:00
Florian Klink
e9db0449e7 refactor(tvix/castore/import): make module, split off fs and error
Move error types and filesystem-specific functions to a separate file,
and keep the fs:: namespace in public exports.

Change-Id: I5e9e83ad78d9aea38553fafc293d3e4f8c31a8c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11486
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-04-20 14:14:19 +00:00