Commit graph

252 commits

Author SHA1 Message Date
Connor Brewster
b0aaff25fa refactor(tvix/castore): extract concurrent blob uploader
The archive ingester has a mechanism for concurrently uploading small
blobs to the blob service in order to hide round trip latency with the
blob service when ingesting many small blobs.

Other ingestion sources like NARs also need a similar mechanism, this
extracts the concurrent blob uploading mechanism into its own struct to
make it more reusable.

Change-Id: I05020419ff4b9ad5829fbfb5cd08d36db983b8c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11693
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-20 15:21:46 +00:00
Yureka
4c062d5c92 fix(castore/directory/objectstore): fix responses for deduplicated dirs
Using remove_node messed up the extraction of nodes from the graph. Use
into_nodes_edges() instead, to remove the nodes without cloning.

Change-Id: Id76c7935d082d6f26192cc3cd490483594f1d1e2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11684
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
2024-05-16 22:06:01 +00:00
Yureka
6aa1d2c4a8 feat(tvix/store): add ObjectStoreDirectoryService
Change-Id: I1636012be2e8ee3ae64f7bc62fd28bfe0cb2bca5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11668
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-16 16:33:06 +00:00
Florian Klink
30995a0990 fix(tvix/castore/grpc): don't use explicit channel
We can just use the `BoxStream` directly, or a `once` with the single
`Directory`.

In the recursive case, we also did not properly close the channel after
the first error.

Change-Id: Ifad56d307fc7861107b6d3cffd28d35631d526e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11635
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-05-13 22:05:59 +00:00
Florian Klink
b26569028c feat(tvix/castore/directory/get_recursive): add 'static to BoxStream
Necessary to directly use this in the GRPC DirectoryService wrapper
directly.

Change-Id: Ic6a0038a40dc30071d145af5035345fcd93288ae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11634
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-13 22:05:59 +00:00
Florian Klink
ed584b9296 feat(tvix/castore/directory/traverse_directory): simplify
Use try_stream! rather than stream!, and a bit more map_err and ok_err
to make things a bit more concise. Once we have proper error types here,
and impl Froms, a lot of the error mapping would disappear entirely.

Change-Id: I5240a6b0ff7818b94c151322774242b2c142e33b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11633
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-05-13 22:05:59 +00:00
Florian Klink
7fd4adc129 feat(tvix/castore/directory/traverse): simplify code
Replace the loop manually driving the iterator with a for … in, and some
of the match with ok_or_else.

Change-Id: I6d7b3ef1bf1c7aa128bd6adef09390b54f79479e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11632
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-13 22:05:59 +00:00
Florian Klink
afcbc1d86d chore(tvix/clippy): configure clippy allow block_in_conditions
… in Cargo.toml.

This gets an imperative `cargo clippy` run to pick up that config,
so `-A clippy::blocks_in_conditions` doesn't need to be explicitly
specified anymore.

Change-Id: I32b6cc50c77c22cba0d816d0db508c2f94b2c383
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11659
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2024-05-13 16:57:03 +00:00
Florian Klink
b22fc8cd9a refactor(tvix/castore/error): drop From<PoisonError<T>> for Error
We don't produce these erorrs anymore, no need to provide a conversion
to it.

Change-Id: I37933e436ad15c5d90b3ac270c4ef5742980513d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11614
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-05-11 15:09:33 +00:00
Florian Klink
25d97c5b55 refactor(tvix/castore/directoryservice/memory): use tokio RwLock
We don't want to block here, and this also means there's no poisoning to
deal with.

Change-Id: Ic375571970c48beace0005ae2c012135086a4d67
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11613
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-11 15:09:33 +00:00
Florian Klink
84114cf02c refactor(tvix/castore/blobservice/memory): use parking_lot RwLock
This one doesn't require us to deal with poisoning, is upgradeable and
the right thing to use when locking access to data, not IO resources.

Change-Id: I78634953a73404500d28f51f1d93a87e215c8149
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11612
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-11 15:09:33 +00:00
Florian Klink
717081ae37 fix(tvix/castore/directory/sled): use spawn_blocking
This does IO, which might take a longer amount of time than what we want
to be blocking the normal executor.

Use spawn_blocking instead. I didn't add it for the constructors, as we
only call these once.

Change-Id: I96231fcff8d10abe90cafde25a099a2db6ea9414
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11617
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-05-11 13:33:59 +00:00
Florian Klink
ebb7b32a2c chore(tvix/castore/blob): remove sled implementation
This never did any chunking, and sled (rightfully) performs really bad
if values get too large.

We switched the default to using the objectstore backend with the local
filesystem a while ago, no need to keep this footgun around anymore.

Change-Id: I2c12672f2ea6a22e40d0cbf9161560baddd73d4a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11616
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-11 13:28:51 +00:00
Florian Klink
bc92f4188e test(tvix-castore/import): add tests for ingest_entries
Change-Id: Ia7906533868fd948509419e0d64b64582575a7fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11591
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-06 15:11:39 +00:00
Florian Klink
4aff40fff8 fix(tvix/castore/import): assert end of stream
Once we break out with the root node, there may be no more elements in
the stream.

Change-Id: I6f5fc5662095aa2b2a56bcad506d25520d9ad00c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11592
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-05-06 15:11:39 +00:00
Florian Klink
75f2a1f97d fix(tvix/castore/import): deal with entry.path() not having a parent
We got away with not properly dealing with this for the archive case,
where everything is contained inside a toplevel dir, but NARs can encode
a single file/symlink.

Properly break if the IngestionEntry path has the ROOT as parent, and
only create filling directories in the other case.

Change-Id: Ib378d0d1040de7c3fe310912a0b0488c55afee83
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11590
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-06 15:11:39 +00:00
Florian Klink
281bd46a43 feat(tvix-castore/import) have IngestionEntry.path() return &Path
There's no need for this to be a &PathBuf.

Change-Id: I2d4126d57cfd8ddaad5dd327943b70b83d45c749
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11589
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-05 14:54:19 +00:00
Florian Klink
ba00f0c695 refactor(tvix/*store): use DS: DirectoryService
We implement DirectoryService for Arc<DirectoryService> and
Box<DirectoryService>, this is sufficient.

Change-Id: I0a5a81cbc4782764406b5bca57f908ace6090737
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11586
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-04 21:27:26 +00:00
Florian Klink
37671d3913 test(tvix/[ca]store): enable bigtable only with "integration" feature
The emulator and bigtable client are quite big. Remove them from the
default //tvix:shell.

Put the tests behind a `integration` feature flag, and add a variant
with that enabled to CI, and drop the bigtable tools from //tvix:shell.

Change-Id: Ie042097a0d6fc26542faa96c139b77298ccb160a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11582
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-03 08:53:09 +00:00
Florian Klink
516c6dc572 refactor(tvix/castore/import): use crate Path[Buf] in IngestionEntry
This explicitly splits ingestion-method-specific path types from the
castore types.

Change-Id: Ia3b16105fadb8d52927a4ed79dc4b34efdf4311b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11563
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-05-02 15:26:29 +00:00
Florian Klink
abc0553eb8 feat(tvix/castore/directory/traverse): use castore Paths
This switches from using std::path::Path to using castore paths.

We can drop some error handling in descend_to, as absolute (or redundant)
paths are not representable.

We however now need to convert from a std::path::Path to our
representation, and decide to accept .. canonicalization, as paths in
EvalIO might contain this. Dealing .. to hop into another store path, if
we encounter this, should be dealt with in a previous step.

Change-Id: I5e94693808420c5d56587c68731252b54755bf93
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11575
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-05-02 15:26:29 +00:00
Florian Klink
4033d4c50f feat(tvix/castore/path): implement PathBuf::from_host_path
This allows converting from std::path::Path to castore PathBufs.

A flag is present to control .. canonicalization, and the usual caveats
about platform-specific differences apply.

Currently only added for unix, we'll carefully consider other platforms
on a case-by-case basis.

Change-Id: If289a92f75a2e5c3eec132b6a91a28d225fc1989
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11577
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-05-02 13:28:29 +00:00
Florian Klink
3a9432f4d8 feat(tvix/castore/path): impl AsRef<Path> for Path
This allows using both Path and PathBuf in a function argument taking
`impl AsRef<Path>`.

Change-Id: Ibd3ba6fac538069d2fe729d1ef399fdef301668f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11574
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-01 19:10:49 +00:00
Florian Klink
ea7b334a62 feat(tvix/castore): expose PathBuf too
Change-Id: I1538725766a73436eba9a92dcb4f3275ace2fa4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11573
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-01 19:10:49 +00:00
edef
f222325339 fix(tvix/castore/path): join, push -> try_{join,push}
These are fallible methods, so they should be named accordingly.

Change-Id: I6dc271c42989dd6500173488190f65381835d6fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11572
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 13:40:24 +00:00
edef
2513120ff5 feat(tvix/castore/path): add PathBuf::{new, with_capacity, push}
Change-Id: Ia64f4bda80e91adbdb804f4f26cef5ace8f5406a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11571
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 13:40:24 +00:00
edef
687291cebc feat(tvix/castore/path): more conversions
Change-Id: I3ee510b444848316df520dc8ca445d0f3c7d607f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11567
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 13:40:24 +00:00
edef
1bb023df91 feat(tvix/castore/path): single-component paths are children of ROOT
The empty path (Path::ROOT) is explicitly a valid path, and "foo" is
simply a child of "". The root itself is the only path without a parent.

Change-Id: Iff00dc8aed89eaf98702b664c0df658bd5a1d88a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11569
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-01 13:40:24 +00:00
edef
aa53338ddb test(tvix/castore/path): test the null byte case
Change-Id: Ia4ceaf56f6cb4d8ed1ad935c77b3898d711da73a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11568
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 13:40:24 +00:00
edef
2d7f4135ec feat(tvix/castore/path): use proto::validate_node_name
Use the shared code for validating node names, since that is what path
components represent.

Change-Id: I12109c1306b224718faa66cf1f2874c78c1436a7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11566
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 13:40:24 +00:00
edef
4b3223a621 chore(tvix/castore/path): drop now-duplicate tests
Since PathBuf doesn't have inherent methods anymore, these just forward
to Path itself.

Change-Id: I30f44adc9994337c367bad985ada0e8fcb98dd6a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11570
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 13:40:24 +00:00
edef
c5b3187002 feat(tvix/castore/path): implement Debug + Display
We implement Debug explicitly, so that we don't just see raw integers.

Change-Id: I11213094728f3e0c674562ee71c092a950041632
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11565
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-01 11:27:59 +00:00
edef
ccb93a65a8 feat(tvix/castore/path): make Path unsized, repr-equivalent to [u8]
Change-Id: Ia1e6a10dba0bcda44e06e4cab7d4e52ca88b5859
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11564
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-01 11:03:12 +00:00
Florian Klink
959c50c4ac feat(tvix/castore/path): add Path{Buf}
This contains Path and PathBuf, representing platform-independent paths
representable by the castore model.

These are always relative, and platform-independent, which distinguishes
them from the ones provided in the standard library.

A subsequent CL will move IngestionEntry (and more) to use them.

Change-Id: Ib85857f4159ebc2f3c00192c95d4e5b54ffd4fcf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11558
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2024-05-01 11:01:30 +00:00
Florian Klink
c9d3946cb5 refactor(tvix/castore/import): restructure error types
Have ingest_entries return an Error type with only three kinds:

 - Error while uploading a specific Directory
 - Error while finalizing the directory upload
 - Error from the producer

Move all ingestion method-specific errors to the individual
implementations.

Change-Id: I2a015cb7ebc96d084cbe2b809f40d1b53a15daf3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11557
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-30 17:12:39 +00:00
Florian Klink
77546d734e refactor(tvix/castore): remove IngestionEntry::Unknown
We shouldn't try to represent non-representable things in the ingestion
entries (only to throw an error).

It's cleaner to throw the error directly in the part producing the
stream.

Change-Id: I6b6f6d8c2f677425210142a39f1829ddeefec812
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11556
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: firefly <firefly@firefly.nu>
2024-04-30 15:53:58 +00:00
Florian Klink
4c5c810c6f refactor(tvix/castore/import): move upload_blob_at_path into fs mod
This is only useful for when we have access to a filesystem, so it
shouldn't be in the root.

Change-Id: I9923aaed1aef9d3a1e8fad41f58821d51c2eb34b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11555
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: firefly <firefly@firefly.nu>
Tested-by: BuildkiteCI
2024-04-30 15:53:58 +00:00
Florian Klink
5e8cfcfcd6 fix(tvix/castore/import): symlink targets are Vec<u8>
These can be arbitrary bytes in theory. Some of our libraries might
be more strict, or inconsistent w.r.t. their representation of path
separators.

Change-Id: I7981b74fc7d3dd79f5589cf2ef52ced7b71dd003
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11551
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2024-04-30 13:18:03 +00:00
Florian Klink
ca64881cb3 docs(tvix/castore): fix tvix_castore::import sub-mod docstrings
The one for `fs` was wrong, and ended up being attached to ingest_path,
and the one for `archive` was missing entirely.

Change-Id: I8a4c32fb5293badb1ea0764c278a88e4ca33c018
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11552
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2024-04-30 10:06:17 +00:00
Alice Carroll
8d49ff3d64 test(tvix): Fix tvix tests on macOS
Prior to this, some tests would not build
or would fail in an obscure way.

Change-Id: I68587cc7592492ebfd71ca02fc7ccc9ff7c0196f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11544
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-30 00:55:34 +00:00
sterni
69e4a78818 chore(3p/sources): Bump channels & overlays
- //tvix: address new clippy lints

- //users/tazjin: Satisfy gonic module's new need for a playlist folder.

- //users/aspen/games: adjust for changed location of df's default
  init.txt and d_init.txt.

Change-Id: I00a2adb506ae866206fb6f88c39c9a6af320380f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11509
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
2024-04-28 16:39:26 +00:00
edef
d93633937c fix(tvix): typo
Change-Id: Ibe4741b8086e9da442232c14cdb337556704cef6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11514
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-25 23:47:49 +00:00
Connor Brewster
d2e67f021e refactor(tvix/castore): add separate Error enum for archives
The `Error` enum for the `imports` crate has both filesystem and archive
specific errors and was starting to get messy.

This adds a separate `Error` enum for archive-specific errors and then
keeps a single `Archive` variant in the top-level import `Error` for all
archive errors.

Change-Id: I4cd0746c864e5ec50b1aa68c0630ef9cd05176c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11498
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
Reviewed-by: flokli <flokli@flokli.de>
2024-04-24 15:41:38 +00:00
Connor Brewster
79698c470c feat(tvix/castore): upload blobs concurrently when ingesting archives
Ingesting tarballs with a lot of small files is very slow because of the
round trip time to the `BlobService`. To mitigate this, small blobs can
be buffered into memory and uploaded concurrently in the background.

Change-Id: I3376d11bb941ae35377a089b96849294c9c139e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11497
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
2024-04-23 17:02:07 +00:00
Connor Brewster
fa69becf4d refactor(tvix/castore): switch to ingest_entries for tarball ingestion
With `ingest_entries` being more generalized, we can now use it for
ingesting the directory entries generated from tarballs.

Change-Id: Ie1f7a915c456045762e05fcc9af45771f121eb43
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11489
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-04-23 15:31:22 +00:00
Florian Klink
5fc403587f refactor(tvix/castore): ingest filesystem entries in parallel
Rather than carrying around an Future in the IngestionEntry::Regular,
simply carry the plain B3Digest.

Code reading through a non-seekable data stream has no choice but to
read and upload blobs immediately, and code seeking through something
seekable (like a filesystem) probably knows better what concurrency to
pick when ingesting, rather than the consuming side.

(Our only) one of these seekable source implementations is now doing
exactly that. We produce a stream of futures, and then use
[StreamExt::buffered] to process more than one, concurrently.

We still keep the same order, to avoid shuffling things and violating
the stream order.

This also cleans up walk_path_for_ingestion in castore/import, as well
as ingest_dir_entries in glue/tvix_store_io.

Change-Id: I5eb70f3e1e372c74bcbfcf6b6e2653eba36e151d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11491
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-04-20 18:54:28 +00:00
Connor Brewster
b0bdeb2e89 feat(tvix/castore): Fix build warnings in release mode
Fixes some build warnings that only happen when building in release mode
which disables `debug_assertions`.

Change-Id: I554d5fce7c869c23cf4aa93179f0ee9f7f7c834e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11490
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
Reviewed-by: flokli <flokli@flokli.de>
2024-04-20 16:47:12 +00:00
Connor Brewster
18ab59ed70 fix(tvix/castore): ensure all directories are present during ingestion
`ingest_entries` requires that all directories referenced by entries in
the ingestion stream have an explicit entry in the stream.

For example, if the stream contains a file with path `foo/bar`, there
must be an entry that comes later in the stream for the directory `foo`.

Change-Id: I61b4fbbb73ea7278715e04271d8073b484e05e61
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11488
Autosubmit: Connor Brewster <cbrewster@hey.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-20 16:46:41 +00:00
Aspen Smith
3107961428 feat(tvix/eval): Implement builtins.fetchTarball
Implement a first pass at the fetchTarball builtin.

This uses much of the same machinery as fetchUrl, but has the extra
complexity that tarballs have to be extracted and imported as store
paths (into the directory- and blob-services) before hashing. That's
reasonably involved due to the structure of those two services.

This is (unfortunately) not easy to test in an automated way, but I've
tested it manually for now and it seems to work:

    tvix-repl> (import ../. {}).third_party.nixpkgs.hello.outPath
    => "/nix/store/dbghhbq1x39yxgkv3vkgfwbxrmw9nfzi-hello-2.12.1" :: string

Co-authored-by: Connor Brewster <cbrewster@hey.com>
Change-Id: I57afc6b91bad617a608a35bb357861e782a864c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11020
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-20 14:58:04 +00:00
Florian Klink
f34e0fa342 feat(tvix/castore/import): only allow normal components in entry paths
Explicitly document and add a debug assertion for that.

It's up to callers to ensure this doesn't happen.

Change-Id: Ib5d154809c2ad2920258e239993d0b790d846dc8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11487
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-20 14:14:19 +00:00