Commit graph

151 commits

Author SHA1 Message Date
Florian Klink
ea6f511241 refactor(tvix/glue): return a parsed Url in NixFetchArgs
The only two consumers (fetchurl, fetchtarball) of these do try to parse
it as URL, so do it in the helper.

Update url_basename to take a &url::URL, not a &str.

Also update the test to use rstest for the fixtures to reduce some
boilerplate there.

Change-Id: I1f85fe2803060dc4423e673cb7b9f9bf799d09b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11875
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-26 04:51:31 +00:00
Florian Klink
540e566900 refactor(tvix/glue): take &CAHash, not CAHash
We use a bit less cloning that way.

Change-Id: I28bf99577e4a481e35fbf99d0724adab5502a1bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11874
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-26 04:51:31 +00:00
Florian Klink
78eb22c54d feat(tvix/glue): handle regular file at builtins.path import
If builtins.path is passed a regular file, no filtering is applied.
We use the just-introduced file_type function in the EvalIO trait for
that.

This means, we don't need to pass through filtered_ingest, and can
assemble the FileNode directly in that specific match case.

This also means, we can explicitly calculate the sha256 flat digest,
and avoid having to pipe through the file contents again (via
blob_to_sha256_hash) to construct the sha256 digest.

Change-Id: I500b19dd9e4b7cc897d88b44547e7851559e5a4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11872
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-26 04:51:31 +00:00
Florian Klink
080654aaf9 feat(tvix/eval): add file_type method
This allows peeking of the type at a given path.

It's necessary, as an open() might not fail until you try to read()
from it, and generally, stat'ing can be faster in some cases.

Change-Id: Ib002da3194a3546ca286de49aac8d1022ec5560f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11871
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-26 04:51:31 +00:00
Florian Klink
28b692fd50 feat(tvix/tvix-store): improve progress bars
Don't show an empty spinner for daemon commands.
Move the bar to the right, so the text is better aligned between spinner
progress and bar progress styles.

Generally, push progress bars a bit more down to the place where we can
track progress. This includes adding one in the upload_blob span.

Introduce another progress style template for transfers, which
interprets the counter as bytes (not just a plain integer), and also a data rate.
Use it for here and in the fetching code, and also make the progress bar
itself a bit less wide.

Change-Id: I15c2ea3d2b24b5186cec19cd3dbd706638497f40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11845
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-17 12:57:34 +00:00
Florian Klink
9d44da01ee fix(tvix-glue/fetchers/decompression): propagate error with ?
Change-Id: I5e9d4f067f09307cb6f990ed6eca48345e2fb342
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11810
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-06-13 21:07:40 +00:00
Florian Klink
b992ca49a6 fix(tvix/glue/tvix_store_io): also populate input sources
These also need to be present in the input nodes of the BuildRequest.

Change-Id: Ie9b957805e42f766002581adc6182a6543c5333b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11802
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-06-13 19:59:28 +00:00
Florian Klink
d947f61d36 fix(tvix/glue/tvix_store_io): disable concurrent fetches for now
We need some shared queue, preventing the same fetches/builds from
getting triggered multiple times unnecessarily.

Change-Id: I7c4a3c66db558f5cccd66865b170242b758e3e02
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11800
Reviewed-by: aspen <root@gws.fyi>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-13 16:19:18 +00:00
Florian Klink
7f29cab1cc fix(tvix/glue/tvix_store_io): distinguish waiting and building
We immediately reported "Building", even though then populated necessary
inputs, which looked a bit odd. Make it clear we're still waiting, and
update the spinner message once we have all inputs we were waiting for.

In the future, we might want to have separate spans for this, so the
timer gets reset, but that's something for later.

Change-Id: Ic22c9a906d0e7e7179c5ee328162401261efc224
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11799
Reviewed-by: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-06-13 16:18:14 +00:00
Florian Klink
99c5a2e8bc feat(tvix/glue): report progress on all fetches, use progress bars
This should also report progress on fetches which we couldn't delay
until actually having to IO into them, like `builtins.fetchurl` calls
without a upfront-provided hash.

While at it, upgrade the progress spinners to progress bars, which
increment if we know the size of the fetch.

Change-Id: Ic3f332286d8bc2177f3d994ba25b165728d4b702
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11797
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
2024-06-13 16:16:42 +00:00
Florian Klink
7ee55c293c fix(tvix/glue/tvix_store_io): use same case for progress messages
"Fetching" was uppercase, "building" was lowercase.
Let's make this consistent.

Change-Id: I11c16f1a7d2057ada4d057e553a4ceaa59597f26
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11796
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-06-13 11:59:58 +00:00
Florian Klink
08f794e7e7 fix(tvix-glue/fetchers): remove unused allow dead code
This isn't unused anymore, we can remove the
`#![allow(dead_code)]`.

Change-Id: Ia2048394a60161fbfd4ac2a2b64d03350526faa4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11795
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-12 23:43:54 +00:00
Florian Klink
ddd88a589b feat(tvix/glue/tvix_store_io): show progress info
In `store_path_to_node`, in case we need to build or fetch something,
render a progress bar, using the spinner for now.
We can upgrade this to a progress *bar* later.

Change-Id: I4a7cf5ef8f639076f176af9b39d276be3f37c8ff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11793
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-06-12 23:43:54 +00:00
Florian Klink
842d6816bf feat(tvix/glue): support builtin:fetchurl
nixpkgs calls <nix/fetchurl.nix> during nixpkgs bootstrap.

This produces a fake derivation with system = builtin
and builder = builtin:fetchurl, and needs to download files from the
internet.

At the end of the Derivation construction, if we have such a derivation,
also synthesize a `Fetch` struct, which we add to the known fetch paths.

This will then cause these fetches to be picked up like all other
fetches in TvixStoreIO.

Change-Id: I72cbca4f85da106b25eda97693a6a6e59911cd57
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10975
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-06-12 22:31:17 +00:00
Florian Klink
5963133a62 feat(tvix/glue/fetchers): add NAR fetching infrastructure
The magic <nix/fetchurl.nix> derivation can cause two other types of
fetch to happen, one that unpacks NAR files, and another one that puts a
file as an executable at the store path root.

This adds the necessary enum type and path calculation logic for it to
the fetcher code.

It also adds code to do the actual NAR fetching. The executable case is
still stubbed out.

Change-Id: I79103fd58c7e22ad7fde34efa5e2d89cad7d5a0e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11790
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-12 10:30:40 +00:00
Florian Klink
6073ef9c6f refactor(tvix/glue/fetchers): move store_path() tests here
Move the part asking a fetch for its store_path() to the place where
this function is defined, and add some more test cases.

Change-Id: I96f326d0d56aa5835f23274b8cd1b1afe3724153
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11789
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-12 10:30:40 +00:00
Florian Klink
e1d3fa240a refactor(tvix/glue/fetchers): use named field for structs
This allows giving more self-speaking names, as well as documenting each
field individually.

Change-Id: Ide164d684b7f819aac279cc8e657c02fc24d093f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11786
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-12 10:30:40 +00:00
Florian Klink
a7bf5b975f test(tvix/glue/fetchurl): add output path tests for fetchurl
These are all derived from the FOD representation of the ATerm
serialization, so no new code necessary to calculate these hashes
correctly.

Change-Id: I8c5dacd2a8de3d5042c9dd56910511c19e1b9b31
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11788
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-12 10:30:30 +00:00
binarycat
beb7f57c73 fix(tvix/eval): handle builtins.split matching the empty string
This prevents the following statements from looping endlessly:

```
builtins.split "(.*)" ""
builtins.split "([abc]*)" "abc"
builtins.split "(.*)" "abc"
builtins.split ".*" ""
```

Cover these (and some more examples) in the test suite.

Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: Ibd339f971e0f4e3e5c229816e2be5a8e3836fec9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11743
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-06-11 14:59:02 +00:00
Aspen Smith
d0ab3c8d15 feat(tvix): Switch to jemalloc
Switch tvix to using jemalloc as the default global allocator on
supported (eg, non-msvc) platforms. This gives a pretty reasonable
performance boost basically for free:

int                     time:   [24.361 µs 24.386 µs 24.418 µs]
                        change: [-19.355% -18.859% -18.527%] (p = 0.00 < 0.05)
                        Performance has improved.

merge small attrs       time:   [37.201 µs 37.328 µs 37.442 µs]
                        change: [-24.609% -24.266% -23.982%] (p = 0.00 < 0.05)
                        Performance has improved.

merge large attrs with small attrs
                        time:   [20.030 ms 20.135 ms 20.251 ms]
                        change: [-4.2811% -3.2549% -2.3807%] (p = 0.00 < 0.05)
                        Performance has improved.

hello outpath           time:   [967.91 ms 974.07 ms 983.33 ms]
                        change: [-5.5008% -4.4552% -3.4099%] (p = 0.00 < 0.05)
                        Performance has improved.

Change-Id: I6c6e6e3295ec2fca01ea28dc37bcb201cd811767
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10851
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
2024-06-06 10:29:33 +00:00
Florian Klink
08e520b7cc fix(tvix/glue): drop tracing-indicatif from deps
We don't use this in tvix-glue currently. A previous version of cl/11747
did, and I forgot to remove it afterwards.

Change-Id: I6191ffc60450f3e79f22a3aca55246e3956ee9cc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11756
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Autosubmit: flokli <flokli@flokli.de>
2024-06-06 10:14:20 +00:00
Florian Klink
20513e7a52 feat(tvix/store/bin): add progress bar infrastructure
This adds the tracing-indicatif crate, and configures it as a layer in
our tracing_subscriber pipeline to emit progress for every span that's
configured so.

It also moves from using std::io::stderr to write logs to using their
writer, to avoid clobbering output.

Progress bar styles are defined in a lazy_static, moving this into a
general tracing is left for later.

This adds some usage of this to the `imports` and `copy` commands.

The output can still be improved a bit - we should  probably split each
task up into a smaller (instrumented) helper functions, so we can create
a progress bar for each task.

Change-Id: I59a1915aa4e0caa89c911632dec59c4cbeba1b89
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11747
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-06-06 09:43:24 +00:00
Ilan Joselevich
34d93f1d96 fix(tvix): make rstest pick up new test case files
Context: https://github.com/la10736/rstest/issues/256

Cargo will now cause a rebuild whenever a new test case file is added.
Previsouly running `cargo test` after adding a new test case resulted in
that case not being picked up and ignored.

Change-Id: Ibfc420b5bfe3f3ee41d3ebd3fb9d248819fa6ed9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11751
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-05 17:52:17 +00:00
Aspen Smith
72b9a126b8 feat(tvix/glue): Implement builtins.storePath
This one's relatively simple - we just check if the store path exists,
and if it does we make a new contextful string containing the store path
as its only context element.

Automatic testing seems tricky for this (I think?) so I tested it
manually:

tvix-repl> builtins.storePath /nix/store/yn46i4xx5alh7gs6fpkxk430i34rp2q9-hello-2.12.1
=> "/nix/store/yn46i4xx5alh7gs6fpkxk430i34rp2q9-hello-2.12.1" :: string

Change-Id: I8a0d9726e4102ab872c53c2419679c2c855a5a18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11696
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
2024-06-05 17:50:15 +00:00
Ilan Joselevich
1b39d5868a feat(tvix): add CI targets for checking crate features powerset
Closes: https://b.tvl.fyi/issues/401

With this change all crate features (and their combinations) will be built and
tested in CI.

From now on, when adding/removing a Cargo feature for a crate,
you will want to add it to the features power set that gets tested in CI.
For each crate there's a default.nix with a `mkFeaturePowerset` invocation,
modify the list to include/remove the feature.
Note that you don't want to add "collection" features,
such as `fs` for tvix-[ca]store or `default`.

Change-Id: I966dde1413d057770787da3296cce9c1924570e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11717
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-03 16:35:51 +00:00
Florian Klink
649a862ae1 feat(tvix/eval): rm NixContext::join, add take_context & IntoIterator
In places where we want to extend context with that from another
NixString, use take_context() to split it off, then call .extend(),
making use of IntoIterator to avoid a bunch of clones.

Change-Id: I2460141a3ed776c64c36132b2203b6a1d710b922
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11705
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
2024-05-23 14:50:38 +00:00
Florian Klink
ec8d79f3db feat(tvix/eval): teach builtins.toXML context
XmlEmitter gains a NixContext field, and `write_typed_value` extends it
with all context elements present in the passed value.

Once all serialization is done, a into_context() function returns the
collected context, so we can construct a NixString with context.

Tests for this live in tvix-glue, as we use builtins.derivation, which
is not present in the tvix-eval crate.

Fixes b/398.

Change-Id: I85feaaa17b753885f8a017a54e419ec4e602af21
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11704
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Alyssa Ross <hi@alyssa.is>
2024-05-23 14:49:07 +00:00
Florian Klink
14766cfe1d refactor(tvix/store): drop calculate_nar from PathInfoService
This shouldn't be part of the PathInfoService trait.

Pretty much none of the PathInfoServices do implement it, and requiring
them to implement it means they also cannot make use of this calculation
already being done by other PathInfoServices.

Move it out into its own NarCalculationService trait, defined somewhere
at tvix_store::nar, and have everyone who wants to trigger nar
calculation use nar_calculation_service directly, which now is an
additional field in TvixStoreIO for example.

It being moved outside the PathInfoService trait doesn't prohibit
specific implementations to implement it (like the GRPC client for the
`PathInfoService` does.

This is currently wired together in a bit of a hacky fashion - as of
now, everything uses the naive implementation that traverses blob and
directoryservice, rather than composing it properly. I want to leave
that up to a later CL, dealing with other parts of store composition
too.

Change-Id: I18d07ea4301d4a07651b8218bc5fe95e4e307208
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11619
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-11 13:33:59 +00:00
Connor Brewster
da9bc274f3 refactor(tvix): remove usage of async-recursion
Rust 1.77 supports async recursion as long as there is some form of
indirection (ie. `Box::pin`). This removes the need to use the
async-recursion crate.

Change-Id: Ic9613ab7f32016f0103032a861edff92e2fb8b41
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11596
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-05-06 16:05:09 +00:00
Florian Klink
aaf258f61e feat(tvix/store): use async nar reader for ingest_nar
Rename read_nar to ingest_nar, and have it use the async nar reader
version, and the ingest_entries machinery.

This means we can now drop all code dealing with manually assembling
castore nodes.

Update our consumer, NixHTTPPathInfoService to use the new API.
As we now accept an AsyncRead, we don't need to do any blocking here
anymore, and can use the same async-compression crate as in the fetching
logic (and support some more compression formats out of the box).

Change-Id: I8646d20bd8603f8da47b5c84bc9e4ac236eb7f1a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11580
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-06 15:24:17 +00:00
Florian Klink
ba00f0c695 refactor(tvix/*store): use DS: DirectoryService
We implement DirectoryService for Arc<DirectoryService> and
Box<DirectoryService>, this is sufficient.

Change-Id: I0a5a81cbc4782764406b5bca57f908ace6090737
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11586
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-04 21:27:26 +00:00
Florian Klink
abc0553eb8 feat(tvix/castore/directory/traverse): use castore Paths
This switches from using std::path::Path to using castore paths.

We can drop some error handling in descend_to, as absolute (or redundant)
paths are not representable.

We however now need to convert from a std::path::Path to our
representation, and decide to accept .. canonicalization, as paths in
EvalIO might contain this. Dealing .. to hop into another store path, if
we encounter this, should be dealt with in a previous step.

Change-Id: I5e94693808420c5d56587c68731252b54755bf93
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11575
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-05-02 15:26:29 +00:00
Florian Klink
c0d5439362 refactor(nix-compat): derivation_or_fod_hash -> hash_derivation_modulo
There's no need for us to come up with our own names for this.
Also update the comments/docstrings a bit, and inline the intermediate
hash_derivation_modulo calculation.

Change-Id: I09dab8ffe1ebfb6601841e98119eee4ff25d8f39
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11578
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-02 10:32:00 +00:00
Florian Klink
c9d3946cb5 refactor(tvix/castore/import): restructure error types
Have ingest_entries return an Error type with only three kinds:

 - Error while uploading a specific Directory
 - Error while finalizing the directory upload
 - Error from the producer

Move all ingestion method-specific errors to the individual
implementations.

Change-Id: I2a015cb7ebc96d084cbe2b809f40d1b53a15daf3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11557
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-30 17:12:39 +00:00
Alice Carroll
8d49ff3d64 test(tvix): Fix tvix tests on macOS
Prior to this, some tests would not build
or would fail in an obscure way.

Change-Id: I68587cc7592492ebfd71ca02fc7ccc9ff7c0196f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11544
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-30 00:55:34 +00:00
Florian Klink
c192cd04b8 feat(tvix/glue/tests): configure <nix NIX_PATH
This is also needed to make `<nix/fetchurl.nix>` available in the
testsuite.

At some point, we might want to classify different types of tests
excepting a different featureset, but for now, enabling this for all of
glue is fine.

Change-Id: Ided450bbb1f8eb7b66d454bd28bd19b17eb318e4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11546
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-29 22:02:03 +00:00
Florian Klink
fab9a07274 feat(tvix/glue/tests): wrap TvixStoreIO with TvixIO
This makes `<nix/fetchurl.nix>` available in the testsuite.

Change-Id: I04657b227c539edec86e5b6033e1016adcfce244
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11537
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-29 21:13:55 +00:00
Florian Klink
499075a84f refactor(tvix/glue/fetchers): rename "nixhash" to just "hash"
Change-Id: I2e40b1bcbd9932469ec2af9bb9ce20bb267566ec
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11536
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-29 17:58:25 +00:00
Florian Klink
298c6876cc docs(tvix/glue/fetchers): update docstring on Tarball
The tarball can also not be compressed.

Change-Id: Idbf1b7168cc9a43826355e3cb8b4140f5f53d780
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11535
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-29 15:44:26 +00:00
Florian Klink
98b85b4580 refactor(tvix/glue): clone less
Prepare the NixAttr to return without an intermediate
Vec<(String, NixString)>, and without into_iter(), and send off the
unmoved Derivation struct to known_paths without having to clone it.

Change-Id: Icdb9f78938e998a27d0313c5d9ab15b93af5821d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11531
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
2024-04-29 13:19:41 +00:00
Florian Klink
26b77b2cf3 refactor(tvix/glue): move decompression into fetchers/ subdir
This is specifically used for the fetcher code (only).
Moving it to there for now.

Change-Id: I1e1d0541b85340ef4ff3a4c6b3fa99b51853f539
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11532
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-29 13:11:36 +00:00
Connor Brewster
d2e67f021e refactor(tvix/castore): add separate Error enum for archives
The `Error` enum for the `imports` crate has both filesystem and archive
specific errors and was starting to get messy.

This adds a separate `Error` enum for archive-specific errors and then
keeps a single `Archive` variant in the top-level import `Error` for all
archive errors.

Change-Id: I4cd0746c864e5ec50b1aa68c0630ef9cd05176c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11498
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
Reviewed-by: flokli <flokli@flokli.de>
2024-04-24 15:41:38 +00:00
Connor Brewster
79698c470c feat(tvix/castore): upload blobs concurrently when ingesting archives
Ingesting tarballs with a lot of small files is very slow because of the
round trip time to the `BlobService`. To mitigate this, small blobs can
be buffered into memory and uploaded concurrently in the background.

Change-Id: I3376d11bb941ae35377a089b96849294c9c139e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11497
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
2024-04-23 17:02:07 +00:00
Florian Klink
e18bc33529 fix(tvix/glue/tvix_store_io): remove early return
Doing the fetch comes up with the root node, but we still need to
descend from there to the desired subpath.

Move things around to ensure the fetch case also only sets root_node.

This logic should probably be moved into smaller, easier to consume
functions.

Change-Id: I6ab9317df794f53d2504029bbc77859e89fef1ed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11507
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-23 14:53:31 +00:00
Florian Klink
72d3f9b914 fix(tvix/glue/fetchers): rename node name for all three types
We also need to rename the node in case it's a directory or symlink at
the root.

Change-Id: I6e9957200f65991645ae3e1755b943200453dfd5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11506
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-04-23 14:41:25 +00:00
Florian Klink
8181817e53 feat(tvix/glue/fetchers): support file:// URLs
Nix supports file:// - URLs for `fetchurl` and `fetchTarball`.

Convert the enums and function arguments to hold a URL type.
reqwest::Url is a re-export of the url crate, but they don't re-export
the parsing errors, and as we want to hold these in our Error types, add
it to Cargo.toml explicitly.

The Fetcher::download function now checks on the scheme, and either
opens the file locally, or does do a HTTP request as before.

Fetch gets its custom debug impl, removing potentially sensitive
username and password out of URLs.

Change-Id: I777db1fe487370e822cbfec4624034aca5e08045
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11504
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-23 12:41:17 +00:00
Florian Klink
dfef3d18d1 test(tvix/glue): add tests for fetchurl and fetchTarball
Change-Id: I53a0590ecf4e5fcb1bfd1d127824211338e28256
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11503
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-23 12:41:05 +00:00
Florian Klink
30950833c9 feat(tvix/glue/store_io): have KnownPaths track fetches too
Have fetcher builtins call queue_fetch() whenever they don't need to
fetch something immediately, and teach TvixStoreIO::store_path_to_node
on how to look up (and call ingest_and persist on our Fetcher).

Change-Id: Id4bd9d639fac9e4bee20c0b1c584148740b15c2f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11501
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-04-23 12:40:55 +00:00
Florian Klink
091de12a9a refactor(tvix/glue): move Fetch[er] into its own types, fetch lazily
We actually want to delay fetching until we actually need the file. A
simple evaluation asking for `.outPath` or `.drvPath` should work even
in a pure offline environment.

Before this CL, the fetching logic was quite distributed between
tvix_store_io, and builtins/fetchers.rs.

Rather than having various functions and conversions between structs,
describe a Fetch as an enum type, with the fields describing the fetch.

Define a store_path() function on top of `Fetch` which can be used to
ask for the calculated store path (if the digest has been provided
upfront).

Have a `Fetcher` struct, and give it a `fetch_and_persist` function,
taking a `Fetch` as well as a desired name, and have it deal with all
the logic of persisting the PathInfos. It also returns a StorePathRef,
similar to the `.store_path()` method on a `Fetch` struct.

In a followup CL, we can extend KnownPaths to track fetches AND
derivations, and then use `Fetcher` when we need to do IO into that
store path.

Change-Id: Ib39a96baeb661750a8706b461f8ba4abb342e777
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11500
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-23 12:30:48 +00:00
Florian Klink
5fc403587f refactor(tvix/castore): ingest filesystem entries in parallel
Rather than carrying around an Future in the IngestionEntry::Regular,
simply carry the plain B3Digest.

Code reading through a non-seekable data stream has no choice but to
read and upload blobs immediately, and code seeking through something
seekable (like a filesystem) probably knows better what concurrency to
pick when ingesting, rather than the consuming side.

(Our only) one of these seekable source implementations is now doing
exactly that. We produce a stream of futures, and then use
[StreamExt::buffered] to process more than one, concurrently.

We still keep the same order, to avoid shuffling things and violating
the stream order.

This also cleans up walk_path_for_ingestion in castore/import, as well
as ingest_dir_entries in glue/tvix_store_io.

Change-Id: I5eb70f3e1e372c74bcbfcf6b6e2653eba36e151d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11491
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-04-20 18:54:28 +00:00