mdebray/tvl-depot
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
tvl-depot/tvix/glue/Cargo.toml

59 lines
1.6 KiB
TOML
Raw Normal View History

refactor(tvix): move tvix glue code into glue crate There's various bits and pieces in tvix-cli that use both the store and evaluator, as well as nix-compat. For example, builtins.derivation, as well as the reference scanning implementation. This "glue code" currently isn't accessible from anywhere else, but it'd be very useful if it were. Move it out into a `glue` crate, and make `tvix-cli` a consumer of it. All the KnownPaths setup and passing around, as well as NIX_PATH handling is also something that should probably be moved into the glue crate as well, but that's something left for a future CL. Change-Id: I080ed3d1825ab23790666486840f301f00856277 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9908 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-03 12:34:37 +01:00
[package]
name = "tvix-glue"
version = "0.1.0"
edition = "2021"
[dependencies]
feat(tvix/store): use async nar reader for ingest_nar Rename read_nar to ingest_nar, and have it use the async nar reader version, and the ingest_entries machinery. This means we can now drop all code dealing with manually assembling castore nodes. Update our consumer, NixHTTPPathInfoService to use the new API. As we now accept an AsyncRead, we don't need to do any blocking here anymore, and can use the same async-compression crate as in the fetching logic (and support some more compression formats out of the box). Change-Id: I8646d20bd8603f8da47b5c84bc9e4ac236eb7f1a Reviewed-on: https://cl.tvl.fyi/c/depot/+/11580 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-04-30 11:17:20 +02:00
async-compression = { version = "0.4.9", features = ["tokio", "gzip", "bzip2", "xz"]}
feat(tvix/glue/derivationStrict): support __structuredAttrs This adds support to handle the __structuredAttrs argument, which can be passed to builtins.derivationStrict. If __structuredAttrs is passed, and set to true, most of the arguments passed to builtins.derivationStrict are not simply coerced to a string and passed down to "environments", but instead kept in a more structured fashion. Inside ATerm, which is what's relevant as far as path calculation is concerned, a virtual `__json` environment variable is present, containing these structured values. Inside Builds, these structured values are not made available as an environment variable, but a JSON file (and source-able bash script). This will need to be respected once we start emitting BuildRequests, and for that we can probably just parse the `__json` key in Derivation.environment again - or keep this additionally in non-serialized form around during Evaluation. No matter what, this is left for a followup CL. The existing handle_derivation_parameters and populate_outputs helper function were removed, as __structuredAttrs causes quite a change in behaviour, and so handling both in the same place makes it more readable. There's some open questions w.r.t. string contexts for structured attrs itself. A TODO is left for this, but at least path calculation for individual structured attrs derivations are correct now. Part of b/366. Change-Id: Ic293822266ced6f8c4826d8ef0d2e098a4adccaa Reviewed-on: https://cl.tvl.fyi/c/depot/+/10604 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-11 14:44:31 +01:00
bstr = "1.6.0"
bytes = "1.4.0"
chore(tvix): bump to data-encoding 2.6.0 Change-Id: I26af403bfa99e5d1cff24641a3dba908e1d06686 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11899 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-30 18:54:27 +02:00
data-encoding = "2.6.0"
feat(tvix/glue): drive builds on IO That's one possible abstraction to drive builds. Whenever IO into a store path is requested, we look up the root node, and in case we don't have it in PathInfoService, but KnownPaths gives us a Derivation for that output path, trigger a build and await the result. This recursively might trigger builds for parent paths if they haven't been built yet. Another option would be to simply expose a PathInfoService interface for a builder too, and loop all building into IO via PathInfoService composition - but let's start with something. Note tvix-cli doesn't have a configurable BuildService yet, it's plugged to the DummyBuildService, so whenever it needs to do a build, it'll fail, but that's how it can be provoked: ``` (builtins.readFile (import <nixpkgs> {}).hello.outPath + "/bin/hello") […] error[E029]: I/O error: /nix/store/cg8a576pz2yfc1wbhxm1zy4x7lrk8pix-hello-2.12.1: builds are not supported with DummyBuildService --> [code]:1:2 | 1 | (builtins.readFile (import <nixpkgs> {}).hello.outPath + "/bin/hello") | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ``` Note how this fails, while pure output path calculation (`(import <nixpkgs> {}).hello.outPath + "/bin/hello")`) still succeeds. Change-Id: Id2075d8a2b18554d0dd608b4b29146a8cd411e7f Reviewed-on: https://cl.tvl.fyi/c/depot/+/10793 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-01-16 20:14:38 +01:00
futures = "0.3.30"
feat(tvix/glue): Add AsyncRead wrapper to decompress streams Add a new AsyncRead wrapper, DecompressedReader, that wraps an underlying AsyncRead, but sniffs the magic bytes at the start of the stream to determine which compression format is being used out of the three that are supported by builtins.fetchTarball, and switches to the correct decompression algorithm adapter dynamically. This will be used in the implementation of builtins.fetchTarball Change-Id: I892a4683d5c93e67d4c173f3d21199bdc6605922 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11019 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-02-23 20:42:52 +01:00
magic = "0.16.2"
refactor(tvix): move tvix glue code into glue crate There's various bits and pieces in tvix-cli that use both the store and evaluator, as well as nix-compat. For example, builtins.derivation, as well as the reference scanning implementation. This "glue code" currently isn't accessible from anywhere else, but it'd be very useful if it were. Move it out into a `glue` crate, and make `tvix-cli` a consumer of it. All the KnownPaths setup and passing around, as well as NIX_PATH handling is also something that should probably be moved into the glue crate as well, but that's something left for a future CL. Change-Id: I080ed3d1825ab23790666486840f301f00856277 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9908 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-03 12:34:37 +01:00
nix-compat = { path = "../nix-compat" }
feat(tvix/glue): Add AsyncRead wrapper to decompress streams Add a new AsyncRead wrapper, DecompressedReader, that wraps an underlying AsyncRead, but sniffs the magic bytes at the start of the stream to determine which compression format is being used out of the three that are supported by builtins.fetchTarball, and switches to the correct decompression algorithm adapter dynamically. This will be used in the implementation of builtins.fetchTarball Change-Id: I892a4683d5c93e67d4c173f3d21199bdc6605922 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11019 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-02-23 20:42:52 +01:00
pin-project = "1.1"
chore(tvix): upgrade to tonic 0.12 / hyper 1.0 Change-Id: Idd8ce48869ddd869d51a10959b920f1290a8a9b3 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11991 Autosubmit: yuka <yuka@yuka.dev> Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-07-20 20:15:42 +02:00
reqwest = { version = "0.12.0", features = ["rustls-tls-native-roots"], default-features = false }
feat(tvix/build): add derivation_to_build_request This function converts from a nix_compat::derivation::Derivation to a BuildRequest. In addition to the Derivation itself, it needs two lookup functions to map input paths to their castore nodes. Change-Id: I0332982f0bc7933a5fda137fe39d5a850639d929 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10236 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-09 11:53:17 +01:00
tvix-build = { path = "../build", default-features = false, features = []}
refactor(tvix): move tvix glue code into glue crate There's various bits and pieces in tvix-cli that use both the store and evaluator, as well as nix-compat. For example, builtins.derivation, as well as the reference scanning implementation. This "glue code" currently isn't accessible from anywhere else, but it'd be very useful if it were. Move it out into a `glue` crate, and make `tvix-cli` a consumer of it. All the KnownPaths setup and passing around, as well as NIX_PATH handling is also something that should probably be moved into the glue crate as well, but that's something left for a future CL. Change-Id: I080ed3d1825ab23790666486840f301f00856277 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9908 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-03 12:34:37 +01:00
tvix-eval = { path = "../eval" }
tvix-castore = { path = "../castore" }
tvix-store = { path = "../store", default-features = false, features = []}
feat(tvix/glue/tvix_store_io): show progress info In `store_path_to_node`, in case we need to build or fetch something, render a progress bar, using the spinner for now. We can upgrade this to a progress *bar* later. Change-Id: I4a7cf5ef8f639076f176af9b39d276be3f37c8ff Reviewed-on: https://cl.tvl.fyi/c/depot/+/11793 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-06-12 23:17:18 +02:00
tvix-tracing = { path = "../tracing" }
refactor(tvix): move tvix glue code into glue crate There's various bits and pieces in tvix-cli that use both the store and evaluator, as well as nix-compat. For example, builtins.derivation, as well as the reference scanning implementation. This "glue code" currently isn't accessible from anywhere else, but it'd be very useful if it were. Move it out into a `glue` crate, and make `tvix-cli` a consumer of it. All the KnownPaths setup and passing around, as well as NIX_PATH handling is also something that should probably be moved into the glue crate as well, but that's something left for a future CL. Change-Id: I080ed3d1825ab23790666486840f301f00856277 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9908 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-03 12:34:37 +01:00
tracing = "0.1.37"
feat(tvix/glue/tvix_store_io): show progress info In `store_path_to_node`, in case we need to build or fetch something, render a progress bar, using the spinner for now. We can upgrade this to a progress *bar* later. Change-Id: I4a7cf5ef8f639076f176af9b39d276be3f37c8ff Reviewed-on: https://cl.tvl.fyi/c/depot/+/11793 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: Connor Brewster <cbrewster@hey.com> Tested-by: BuildkiteCI
2024-06-12 23:17:18 +02:00
tracing-indicatif = "0.3.6"
refactor(tvix): move tvix glue code into glue crate There's various bits and pieces in tvix-cli that use both the store and evaluator, as well as nix-compat. For example, builtins.derivation, as well as the reference scanning implementation. This "glue code" currently isn't accessible from anywhere else, but it'd be very useful if it were. Move it out into a `glue` crate, and make `tvix-cli` a consumer of it. All the KnownPaths setup and passing around, as well as NIX_PATH handling is also something that should probably be moved into the glue crate as well, but that's something left for a future CL. Change-Id: I080ed3d1825ab23790666486840f301f00856277 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9908 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-03 12:34:37 +01:00
tokio = "1.28.0"
feat(tvix/glue): Add AsyncRead wrapper to decompress streams Add a new AsyncRead wrapper, DecompressedReader, that wraps an underlying AsyncRead, but sniffs the magic bytes at the start of the stream to determine which compression format is being used out of the three that are supported by builtins.fetchTarball, and switches to the correct decompression algorithm adapter dynamically. This will be used in the implementation of builtins.fetchTarball Change-Id: I892a4683d5c93e67d4c173f3d21199bdc6605922 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11019 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-02-23 20:42:52 +01:00
tokio-tar = "0.3.1"
feat(tvix/glue): Implement builtins.fetchurl Implement the fetchurl builtin, and lay the groundwork for implementing the fetchTarball builtin (which works very similarly, and is implemented using almost the same code in C++ nix). An overview of how this works: 1. First, we check if the store path that *would* result from the download already exists in the store - if it does, we just return that 2. If we need to download the URL, TvixStoreIO has an `http_client: reqwest::Client` field now which we use to make the request 3. As we're downloading the blob, we hash the data incrementally into a SHA256 hasher 4. We compare the hash against the expected hash (if any) and bail out if it doesn't match 5. Finally, we put the blob in the store and return the store path Since the logic is very similar, this commit also implements a *chunk* of `fetchTarball` (though the actual implementation will likely include a refactor to some of the code reuse here). The main thing that's missing here is caching of downloaded blobs when fetchurl is called without a hash - I've opened b/381 to track the TODO there. Adding the `SSL_CERT_FILE` here is necessary to teach reqwest how to load it during tests - see 1c16dee20 (feat(tvix/store): use reqwests' rustls-native-roots feature, 2024-03-03) for more info. Change-Id: I83c4abbc7c0c3bfe92461917e23d6d3430fbf137 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11017 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: aspen <root@gws.fyi>
2024-02-23 16:09:20 +01:00
tokio-util = { version = "0.7.9", features = ["io", "io-util", "compat"] }
refactor(tvix): move tvix glue code into glue crate There's various bits and pieces in tvix-cli that use both the store and evaluator, as well as nix-compat. For example, builtins.derivation, as well as the reference scanning implementation. This "glue code" currently isn't accessible from anywhere else, but it'd be very useful if it were. Move it out into a `glue` crate, and make `tvix-cli` a consumer of it. All the KnownPaths setup and passing around, as well as NIX_PATH handling is also something that should probably be moved into the glue crate as well, but that's something left for a future CL. Change-Id: I080ed3d1825ab23790666486840f301f00856277 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9908 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-03 12:34:37 +01:00
thiserror = "1.0.38"
feat(tvix/glue/derivationStrict): support __structuredAttrs This adds support to handle the __structuredAttrs argument, which can be passed to builtins.derivationStrict. If __structuredAttrs is passed, and set to true, most of the arguments passed to builtins.derivationStrict are not simply coerced to a string and passed down to "environments", but instead kept in a more structured fashion. Inside ATerm, which is what's relevant as far as path calculation is concerned, a virtual `__json` environment variable is present, containing these structured values. Inside Builds, these structured values are not made available as an environment variable, but a JSON file (and source-able bash script). This will need to be respected once we start emitting BuildRequests, and for that we can probably just parse the `__json` key in Derivation.environment again - or keep this additionally in non-serialized form around during Evaluation. No matter what, this is left for a followup CL. The existing handle_derivation_parameters and populate_outputs helper function were removed, as __structuredAttrs causes quite a change in behaviour, and so handling both in the same place makes it more readable. There's some open questions w.r.t. string contexts for structured attrs itself. A TODO is left for this, but at least path calculation for individual structured attrs derivations are correct now. Part of b/366. Change-Id: Ic293822266ced6f8c4826d8ef0d2e098a4adccaa Reviewed-on: https://cl.tvl.fyi/c/depot/+/10604 Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-11 14:44:31 +01:00
serde = "1.0.195"
serde_json = "1.0"
feat(tvix/glue): handle passAsFile This extends derivation_to_build_request to handle passAsFile the same way Nix does, and adds a unit test for it. I opted to making this function fallible (if passAsFile contains a non-existent env var), rather than pushing all of this into the Derivation validate function. Change-Id: I75b635f1f6c0c78d72b9a8fc7824f77e97b69951 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10522 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-02 13:17:02 +01:00
sha2 = "0.10.8"
refactor(tvix/glue): move Fetch[er] into its own types, fetch lazily We actually want to delay fetching until we actually need the file. A simple evaluation asking for `.outPath` or `.drvPath` should work even in a pure offline environment. Before this CL, the fetching logic was quite distributed between tvix_store_io, and builtins/fetchers.rs. Rather than having various functions and conversions between structs, describe a Fetch as an enum type, with the fields describing the fetch. Define a store_path() function on top of `Fetch` which can be used to ask for the calculated store path (if the digest has been provided upfront). Have a `Fetcher` struct, and give it a `fetch_and_persist` function, taking a `Fetch` as well as a desired name, and have it deal with all the logic of persisting the PathInfos. It also returns a StorePathRef, similar to the `.store_path()` method on a `Fetch` struct. In a followup CL, we can extend KnownPaths to track fetches AND derivations, and then use `Fetcher` when we need to do IO into that store path. Change-Id: Ib39a96baeb661750a8706b461f8ba4abb342e777 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11500 Reviewed-by: raitobezarius <tvl@lahfa.xyz> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2024-04-22 13:02:48 +02:00
sha1 = "0.10.6"
md-5 = "0.10.6"
feat(tvix/glue/fetchers): support file:// URLs Nix supports file:// - URLs for `fetchurl` and `fetchTarball`. Convert the enums and function arguments to hold a URL type. reqwest::Url is a re-export of the url crate, but they don't re-export the parsing errors, and as we want to hold these in our Error types, add it to Cargo.toml explicitly. The Fetcher::download function now checks on the scheme, and either opens the file locally, or does do a HTTP request as before. Fetch gets its custom debug impl, removing potentially sensitive username and password out of URLs. Change-Id: I777db1fe487370e822cbfec4624034aca5e08045 Reviewed-on: https://cl.tvl.fyi/c/depot/+/11504 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI
2024-04-22 19:05:49 +02:00
url = "2.4.0"
feat(tvix/eval): implement `builtins.filterSource` We add a new set of builtins called `import_builtins`, which will contain import-related builtins, such as `builtins.path` and `builtins.filterSource`. Both can import paths into the store, with various knobs to alter the result, e.g. filtering, renaming, expected hashes. We introduce `filtered_ingest` which will drive the filtered ingestion via the Nix function via the generator machinery, and then we register the root node to the path info service inside the store. `builtins.filterSource` is very simple, `builtins.path` is a more complicated model requiring the same logic albeit more sophisticated with name customization, file ingestion method and expected SHA-256. Change-Id: I1083f37808b35f7b37818c8ffb9543d9682b2de2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10654 Autosubmit: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-01-17 07:45:55 +01:00
walkdir = "2.4.0"
refactor(tvix): move service addrs into shared clap struct Change-Id: I7cab29ecfa1823c2103b4c47b7d784bc31459d55 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12008 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de> Autosubmit: yuka <yuka@yuka.dev>
2024-07-21 16:41:09 +02:00
clap = { version = "4.4.0", default-features = false }
refactor(tvix): move tvix glue code into glue crate There's various bits and pieces in tvix-cli that use both the store and evaluator, as well as nix-compat. For example, builtins.derivation, as well as the reference scanning implementation. This "glue code" currently isn't accessible from anywhere else, but it'd be very useful if it were. Move it out into a `glue` crate, and make `tvix-cli` a consumer of it. All the KnownPaths setup and passing around, as well as NIX_PATH handling is also something that should probably be moved into the glue crate as well, but that's something left for a future CL. Change-Id: I080ed3d1825ab23790666486840f301f00856277 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9908 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-11-03 12:34:37 +01:00
[dependencies.wu-manber]
git = "https://github.com/tvlfyi/wu-manber.git"
[dev-dependencies]
feat(tvix/glue): add nixpkgs eval benchmark This adds a criterion.rs-based testbench into tvix-glue. It can be invoked by running `cargo bench` from inside the `tvix-glue` crate. `target/criterion/report/index.html` contains nice graphs. It's able to diff against the previous run, so you can invoke `cargo bench` before and after a certain change to reason about the impact in evaluation performance. Currently, we need to create a bunch of Evaluator resources inside the benchmark loop itself, which is a bit annoying, as it leaks into the things we benchmark. This should become better with b/262. Fixes b/322. Change-Id: I91656a308887baa1d459ed54d58baae919a4aaf2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10245 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-09 17:45:39 +01:00
criterion = { version = "0.5", features = ["html_reports"] }
test(tvix/glue/known_paths): add tests Change-Id: I02dc891c49ed650c66c403107767b11b6d5e2734 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10949 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: Peter Kolloch <info@eigenvalue.net> Autosubmit: flokli <flokli@flokli.de>
2024-02-17 07:10:13 +01:00
hex-literal = "0.4.1"
feat(tvix/build): add derivation_to_build_request This function converts from a nix_compat::derivation::Derivation to a BuildRequest. In addition to the Derivation itself, it needs two lookup functions to map input paths to their castore nodes. Change-Id: I0332982f0bc7933a5fda137fe39d5a850639d929 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10236 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-09 11:53:17 +01:00
lazy_static = "1.4.0"
feat(tvix): Jemalloc -> MiMalloc Use the faster and newer MiMalloc memory allocator for all endpoints in the workspace. Change-Id: Ic60237284ed168e46ec6e8f28e2710bae4385c6f Reviewed-on: https://cl.tvl.fyi/c/depot/+/12149 Tested-by: BuildkiteCI Reviewed-by: aspen <root@gws.fyi>
2024-08-07 23:21:26 +02:00
mimalloc = "0.1.43"
feat(tvix/eval): implement `builtins.filterSource` We add a new set of builtins called `import_builtins`, which will contain import-related builtins, such as `builtins.path` and `builtins.filterSource`. Both can import paths into the store, with various knobs to alter the result, e.g. filtering, renaming, expected hashes. We introduce `filtered_ingest` which will drive the filtered ingestion via the Nix function via the generator machinery, and then we register the root node to the path info service inside the store. `builtins.filterSource` is very simple, `builtins.path` is a more complicated model requiring the same logic albeit more sophisticated with name customization, file ingestion method and expected SHA-256. Change-Id: I1083f37808b35f7b37818c8ffb9543d9682b2de2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10654 Autosubmit: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-01-17 07:45:55 +01:00
nix = { version = "0.27.1", features = [ "fs" ] }
feat(tvix/glue): introduce test suite for context strings This is an additional test suite on the top of the Nix ones for context strings matters. It already smoked out multiple mistakes and potential bugs and non-deterministic result from the evaluator. It uses a similar technology as the one in the tvix-eval albeit we instantiate a fully fledged evaluator with in-memory store. We copy the files instead of symlinking them because crates are built in isolation, so symlinks cannot work. Change-Id: I63ae225ce4f83c6e2c8ccd60d779c2f8eb9d08fb Reviewed-on: https://cl.tvl.fyi/c/depot/+/10619 Autosubmit: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-01-14 01:41:16 +01:00
pretty_assertions = "1.4.0"
chore(tvix): bump rstest to 0.19.0 Change-Id: Ib2f5e84fdb8be1210b3507da67d4fe84f061651e Reviewed-on: https://cl.tvl.fyi/c/depot/+/11387 Tested-by: BuildkiteCI Reviewed-by: Brian Olsen <me@griff.name> Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-10 13:37:45 +02:00
rstest = "0.19.0"
feat(tvix/glue): add some import tests This creates a directory with a .keep file inside, and uses "${path/to/there}" to coerce it to a string (and import it into the store), ensuring it calculates the same store paths as Nix does. Change-Id: Ie14ae075104ce278bc4f2cce93aab5762a2734d1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10343 Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de> Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-13 10:34:17 +01:00
tempfile = "3.8.1"
feat(tvix/glue): Add refscanner pattern and AsyncRead This splits the existing ReferenceScanner into a ReferenceScanner and ReferencePattern as well as adds an AsyncRead implementation that can do a scan while you read from it. The reason to split the scanner in two is that generating the pattern is expensive and when ingesting build results with multiple outputs you want to do several independant scans that look for the same pattern. The reader is for scanning files without having to load the entire file into memory. Change-Id: I993f5a32308c12d9035840f8e04fe82e8dc1d962 Reviewed-on: https://cl.tvl.fyi/c/depot/+/12052 Autosubmit: Brian Olsen <me@griff.name> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-07-22 15:44:42 +02:00
tokio-test = "0.4.3"
feat(tvix/glue): add nixpkgs eval benchmark This adds a criterion.rs-based testbench into tvix-glue. It can be invoked by running `cargo bench` from inside the `tvix-glue` crate. `target/criterion/report/index.html` contains nice graphs. It's able to diff against the previous run, so you can invoke `cargo bench` before and after a certain change to reason about the impact in evaluation performance. Currently, we need to create a bunch of Evaluator resources inside the benchmark loop itself, which is a bit annoying, as it leaks into the things we benchmark. This should become better with b/262. Fixes b/322. Change-Id: I91656a308887baa1d459ed54d58baae919a4aaf2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10245 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-09 17:45:39 +01:00
feat(tvix/glue): introduce test suite for context strings This is an additional test suite on the top of the Nix ones for context strings matters. It already smoked out multiple mistakes and potential bugs and non-deterministic result from the evaluator. It uses a similar technology as the one in the tvix-eval albeit we instantiate a fully fledged evaluator with in-memory store. We copy the files instead of symlinking them because crates are built in isolation, so symlinks cannot work. Change-Id: I63ae225ce4f83c6e2c8ccd60d779c2f8eb9d08fb Reviewed-on: https://cl.tvl.fyi/c/depot/+/10619 Autosubmit: raitobezarius <tvl@lahfa.xyz> Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2024-01-14 01:41:16 +01:00
[features]
default = ["nix_tests"]
# Enables running the Nix language test suite from the original C++
# Nix implementation (at version 2.3) against Tvix.
nix_tests = []
feat(tvix/glue): add nixpkgs eval benchmark This adds a criterion.rs-based testbench into tvix-glue. It can be invoked by running `cargo bench` from inside the `tvix-glue` crate. `target/criterion/report/index.html` contains nice graphs. It's able to diff against the previous run, so you can invoke `cargo bench` before and after a certain change to reason about the impact in evaluation performance. Currently, we need to create a bunch of Evaluator resources inside the benchmark loop itself, which is a bit annoying, as it leaks into the things we benchmark. This should become better with b/262. Fixes b/322. Change-Id: I91656a308887baa1d459ed54d58baae919a4aaf2 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10245 Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-09 17:45:39 +01:00
[[bench]]
name = "eval"
harness = false
Reference in a new issue Copy permalink