mdebray/tvl-depot
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
tvl-depot/tvix/cli/Cargo.toml

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

26 lines
526 B
TOML
Raw Normal View History

refactor(tvix): split binary (REPL etc.) out from evaluator library The tvix-eval project is independent from any *uses* of the evaluator, such as the tvix-repl. This functionality has been split out into a separate "tvix-cli" crate. Note that this doesn't have to mean that this CLI crate is the "final" CLI crate for tvix, the point of this is not "getting the CLI structure right" but rather "getting the evaluator structure right". This reshuffling is part of restructuring the way that functionality like store communication is injected into language evaluation. Note that at this commit the new CLI crate is not at feature-parity. Change-Id: Id0af03dc8e07ef09a9f882a89612ad555eca8f93 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7541 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-08 22:19:22 +01:00
[package]
name = "tvix-cli"
version = "0.1.0"
edition = "2021"
chore(tvix/cli): rename binary to just 'tvix' Change-Id: If735c81a04f60dcd77f1c40b595b52ed88bbbe95 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7580 Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi> Reviewed-by: flokli <flokli@flokli.de>
2022-12-12 19:40:29 +01:00
[[bin]]
name = "tvix"
path = "src/main.rs"
refactor(tvix): split binary (REPL etc.) out from evaluator library The tvix-eval project is independent from any *uses* of the evaluator, such as the tvix-repl. This functionality has been split out into a separate "tvix-cli" crate. Note that this doesn't have to mean that this CLI crate is the "final" CLI crate for tvix, the point of this is not "getting the CLI structure right" but rather "getting the evaluator structure right". This reshuffling is part of restructuring the way that functionality like store communication is injected into language evaluation. Note that at this commit the new CLI crate is not at feature-parity. Change-Id: Id0af03dc8e07ef09a9f882a89612ad555eca8f93 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7541 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-08 22:19:22 +01:00
[dependencies]
refactor(tvix/nix-compat): absorb //tvix/derivation Put this in its src/derivation. Change-Id: Ic047ab1c2da555a833ee454e10ef60c77537b617 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7967 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-01-31 14:45:42 +01:00
nix-compat = { path = "../nix-compat" }
chore(tvix/store): add fuse feature This brings in fuse (via the `fuser` crate), and adds pkg-config and libfuse to the dev shell, so `cargo build` can link against it. Change-Id: I0d11607490e27d946bdf92b0b9e45f9ab644ba74 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8664 Autosubmit: flokli <flokli@flokli.de> Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-05-28 09:22:08 +02:00
tvix-store = { path = "../store", features = []}
refactor(tvix): split binary (REPL etc.) out from evaluator library The tvix-eval project is independent from any *uses* of the evaluator, such as the tvix-repl. This functionality has been split out into a separate "tvix-cli" crate. Note that this doesn't have to mean that this CLI crate is the "final" CLI crate for tvix, the point of this is not "getting the CLI structure right" but rather "getting the evaluator structure right". This reshuffling is part of restructuring the way that functionality like store communication is injected into language evaluation. Note that at this commit the new CLI crate is not at feature-parity. Change-Id: Id0af03dc8e07ef09a9f882a89612ad555eca8f93 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7541 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-08 22:19:22 +01:00
tvix-eval = { path = "../eval" }
refactor(tvix/{cli,store}): move TvixStoreIO to tvix-cli crate This trait is eval-specific, there's no point in dealing with these things in tvix-store. This implements the EvalIO interface for a Tvix store. The proper place for this glue code (for now) is tvix-cli, which knows about both tvix-store and tvix-eval. There's one annoyance with this move: The `tvix-store import` subcommand previously also used the TvixStoreIO implementation (because it conveniently did what we wanted). Some of this code had to be duplicated, mostly logic to calculate the NAR-based output path and create the PathInfo object. Some, but potentially more of this can be extracted into helper functions in a shared crate, and then be used from both TvixStoreIO in tvix-cli as well as the tvix-store CLI entrypoint. Change-Id: Ia7515e83c1b54f95baf810fbd8414c5521382d40 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9212 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-09-02 20:16:35 +02:00
bytes = "1.4.0"
chore(tvix): upgrade to clap 4.0 In //tvix/eval: * criterion bumped to 4.0, which at least depends on clap 3.x instead of 2.x, which is less incompatible In //tvix/cli: * no changes required In //tvix/nix_cli: * some minor changes for compatibility with clap 4.0, no functionality changes Change-Id: If793f64b59fcaa2402d3d483ddbab4092f32df03 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7588 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-16 12:54:22 +01:00
clap = { version = "4.0", features = ["derive", "env"] }
refactor(tvix/{cli,store}): move TvixStoreIO to tvix-cli crate This trait is eval-specific, there's no point in dealing with these things in tvix-store. This implements the EvalIO interface for a Tvix store. The proper place for this glue code (for now) is tvix-cli, which knows about both tvix-store and tvix-eval. There's one annoyance with this move: The `tvix-store import` subcommand previously also used the TvixStoreIO implementation (because it conveniently did what we wanted). Some of this code had to be duplicated, mostly logic to calculate the NAR-based output path and create the PathInfo object. Some, but potentially more of this can be extracted into helper functions in a shared crate, and then be used from both TvixStoreIO in tvix-cli as well as the tvix-store CLI entrypoint. Change-Id: Ia7515e83c1b54f95baf810fbd8414c5521382d40 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9212 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-09-02 20:16:35 +02:00
data-encoding = "2.3.3"
refactor(tvix): split binary (REPL etc.) out from evaluator library The tvix-eval project is independent from any *uses* of the evaluator, such as the tvix-repl. This functionality has been split out into a separate "tvix-cli" crate. Note that this doesn't have to mean that this CLI crate is the "final" CLI crate for tvix, the point of this is not "getting the CLI structure right" but rather "getting the evaluator structure right". This reshuffling is part of restructuring the way that functionality like store communication is injected into language evaluation. Note that at this commit the new CLI crate is not at feature-parity. Change-Id: Id0af03dc8e07ef09a9f882a89612ad555eca8f93 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7541 Autosubmit: tazjin <tazjin@tvl.su> Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI
2022-12-08 22:19:22 +01:00
dirs = "4.0.0"
refactor(tvix/{cli,store}): move TvixStoreIO to tvix-cli crate This trait is eval-specific, there's no point in dealing with these things in tvix-store. This implements the EvalIO interface for a Tvix store. The proper place for this glue code (for now) is tvix-cli, which knows about both tvix-store and tvix-eval. There's one annoyance with this move: The `tvix-store import` subcommand previously also used the TvixStoreIO implementation (because it conveniently did what we wanted). Some of this code had to be duplicated, mostly logic to calculate the NAR-based output path and create the PathInfo object. Some, but potentially more of this can be extracted into helper functions in a shared crate, and then be used from both TvixStoreIO in tvix-cli as well as the tvix-store CLI entrypoint. Change-Id: Ia7515e83c1b54f95baf810fbd8414c5521382d40 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9212 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-09-02 20:16:35 +02:00
rustyline = "10.0.0"
chore(tvix/*): bump to smol_str 0.2.0 Change-Id: Ic9ac1b6fecb564eafb41b265bf317cd385fdc170 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8560 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-05-11 17:06:32 +02:00
smol_str = "0.2.0"
fix(tvix/cli): handle SRI hashes in outputHash Instead of being called with `md5`, `sha1`, `sha256` or `sha512`, `fetchurl.nix` (from corepkgs / `<nix`) can also be called with a `hash` attribute, being an SRI hash. In that case, `builtin.derivation` is called with `outputHashAlgo` being an empty string, and `outputHash` being an SRI hash string. In other cases, an SRI hash is passed as outputHash, but outputHashAlgo is set too. Nix does modify these values in (single, fixed) output specification it serializes to ATerm, but keeps it unharmed in `env`. Move this into a construct_output_hash helper function, that can be tested better in isolation. Change-Id: Id9d716a119664c44ea7747540399966752e20187 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7933 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
2023-01-26 14:18:12 +01:00
ssri = "7.0.0"
refactor(tvix/cli): describe errors with thiserror This is much less code, and makes it much easier to read. Change-Id: I9028f226105f905c2cc2cabd33907ff493e26225 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7938 Reviewed-by: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI Autosubmit: flokli <flokli@flokli.de>
2023-01-26 23:42:10 +01:00
thiserror = "1.0.38"
refactor(tvix/{cli,store}): move TvixStoreIO to tvix-cli crate This trait is eval-specific, there's no point in dealing with these things in tvix-store. This implements the EvalIO interface for a Tvix store. The proper place for this glue code (for now) is tvix-cli, which knows about both tvix-store and tvix-eval. There's one annoyance with this move: The `tvix-store import` subcommand previously also used the TvixStoreIO implementation (because it conveniently did what we wanted). Some of this code had to be duplicated, mostly logic to calculate the NAR-based output path and create the PathInfo object. Some, but potentially more of this can be extracted into helper functions in a shared crate, and then be used from both TvixStoreIO in tvix-cli as well as the tvix-store CLI entrypoint. Change-Id: Ia7515e83c1b54f95baf810fbd8414c5521382d40 Reviewed-on: https://cl.tvl.fyi/c/depot/+/9212 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: flokli <flokli@flokli.de>
2023-09-02 20:16:35 +02:00
tracing = "0.1.37"
refactor(tvix/cli): use Wu-Manber string scanning for drv references Switch out the string-scanning algorithm used in the reference scanner. The construction of aho-corasick automata made up the vast majority of runtime when evaluating nixpkgs previously. While the actual scanning with a constructed automaton is relatively fast, we almost never scan for the same set of strings twice and the cost is not worth it. An algorithm that better matches our needs is the Wu-Manber multiple string match algorithm, which works efficiently on *long* and *random* strings of the *same length*, which describes store paths (up to their hash component). This switches the refscanner crate to a Rust implementation[0][1] of this algorithm. This has several implications: 1. This crate does not provide a way to scan streams. I'm not sure if this is an inherent problem with the algorithm (probably not, but it would need buffering). Either way, related functions and tests (which were actually unused) have been removed. 2. All strings need to be of the same length. For this reason, we truncate the known paths after their hash part (they are still unique, of course). 3. Passing an empty set of matches, or a match that is shorter than the length of a store path, causes the crate to panic. We safeguard against this by completely skipping the refscanning if there are no known paths (i.e. when evaluating the first derivation of an eval), and by bailing out of scanning a string that is shorter than a store path. On the upside, this reduces overall runtime to less 1/5 of what it was before when evaluating `pkgs.stdenv.drvPath`. [0]: Frankly, it's a random, research-grade MIT-licensed crate that I found on Github: https://github.com/jneem/wu-manber [1]: We probably want to rewrite or at least fork the above crate, and add things like a three-byte wide scanner. Evaluating large portions of nixpkgs can easily lead to more than 65k derivations being scanned for. Change-Id: I08926778e1e5d5a87fc9ac26e0437aed8bbd9eb0 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8017 Tested-by: BuildkiteCI Reviewed-by: flokli <flokli@flokli.de>
2023-02-02 13:51:59 +01:00
[dependencies.wu-manber]
fix(tvix/cli): use tvlfyi/wu-manber fork for refscanner Our fork fixes a small bug (https://github.com/jneem/wu-manber/pull/1) but it's not clear whether upstream will accept patches, so for now lets point this directly at our fork. Change-Id: Iccdcedae3e9a8b783241431787c952561d032694 Reviewed-on: https://cl.tvl.fyi/c/depot/+/8031 Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2023-02-04 10:05:13 +01:00
git = "https://github.com/tvlfyi/wu-manber.git"
Reference in a new issue Copy permalink