Commit graph

135 commits

Author SHA1 Message Date
Florian Klink
8ea93bb646 refactor(tvix/cli/derivation): use if let to destructure
We only do logic here if algo and hash_mode are Some(_)
(and there's an `out` output).

The fact we don't do anything in all in other cases is a bit hidden at
the bottom. Use if let for the destructuring, and drop the other case.

Change-Id: Icc0e38e62947d52d48ef610f754749737977fca9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7966
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 13:26:11 +00:00
Florian Klink
6fa91349a9 refactor(tvix/cli): remove unneeded clone
Change-Id: I6f4cb24bdd636af8918a2ade44075af92161c97d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7965
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 13:26:11 +00:00
Florian Klink
4b3ccd205a refactor(tvix/cli): absorb construct_output_hash
This helper function only was created because
populate_output_configuration was hard to test before cl/7939.

With that out of the way, we can pull it in.

Change-Id: I64b36c0eb34343290a8d84a03b0d29392a821fc7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7961
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-31 08:52:00 +00:00
Vincent Ambo
e4bb750b3b refactor(tvix/cli): force outside of output configuration helper
Change-Id: I28357fe131cefedcef9761b08a72f675f4a10789
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7939
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 08:51:03 +00:00
Vincent Ambo
124af9e5d5 refactor(tvix/cli): add helper method for strong string coercion
This is repetitive and error prone (e.g. switching around
to_string/as_str has drastic consequences) due to the ToString
overloads.

Change-Id: I9b16a2e0e05e4c21e83f43e9f603746eb42e53f7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7947
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-01-29 17:39:36 +00:00
Florian Klink
e3c2b3650a refactor(tvix/cli): describe errors with thiserror
This is much less code, and makes it much easier to read.

Change-Id: I9028f226105f905c2cc2cabd33907ff493e26225
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7938
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-01-27 14:08:15 +00:00
Florian Klink
a94a1434cc fix(tvix/cli): handle SRI hashes in outputHash
Instead of being called with `md5`, `sha1`, `sha256` or `sha512`,
`fetchurl.nix` (from corepkgs / `<nix`) can also be called with a `hash`
attribute, being an SRI hash.

In that case, `builtin.derivation` is called with `outputHashAlgo` being
an empty string, and `outputHash` being an SRI hash string.

In other cases, an SRI hash is passed as outputHash, but outputHashAlgo
is set too.

Nix does modify these values in (single, fixed) output specification it
serializes to ATerm, but keeps it unharmed in `env`.

Move this into a construct_output_hash helper function, that can be
tested better in isolation.

Change-Id: Id9d716a119664c44ea7747540399966752e20187
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7933
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-27 14:06:13 +00:00
Vincent Ambo
f22b9cb0d7 feat(tvix/cli): faux implementation of builtins.toFile
This adds an implementation of this builtin which correctly calculates
paths, but does not actually write anything to the store or verify
references.

Change-Id: Ie9764cbc1d13a73d8dc9350910304e2b7cad3fe8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7910
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-27 12:21:41 +00:00
Vincent Ambo
de10a924f2 feat(tvix/cli): implement builtins.derivation
This uses the actual upstream Nix code for
`builtins.derivation` (which is not a primop in C++ Nix) to implement
`builtins.derivation` as a wrapper around `builtins.derivationStrict`.

We're doing it this way to ensure that our thunking logic is correct.
An initial Rust-native rewrite (see e.g. cl/7363) is pretty difficult
to debug while there are still other issues to root out, but
eventually we might want to turn this into native code.

Change-Id: I5845e18073e103b8670e40648bd7fd9b511058e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7902
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-27 12:21:41 +00:00
Vincent Ambo
8a9aa018dc feat(tvix/cli): implement builtins.derivationStrict
Implements the logic for converting an evaluator value supplied as
arguments to builtins.derivationStrict into an actual,
fully-functional derivation struct.

This skips the implementation of structuredAttrs, which are left for a
subsequent commit.

Note: We will need to port some eval tests over to CLI to test this
correct, which will be done in a separate commit later on.

Change-Id: I0db69dcf12716180de0eb0b126e3da4683712966
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7756
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-27 12:21:41 +00:00
Vincent Ambo
3d7c371e22 feat(tvix/cli): add helper for handling special drv parameters
Adds a helper function which handles special parameters to
`builtins.derivation` that are not just blindly passed through to the
builder environment, but populate other specific fields of the
derivation (outside of the ones handled by other, more complex helpers
from previous commits).

Change-Id: I82d1edf9af714fc4591e9071c0b83ece83be7eee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7901
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-27 12:21:41 +00:00
Vincent Ambo
dfc50c9ef5 feat(tvix/cli): add helper for populating drv output configuration
This threads through the fields that control whether a derivation is a
fixed-output derivation or not.

Change-Id: I49739de178fed9f258291174ca1a2c15a7cf5c2a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7900
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-27 12:21:41 +00:00
Vincent Ambo
fdca93d6ed feat(tvix/cli): add helper for populating derivation inputs
This adds a helper function which takes the output of the reference
scanner used on derivation inputs and populates the `input_sources`
and `input_derivations` field of the derivation accordingly.

Note that we have a divergence from C++ Nix here, as we do not
populate the entire FS closure of a literally referred derivation (and
our standing theory is that this is unnecessary for nixpkgs).

Change-Id: Id0f605dd8c0a82973c56605c2b8f478fc17777d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7899
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-27 12:21:41 +00:00
Vincent Ambo
c6811f0cea feat(tvix/cli): add helper for populating derivation outputs
Adds a small helper function which uses a Nix value supplied to
`builtins.derivation{Strict}` to populate the `outputs` field of the
`Derivation` struct.

Change-Id: Iccc7a4f293b3d913140aed576a573a8992241e46
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7898
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-27 12:21:41 +00:00
Vincent Ambo
2e66f7da92 fix(tvix/cli): correctly trim cppnix output in NixCompatIO
We only stripped one of the two uses of this string, leading to
extraneous newlines in the refscanner.

Change-Id: I25d9119be082c487352f0cf66b97ecdcc3e1de06
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7932
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-26 10:29:55 +00:00
Vincent Ambo
259faea2b2 feat(tvix/cli): add errors module with drv construction errors
These will be threaded through to eval through the new `TvixError`
variant.

Change-Id: Ia0d3f8710dcf26bb95015cd2a6a2b2911f06343f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7842
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-20 15:39:51 +00:00
Vincent Ambo
0cbdfffea2 refactor(tvix/cli): consistently assert type unity in known_paths
No situation should be allowed in which a path is inserted into
known_paths with different types twice, which we previously enforced
only for some path types.

Change-Id: I8cb47d4b29c0aab3c58694f8b590e131deba7043
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7843
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-17 10:45:09 +00:00
Vincent Ambo
79e54f46ce feat(tvix/cli): add replacement strings tracking to KnownPaths
Replacement strings are some weird internal feature of Nix that is
required for calculating derivation hashes. We need to track these
like other paths, as they need to be re-used on builds with
dependencies on values from previous builds.

Change-Id: Ie955b3fb5ae3685cfadfbe4d06ea6b5e219590c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7828
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-17 10:31:42 +00:00
Vincent Ambo
499e72c1cb feat(tvix/cli): track known plain paths in NixCompatIO
When adding things to a C++ Nix store, ensure that the path is tracked
in the tracker.

Since the mechanism for propagating the tracker instance isn't
finalised yet, I've opted to take an Rc<RefCell> parameter for it. How
exactly that ends up there is going to become clear in the next
commits, but for now it's just instantiated in main with
Default::default.

Change-Id: I90f0b44f2d4f292dedc98ff1aa39041d279b61fd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7833
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-17 10:31:42 +00:00
Vincent Ambo
9cb3daee20 refactor(tvix/cli): reference scanner owns all the strings
This gets very complex very quickly otherwise, as all the construction
paths for a reference scanner and all the access patterns for the
KnownPaths structure are not yet fully understood.

Change-Id: Ibadf1f18b476695f3c286fc6896ae557760edf63
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7827
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-17 10:31:42 +00:00
Vincent Ambo
f27f5ef0c9 feat(tvix/cli): add known_paths module
This module implements types used to track the set of known paths in
the context of an evaluation.

These are used to determine the build references of a derivation.

Change-Id: I81e15ae33632784e699128916485751613b231a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7816
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-17 10:20:38 +00:00
Vincent Ambo
e63bff5545 feat(tvix/refscan): implement reference scanning over data streams
Using yet more machinery from the pretty comprehensive aho_corasick
crate, this makes it possible to pass anything implementing `io::Read`
to the `ReferenceScanner` to accumulate matches.

Change-Id: I5b0e28eb44ea4df24010f40831e29f2cbb8c1f80
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7810
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-11 16:03:51 +00:00
Vincent Ambo
3045645df0 feat(tvix/cli): implement initial refscan module
This module implements a ReferenceScanner struct which uses the
aho_corasick crate to scan string inputs for known, non-overlapping
candidates (store paths, in our case).

I experimented with several different APIs, and landed on this version
with an initial accumulator in the scanner. The scanner is
instantiated from the candidates and "fed" all the strings, then
consumed by the caller to retrieve the result.

Right now only things that look vaguely like bytestrings can be fed to
the scanner, there is no streaming support in the API yet.

Change-Id: I7782f0f0df5fc64bccd813aa14712f5525b0168c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7808
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-11 16:03:51 +00:00
Vincent Ambo
0e421a16c5 feat(tvix/cli): add --compile-only "linter" functionality
With this, tvix/cli can be run on files and only produce compiler
errors and warnings.

Change-Id: I5dd9229fc065647787daafd17d7c1540579a1d98
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7764
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-06 12:23:55 +00:00
Vincent Ambo
3d238c350b refactor(tvix/eval): streamline construction of globals/builtins
Previously the construction of globals (a compiler-only concept) and
builtins (a (now) user-facing API) was intermingled between multiple
different modules, and kind of difficult to understand.

The complexity of this had grown in large part due to the
implementation of `builtins.import`, which required the notorious
"knot-tying" trick using Rc::new_cyclic (see cl/7097) for constructing
the set of globals.

As part of the new `Evaluation` API users should have the ability to
bring their own builtins, and control explicitly whether or not impure
builtins are available (regardless of whether they're compiled in or
not).

To streamline the construction and allow the new API features to work,
this commit restructures things by making these changes:

1. The `tvix_eval::builtins` module is now only responsible for
   exporting sets of builtins. It no longer has any knowledge of
   whether or not certain sets (e.g. only pure, or pure+impure) are
   enabled, and it has no control over which builtins are globally
   available (this is now handled in the compiler).

2. The compiler module is now responsible for both constructing the
   final attribute set of builtins from the set of builtins supplied
   by a user, as well as for populating its globals (that is
   identifiers which are available at the top-level scope).

3. The `Evaluation` API now carries a `builtins` field which is
   populated with the pure builtins by default, and can be extended by
   users.

4. The `import` feature has been moved into the compiler, as a
   special case. In general, builtins no longer have the ability to
   reference the "fix point" of the globals set.

This should not change any functionality, and in fact preserves minor
differences between Tvix/Nix that we already had (such as
`builtins.builtins` not existing).

Change-Id: Icdf5dd50eb81eb9260d89269d6e08b1e67811a2c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7738
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-04 12:28:43 +00:00
Vincent Ambo
e3f81e8d3b feat(tvix/cli): add :d REPL-prefix to print explanations of values
Change-Id: I1dd8de8d996e07840f9b0aaebf932b812103a43a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7593
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-22 19:09:40 +00:00
Vincent Ambo
ea7d63e177 feat(tvix/cli): implement NixCompatIO helper type
This type allows for temporarily compatibility with the C++ Nix store,
specifically (for now) it gives us the store directory used by Nix and
imports files the same way.

Change-Id: I4767794ef2863eba49661315c63c4e17de946d60
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7587
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-22 19:09:40 +00:00
Vincent Ambo
29790147b6 fix(tvix/cli): use tvix_eval::StdIO in CLI
... until we have a store-I/O layer, or something that intercepts the
store-related stuff appropriately.

Change-Id: I22f63435b3f9e118e3faeb2924fda8373a23ea7f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7568
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-21 22:37:11 +00:00
Vincent Ambo
8b3d03db92 chore(tvix/cli): rename binary to just 'tvix'
Change-Id: If735c81a04f60dcd77f1c40b595b52ed88bbbe95
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7580
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
2022-12-21 21:48:08 +00:00
Vincent Ambo
f467df06dc chore(tvix/cli): re-add observer flags
Users can again pass flags for dumping the AST, bytecode, and runtime
trace.

With this commit the CLI is at feature-parity with what we had before,
but entirely through the new API.

Change-Id: I30fe26f243224b25d1e4f828fec607325ef88306
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7550
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-12-21 13:23:38 +00:00
Vincent Ambo
60d9fba56d chore(tvix/cli): re-add NIX_PATH handling
Change-Id: I5595d6f5141ed4a533ca44a46264dc604fca6be1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7549
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-12-21 13:23:38 +00:00
Vincent Ambo
36eaa9b3b3 chore(tvix/cli): re-add --raw argument
Change-Id: I82f469f7f186c754c40bd941587b30aa3fc48045
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7548
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-21 13:23:38 +00:00
Vincent Ambo
6d3d237923 refactor(tvix/eval): consume self in Evaluation::evaluate
This simplifies lifetime management for observers in callers of tvix_eval.

Change-Id: I2f47c8d89f22b1c766526e5d1483c0d026b500ae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7546
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-21 13:23:38 +00:00
Vincent Ambo
257c67f204 chore(tvix): upgrade to clap 4.0
In //tvix/eval:

* criterion bumped to 4.0, which at least depends on clap 3.x instead
  of 2.x, which is less incompatible

In //tvix/cli:

* no changes required

In //tvix/nix_cli:

* some minor changes for compatibility with clap 4.0, no functionality
  changes

Change-Id: If793f64b59fcaa2402d3d483ddbab4092f32df03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7588
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-21 13:23:38 +00:00
Vincent Ambo
d9e2bec953 refactor(tvix): split binary (REPL etc.) out from evaluator library
The tvix-eval project is independent from any *uses* of the evaluator,
such as the tvix-repl.

This functionality has been split out into a separate "tvix-cli"
crate. Note that this doesn't have to mean that this CLI crate is the
"final" CLI crate for tvix, the point of this is not "getting the CLI
structure right" but rather "getting the evaluator structure right".

This reshuffling is part of restructuring the way that functionality
like store communication is injected into language evaluation.

Note that at this commit the new CLI crate is not at feature-parity.

Change-Id: Id0af03dc8e07ef09a9f882a89612ad555eca8f93
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7541
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-21 13:09:29 +00:00