Commit graph

1222 commits

Author SHA1 Message Date
Vincent Ambo
2c07ff0f8c docs(tvix): add more information to README
The README was very sparse before and we've actually had people email
us (as it says to contact us) just to ask what Tvix *is*. This should
answer some questions!

Change-Id: I0f248cb060eccfe086468afed1d648652b35dfd1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8018
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
2023-02-02 16:25:52 +00:00
Vincent Ambo
503e6810e7 fix(tvix/eval): unsafeDiscardStringContext is a no-op
... not just a TODO.

Most use-cases of unsafeDiscardStringContext are for cases where a
string is processed in some ways and no longer contains a "physical"
reference, but still has its context attached in C++ Nix.

We don't need to do this. This does diverge in behaviour in use-cases
related to build scheduling, but that whole behaviour will be
different in Tvix.

Change-Id: I4056d4c09f62d44d6bd52b791db03fe5556672b5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8016
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-02 14:30:32 +00:00
Vincent Ambo
75d1b72b7c refactor(tvix/eval): import_cache can be a HashMap
... instead of a BTreeMap, as we do not need ordering guarantees here.
HashMaps are noticeably faster here (especially as we've been sorting
essentially random data!).

Change-Id: Ie92d74286df9f763c04c9b226ef1066ee8484c13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8014
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-02-02 14:25:30 +00:00
Vincent Ambo
98ea614c6e chore(tvix/eval): elaborate on internal types in Value::type_of
This aids in debugging quite substantially.

Change-Id: Ic43232aa6165ae1c3db7ac2701938e1dfeeb418c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8013
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
2023-02-02 09:29:43 +00:00
Florian Klink
6abd1e2dae chore(tvix): update Cargo.lock
Change-Id: I08b2dc2393819c4f0d3871fe05bdbd29e89a6d75
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8010
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-01 23:06:02 +00:00
Vincent Ambo
dc1e86001c fix(tvix): add dummy target to attach extra-step to
Change-Id: I594a6652e2efe7aa6e35c7cdd84fc3097660614f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8009
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-02-01 17:34:08 +00:00
Vincent Ambo
87de3fff52 chore(tvix): mirror exported repository
Pushes a mirror of tvix to github/tvlfyi/tvix

Change-Id: I86e7c1b1dc1fcc1012fa72234588b26ac2554056
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8007
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-02-01 17:21:05 +00:00
Florian Klink
5ac33a227f docs(tvix): update README to document usage inside tvix view
If the tvix view is cloned through josh, you don't use mg, but a
`shell.nix` is provided.

Also, add the `git clone` command, so people browsing tvix source code
in the browser know where to clone from.

Change-Id: I18483d6a52953f9f4eafd1533ea69afb0e329b04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8001
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-01 17:11:50 +00:00
Florian Klink
f5ccb65f26 feat(tvix/nix-compat/derivation): Display -> to_aterm_string()
Instead of implementing `std::fmt::Display for Derivation` and relying
on the `to_string` method, introduce a `to_aterm_string()` method, which
does the same thing, but makes it clearer what we're producing, rather
than just calling `to_string()``.

Change-Id: I21823de9096a0f2c2eb6f4591e48c1aa9fd94161
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7998
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-01 16:31:56 +00:00
Florian Klink
99d5cf822a refactor(tvix/cli): use nixhash module for output hash calculation
This covers all the weird corner cases.

Change-Id: I85637e82e8929828064ab562dc8a1c8bf161fffa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7991
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-01 15:29:48 +00:00
Florian Klink
265dd36b98 feat(tvix/nix-compat): add nixhash module
This module takes care of parsing various hashes and algorithms.

It will get used to modify derivation output hashes in the next CL.

Change-Id: Idc07c401dbb7510f49883ac02b8379b9a5d930c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7990
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-01 15:29:48 +00:00
Florian Klink
366d2c5e92 refactor(tvix/nix-compat/derivation): remove module inception
The module `src/derivation/derivation.rs` is a sign of module inception.

Move the Derivation struct definiton up into `src/derivation/mod.rs`,
and some of the helpers in a `util.rs`.

Change-Id: Ib24a5f8a27bdd45df8b1fa2b3482a79b33cab8d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7997
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-02-01 13:43:25 +00:00
Florian Klink
2551adb4f1 refactor(tvix/nix-compat): operator precedence can trip the unwary
warning: operator precedence can trip the unwary
  --> nix-compat/src/nixbase32.rs:41:23
   |
41 |                 c |= ((input[i + 1] as u16) << 8 - j as u16) as u8
   |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: consider parenthesizing your expression: `(input[i + 1] as u16) << (8 - j as u16)`
   |
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#precedence
   = note: `#[warn(clippy::precedence)]` on by default

Change-Id: I091071d649abf4ed38f5f4e39a0c5d21a0459bff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7996
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-01 13:43:24 +00:00
Florian Klink
361ffd7903 refactor(tvix/derivation): don't create deref'd immediately ref
clippy says:

> This expression creates a reference which is immediately dereferenced
> by the compiler

Change-Id: Ic2c093b043ebee9ae80912075083107e4d216cf1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7995
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-01 13:38:21 +00:00
Florian Klink
ce4d87b129 refactor(tvix/nix-compat): remove unneeded return
Change-Id: I97b9fb3ad33d2f51baf29486d4eff11f8dedcbab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7994
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-02-01 13:38:19 +00:00
Florian Klink
894d17b29b refactor(tvix/nix-compat): const is always 'static
Change-Id: If33bbf47a6cc376f2c0935ea4331a819223b2f00
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7993
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-01 13:38:18 +00:00
Vincent Ambo
759f9dbf39 feat(tvix/cli): implement builtins.placeholder
This doesn't require any other corresponding handling *yet*, as the
actual replacements happen in the builder logic (which we delegate to
cppnix at the moment).

Change-Id: I034147c933f05ae427c7a8794647132d108d0ede
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7972
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-02-01 10:01:40 +00:00
Florian Klink
5f62cd1ad5 refactor(tvix/serde): allow dead_code in struct
This is an example, which uses the debug trait to print all field
values.

Silence the compiler warning about unused fields.

Change-Id: I5f1216c77819003302e83ba1af1ff13c924f3b38
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7971
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-01-31 15:35:46 +00:00
Florian Klink
a7e20bbef3 docs(tvix/nix-compat/store_path): fix docstrings
Change-Id: I19261233e338b406a87fe680675eb4dfd853fe11
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7970
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-01-31 15:35:45 +00:00
Vincent Ambo
ddf6e6cf47 fix(tvix/eval): allow builtins.toXML to serialise any function
This adds a fake argument name to builtins.toXML which allows toXML to
serialise any value instead of panicking on functions. We do still
have to fix the value itself, eventually, though.

Change-Id: I2e330ecddcd80442b4fac5eced64431ac86123ba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7962
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-31 15:31:30 +00:00
Florian Klink
0db73cb2bd feat(tvix/store): add write_nar function
This adds a function that consumes a [proto::node::Node] pointing to the
root of a (store) path, and writes the contents in NAR serialization to
the passed [std::io::Write].

We need this in various places:

 - tvix-store's calculate_nar() RPC method needs to render a NAR stream
   to get the nar hash, which is necessary to give things imported in
   the store a "NAR-based" store path.

 - communication with (remote) Nix (via daemon protocol) needs a NAR
   representation.

 - Things like nar-bridge, exposing a NAR/NARInfo HTTP interface need a
   NAR representation.

Change-Id: I7fb2e0bf01814a1c09094c0e35394d9d6b3e43b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7956
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 15:28:22 +00:00
Florian Klink
a23b7e17c0 refactor(tvix/nix-compat): absorb nar writer
Expose it at `nix_compat::nar::writer`.

Change-Id: I98a8acfa6277074f24608d64fb6c0082b41d10f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7969
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 15:18:39 +00:00
Florian Klink
1e28cc7ede docs(tvix/nix-compat/derivation): fix docstrings
Change-Id: I3fd02e62c8a24fa7f27461b17cdd824b060d9bdb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7968
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-01-31 15:16:40 +00:00
Florian Klink
2d24c5f260 refactor(tvix/nix-compat): absorb //tvix/derivation
Put this in its src/derivation.

Change-Id: Ic047ab1c2da555a833ee454e10ef60c77537b617
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7967
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-01-31 15:16:39 +00:00
Florian Klink
9e809e21cc feat(tvix/store): implement iteration over Directory nodes
This provides a `Directory.nodes()` function, returning an iterator over
all three node types in an ordered fashion.

Change-Id: Ib98696c03a9db8b6c613d6e2bf5587c1ae35133f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7955
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 13:29:18 +00:00
Florian Klink
1ee6bd06e3 feat(tvix/store/proto): implement get_name for node::Node, pub trait
Make the trait public, so consumers can use it.

Also, implement it for node::Node, so we can later use this to access
the name from all three if we don't care about the enum type.

Change-Id: Iae530a16b705493645e61947852c03273876cc55
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7963
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 13:29:18 +00:00
Florian Klink
c27bacd905 refactor(tvix): introduce nix-compat crate
Move nixbase32 and store_path into this.

This allows //tvix/cli to not pull in //tvix/store for now.

Change-Id: Id3a32867205d95794bc0d33b21d4cb3d9bafd02a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7964
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-31 13:26:18 +00:00
Florian Klink
8ea93bb646 refactor(tvix/cli/derivation): use if let to destructure
We only do logic here if algo and hash_mode are Some(_)
(and there's an `out` output).

The fact we don't do anything in all in other cases is a bit hidden at
the bottom. Use if let for the destructuring, and drop the other case.

Change-Id: Icc0e38e62947d52d48ef610f754749737977fca9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7966
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 13:26:11 +00:00
Florian Klink
6fa91349a9 refactor(tvix/cli): remove unneeded clone
Change-Id: I6f4cb24bdd636af8918a2ade44075af92161c97d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7965
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 13:26:11 +00:00
Vincent Ambo
4da1e64c03 feat(tvix/serde): add an example application to the project
This shows how people can use tvix_serde to deserialise configuration
structs for their programs from Nix code.

Change-Id: I71bf4e03dce19dddafe67dd729b4e4b10719a739
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7945
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-31 13:11:28 +00:00
Vincent Ambo
7702941dd9 test(tvix/eval): add tests for internal formals dependencies
Formals can depend on each other when using another formal as a
default value.

This test ensures that the compiler's declaration and initialisation
order of formals is consistent with what actually happens in the VM.

Change-Id: Ibdabe262554e8066d67fac1ebc3b5a48ef626e18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7948
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-31 10:25:02 +00:00
Florian Klink
4b3ccd205a refactor(tvix/cli): absorb construct_output_hash
This helper function only was created because
populate_output_configuration was hard to test before cl/7939.

With that out of the way, we can pull it in.

Change-Id: I64b36c0eb34343290a8d84a03b0d29392a821fc7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7961
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-31 08:52:00 +00:00
Vincent Ambo
e4bb750b3b refactor(tvix/cli): force outside of output configuration helper
Change-Id: I28357fe131cefedcef9761b08a72f675f4a10789
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7939
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 08:51:03 +00:00
Florian Klink
8ec8d03175 fix(tvix/store/nixbase32): fix encoder/decoder
Replace our data_encoding usage with the implementation taken from
https://github.com/nix-community/go-nix/tree/master/pkg/nixbase32

Also uncomment some of the unit tests, and add a regression test for a
NIXBASE32.encode with a 32 bytes sequence.

The previous implementation of NIXBASE32.encode is wrong in that case:

```
❯ nix hash to-base32 sha256-s6JN6XqP28g1uYMxaVAQMLiXcDG8tUs7OsE3QPhGqzA=
0c5b8vw40dy178xlpddw65q9gf1h2186jcc3p4swinwggbllv8mk
❯ echo -n s6JN6XqP28g1uYMxaVAQMLiXcDG8tUs7OsE3QPhGqzA= | base64 -d | hexdump -C
00000000  b3 a2 4d e9 7a 8f db c8  35 b9 83 31 69 50 10 30  |..M.z...5..1iP.0|
00000010  b8 97 70 31 bc b5 4b 3b  3a c1 37 40 f8 46 ab 30  |..p1..K;:.7@.F.0|
00000020
```

Change-Id: I0468b62bbbab390f8d7d3812e657e5d59dceed59
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7934
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-30 11:46:30 +00:00
Florian Klink
85e563c554 refactor(tvix/store/tests): move Directory tests into directory.rs
There's gonna be more tests coming, let's move them into a separate file
everything Directory-related into a separate module.

Change-Id: I78a0f263925528907a22724b028f75cce644b329
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7954
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-30 09:45:11 +00:00
Florian Klink
a2c33f517c feat(tvix/store/proto): implement get_name for all nodes
Also add a `NamedNode` trait. We'll later use this to access names from
all three individually.

Change-Id: Icb5afd6fa5a0d834e9908294382de9892a5a6440
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7953
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-30 09:45:11 +00:00
Florian Klink
9c18888715 docs(tvix/store/protos): add docstring for Directory::digest()
Change-Id: I361dbca444a267fea28cd212d563ee2d03497c16
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7952
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-30 09:45:11 +00:00
Florian Klink
42d3c06988 docs(tvix/store/protos): fix docstring
Change-Id: Idf835e9648ab6c95c29960f3f8176153d4888b27
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7951
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-30 09:45:11 +00:00
Vincent Ambo
124af9e5d5 refactor(tvix/cli): add helper method for strong string coercion
This is repetitive and error prone (e.g. switching around
to_string/as_str has drastic consequences) due to the ToString
overloads.

Change-Id: I9b16a2e0e05e4c21e83f43e9f603746eb42e53f7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7947
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-01-29 17:39:36 +00:00
Florian Klink
e3c2b3650a refactor(tvix/cli): describe errors with thiserror
This is much less code, and makes it much easier to read.

Change-Id: I9028f226105f905c2cc2cabd33907ff493e26225
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7938
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-01-27 14:08:15 +00:00
Florian Klink
a94a1434cc fix(tvix/cli): handle SRI hashes in outputHash
Instead of being called with `md5`, `sha1`, `sha256` or `sha512`,
`fetchurl.nix` (from corepkgs / `<nix`) can also be called with a `hash`
attribute, being an SRI hash.

In that case, `builtin.derivation` is called with `outputHashAlgo` being
an empty string, and `outputHash` being an SRI hash string.

In other cases, an SRI hash is passed as outputHash, but outputHashAlgo
is set too.

Nix does modify these values in (single, fixed) output specification it
serializes to ATerm, but keeps it unharmed in `env`.

Move this into a construct_output_hash helper function, that can be
tested better in isolation.

Change-Id: Id9d716a119664c44ea7747540399966752e20187
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7933
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-27 14:06:13 +00:00
Florian Klink
bda5fc58d0 feat(tvix/derivation): derive PartialEq for error
Allows easier to use these errors in test cases.

Change-Id: I8abee3b522909ed4aa4b066499cc48eececbc036
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7937
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-27 14:06:12 +00:00
Vincent Ambo
f22b9cb0d7 feat(tvix/cli): faux implementation of builtins.toFile
This adds an implementation of this builtin which correctly calculates
paths, but does not actually write anything to the store or verify
references.

Change-Id: Ie9764cbc1d13a73d8dc9350910304e2b7cad3fe8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7910
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-27 12:21:41 +00:00
Vincent Ambo
de10a924f2 feat(tvix/cli): implement builtins.derivation
This uses the actual upstream Nix code for
`builtins.derivation` (which is not a primop in C++ Nix) to implement
`builtins.derivation` as a wrapper around `builtins.derivationStrict`.

We're doing it this way to ensure that our thunking logic is correct.
An initial Rust-native rewrite (see e.g. cl/7363) is pretty difficult
to debug while there are still other issues to root out, but
eventually we might want to turn this into native code.

Change-Id: I5845e18073e103b8670e40648bd7fd9b511058e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7902
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-27 12:21:41 +00:00
Vincent Ambo
8a9aa018dc feat(tvix/cli): implement builtins.derivationStrict
Implements the logic for converting an evaluator value supplied as
arguments to builtins.derivationStrict into an actual,
fully-functional derivation struct.

This skips the implementation of structuredAttrs, which are left for a
subsequent commit.

Note: We will need to port some eval tests over to CLI to test this
correct, which will be done in a separate commit later on.

Change-Id: I0db69dcf12716180de0eb0b126e3da4683712966
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7756
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-27 12:21:41 +00:00
Vincent Ambo
3d7c371e22 feat(tvix/cli): add helper for handling special drv parameters
Adds a helper function which handles special parameters to
`builtins.derivation` that are not just blindly passed through to the
builder environment, but populate other specific fields of the
derivation (outside of the ones handled by other, more complex helpers
from previous commits).

Change-Id: I82d1edf9af714fc4591e9071c0b83ece83be7eee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7901
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-27 12:21:41 +00:00
Vincent Ambo
dfc50c9ef5 feat(tvix/cli): add helper for populating drv output configuration
This threads through the fields that control whether a derivation is a
fixed-output derivation or not.

Change-Id: I49739de178fed9f258291174ca1a2c15a7cf5c2a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7900
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-27 12:21:41 +00:00
Vincent Ambo
fdca93d6ed feat(tvix/cli): add helper for populating derivation inputs
This adds a helper function which takes the output of the reference
scanner used on derivation inputs and populates the `input_sources`
and `input_derivations` field of the derivation accordingly.

Note that we have a divergence from C++ Nix here, as we do not
populate the entire FS closure of a literally referred derivation (and
our standing theory is that this is unnecessary for nixpkgs).

Change-Id: Id0f605dd8c0a82973c56605c2b8f478fc17777d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7899
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-27 12:21:41 +00:00
Vincent Ambo
c6811f0cea feat(tvix/cli): add helper for populating derivation outputs
Adds a small helper function which uses a Nix value supplied to
`builtins.derivation{Strict}` to populate the `outputs` field of the
`Derivation` struct.

Change-Id: Iccc7a4f293b3d913140aed576a573a8992241e46
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7898
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-27 12:21:41 +00:00
Vincent Ambo
1028aff105 chore(tvix/eval): remove dead comment
Change-Id: Icedb7f272e5067569b8dbf1c2d8b0fdd352b8e12
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7936
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
2023-01-26 23:30:43 +00:00
Florian Klink
54b4fb8b16 refactor(tvix/derivation): align error messages with rust style
Change-Id: I9ccd4c043bdddefee98a2c0b3d6eb7d9cb53c454
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7935
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-01-26 16:30:33 +00:00
Vincent Ambo
2e66f7da92 fix(tvix/cli): correctly trim cppnix output in NixCompatIO
We only stripped one of the two uses of this string, leading to
extraneous newlines in the refscanner.

Change-Id: I25d9119be082c487352f0cf66b97ecdcc3e1de06
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7932
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-26 10:29:55 +00:00
sterni
daa3721f73 docs(tvix): fix minor spelling problems in pointer equality document
Main one is the its-it's mistake in the last paragraph, the rest was
suggested by LanguageTool.

Change-Id: If1b87a11f480452f312fc2759be7ded782d0a522
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7930
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2023-01-25 14:30:50 +00:00
sterni
8a8325fb9d docs(tvix/eval): builtins.add is not equivalent to +
While it is in the given example, i.e. for integer addition, to claim
that they are equivalent is a bit misleading: builtins.add is less
overloaded than +, i.e. builtins.add "foo" "bar" will fail whereas
"foo" + "bar" performs string concatenation.

Change-Id: Ib52d530d1ab289b367565b286f06a76dd518d4fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7929
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-25 14:30:50 +00:00
Florian Klink
669496f0ba test(tvix/eval): add test for total_fmt_float
Change-Id: If6c478ee3d2e4ecf5ef92289614f86535ad05cb7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7927
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-01-25 11:24:20 +00:00
Vincent Ambo
164005656d refactor(tvix/eval): extract float formatting into a helper
This keeps the actual TotalDisplay implementation readable, as this
float formatting code suddenly made up the majority of its implementation.

Change-Id: I2c0d00e4a691e0b8ffbc72680f680e16feef4bee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7925
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-25 11:24:19 +00:00
Florian Klink
e0b05c0fa6 feat(tvix/eval): implement builtins.fromTOML
This allows parsing TOML from Tvix. We can enable the eval-okay-fromTOML
testcase from nix_tests. It uses the `toml` crate, and the serde
integration it brings with it.

Change-Id: Ic6f95aacf2aeb890116629b409752deac49dd655
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7920
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-25 07:49:44 +00:00
Florian Klink
1facd889bb feat(tvix/eval): use lexical-core to format float
Apparently our naive implementation of float formatting, which simply
used {:.5}, and trimmed trailing "0" strings not sufficient.

It wrongly trimmed numbers with zeroes but no decimal point, like
`10000` got trimmed to `1`.

Nix uses `std::to_string` on the double, which according to
https://en.cppreference.com/w/cpp/string/basic_string/to_string
is equivalent to `std::sprintf(buf, "%f", value)`.

https://en.cppreference.com/w/cpp/io/c/fprintf mentions this is treated
like this:

> Precision specifies the exact number of digits to appear after
> the decimal point character. The default precision is 6. In the
> alternative implementation decimal point character is written even if
> no digits follow it. For infinity and not-a-number conversion style
> see notes.

This doesn't seem to be the case though, and Nix uses scientific
notation in some cases.

There's a whole bunch of strategies to determine which is a more compact
notation, and which notation should be used for a given number.
https://github.com/rust-lang/rust/issues/24556 provides some pointers
into various rabbit holes for those interested.

This gist seems to be that currently a different formatting is not
exposed in rust directly, at least not for public consumption.

There is the
[lexical-core](https://github.com/Alexhuszagh/rust-lexical) crate
though, which provides a way to format floats with various strategies
and formats.

Change our implementation of `TotalDisplay` for the `Value::Float` case
to use that. We still need to do some post-processing, because Nix
always adds the sign in scientific notation (and there's no way to
configure lexical-core to do that), and lexical-core in some cases keeps
the trailing zeros.

Even with all that in place, there as a difference in `eval-okay-
fromjson.nix` (from tvix-tests), which I couldn't get to work. I updated
the fixture to a less problematic number.

With this, the testsuite passes again, and does for the upcoming CL
introducing builtins.fromTOML, and enabling the nix testsuite bits for
it, too.

Change-Id: Ie6fba5619e1d9fd7ce669a51594658b029057acc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7922
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-25 07:49:44 +00:00
Vincent Ambo
9ecf668452 chore(tvix/eval): delete "useless parenthesis" warning/optimisation
Two main reasons:

1. Traversing the structure to do this optimisation is
   actually *slower* than not optimising it.

2. There are literally hundreds of thousands of incidences of this in
   nixpkgs, and with some of the weird code there some of
   these (functionally) useless parens are actually required for
   readability reasons.

Change-Id: I1044b1c5f9fe20df4b6085851fc3b191277c65dc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7917
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-23 17:59:06 +00:00
Vincent Ambo
d2f1a290a4 fix(tvix/eval): force functors before applying them
call_value in the VM expects the callable to be forced when calling
it, which was not the case for functors.

Change-Id: Id55a2fe32a9573be42aef8669e268df519a989cd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7909
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-01-23 17:27:53 +00:00
Vincent Ambo
9f75973e4a refactor(tvix/derivation): relax bounds for reference iterator
This makes it easier to pass owned strings if that's what we have.

Change-Id: Ia7351ff2681292d16534ec50fe60b926b683bb9a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7908
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-23 15:08:32 +00:00
Florian Klink
b0d6e748ac refactor(tvix/derivation): remove DOT_FILE_EXT const
This is used in few enough places to just inline it. It felt a bit alien
in the ATerm construction aswell.

write.rs now pleasantly only includes tokens that occur in the ATerm
representation.

Change-Id: I524f8d6c1ce9057ff7fd16c6c3efd98467040a44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7911
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-23 13:53:45 +00:00
Florian Klink
fc177af0c1 refactor(tvix/derivation): pass fingerprint to build_store_path
All of these occurences are actually sha256 digests of some fingerprint
- which means there's not a lot of need to do the hashing outside.
Instead of passing in a digest, keep the sha256 hasher in the function,
and pass in a fingerprint.

This makes it much easier to construct fingerprints using format!() in
all consumers, because we need don't need to juggle with the hasher
anymore.

Change-Id: I2dc3af2cab6cf06f55ae6cbd9a8be95faf2a07b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7907
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-23 13:40:46 +00:00
Vincent Ambo
5d856ac2e9 feat(tvix/derivation): add path_with_references
This allows the calculation of a store path for a plain string that
potentially contains references. These paths are used for
`builtins.toFile` (and potentially other features of C++ Nix).

Change-Id: Ic507c7f264f362b5e6e628255869e5a4fbe4d788
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7906
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-23 12:55:24 +00:00
Vincent Ambo
be7f64ac87 refactor(tvix/derivation): minor structure simplification
Fixes some module comments and embeds the `compress_hash` function in
the `derivation` module, as it was not used outside of this module
anyways.

Change-Id: I6c5c92b3f0c03c2cdcbcfc2f813909a968c4d44c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7905
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-23 12:55:24 +00:00
Vincent Ambo
5719763fd3 feat(tvix/eval): support builtins implemented in Nix itself
This makes it possible to inject builtins into the builtin set that
are written in Nix code, and which at runtime are represented by a
thunk that will compile them the first time they are used.

Change-Id: Ia632367328f66fb2f26cb64ae464f8f3dc9c6d30
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7891
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-22 20:48:53 +00:00
Vincent Ambo
8513a58b37 docs(tvix/eval): update some outdated comments
These don't apply anymore since the "antidote-CL".

Change-Id: I40ee73ef43d44bbfc650a8fe6c2b33263dd06959
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7890
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-22 19:39:00 +00:00
Florian Klink
1a84e3e6d8 chore(tvix/store): add fastcdc crate
This is used for content-defined chunking.

Change-Id: I10345372cecb9a643cc51ca45aa5b77d2a05198a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7889
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-21 15:12:45 +00:00
Florian Klink
4614db2f83 docs(tvix): link to IRC channel and mailing list
I omitted the link to the commit history. Most people see this README
through some git interface anyways, so it felt a bit redundant.

Change-Id: I92cf6d08b83ef680fe37df29d3d546cad020955a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7888
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-01-21 15:08:38 +00:00
Florian Klink
758b94f47c docs(tvix): move most of //tvix/eval README up to //tvix
While moving the CLI out of the evaluator, we forgot to update the
README in //tvix/eval. Move this up to //tvix, so people know where
to start.

Keep the instructions on how to build only `//tvix/eval` in `//tvix/
eval/README.md`.

Change-Id: Ie2755e8b5a0056225dbf3a0ee040f70f7f6a1f27
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7887
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-21 15:08:38 +00:00
Florian Klink
90979d39f3 feat(tvix/store/directory): validate Directory and sizes
This calls out to Directory::validate() for all received Directory
messages, and also makes sure the sizes we refer a Directory message as
matches the sizes that have been calculated.

Change-Id: I316f9191d5872ee4ba6d78b9a4326f069b22fa63
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7882
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-21 10:48:17 +00:00
Florian Klink
d8e0fa8e5e feat(tvix/store/directory): deduplicate Directory messages
We can omit sending Directory messages to clients that have already been
sent in the same stream.

We can also omit storing a Directory message if we already have it -
they're content-addressed anyways.

Change-Id: Iba44565e07157a83a033177a2ffbdddced64ba5c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7881
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-21 10:48:17 +00:00
Florian Klink
e719da53be feat(tvix/store): add SledDirectoryService
Change-Id: I8e97a59da2c6623f4cca746efa741097a6dbe5b1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7871
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-21 10:48:17 +00:00
Adam Joseph
22b9e6ff09 refactor(tvix/eval): administer antidote for poison
The codebase contains a lot of complexity and odd roundabout
handling for shadowing globals.  I'm pretty sure none of this is
necessary, and all of it disappears if you simply make the globals
part of the ordinary identifier resolution chain, with their own
scope up above the root scope.  Then the ordinary shadowing routines
do the right thing, and no special cases or new terminology are
required.

This commit does that.

Note by tazjin: This commit was originally abandoned when Adam decided
not to take away reviewer bandwidth for this at the time (eval was
still in a much earlier stage). As we've recently done some
significant refactoring of globals initialisation this came up again,
and it seems we can easily cover the use-cases of the poison tracking
in other ways now, so I've rebased, updated and resurrected the CL.

Co-Authored-By: Vincent Ambo <tazjin@tvl.su>
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Ib3309a47a7b31fa5bf10466bade0d876b76ae462
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7089
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-21 10:19:15 +00:00
Florian Klink
ab8486e5b8 chore(tvix/store): add tonic-mock
Upstream seems to be dead, so we're using https://github.com/tyrchen/
tonic-mock/pull/3 here.

According to https://github.com/tyrchen/tonic-mock/pull/1#issuecomment-
1241164173, we might not need this crate at all, but for now, it gets
the job done and is less code to write in the tests.

Change-Id: Ia77fa19b998a5bbabd0311cc714b85a2ee30f36a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7869
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-21 09:34:15 +00:00
Florian Klink
a8b13a0b57 refactor(tvix/store): simplify test a bit
Import more things, and use expect_err to unpack the response.

Change-Id: Ia319dd4d126b8d0e1df585234710d825a33a0002
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7868
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-21 09:34:15 +00:00
Vincent Ambo
7442558b33 refactor(tvix/eval): keep globals alive through VM struct
This forces users to pass the fully constructed set of globals to the
VM, making it harder to accidentally "lose" the set while weak
references to it still exist.

This doesn't modify any functionality, but is laying the foundation
for simplifying some of the builtins behaviour that has grown more
complex again.

Change-Id: I5120f97861c65dc46d90b8a4e2c92ad32cc53e03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7877
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-20 22:48:13 +00:00
Vincent Ambo
6d03e31060 docs(tvix/eval): add doc comments to VM fields
Change-Id: Ia4857c217de15aec8b61e1abd39e22c50e2d816a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7876
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-01-20 22:48:13 +00:00
Vincent Ambo
259faea2b2 feat(tvix/cli): add errors module with drv construction errors
These will be threaded through to eval through the new `TvixError`
variant.

Change-Id: Ia0d3f8710dcf26bb95015cd2a6a2b2911f06343f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7842
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-20 15:39:51 +00:00
Vincent Ambo
7d0456fa0e feat(tvix/eval): let builtin macro capture external state
This adds a feature to the `#[builtins]` macro which lets users
specify an additional state type to (optionally) thread through to
builtins when constructing them.

This makes it possible for builtins-macro users to pass external state
handles (specifically, in our case, known path tracking) into a set of
builtins.

Change-Id: I3ade20d333fc3ba90a80822cdfa5f87a9cfada75
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7840
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-20 15:39:51 +00:00
Vincent Ambo
f12f938166 refactor(tvix/eval): directly return builtin tuples from macro
All invocations of the builtin macro had to previously filter through
the `builtin_tuple` function, but it's more sensible to directly
return these from the macro.

Change-Id: I45600ba84d56c9528d3e92570461c319eea595ce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7825
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-20 15:39:51 +00:00
Vincent Ambo
972c867b36 feat(tvix/eval): add error contexts to annotate error kinds
This makes it possible for users to add additional context to an
error, which will then be rendered as an additional secondary span in
the formatted error output.

We should strive to do this basically anywhere errors are raised that
can occur multiple times, *especially* during type casts. This was
triggered by me debugging a type cast error attached to a fairly
large-ish span (a builtin invocation).

Change-Id: I51be41fabee00cf04de973935daf34fe6424e76f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7849
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-20 15:39:51 +00:00
Florian Klink
9822fa387a feat(tvix/derivation): more checks for output hashes and encoding
Change-Id: Idebad60c3bf9daf94d04a36bb73ac0dd767f9e79
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7856
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-18 17:13:23 +00:00
Florian Klink
083e24bbb1 feat(tvix/derivation): add validate_output_paths flag for validate
This allows calling validate() on a derivation that doesn't have its
output paths already calculated yet. It allows offloading some of the
error checking in builtins.derivation* to be offloaded to that function.

Change-Id: Ib4aeadc0eb6583ef8cd765f33e9a9ec32be62729
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7848
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-18 17:13:23 +00:00
Florian Klink
b1e8fe7212 feat(tvix/derivation): check for other invalid output names
This uses the exposed StorePath::validate_name method to check for other
invalid output names (for which it would not be possible to construct a
store path of).

Change-Id: Ia3f65e19a07ef164f9f64013a5f37cbac99eb8e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7855
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-18 17:13:23 +00:00
Florian Klink
4a256dda09 docs(tvix/store): fix docstring for StorePath::to_absolute_path
Change-Id: I8b36c85fe11e4fb62e5d28f6900cd80d89a5cc41
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7854
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-18 17:13:23 +00:00
Florian Klink
0f7e4bec66 feat(tvix/store): rename and expose StorePath::validate_name
Change-Id: I8a16c214c7c644756d9d54187beba8c80ccfb39c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7853
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-18 17:13:23 +00:00
Florian Klink
6e7f06d942 refactor(tvix/derivation): use DerivationError in Output::validate
Change-Id: I7dbd3b8ff9ef92acddde2e579fb24b8311c34d8f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7852
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-18 17:13:23 +00:00
Florian Klink
eebb3ce028 refactor(tvix/derivation): rename {ValidateDerivation,}Error
This is now used in more than just validate().

Change-Id: I69c3ad6cb5f3ad60a636fe2ea05d432aebe8e53b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7851
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-18 17:13:23 +00:00
Florian Klink
0aad4e2246 feat(tvix/derivation): also fail if output name is called drv
`drv` is an invalid output name too, as this would cause a
`builtins.derivation` call to return an attrset with a `drvPath` key
(which already exists) and has a different meaning.

Also handle errors during store path construction, and return our own
error type, instead of the ParseStorePathError.

Change-Id: Ib7952dde1d5cf18a0e210928df7c57b5939b7678
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7850
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-01-18 17:13:23 +00:00
Vincent Ambo
0cbdfffea2 refactor(tvix/cli): consistently assert type unity in known_paths
No situation should be allowed in which a path is inserted into
known_paths with different types twice, which we previously enforced
only for some path types.

Change-Id: I8cb47d4b29c0aab3c58694f8b590e131deba7043
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7843
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-17 10:45:09 +00:00
Vincent Ambo
bf6f6a0b3f refactor(tvix/eval): non-hacky suspended native thunks
Instead of having a representation of suspended native thunks that
involves constructing a fake code chunk, make these thunks a
first-class part of the internal thunk representation.

The previous code was not that simple to understand, and actually
contained a critical bug which could lead to Tvix crashes. This
version fixes the particular instance of that bug, but instead
uncovers another (b/238) which can still lead to Tvix crashes.

Fixes: b/237.
Change-Id: I771d03864084d63953bdbb518fec94487481f839
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7750
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-17 10:38:06 +00:00
Vincent Ambo
559a09c5e6 refactor(tvix/eval): remove Box in new_suspended_native
This is unnecessary, Rc already provides all the boxing we need.

Change-Id: I08cf0939c48da43f04c847526c7e5dae5336d528
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7749
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-01-17 10:38:06 +00:00
Vincent Ambo
b9646ab40c feat(tvix/eval): add builtins to builtins
This is a somewhat terrifying hack that enables us to support
`builtins.builtins`, by running a "fake compilation" inside of a
suspended native thunk that can resolve the weak pointer to the
globals.

With this implementation, the thunk at `builtins.builtins` actually
resolves to the "real" `builtins` (verified with a new test).

This is kind of ugly, and it's something users shouldn't use, but
bubbling a warning out of this is difficult at the moment due to a
little bit of trickery with how the spans in suspended native thunks
work (they don't) (see b/237, b/238)

Change-Id: I67d0e93246dd5b279c960aeda00402031aa12af3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7748
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-17 10:38:06 +00:00
Vincent Ambo
79e54f46ce feat(tvix/cli): add replacement strings tracking to KnownPaths
Replacement strings are some weird internal feature of Nix that is
required for calculating derivation hashes. We need to track these
like other paths, as they need to be re-used on builds with
dependencies on values from previous builds.

Change-Id: Ie955b3fb5ae3685cfadfbe4d06ea6b5e219590c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7828
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-17 10:31:42 +00:00
Vincent Ambo
499e72c1cb feat(tvix/cli): track known plain paths in NixCompatIO
When adding things to a C++ Nix store, ensure that the path is tracked
in the tracker.

Since the mechanism for propagating the tracker instance isn't
finalised yet, I've opted to take an Rc<RefCell> parameter for it. How
exactly that ends up there is going to become clear in the next
commits, but for now it's just instantiated in main with
Default::default.

Change-Id: I90f0b44f2d4f292dedc98ff1aa39041d279b61fd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7833
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-17 10:31:42 +00:00
Vincent Ambo
9cb3daee20 refactor(tvix/cli): reference scanner owns all the strings
This gets very complex very quickly otherwise, as all the construction
paths for a reference scanner and all the access patterns for the
KnownPaths structure are not yet fully understood.

Change-Id: Ibadf1f18b476695f3c286fc6896ae557760edf63
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7827
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-17 10:31:42 +00:00
Vincent Ambo
940251b87f refactor(tvix/value): use proptest strategies from imbl crate
Instead of going through Vec/BTreeMap for generating our internal
types, use the proptest strategies from imbl.

The one thing I couldn't figure out in the previous implementation is
where the ranges/sizes of generated collections came from. The
strategies in proptest use different types (Range, with an unknown
default value, and SizeRange with 0..100). I've opted to specify
0..100 directly, but we can probably make it configurable.

Change-Id: I749bc4c703fe424099240cab822b1642e5216361
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7791
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-17 10:20:38 +00:00
Vincent Ambo
f27f5ef0c9 feat(tvix/cli): add known_paths module
This module implements types used to track the set of known paths in
the context of an evaluation.

These are used to determine the build references of a derivation.

Change-Id: I81e15ae33632784e699128916485751613b231a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7816
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-17 10:20:38 +00:00
Florian Klink
d644ed389a refactor(tvix/derivation): expose proper ValidateDerivationError
Use proper errors, instead of anyhow.

Change-Id: I6db14c72a6319b389b0136aac7b84f50a30fb366
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7847
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-16 23:04:50 +00:00
Florian Klink
95cad95b93 feat(tvix/derivation): make input_sources a BTreeSet
These need to be sorted anyways, so let's use the correct data structure for it.

Change-Id: I009c9989d7647dc1df716170f3680c981db6e4b2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7846
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-16 16:45:56 +00:00
Florian Klink
f6b4abac35 chore(tvix/derivation): drop TODOs related to stripping outputs
Different to the go-nix implementation, we don't have a custom "writing
mode" where output paths are omitted from output, but we simply run the
serialization with these fields unpopulated (during construction).

This means, there's no need for a custom writing mode that strips
outputs, so we can drop the comments.

Change-Id: Ic0aafd4e34e0294603490cfce2b1aef4085ff34b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7845
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-01-16 16:45:56 +00:00
Vincent Ambo
67addf3b3d feat(tvix/eval): add error variant for threading through errors
This variant is required for external builtins (which in our case
includes `derivation`) to thread through reasonable error messages.

This has some potential for improvement, but it's an improvement over
the status quo of panicking in the external builtins when no
appropriate error is available.

Change-Id: I7e4bdb0a156c7717092dde30aa4785192182dc66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7841
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-16 13:43:14 +00:00
Vincent Ambo
d365b09226 feat(tvix/eval): implement builtins.toXML
Change-Id: I009efc53a8e98f0650ae660c4decd8216e8a06e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7835
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-16 13:43:14 +00:00
Vincent Ambo
1786b4c835 chore(tvix/eval): add other required items to public API
External implementors of builtins must be able to force values, which
necessitates publishing a bunch more items from the crate.

Change-Id: I8f6b8ae88156aae417dbe630a698d123d0c1c8d4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7830
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-16 13:43:14 +00:00
sterni
ca0bd84589 docs(tvix): add reconstructed changelog for the Nix language
This was put together by digging through the C++ Nix git history as
described in the second paragraph. Note that this is only based on
changes given when the language version was actually increased.

There is no guarantee that there have been no other changes inbetween
that would have warranted an increase as well.

Change-Id: I4ddee0d4ecafa1b3e5e1a867e9700d6c32e936ad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7836
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-16 09:32:49 +00:00
Vincent Ambo
43d83274e6 chore(tvix/store): derive Clone for StorePath
Change-Id: Id325afa766b0bd2a3d04cf538f13e77692a22687
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7829
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-14 09:48:37 +00:00
Vincent Ambo
a35dadf9f0 refactor(tvix/derivation): use BTreeSet for derivation outputs
When constructing derivations inside builtins.derivationStrict, we'd
have to very frequently check whether certain outputs have already
been inserted into the derivation inputs.

Using a set type is much easier, especially as this has to be ordered
and the source data that is being inserted also comes from a set,
which might let us pass this more efficiently in the future.

Note that the validate function no longer checks the order of the
entries, as that is now guaranteed by the type.

Change-Id: I2fbb984facba3e668075f6f8df8992092368c63d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7826
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-13 18:01:09 +00:00
Aaqa Ishtyaq
158f4d1d69 fix(tvix/eval): len_without_is_empty clippy warn
This CL addresses clippy warning len_without_is_empty
which expects `.is_empty()` method to be present when
implementing `.len()` method for an item.

Change-Id: I8878db630b9ef5853649a906b764a33299bb5dc8
Signed-off-by: Aaqa Ishtyaq <aaqaishtyaq@gmail.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7806
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-12 18:06:49 +00:00
Vincent Ambo
09654ffb80 docs(tvix): add build-references / string-context document
This explains my current thinking on string contexts. Thanks to
everyone who gave input so far.

Change-Id: I773219402a79a9d4753b4e7cfbf3a4a751a993a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7807
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-12 11:01:40 +00:00
Vincent Ambo
02d35e4aa6 feat(tvix/eval): implement builtins.toJSON
Implements `Serialize` for `tvix_eval::Value`. Special care is taken
with serialisation of attribute sets, and forcing of thunks.

The tests should cover both cases well.

Change-Id: I9bb135bacf6f87bc6bd0bd88cef0a42308e6c335
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7803
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-01-12 10:42:44 +00:00
Florian Klink
43f6aec384 feat(tvix/store): implement PathInfoService with sled
This uses [sled](https://github.com/spacejam/sled) to store PathInfo
objects.

Change-Id: I12e8032e5562af8f884efa23a78049fd1108fdbc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7726
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-12 10:11:57 +00:00
Florian Klink
cfa42fd19a chore(tvix/store): update ParseStorePathError messages
Change-Id: I49799393a1e35b4475566819111beb57a628c555
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7801
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-12 10:11:57 +00:00
Florian Klink
ed3b28aa02 chore(tvix/store): update ValidatePathInfoError messages
Change-Id: I8f16e25d998a74ac110ae99eec0edbbfd720c8dd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7800
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-12 10:11:57 +00:00
Vincent Ambo
e63bff5545 feat(tvix/refscan): implement reference scanning over data streams
Using yet more machinery from the pretty comprehensive aho_corasick
crate, this makes it possible to pass anything implementing `io::Read`
to the `ReferenceScanner` to accumulate matches.

Change-Id: I5b0e28eb44ea4df24010f40831e29f2cbb8c1f80
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7810
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-11 16:03:51 +00:00
Vincent Ambo
3045645df0 feat(tvix/cli): implement initial refscan module
This module implements a ReferenceScanner struct which uses the
aho_corasick crate to scan string inputs for known, non-overlapping
candidates (store paths, in our case).

I experimented with several different APIs, and landed on this version
with an initial accumulator in the scanner. The scanner is
instantiated from the candidates and "fed" all the strings, then
consumed by the caller to retrieve the result.

Right now only things that look vaguely like bytestrings can be fed to
the scanner, there is no streaming support in the API yet.

Change-Id: I7782f0f0df5fc64bccd813aa14712f5525b0168c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7808
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-11 16:03:51 +00:00
Aaqa Ishtyaq
9382afdb0d fix(tvix/eval): address useless_format clippy warn
This CL address clippy warnings related to use of 'format!' macro
to return unmodified 'String'.

Change-Id: I88726e59d8f39f6a455a8c1f48075b52d167e489
Signed-off-by: Aaqa Ishtyaq <aaqaishtyaq@gmail.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7804
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-10 19:13:45 +00:00
Ryan Lahfa
805219a2fa feat(tvix/eval): implement serde::Deserialize for Value
Co-Authored-By: Vincent Ambo <tazjin@tvl.su>

Change-Id: Ib6f7d1f4f4faac36b44f5f75cccc57bf912cf606
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7626
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-10 09:53:21 +00:00
Vincent Ambo
c011a6130c refactor(tvix/eval): impl Display for ErrorKind
This just shuffles the Display implementations around so that
ErrorKind itself is displayable, which is useful in some situations
where errors under construction need to be type-converted.

Change-Id: I7b633d03d0dc34f345c4f20676e0023ecb1db0c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7802
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-01-10 09:52:52 +00:00
Vincent Ambo
681800b438 chore(tvix/derivation): expose necessary types in public API
Change-Id: Ia05b8eb3054a39897d3ec5783daecfe4bcf054e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7779
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-01-09 19:48:20 +00:00
Vincent Ambo
1d59d3ba8f fix(tvix/eval): fix last uses of Vec<Value> -> NixList in builtins
Change-Id: I0d71b82eb7ddc1e457b0996b0668006f55f56751
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7790
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-01-08 11:11:52 +00:00
Vincent Ambo
a149a1ee06 fix(tvix/eval): fix typo'd function name in tests
Caught by sterni on cl/7783.

Change-Id: I15d57b893ef22538fdd7e809f3b92861dd2bc1af
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7789
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-07 14:48:42 +00:00
Florian Klink
1e2c859840 feat(tvix/derivation): derive Default for Derivation
Some of the fields in a Derivation struct stay empty, and manually
creating BTreeMap or vec for it is annoying.

Derive Default instead, so we can use the defaults instead of writing
more by hand.

Change-Id: I5d41b4b55c8187cb101eb4266451a470008e0067
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7788
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-07 08:05:41 +00:00
Vincent Ambo
e0c47f32af refactor(tvix/eval): use builtins macro for placeholders
Change-Id: I30bc475e3e36a163fa169083481cdd4b4d0ca456
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7785
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-01-06 20:53:02 +00:00
Vincent Ambo
6b6bd30772 refactor(tvix/eval): move mocked builtins.derivation to tests
This placeholder should not live in the main crate anymore as we will
be injecting the real one from outside of eval, but there are still
language tests that depend on a (simple, mockable) version of it.

Change-Id: I68ea169db15cbdbeed320930d3069e21e376c90d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7783
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-06 20:53:02 +00:00
Florian Klink
396fdde0c0 refactor(tvix/derivation): use StorePath::to_absolute_path()
Instead of concatenating STORE_DIR manually, use
StorePath::to_absolute_path() that does it for us.

Change-Id: Ia288851a05b4e339679db268f3dd7924e7b65586
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7782
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-06 17:59:35 +00:00
Vincent Ambo
1ebda9e13e fix(tvix/derivation): fix build after StorePath rename
This project was not previously covered by CI (fixed in this commit),
so we didn't catch breakage due to a renamed module.

This was noticed while rebasing a CL that has a dependency on this
crate in its Nix build.

Change-Id: Ic48570b9313e5f73e14daab50cf7ea70918c94d1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7778
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-06 17:57:06 +00:00
Vincent Ambo
3e03e59893 refactor(tvix/store): move protobuf build config one level up
This embeds the build config directly at the point where `Cargo.nix`
is imported, making it transparent to library consumers.

Change-Id: I5586e12f02ed14587c32d9ef7d93f079366fb127
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7780
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-06 17:57:06 +00:00
Florian Klink
144d010515 feat(tvix/store): document StorePath a bit more.
Change-Id: Ifab28d97ddc22a2073c5df5e6e2cefb51b4b9191
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7777
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-06 15:47:46 +00:00
Florian Klink
c89af03a03 refactor(tvix/store): rename NixPath to StorePath
As discussed in #tvl, this is a more common term for it.

Change-Id: I9b904222b8c076f82192c9b7f0b42be171614ab7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7776
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-06 15:39:35 +00:00
Florian Klink
999afd4be2 test(tvix/derivation): add output_path_construction test
This exercises the output path calculation functions like a constructing
client (an implementation of builtins.derivation) would do.
It first assembles the bar derivation, does the output path calculation
on it, then continues with the foo derivation.

The code ensures the resulting Derivations match our fixtures.

Change-Id: If93f89c6622fac1c1941085083931b6f657c04bc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7775
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-06 15:39:35 +00:00
Florian Klink
f73ab76fce feat(tvix/store): add NixPath::to_absolute_path
This provides a function returning a string starting with the store
path prefix, the counterpart of `from_absolute_path`.

Change-Id: I4947f3b00171fe70357c62b3d64dc769b69e7a44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7774
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-06 15:39:35 +00:00
Florian Klink
95bec264d5 feat(tvix/derivation): implement output path calculation
This implement output path calculation for fixed outputs, both fixed-
output and non-fixed-output.

Change-Id: I0a77b99f2ba6b39467cc5dd589ce152a40387f9a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7761
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: jrhahn <mail.jhahn@gmail.com>
2023-01-06 12:25:02 +00:00
Florian Klink
9df9a2f1ab feat(tvix/derivation): add get_fixed_output() helper function
This will return the fixed output of a derivation (and its hash), or
None if the Derivation is not fixed-output.

It will simplify the logic in the output path calculation a bit.

Change-Id: I1066cc18ee4fc419421a8c5995c93ba91b35588f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7760
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-06 12:25:02 +00:00
Florian Klink
d4603fc0af refactor(derivation): return NixPath in calculate_derivation_path
This moves all the hash compression logic into a common helper function.

Also update the docstring, which said "path" here, which could have been
confused with output paths.

Change-Id: Iedfb59aeb24f7638afac669dcd18d57b6cfaece2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7759
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-06 12:25:02 +00:00
Vincent Ambo
87c80895cd feat(tvix/eval): skip & warn for useless parenthesis
Change-Id: I567ca0682012b9d09f1217e57a104ac5671f8d82
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7771
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: flokli <flokli@flokli.de>
2023-01-06 12:23:55 +00:00
Vincent Ambo
aadf71a6ed feat(tvix/eval): warn on empty let-bindings
Change-Id: Ib6ef7ce514abbd3e372dfe9df7137aa36dbda9d4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7770
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-06 12:23:55 +00:00
Vincent Ambo
61b8a9b2ba refactor(tvix/eval): short-circuit on empty attrs in compiler
This is marginally more efficient and has simpler bytecode.

Change-Id: Iad37c9aeef24583e8f696911bcd83d43639f2e36
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7769
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-06 12:23:55 +00:00
Vincent Ambo
4e98730f38 feat(tvix/eval): warn about empty inherits
Change-Id: I82bec6fe2210bcb88c46fd2fdf3e26bd613d1c1f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7768
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-01-06 12:23:55 +00:00
Vincent Ambo
e8dcdceb34 fix(tvix/eval): compile but don't emit dead code
This adds a mechanism to the compiler to compile an expression without
emitting any code. This allows for detected dead code to still be
compiled to detect errors & warnings inside of it.

Change-Id: Ie78479173570e9c819d8f32ae683ce34234a4c5d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7767
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-06 12:23:55 +00:00
Vincent Ambo
6a8541e35a feat(tvix/eval): implement initial compiler::optimiser module
This optimiser can rewrite some expressions into more efficient forms,
and warn users about those cases.

As a proof-of-concept, only some simple boolean comparisons are
supported for now.

Change-Id: I7df561118cfbad281fc99523e859bc66e7a1adcb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7766
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-06 12:23:55 +00:00
Vincent Ambo
36e5a4cc07 refactor(tvix/eval): take owned ast::Expr in Compiler::compile
This adds a very minimal amount of additional Rc-increments (~1 per
compilation), but makes it a lot easier to add an AST-optimising
compiler pass without incurring a lot of extra cost.

Change-Id: I57208bdfc8882e3ae21c5850e14aa380d3ccea36
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7765
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-06 12:23:55 +00:00
Vincent Ambo
0e421a16c5 feat(tvix/cli): add --compile-only "linter" functionality
With this, tvix/cli can be run on files and only produce compiler
errors and warnings.

Change-Id: I5dd9229fc065647787daafd17d7c1540579a1d98
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7764
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-06 12:23:55 +00:00
Vincent Ambo
5926a05f46 feat(tvix/eval): add Evaluation::compile_only method
This would make it possible to implement something like a linter based
on the tvix-eval compiler warnings.

Change-Id: I1feb4e7c4a44be7d1204b0a962ab522fd32b93c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7763
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-06 12:23:55 +00:00
Vincent Ambo
27fecee1bc fix(tvix/eval): don't increase with_stack_size in scope inherits
There was probably a misunderstanding somewhere about the
with_stack_size being related to how far away it is from the with, but
it is about whether there is a with at all.

This broke a warning (`UselessInherit`), and may actually have let to
more inefficient codegen in some cases.

Change-Id: I08338ea59ae39dad01ca8a4e09d934a936cdea2f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7762
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-06 12:23:55 +00:00
Vincent Ambo
88432235ae fix(tvix/eval): VM & Builtin* types have to be public
... without them, using the new Builtins API is basically impossible
for library consumers.

Change-Id: Ice0557a2e55e12d812f51bf5a99e6b8c91ad1b91
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7755
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-06 12:15:23 +00:00
sterni
5946b47ab7 feat(tvix/eval): add builtins.{null,true,false}
Code probably rarely relies on these, but it's not hard to support them.

Change-Id: I8499fec34efaf031f9c013bbd370a13db929a2a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7772
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
2023-01-06 12:14:42 +00:00
sterni
b7090ec874 test(tvix/eval): add test for builtins parity
This will eventually force us to have a base builtins set in common with
C++ Nix, i.e. all 2.3 builtins except the controversial
builtins.valueSize.

Change-Id: I2c767f07d6a14711911658e87da9f18ede57a143
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7747
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-06 12:00:38 +00:00
Florian Klink
95c9c2ae8b feat(tvix/derivation): derive Clone, Debug, Eq and PartialEq
This allows juggling with Derivation structs in unit tests, and makes it
very easy to compare them for equality.

Change-Id: I1faf2ec1eefb1a40fcee3c29f04ec47d29f22691
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7758
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-04 21:58:02 +00:00
Florian Klink
cc626d686c feat(tvix/derivation): implement Derivation::validate()
Change-Id: I87dfadda872439e108e5f678a5da63dd5b1915d1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7732
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-04 21:58:02 +00:00
Florian Klink
407a9cd90f chore(tvix/derivation): align json serialization with nix output
Use the #[serde(rename = "…")] field attributes to match the field names
that Nix uses in its JSON output (nix show-derivation).

This allows us to just re-use the exact same fixtures from go-nix,
without manual post-massaging.

Change-Id: Ifd5c08e43cd4f50d5e02903eccd8cb37230b70a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7757
Reviewed-by: jrhahn <mail.jhahn@gmail.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-04 21:58:02 +00:00
Florian Klink
77cc6a1f78 refactor(tvix/derivation): make output hashes an Option<Hash>
This conveys better if an output of a Derivation is fixed-output or not,
and provides a Hash struct that can be used to store the algo and
digest.

In case it's not, this can simply be None. The serde field attributes
have been updated to still accept the same JSON.

We currently still store the hash algo and digest as strings, mostly
because the only thing populating it so far is the example JSONs.
We might want to update this, once actual Nix code populates this.

While updating write.rs, I pushed some of the Vec<String> to [&str]
conversions inline, and made it a Vec<&str>, because it was annoying to
juggle with.

Change-Id: Ia9cd0568fe179ac22a4a636237f22ab4ad92b95b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7746
Tested-by: BuildkiteCI
Reviewed-by: jrhahn <mail.jhahn@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-04 20:23:29 +00:00
Florian Klink
468dc5cd0c feat(tvix/derivation): make struct members public
Allow others to peek into values. We should probably still restrict
this further at a later point.

Change-Id: I2831432038aa87c3c7dcc85af4fa76a4fe0eadff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7745
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-04 20:23:29 +00:00
Florian Klink
f00e7fa92d refactor(tvix/derivation): use tvix_store::nixpath::STORE_DIR
There's now a common constant for this, so we don't need to redefine
it here.

Change-Id: I48b096f5e623e5fc5bbd355db83b674766935f94
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7753
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-04 20:23:29 +00:00
Florian Klink
8449f6cd0c feat(tvix/store): implement Nixpath::from_absolute_path
This allows constructing a NixPath from an absolute path. It pops off
the STORE_DIR prefix and the trailing slash and returns an error if it
couldn't be found.

Change-Id: Ib540e353c63cc247ac15e20414b0db2caf695ef4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7751
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-04 20:23:29 +00:00
Vincent Ambo
34be6466d4 feat(tvix/serde): implement enum deserialisation
Implements externally tagged enum deserialisation. Other serialisation
methods are handled by serde internally using the existing methods.

See the tests for examples.

Change-Id: Ic4a9da3b5a32ddbb5918b1512e70c3ac5ce64f04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7721
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
2023-01-04 17:21:40 +00:00
Vincent Ambo
0e88eb83ef feat(tvix/serde): add newtype & tuple deserialisation
Only missing enums at this point, but they're a bit of a beast.

Change-Id: I4ad47c034851f9a8794c81f39a5149a8ac1826e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7716
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
2023-01-04 17:21:40 +00:00
Vincent Ambo
0c17718dd1 refactor(tvix/eval): avoid unnecessary pop/push in OpForce
Change-Id: Ic7559eaa43aa0dcc97babb7669770c0f7f959f1b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7754
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-01-04 16:30:03 +00:00
sterni
9b8ba915c8 fix(tvix/eval): ' is allowed in nonfirst position in Nix identifiers
With this is_valid_nix_identifier should line up with the upstream lexer
definition:

    ID          [a-zA-Z\_][a-zA-Z0-9\_\'\-]*

While we're working on this, add a simple test checking the various
formatting rules. Interestingly, it would not be suitable as an identity
test, since you have to write

    { "assert" = null; }

in order to avoid an evaluation error, but C++ Nix is happy to print
this as

    { assert = null; }

– maybe should be considered to be a bug.

Change-Id: I0a4e1ccb5033a80f3767fb8d1c4bba08d303c5d8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7744
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-04 12:36:28 +00:00
Vincent Ambo
3d238c350b refactor(tvix/eval): streamline construction of globals/builtins
Previously the construction of globals (a compiler-only concept) and
builtins (a (now) user-facing API) was intermingled between multiple
different modules, and kind of difficult to understand.

The complexity of this had grown in large part due to the
implementation of `builtins.import`, which required the notorious
"knot-tying" trick using Rc::new_cyclic (see cl/7097) for constructing
the set of globals.

As part of the new `Evaluation` API users should have the ability to
bring their own builtins, and control explicitly whether or not impure
builtins are available (regardless of whether they're compiled in or
not).

To streamline the construction and allow the new API features to work,
this commit restructures things by making these changes:

1. The `tvix_eval::builtins` module is now only responsible for
   exporting sets of builtins. It no longer has any knowledge of
   whether or not certain sets (e.g. only pure, or pure+impure) are
   enabled, and it has no control over which builtins are globally
   available (this is now handled in the compiler).

2. The compiler module is now responsible for both constructing the
   final attribute set of builtins from the set of builtins supplied
   by a user, as well as for populating its globals (that is
   identifiers which are available at the top-level scope).

3. The `Evaluation` API now carries a `builtins` field which is
   populated with the pure builtins by default, and can be extended by
   users.

4. The `import` feature has been moved into the compiler, as a
   special case. In general, builtins no longer have the ability to
   reference the "fix point" of the globals set.

This should not change any functionality, and in fact preserves minor
differences between Tvix/Nix that we already had (such as
`builtins.builtins` not existing).

Change-Id: Icdf5dd50eb81eb9260d89269d6e08b1e67811a2c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7738
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-04 12:28:43 +00:00
Jürgen Hahn
6f993b8bde feat(tvix/derivation): add nix drv path generation to Derivation
This adds a function to generate the derivation path. The computation
is based on the Go implementation.

Change-Id: Iae89db4976f5fd9208f0453f73688689a245cd66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7729
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-04 12:24:00 +00:00
Jürgen Hahn
79c05f3810 docs(feat/derivation): add docstring to formatter
This is just to clarify that the formatted result is the ATerm representation

Change-Id: I98fd0b1d2daa3cf6fdbe526ae1e0bd100ff62df3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7742
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-04 12:10:43 +00:00
Jürgen Hahn
abd539ddb8 feat(tvix/derivation) Add fmt::Display implementation for Derivation
This adds the implementation of fmt::Display for Derivation so that we can
easily store the formatted content as a string. Internally, we use the
serialization function to generate the string.

Change-Id: I6caca0d6c1bea3ca44b6c535c5b1d5d66d8413b7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7741
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-04 11:38:36 +00:00
Vincent Ambo
00dab6142e test(tvix/serde): add initial set of deserialisation tests
Change-Id: I0a9779edf0296c25d37fb5f75f8fc5852fe00121
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7715
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
2023-01-03 13:37:12 +00:00
Vincent Ambo
4350be34d1 feat(tvix/serde): handle nested data structures
Change-Id: I543fc05d31bbb9ad2edb887bce4510e9a1cdb102
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7714
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
2023-01-03 13:37:12 +00:00
Florian Klink
8e2b1de3d5 feat(tvix/store/nixpath): DIGEST_SIZE public, use more consts
Rename PATH_HASH_SIZE to DIGEST_SIZE.

It's a digest, not hash (we don't necessarily have the internal hash
state anymore), and the fact it's about (Nix)Paths is already visible
from the module name.

Also expose ENCODED_DIGEST_SIZE, so we don't need to do the calculation
inside from_string() method, and it becomes more clear this is a
constant.

Change-Id: I0e7577dd7a6e503039037b986313b214e995d826
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7725
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: jrhahn <mail.jhahn@gmail.com>
Tested-by: BuildkiteCI
2023-01-03 13:03:07 +00:00
Florian Klink
2df642231c feat(tvix/store/nixpath): expose digest and name fields
These can be accessed directly.

Change-Id: I71dc84f982820d53f319efefbed9b9055034954d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7724
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-03 13:03:07 +00:00
Florian Klink
0b56d9f21b feat(src/proto): add PathInfo.validate()
This provides validation of PathInfo messages, and ensures the output
hashes are properly parsed from the root node names.

NixPath already has a more extensive test suite for various wrong
NixPaths, so it's omitted from here.

Change-Id: I5d69118df5816daabb521ddb19d178bddd1caacf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7684
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-03 13:03:07 +00:00
Florian Klink
ceb2c0ba89 chore(tvix/store): make importable
This allows other crates to import tvix_store.

Rename the bin crate to tvix-store-bin, to avoid having multiple crates
with the same name (https://github.com/rust-lang/cargo/issues/6313)

Change-Id: I857768d6115640dbf102e79ed03e8474090df2fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7728
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-03 13:03:07 +00:00
Florian Klink
d0bbc8c821 chore(tvix/store/nixbase32): address clippy
Change-Id: Ib47a55e39ed752492b9732439de5f327a7fa601e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7723
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-03 12:06:55 +00:00
Florian Klink
ecd943ea6a chore(tvix/store/main): address clippy
Change-Id: Ie400a7109d3aa6b66153c0ab25a38abf06984cbf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7722
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-03 12:06:54 +00:00
Vincent Ambo
90c32eec7a feat(tvix/serde): initial Nix->serde::Deserialize impl
This will make it possible fairly easily use Nix to represent
arbitrary data structures, e.g. for using Nix as a config language.

Only pure Nix (i.e. no `import` etc.) is supported for now.

Not all types, specifically no struct traversal, are implemented in
this commit.

Change-Id: I9ac91a229a0d12bf818e6e3249f3e5a691599a2c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7712
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-02 22:24:43 +00:00
Vincent Ambo
49ee3e3b14 chore(tvix/eval): implement std::error::Error for tvix_eval::Error
This makes it easier to interface this error with other crates.

Change-Id: I4947ea6097608f8c0427fb94a819ef748d94ea4b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7711
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-02 22:24:43 +00:00
Jürgen Hahn
31973890a9 refactor(tvix/derivation): refactor the derivation serialization
This refactors the code to serialize a derivation. The original code
has beed moved to seperate crates for better code structure.

Change-Id: I3b1a6b134428fcbc9930c330bced8ec3610cfb4c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7733
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-02 20:55:14 +00:00
Florian Klink
319c03f634 feat(tvix/store): add logging with tracing
This uses [tracing](https://github.com/tokio-rs/tracing) for logs/
tracing.

Annotate all method handlers with an instrument macro, and warn! a
message for them being unimplemented.

Co-Authored-By: Márton Boros <martonboros@gmail.com>
Change-Id: Id42a41db33782d82abfb8dc0e49a8915000e5d89
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7665
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-30 20:25:09 +00:00
Florian Klink
0bf2b0ef11 feat(tvix/store): implement reflection
This implements grpc.reflection.v1alpha.ServerReflection, and will make tools
like evans automatically discover available services, without having to
specify the path to the .proto files client-side.

It's behind a reflection feature flag, which is enabled by default.

Change-Id: Icbcb5eb05ceede5b9952e38a2ba72eaa6fa8a437
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7435
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-30 20:25:09 +00:00
edef
51243007f6 refactor(tvix/store): make nixbase32 reversal more idiomatic
Change-Id: Ibe550f9573b0f45ee95453b50c7510e49b07c719
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7685
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-12-30 16:40:17 +00:00
Jürgen Hahn
60bdf619a3 feat(tvix/store): refactor digest conversion
This refactors how the original digest type (Vec<u8>) is converted
to [u8; PATH_HASH_SIZE].

Change-Id: I9441470a3a199620fcf328f2b7c890ca6ae93bde
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7710
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2022-12-30 16:02:38 +00:00
Florian Klink
58f5ff2c17 feat(tvix/store): initial dummy implementation
This replaces the hello world example from tvix-store with an actual
gRPC endpoint, implementing all of BlobService, DirectoryService and
PathInfoService.

All RPC methods currently respond with the unimplemented gRPC status.

Co-Authored-By: Márton Boros <martonboros@gmail.com>
Change-Id: Ieba333cca44dc1e3f2ffbe676ba7a99e672b9bfb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7664
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-30 15:21:38 +00:00
Florian Klink
d22a9c8610 chore(tvix/store/protos): more idiomatic go
Pointed out by edef in
https://cl.tvl.fyi/c/depot/+/7648/comment/4551ba4b_e89ade36/#, thanks!

Change-Id: I6b0d317bb0210521622483cdf4cb557bc637a100
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7709
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-30 15:18:02 +00:00
Florian Klink
4036740ca7 feat(tvix/store): parameterise validate_digest error
Similar to cl/7682, we also want to make that error configurable.

Change-Id: I64f1a4570b3d75af4741abe10c2855959766e107
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7708
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-30 14:43:19 +00:00
Jürgen Hahn
5520bf3682 feat(tvix/store): add nixpath
This implements the NixPath structure. NixPath allow to parse a string to a nix path. If the parsing fails, a DecodeError will be raised.

Change-Id: I28363cdcfb27f04bf21a11c0d130b461667e3720
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7706
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-30 14:39:59 +00:00
Florian Klink
357c4d4836 feat(tvix/store): add nixbase32 mod
This implements the nix-specific base32 encoding and decoding, exposing
a subset of the API that the data-encoding crate provides.

Nix uses a custom alphabet, no padding, and encodes bytes in reverse
order. The latter one is the reason we can't just use the data-encoding
crate directly.

Three odd corner case tests ported over from go-nix failed. We opened
b/235 to further investigate.

Change-Id: I73fab6ddd67177d882e4c3f2b48761c95853d558
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7683
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2022-12-30 13:31:46 +00:00
Florian Klink
5ba47a2bc3 feat(tvix/store): parameterise validate_node_name error
We'll be using validate_node_name in other places in a bit, where
returning a ValidateDirectoryError is not appropriate.

Accept a function mapping a string to error as a second argument, and
pass ValidateDirectoryError::InvalidName at the current call sites.

Change-Id: I45cbb0deb4763061ad912c6b18a112c727795a17
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7682
Tested-by: BuildkiteCI
Reviewed-by: jrhahn <mail.jhahn@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2022-12-29 21:55:45 +00:00
Florian Klink
e97a429e73 fix(tvix/store): fix typo
Change-Id: I351a2bd4e007443aef1b97f50aecffc095ac60b1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7681
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: jrhahn <mail.jhahn@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-29 21:51:43 +00:00
Vincent Ambo
86361f0f4a refactor(tvix/eval): remove extra Rc<..> around Value::Attrs
The `im::OrdMap` is already small and cheap to copy while sharing
memory, so this is not required anymore.

Only the `KV` variant may have slightly larger content, but in
practice this doesn't seem to make a difference when comparing the two
variants and this one is less complicated.

Change-Id: I64a563b209a2444125653777551373cb2989ca7d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7677
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-12-29 17:44:56 +00:00
Vincent Ambo
91465dc78e refactor(tvix/eval): persistent, memory-sharing OrdMap for NixAttrs
This uses the `im::OrdMap` for `NixAttrs` to enable sharing of memory
between different iterations of a map.

This slightly speeds up eval, but not significantly. Future work might
include benchmarking whether using a `HashMap` and only ordering in
cases where order is actually required would help.

This switches to a fork of `im` that fixes some bugs with its OrdMap
implementation.

Change-Id: I2f6a5ff471b6d508c1e8a98b13f889f49c0d9537
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7676
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-12-29 17:44:56 +00:00
Vincent Ambo
610c44ec1e refactor(tvix/eval): use im::Vector directly where possible
The conversion from im::Vector -> Vec is cheaper for NixList
construction (of course), so where possible we should make use of
that.

This updates most builtins dealing with lists to use Vector directly,
and marks the function constructing NixList from Vec as deprecated so
that we get appropriate warnings in places where it's still in use.

These places are currently inside of JSON serialisation logic which is
in flux right now, so lets leave them as-is until it's stabilised.

Change-Id: I037f12a2800f2576db4d9526bd935efd079163f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7671
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-12-29 16:33:14 +00:00
Jürgen Hahn
bb185b2c6e feat(tvix/derivation): serialize Nix Derivation
This adds a Derivation structure and allows to write it to a structure that implements std::fmt:Write.
The implementation is based on the go-nix version.

Change-Id: Ib54e1202b5c67f5d206b21bc109a751e971064cf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7659
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-29 14:38:45 +00:00
Vincent Ambo
5d73c06b1a refactor(tvix/eval): use im::Vector for NixList representation
This is a persistent, structurally sharing data structure which is
more efficient in some of our use-cases. I have verified the
efficiency improvement using `hyperfine` repeatedly over expressions
on nixpkgs.

Lists are not the most performance-critical structure in Nix (that
would be attribute sets), but we can already see a small (~5-10%)
improvement.

Note that there are a handful of cases where we still go via `Vec`
that need to be fixed, most notable for `builtins.sort` which can not
currently be implemented directly using `im::Vector` because of a
restrictive type bound.

Change-Id: I237cc50cbd7629a046e5a5e4601fbb40355e551d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7670
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-12-29 12:27:59 +00:00
Florian Klink
6ab8320f07 chore(tvix/store): address clippy warnings
The only warnings left are unused warnings, but that'll change once we
have a real implementation, and not just tests.

Change-Id: I28912281b5e66735be37e999cc8ef4b8b09028fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7669
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-29 12:07:57 +00:00
Florian Klink
b0879917d4 chore(tvix/store): trim Cargo.toml
lazy_static is only used in tests, and anyhow isn't used at all (yet).

This can be dropped.

Change-Id: Ic41ff3f9bb93cfa600c3485e85464f78a3976504
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7668
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-29 12:07:57 +00:00
Florian Klink
ff71366f41 chore(tvix/store): move tests into separate file
Move them from the bottom of src/proto.rs to its own src/tests/mod.rs.

Also drop the test_ prefix, this is not golang.

Change-Id: I2e0b6b9812264f3d9721c0766936f08157fadc66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7667
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-29 12:07:57 +00:00
sterni
56555b211e docs(tvix/eval): sketch in place list/attr set update idea
Change-Id: Ic7debbd8cbd3acdf5f3947288f2aa2964bd163a0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7660
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-29 09:52:17 +00:00
Florian Klink
c4ee942b1c feat(tvix/store/protos): rename Get to Read, add Stat method
Stat exposes metadata about a given blob,
such as more granular chunking, baos.
It implicitly allows checking for existence too, as asking this for a
non-existing Blob will return a Status::not_found grpc error.

The previous version returned a Status::not_found error on the Get
request too, but there was no chance to prevent the server from starting
to stream (except sending an immediate cancellation).

Being able to check whether something exists in a BlobStore helps to
prevent from uploading in first place.

The granular chunking bits are an optional optimization - if the
BlobStore implements no more granular chunking, the Stat response can
simply contain a single chunk.

Read returns a stream of BlobChunk, which is just a stream of bytes -
not necessarily using the chunking that's returned in the reply of a
Stat() call. It can be used to read blobs or chunks.

Change-Id: I4b6030ef184ace5484c84ca273b49d710433731d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7652
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-28 13:55:18 +00:00
Florian Klink
1c15154b83 feat(tvix/store): make blobstore stream chunks
This changes the RPC methods to return/consume a stream of chunks, instead of a
very big message containing the whole blob, to keep message sizes in manageable
sizes (less than 4MiB).

Change-Id: I2a3a50f07b059d8a2f5196860254adff98c8a352
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7651
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-28 13:55:18 +00:00
Florian Klink
f879993cc4 feat(tvix): add evans to shell
Change-Id: I7ec8bae236330ec1e38e82da67b47aef2815d4b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7437
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-28 13:55:18 +00:00
Florian Klink
d973c9772d chore(tvix/store/protos): add PathInfoService::CalculateNAR()
Expose the NAR calculation to a separate `CalculateNAR` method, which
responds with the NAR size and sha256 hash.

Contrary to what cl/7618 and cl/7620 initially did, don't add different
other request types.

In the CalculateNARResponse message, there's now some duplication in the
(optional) `narinfo` field of a PathInfo, but I'm not entirely sure if
we want to drop the fields from there yet.

Change-Id: Id797c56e17efedac115fbd43de9dfde9fa1db140
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7663
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-28 13:55:18 +00:00
Florian Klink
ce7be00994 chore(tvix/store/protos): drop non-sha256 NAR hashes
Nix actually doesn't support anything else than sha256 NAR hashes, so
there's little reason to communicate anything else in here.

Co-Authored-By: edef <edef@unfathomable.blue>
Change-Id: I760370bd6cabd02028e001a74c454ef9296eb600
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7619
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-28 13:55:18 +00:00
Florian Klink
27bfc8cfcd chore(tvix/store/protos): move (Root)Node into separate message
This is gonna get used in another place in a second.

Co-Authored-By: edef <edef@unfathomable.blue>
Change-Id: I347c11c8d24379628b7ed09d2c90670c576e686a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7617
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-28 13:55:18 +00:00
Florian Klink
7163d3ad37 feat(tvix/store): implement Directory::validate()
Change-Id: I4c6ae79d705b8e19a3e2ed54812366e88935d7a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7650
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-28 10:58:50 +00:00
Florian Klink
0c7e545fd0 chore(tvix/store): use DUMMY_DIGEST, too
Missed from cl/7649.

Change-Id: Id2d382567f91cba9cdd792fe44bc0a83d040ec69
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7656
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-28 10:58:50 +00:00
Florian Klink
f930c91086 chore(tvix/store): use Directory::default in some more places
Change-Id: Iebad5e0b38f9506933fa855e1bfb2611d27ff572
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7655
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-28 10:58:50 +00:00
Florian Klink
35b18b2fdb chore(tvix/store): use lazy_static and dummy digests
While there's currently nothing in here checking the size of the digest,
we should use something that passes the to-be-introduced validate()
function.

Change-Id: I0c515d9e3afc79292dedebce659a32485aa3d936
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7649
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-27 21:31:33 +00:00
Florian Klink
dfd9286f68 feat(tvix/store/protos): implement Directory.Validate()
Validate thecks the Directory message for invalid data, such as:
- violations of name restrictions
- invalid digest lengths
- not properly sorted lists
- duplicate names in the three lists

Change-Id: I8d43a13797793c64097e526ef3bd482c9606c87b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7648
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-27 21:31:33 +00:00
Florian Klink
c3fb6d2218 feat(tvix/store/protos): implement Size() and Digest() for Directory
This adds Size() and Digest() functions for the golang version.

Change-Id: If71445a9bb26100bb4076ac4f5c96945b33919f9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7325
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-27 21:31:33 +00:00
Florian Klink
a91e6334ff feat(txix/store/protos): init go.mod, generate .pb.go files
This allows importing the generated .pb.go files into other go projects.

I initially looked at buildGo.protos, but it doesn't work for multi-.proto
files, and actually having LSP support for the generated structs is nice, too.

Change-Id: Idbd448008010790a10a0ea42e4059dbb609eaf1a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7322
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-26 11:38:52 +00:00
Florian Klink
2b9330911a feat(tvix/store/protos): add go_package option
Change-Id: I0898b8a0a78e704219da38e5acaabef1e640d4e4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7321
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
2022-12-26 11:38:52 +00:00
Florian Klink
e05e8bdd03 docs(tvix/store): add README, document services and store model
These are intended to help digest the protocol definitions for tvix-
store, and how they tie into the whole concept.

Co-Authored-By: Vincent Ambo <mail@tazj.in>
Change-Id: Ic1ba3ba41ef599209453f15d0ac2e07a6144bcca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7439
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-26 11:38:52 +00:00
Florian Klink
3676ce8d6d feat(tvix/store): use tonic-build for protos
While prost-build already exposes protobuf message types as structs, we
actually need tonic-build too, to be able to get traits for all the RPC
services defined in the proto files.

Change-Id: I7f4c08454bf0d280d577975c7cdae13ccc2d933b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7320
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-26 11:38:52 +00:00
Vincent Ambo
71174f6626 fix(tvix/eval): fix current clippy warnings
It's been a while since the last time, so quite a lot of stuff has
accumulated here.

Change-Id: I0762827c197b30a917ff470fd8ae8f220f6ba247
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7597
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-25 18:25:06 +00:00
Adam Joseph
67d508f2ec refactor(tvix/eval): non-recursive thunk forcing
Introduces continuation-passing-based trampolining of thunk forcing to
avoid recursing when forcing deeply nested expressions.

This is required for evaluating large expressions.

This change was extracted out of cl/7362.

Co-authored-by: Vincent Ambo <tazjin@tvl.su>
Co-authored-by: Griffin Smith <grfn@gws.fyi>
Change-Id: Ifc1747e712663684b2fff53095de62b8459a47f3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7551
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-25 18:17:46 +00:00
Ryan Lahfa
4cda236c0c feat(tvix/eval): implement From<f64> for Value
Change-Id: I287282a195d6f752260242739332b2357791974a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7625
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-12-25 15:47:40 +00:00
Vincent Ambo
712f1167ad feat(tvix/eval): display function names in documentation
... if they are known. We currently do not propagate names correctly
for curried functions.

Change-Id: I19d57fb30a5c0000ccdf690b91076f6b2191de23
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7596
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-12-22 19:09:40 +00:00
Vincent Ambo
e3f81e8d3b feat(tvix/cli): add :d REPL-prefix to print explanations of values
Change-Id: I1dd8de8d996e07840f9b0aaebf932b812103a43a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7593
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-22 19:09:40 +00:00
Vincent Ambo
e306d1d1a1 feat(tvix/eval): add Value::explain method
This value creates a human-readable explanation of a value. This can
be used to implement documentation related functionality.

For some values, the amount of information displayed can be expanded
quite a bit.

Change-Id: Ie8c400feae909e7680af163596f99060262e4241
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7592
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-22 19:09:40 +00:00
Vincent Ambo
ea7d63e177 feat(tvix/cli): implement NixCompatIO helper type
This type allows for temporarily compatibility with the C++ Nix store,
specifically (for now) it gives us the store directory used by Nix and
imports files the same way.

Change-Id: I4767794ef2863eba49661315c63c4e17de946d60
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7587
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-22 19:09:40 +00:00
Florian Klink
40a17acdc1 feat(tvix): add crate2nixGenerate target
`mg run //tvix:crate2nixGenerate --` runs crate2nix generate and
depotfmt's it afterwards.

This removes a frequent point of friction, because the Cargo.nix emitted
by crate2nix needs another formatter.

Change-Id: I649495980718cd3847d4cff77c9d4bfcb599387c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7612
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-12-22 15:26:41 +00:00
Florian Klink
2e04f68df8 feat(tvix/proto): add rpc_directory.proto
This provides an interface to retrieve and upload single Directory
messages, or a DAG of them.

Change-Id: Id9e7084bd697d843649a122da2c992a3e36d808c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7137
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2022-12-22 14:35:32 +00:00
Florian Klink
870cd533dd feat(tvix/proto): add rpc_pathinfo
This defines a service that can be used to upload and retrieve metadata
of nix paths.

Change-Id: Id86eb531ce4ae316adb15934b0d1386a14ba2132
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7136
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2022-12-22 14:35:32 +00:00
Vincent Ambo
22ed59fabd chore(tvix/eval): fix a broken comment
Change-Id: I0a6edb51685f94d4712089ae805170da3fb7faae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7609
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-21 23:29:26 +00:00
Vincent Ambo
908cebf35c fix(tvix/builtin-macros): parse multi-line docstrings correctly
Having a multi-line docstring yields multiple doc-attributes in order,
however we were previously discarding all but the first one.

This reduces them into a single string instead, which can then be
displayed as multi-line documentation.

Change-Id: I1f237956cdea2e4c746d3f13744e0373c1c645a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7594
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-21 23:05:22 +00:00
Vincent Ambo
270b1084e8 feat(tvix/eval): use EvalIO::import_path when coercing paths
This "ties the knot" of importing files into a store when referring
to them through path literals, e.g. inside of strings.

I'm not yet sure if this interface is sufficient for
builtins.path (which we haven't implemented at all yet), but it's
enough to wire up eval & store initially.

In the default implementations nothing interesting happens in this
function at all.

Change-Id: Ie01ff4161617d1e743a68dbd1a5e54c1b40c0990
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7582
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-21 22:59:18 +00:00
Vincent Ambo
edd13573f5 refactor(tvix/eval): use light spans in builtins.import
Change-Id: I05732073155b430575babb6f076bf465aef98857
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7581
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-12-21 22:59:18 +00:00
Adam Joseph
947a56c4b6 feat(tvix/eval): builtins.storeDir
Returns the store directory through EvalIO::store_dir.

Note that this is _optional_ in Tvix, as an evaluation can occur in a
context where there simply is no store directory. In those contexts,
`builtins.storeDir` returns `null` in Tvix.

This would only happen in contexts like Tvixbolt (or completely
unrelated use-cases) in practice.

Co-Authored-By: Vincent Ambo <tazjin@tvl.su>
Change-Id: I5a752c7e89b2f75bd7efb082dbfa5b25e3b1ff3b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7452
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-12-21 22:59:18 +00:00
Vincent Ambo
4cb9ada0df refactor(tvix/eval): use EvalIO::read_dir for equivalent builtin
Change-Id: I6d782c07166f51587d2f1d06607823268debb5d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7574
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-21 22:59:18 +00:00
Vincent Ambo
51deadd983 refactor(tvix/eval): use EvalIO::path_exists for the builtin
Change-Id: I49822ce30137777865e7370ee86666636e277b35
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7573
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-12-21 22:53:02 +00:00
Vincent Ambo
ec3e38c2d6 chore(tvix/eval): gate tvix_eval::StdIO behind the impure feature
This shouldn't be available if we've built a "pure" crate.

Change-Id: I7c85827ee212890252ff7e0b6242e2c52618cba5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7572
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-12-21 22:53:02 +00:00
Vincent Ambo
29790147b6 fix(tvix/cli): use tvix_eval::StdIO in CLI
... until we have a store-I/O layer, or something that intercepts the
store-related stuff appropriately.

Change-Id: I22f63435b3f9e118e3faeb2924fda8373a23ea7f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7568
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-21 22:37:11 +00:00
Vincent Ambo
0ef3c2fc2b refactor(tvix/eval): use EvalIO::read_to_string in impure builtins
With this change, the behaviour of reading a string from a file path
is controlled by the provided `EvalIO` structure.

This is a huge step towards abstracting away I/O behaviour correctly.

Change-Id: Ifde8e46cd863b16e0301dca45a434ad27560399f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7567
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-12-21 22:37:11 +00:00
Vincent Ambo
c3c4d752c9 feat(tvix/eval): add EvalIO to public crate API
This lets users set the `io_handle` field on an `Evaluation`, which is
then propagated to the VM.

Change-Id: I616d7140724fb2b4db47c2ebf95451d5303a487a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7566
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-21 22:37:11 +00:00
Vincent Ambo
25fc6b7c25 feat(tvix/eval): introduce initial EvalIO trait
This trait is going to be used to abstract filesystem interactions in
Tvix.

For now, it only contains a `read_to_string` method that closely
mirrors `std::fs::read_to_string`.

As a first step, to see how this works in practice, we will thread
through only this function to the various relevant parts.

Two implementations are provided in tvix-eval itself: A dummy
implementation (which just returns ErrorKind::NotImplemented for all
operations), and a std implementation which delegates to `std`
functions.

Change-Id: Ied3e3bf4bd0e874dd84e166190e3873a0f923ddb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7565
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-21 22:37:11 +00:00
Vincent Ambo
bf286a54bc refactor(tvix/eval): add a LightSpan type for lighter span tracking
This type carries the information required for calculating a
span (i.e. the chunk and offset), instead of the span itself. The span
is then only calculated in cases where it is required (when throwing
errors).

This reduces the eval time for
`builtins.length (builtins.attrNames (import <nixpkgs> {}))` by *one
third*!

The data structure in chunks that carries span information reduces
in-memory size by trading off the speed of retrieving span
information. This is because the span information is only actually
required when throwing errors (or emitting warnings).

However, somewhere along the way we grew a dependency on carrying span
information in thunks (for correctly reporting error chains). Hitting
the code paths for span retrieval was expensive, and carrying the
spans in a different way would still be less cache-efficient. This
change is the best tradeoff I could come up with.

Refs: b/229.
Change-Id: I27d4c4b5c5f9be90ac47f2db61941e123a78a77b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7558
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-21 22:21:25 +00:00
Vincent Ambo
8b3d03db92 chore(tvix/cli): rename binary to just 'tvix'
Change-Id: If735c81a04f60dcd77f1c40b595b52ed88bbbe95
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7580
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
2022-12-21 21:48:08 +00:00
Adam Joseph
da7c331d2c feat(tvix/eval): add thunks with suspended native Rust code
Having thunks which, when forced, execute native Rust code rather
than interpreted opcodes lets us avoid having to bundle
`src/libexpr/primops/derivation.nix` like cppnix does by implementing
it in Rust instead.

Change-Id: If91d77a6736234321eee87ba4b4777eed5a3fe1c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7450
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-21 21:48:08 +00:00
Vincent Ambo
87995ed355 refactor(tvix/eval): add name-based index over compiler's locals
Instead of finding locals by doing 2x O(n) walks over the compiler's
locals list, use a secondary name-based index for resolving locals by
name.

Previously, almost 60% (!!) of eval time on some expressions over
nixpkgs was spent in `Local::has_name`. This function doesn't even
exist anymore now, and eval speed about doubles as a result.

Note that this doesn't exactly make the locals code easier to read,
but I'm also not sure what we can simplify in there in general.

This fixes b/227.

Change-Id: I29ce5eb9452b02d3b358c673e1f5cf8082e2fef9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7560
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-21 21:48:08 +00:00
Adam Joseph
922bf7aca9 feat(tvix/eval): remove derive(Copy) from Upvalues
Change-Id: I0fa069fbeff6718a765ece948c2c1bce285496f7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7449
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-21 14:50:04 +00:00
Adam Joseph
e04b1697e4 feat(tvix/eval): wrap Closure in Rc<> to match cppnix semantics
Change-Id: I595087eff943d38a9fc78a83d37e207bb2ab79bc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7443
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-21 14:50:04 +00:00
Vincent Ambo
b3c34c3c61 docs(tvix/nar): document how to use NAR writer
This is done in the form of some comments on the functions, as well as
a functional doctest that writes a single file to a NAR.

Change-Id: Ic97ebd439e91d6b076685807fe70de098ec02575
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7599
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2022-12-21 14:31:54 +00:00
Vincent Ambo
a95dea719f chore(tvix/nar): add CI build target
Change-Id: Ic73a391da9a733cade26114ab1127907c8d62a57
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7598
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-21 14:08:30 +00:00
Vincent Ambo
f467df06dc chore(tvix/cli): re-add observer flags
Users can again pass flags for dumping the AST, bytecode, and runtime
trace.

With this commit the CLI is at feature-parity with what we had before,
but entirely through the new API.

Change-Id: I30fe26f243224b25d1e4f828fec607325ef88306
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7550
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-12-21 13:23:38 +00:00
Vincent Ambo
60d9fba56d chore(tvix/cli): re-add NIX_PATH handling
Change-Id: I5595d6f5141ed4a533ca44a46264dc604fca6be1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7549
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-12-21 13:23:38 +00:00
Vincent Ambo
36eaa9b3b3 chore(tvix/cli): re-add --raw argument
Change-Id: I82f469f7f186c754c40bd941587b30aa3fc48045
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7548
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-21 13:23:38 +00:00
Vincent Ambo
6d3d237923 refactor(tvix/eval): consume self in Evaluation::evaluate
This simplifies lifetime management for observers in callers of tvix_eval.

Change-Id: I2f47c8d89f22b1c766526e5d1483c0d026b500ae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7546
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-21 13:23:38 +00:00
Vincent Ambo
59e695a9d9 feat(tvix/eval): add observer configuration to public API
With this change, it should be possible to have both existing
use-cases (CLI & Tvixbolt) use the same API.

Change-Id: I2195264f08cc892177b559a28660dc5f98e48e41
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7545
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-12-21 13:23:38 +00:00
Vincent Ambo
9bc1e6ef05 feat(tvix/eval): add configuration of Nix search path to public API
This is required for passing through NIX_PATH from the CLI.

Change-Id: If129df79ef9c3ffab31408adb85679909276c4f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7544
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-12-21 13:23:38 +00:00
Vincent Ambo
9d6ee5b6a6 fix(tvix/eval): use test-generator fork that supports workspaces
This should make no difference in Nix builds, but allows running tests
locally again with `cargo test` for //tvix/eval.

Change-Id: I97d61840143d5c14db61d5862781bf635f9a28e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7590
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2022-12-21 13:23:38 +00:00
Vincent Ambo
257c67f204 chore(tvix): upgrade to clap 4.0
In //tvix/eval:

* criterion bumped to 4.0, which at least depends on clap 3.x instead
  of 2.x, which is less incompatible

In //tvix/cli:

* no changes required

In //tvix/nix_cli:

* some minor changes for compatibility with clap 4.0, no functionality
  changes

Change-Id: If793f64b59fcaa2402d3d483ddbab4092f32df03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7588
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-21 13:23:38 +00:00
Vincent Ambo
d9e2bec953 refactor(tvix): split binary (REPL etc.) out from evaluator library
The tvix-eval project is independent from any *uses* of the evaluator,
such as the tvix-repl.

This functionality has been split out into a separate "tvix-cli"
crate. Note that this doesn't have to mean that this CLI crate is the
"final" CLI crate for tvix, the point of this is not "getting the CLI
structure right" but rather "getting the evaluator structure right".

This reshuffling is part of restructuring the way that functionality
like store communication is injected into language evaluation.

Note that at this commit the new CLI crate is not at feature-parity.

Change-Id: Id0af03dc8e07ef09a9f882a89612ad555eca8f93
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7541
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-21 13:09:29 +00:00
Vincent Ambo
1138fbcaad refactor(tvix/eval): use new public API in test code
This removes internal uses of the previous crate::eval module, which
is being removed.

Change-Id: I5fb3c53460a9c5381853d0258f9ed074ab23c630
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7543
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: grfn <grfn@gws.fyi>
2022-12-21 13:09:28 +00:00