Commit graph

53 commits

Author SHA1 Message Date
Aspen Smith
201173afac fix(tvix): Represent strings as byte arrays
C++ nix uses C-style zero-terminated char pointers to represent strings
internally - however, up to this point, tvix has used Rust `String` and
`str` for string values. Since those are required to be valid utf-8, we
haven't been able to properly represent all the string values that Nix
supports.

To fix that, this change converts the internal representation of the
NixString struct from `Box<str>` to `BString`, from the `bstr` crate -
this is a wrapper around a `Vec<u8>` with extra functions for treating
that byte vector as a "morally string-like" value, which is basically
exactly what we need.

Since this changes a pretty fundamental assumption about a pretty core
type, there are a *lot* of changes in a lot of places to make this work,
but I've tried to keep the general philosophy and intent of most of the
code in most places intact. Most notably, there's nothing that's been
done to make the derivation stuff in //tvix/glue work with non-utf8
strings everywhere, instead opting to just convert to String/str when
passing things into that - there *might* be something to be done there,
but I don't know what the rules should be and I don't want to figure
them out in this change.

To deal with OS-native paths in a way that also works in WASM for
tvixbolt, this also adds a dependency on the "os_str_bytes" crate.

Fixes: b/189
Fixes: b/337
Change-Id: I5e6eb29c62f47dd91af954f5e12bfc3d186f5526
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10200
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
2024-01-31 14:51:49 +00:00
Florian Klink
43b9e25025 refactor(tvix/eval): move Evaluation::{default,new_pure}() again
Have a Evaluation::new() function that's used to set up the Evaluation
struct initially - which is also used by both new_pure and new_impure
internally.

It's generic over the exact type of IO, making it easier to instantiate
Evaluation with non-tvix-eval EvalIO implementations, that might not be
in a Box.

Change-Id: Ibf728da24aca59639c5b6df58d00ae98c99a63f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10640
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-18 14:24:40 +00:00
Ryan Lahfa
75cc52ddb1 fix(tvix/eval): getContext merges underlying values
Previously, we were assembling very naively an attribute set composed of context we saw.

But it was forgetting that `"${drv}${drv.drvPath}"` would contain 2 contexts with the same key, but
with different values, one with `outputs = [ "out" ];` and `allOutputs = true;`.

Following this reasoning and comparing with what Nix does, we ought to merge underlying values systematically.

Hence, I bring `itertools` to perform a group by on the key and merge everything on the fly, it's not
beautiful but it's the best I could find, notice that I don't use
`group_by` but I talk about group by, that is, because `group_by` is a
`group_by_consecutive`, see
https://github.com/rust-itertools/itertools/issues/374.

Initially, I tried to do it without a `into_grouping_map_by`, it was akin to assemble the final `NixAttrs` directly,
it was less readable and harder to pull out because we don't have a lot of in-place mutable functions on
our data structures.

Change-Id: I9933c9bd88ffe04de50dda14f21879b60d8b8cd4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10620
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-01-17 17:25:24 +00:00
Florian Klink
4fba57c2c9 refactor(tvix/eval): remove code and location from struct
Instead, it's passed in the evaluate/compile_only functions, which feels
more naturally. It lets us set up the Evaluation struct long before
we actually feed it with data to evaluate.

Now that Evaluation::new() would be accepting an empty list of
arguments, we can simply implement Default, making things a bit more
idiomatic.

Change-Id: I4369658634909a0c504fdffa18242a130daa0239
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10475
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-12-31 13:15:24 +00:00
Vincent Ambo
eba5c1757a chore(3p/sources): bump channels & overlays (2023-12-29)
* all: update wasm-bindgen to 0.2.89 in WASM projects
* users/grfn: explicitly set pinentry for gpg-agent
* 3p/crate2nix: drop patches that were merged upstream
* 3p/rust-crates: fix one more package name that was broken by crates.io
* 3p/overlays: bump telega backend to new required version

The update for agenix has been dropped. It caused strange build errors
with messages like these:

    patching script interpreter paths in /nix/store/0g0wpa3vxfb4w461s6ny3s1wr08faj73-agenix-0.15.0
    /nix/store/0g0wpa3vxfb4w461s6ny3s1wr08faj73-agenix-0.15.0/bin/agenix: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/q8qq40xg2grfh9ry1d9x4g7lq4ra7n81-bash-5.2-p21/bin/bash"
    stripping (with command strip and flags -S -p) in  /nix/store/0g0wpa3vxfb4w461s6ny3s1wr08faj73-agenix-0.15.0/bin
    Running phase: installCheckPhase
    no Makefile or custom installCheckPhase, doing nothing
    agenix version: 0.15.0
    error: creating directory '/nix/var': Permission denied
    There is no rule for secret1.age in ./secrets.nix.
    /nix/store/d4jf1cbbk494zwgbqz31pxgigpsbh6w2-stdenv-linux/setup: line 138: test: =: unary operator expected
    /nix/store/d4jf1cbbk494zwgbqz31pxgigpsbh6w2-stdenv-linux/setup: line 131: pop_var_context: head of shell_variables not a function context
    builder for '/nix/store/0ivvf44hxy0zv4gg8nvchdkp895xw5ri-agenix-0.15.0.drv' failed with exit code 2

I can't be bothered to deal with that right now.

Change-Id: Ia052af0d97dbe9ef0c0d4f3e2214ac00ca8645a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10458
Reviewed-by: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
2023-12-29 21:52:10 +00:00
Vincent Ambo
e36872e689 chore(3p/sources): bump nixpkgs & channels (2023-11-12)
* update wasm-bindgen in all Rust-wasm projects
* remove stable overlays that work again in unstable
* add texlive to stable overlays (see linked nixpkgs PR)
* bump tdlib to 1.8.18, new minimum for telega.el

Change-Id: Ib8e202de7dfbc35115fda31d0a98b6314b2adf17
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10010
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
2023-11-12 18:19:57 +00:00
Vincent Ambo
ad28f78741 chore(3p/sources): Bump channels & overlays
* web/pwcrypt: bump wasm-bindgen
* corp/tvixbolt: bump wasm-bindgen
* corp/rih/frontend: bump wasm-bindgen
* corp/predlozhnik: bump wasm-bindgen
* 3p/overlays: set hiPrio on nixpkgs-review package

  There is some upstream bug causing a conflict with the ZSH
  completion files generated by home-manager.

Change-Id: Ibe5de5564d3214d48469abe175cbebe5356acf74
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9046
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-08-08 21:56:24 +00:00
Florian Klink
432222f098 feat(tvix/store/proto): use Bytes instead of Vec<u8>
Makes use of https://github.com/tokio-rs/prost/pull/341, which makes our
bytes field cheaper to clone.

It's a bit annoying to configure due to
https://github.com/hyperium/tonic/issues/908, but the workaround does
get the job done.

Change-Id: I25714600b041bb5432d3adf5859b151e72b12778
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8975
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-07-22 09:03:02 +00:00
Vincent Ambo
9c7d1361c5 feat(web/static): serve terminalcss from static.tvl.{fyi|su}
Change-Id: I3cdacc1429f86e044fc77c9b68477d577a0f67c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8779
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-15 10:09:52 +00:00
Florian Klink
6a30eb69e0 chore(tvix/*): bump to smol_str 0.2.0
Change-Id: Ic9ac1b6fecb564eafb41b265bf317cd385fdc170
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8560
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-05-11 18:19:20 +00:00
Vincent Ambo
2696839770 feat(tvix/eval): introduce generators module
This module contains the request/response types for generators
requesting actions from the VM.

For most of these, an async helper function is added that will be used
inside of generator functions to make use of these requests/responses
instead of constructing them directly.

Change-Id: I1e085f88adaf784a34867957a0e82532d3a83d7c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8148
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-07 22:04:59 +00:00
Vincent Ambo
0a27344953 chore(3p/sources): bump nixpkgs & overlays (2023-02-07)
Included fixes:

* //3p/overlays: tdlib override no longer needed (bump has landed upstream)
* //corp/{predlozhnik,tvixbolt}: bump wasm-bindgen to match nixpkgs

Home-manager has not been bumped as it has introduced an
incompatibility with Nix 2.3

Change-Id: I96ac3462b82c73db1ba23be03d7968f10abc9b53
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8033
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-02-07 13:46:35 +00:00
Florian Klink
c13759e89f feat(corp/tvixbolt): link to tvl.su too
Make it clear that Tvixbolt is a project of TVL LLC, and link to the
community website too.

See https://b.tvl.fyi/issues/248

Change-Id: Iefefe0263fa5ef01587d49c5a130a38b78ca7981
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8019
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-02-03 19:04:21 +00:00
Florian Klink
90f847ad32 chore(corp/tvixbolt): update Cargo.lock
Change-Id: Ie6bb2df16f79f7d977a7e95187a6b81e3ac0108f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8011
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-02-01 23:06:02 +00:00
Florian Klink
e0b05c0fa6 feat(tvix/eval): implement builtins.fromTOML
This allows parsing TOML from Tvix. We can enable the eval-okay-fromTOML
testcase from nix_tests. It uses the `toml` crate, and the serde
integration it brings with it.

Change-Id: Ic6f95aacf2aeb890116629b409752deac49dd655
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7920
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-25 07:49:44 +00:00
Florian Klink
1facd889bb feat(tvix/eval): use lexical-core to format float
Apparently our naive implementation of float formatting, which simply
used {:.5}, and trimmed trailing "0" strings not sufficient.

It wrongly trimmed numbers with zeroes but no decimal point, like
`10000` got trimmed to `1`.

Nix uses `std::to_string` on the double, which according to
https://en.cppreference.com/w/cpp/string/basic_string/to_string
is equivalent to `std::sprintf(buf, "%f", value)`.

https://en.cppreference.com/w/cpp/io/c/fprintf mentions this is treated
like this:

> Precision specifies the exact number of digits to appear after
> the decimal point character. The default precision is 6. In the
> alternative implementation decimal point character is written even if
> no digits follow it. For infinity and not-a-number conversion style
> see notes.

This doesn't seem to be the case though, and Nix uses scientific
notation in some cases.

There's a whole bunch of strategies to determine which is a more compact
notation, and which notation should be used for a given number.
https://github.com/rust-lang/rust/issues/24556 provides some pointers
into various rabbit holes for those interested.

This gist seems to be that currently a different formatting is not
exposed in rust directly, at least not for public consumption.

There is the
[lexical-core](https://github.com/Alexhuszagh/rust-lexical) crate
though, which provides a way to format floats with various strategies
and formats.

Change our implementation of `TotalDisplay` for the `Value::Float` case
to use that. We still need to do some post-processing, because Nix
always adds the sign in scientific notation (and there's no way to
configure lexical-core to do that), and lexical-core in some cases keeps
the trailing zeros.

Even with all that in place, there as a difference in `eval-okay-
fromjson.nix` (from tvix-tests), which I couldn't get to work. I updated
the fixture to a less problematic number.

With this, the testsuite passes again, and does for the upcoming CL
introducing builtins.fromTOML, and enabling the nix testsuite bits for
it, too.

Change-Id: Ie6fba5619e1d9fd7ce669a51594658b029057acc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7922
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-25 07:49:44 +00:00
Aaqa Ishtyaq
59c3383b10 fix(corp/tvixbolt): set output to eval value.
This CL fixes the bug where output of a nix
evaluation is not set.

Change-Id: I8ae2759a7ec26e1de2e57dd43302129347a8c302
Signed-off-by: Aaqa Ishtyaq <aaqaishtyaq@gmail.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7896
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-22 16:35:49 +00:00
Vincent Ambo
d365b09226 feat(tvix/eval): implement builtins.toXML
Change-Id: I009efc53a8e98f0650ae660c4decd8216e8a06e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7835
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-16 13:43:14 +00:00
Vincent Ambo
91465dc78e refactor(tvix/eval): persistent, memory-sharing OrdMap for NixAttrs
This uses the `im::OrdMap` for `NixAttrs` to enable sharing of memory
between different iterations of a map.

This slightly speeds up eval, but not significantly. Future work might
include benchmarking whether using a `HashMap` and only ordering in
cases where order is actually required would help.

This switches to a fork of `im` that fixes some bugs with its OrdMap
implementation.

Change-Id: I2f6a5ff471b6d508c1e8a98b13f889f49c0d9537
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7676
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-12-29 17:44:56 +00:00
Vincent Ambo
5d73c06b1a refactor(tvix/eval): use im::Vector for NixList representation
This is a persistent, structurally sharing data structure which is
more efficient in some of our use-cases. I have verified the
efficiency improvement using `hyperfine` repeatedly over expressions
on nixpkgs.

Lists are not the most performance-critical structure in Nix (that
would be attribute sets), but we can already see a small (~5-10%)
improvement.

Note that there are a handful of cases where we still go via `Vec`
that need to be fixed, most notable for `builtins.sort` which can not
currently be implemented directly using `im::Vector` because of a
restrictive type bound.

Change-Id: I237cc50cbd7629a046e5a5e4601fbb40355e551d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7670
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-12-29 12:27:59 +00:00
Vincent Ambo
32dba980eb refactor(tvixbolt): use new tvix_eval API
As expected, this ends up being significantly nicer to use than the
previous API.

While doing this, I've combined the error fields into one. This is
because there would only ever be one of those anyways, and combining
them ensures that we have consistent formatting (for example,
parser errors would previously not be run through the pretty
formatter but are now).

Change-Id: I6074ec8a4a3901ea82d5d07174b76a345210967b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7547
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-21 13:23:38 +00:00
Vincent Ambo
3aca3d3bba refactor(tvix): build Rust projects using crate2nix
Introduces granular dependency builds using crate2nix, bootstrapped
off the generated configuration from the newly introduced
workspace (see cl/7533).

This commit checks in the generated Cargo.nix file which can be
regenerated with a parameterless invocation of `crate2nix generate` in
`//tvix`. I tried generating this in IFD, but it turned out to be
harder than what seemed worthwhile for now.

In this setup, the various build targets for Rust projects end up
being attributes of the imported `Cargo.nix` file at the `tvix.crates`
attribute. These still lack configuration, however, which has been
fixed in the various `default.nix` files of individual projects.

Note that we (temporarily) lose the ability to build tvix-eval's
benchmarks in CI. I haven't figured out what magic incantation summons
them from the void again ...

The `eval-okay-readDir` tests from both test suites have been disabled
because they fail for unknown reasons when run in this new derivation.
Somebody will have to debug it!

Change-Id: I2014614ccb9c8951aedbd71df7966ca191a13695
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7538
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2022-12-15 17:26:45 +00:00
Vincent Ambo
f282984c25 fix(tvix/eval): use rnix-parser from crates.io
A few weeks ago, oberblastmeister did a release to crates.io so we can
stop importing it via GitHub.

Change-Id: I9d5fa5cd281685779c71b12fed45ed201a1db17e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7532
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-12-06 18:29:50 +00:00
Adam Joseph
8616f13a71 feat(tvix/eval): builtins.import without RefCell
CL/6867 added support for builtins.import, which required a cyclic
reference import->globals->builtins->import.  This was implemented
using a RefCell, which makes it possible to mutate the builtins
during evaluation.  The commit message for CL/6867 expressed a
desire to eliminate this possibility:

  This opens up a potentially dangerous footgun in which we could
  mutate the builtins at runtime leading to different compiler
  invocations seeing different builtins, so it'd be nice to have
  some kind of "finalised" status for them or some such, but I'm not
  sure how to represent that atm.

This CL replaces the RefCell with Rc::new_cyclic(), making the
globals/builtins immutable once again.  At VM runtime (once opcodes
start executing) everything is the same as before this CL, except
that the Rc<RefCell<>> introduced by CL/6867 is turned into an
rc::Weak<>.

The function passed to Rc::new_cyclic works very similarly to
overlays in nixpkgs: a function takes its own result as an argument.
However instead of laziness "breaking the cycle", Rust's
Rc::new_cyclic() instead uses an rc::Weak.  This is done to prevent
memory leaks rather than divergence.

This CL also resolves the following TODO from CL/6867:

  // TODO: encapsulate this import weirdness in builtins

The main disadvantage of this CL is the fact that the VM now must
ensure that it holds a strong reference to the globals while a
program is executing; failure to do so will cause a panic when the
weak reference in the builtins is upgrade()d.

In theory it should be possible to create strong reference cycles
the same way Rc::new_cyclic() creates weak cycles, but these cycles
would cause a permanent memory leak -- without either an rc::Weak or
RefCell there is no way to break the cycle.  At some point we will
have to implement some form of cycle collection; whatever library we
choose for that purpose is likely to provide an "immutable strong
reference cycle" primitive similar to Rc::new_cyclic(), and we
should be able to simply drop it in.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I34bb5821628eb97e426bdb880b02e2097402adb7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7097
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-27 21:36:01 +00:00
Vincent Ambo
a8f7383fcb refactor(tvixbolt): use details elements for toggling outputs
For optional outputs (runtime trace & AST) this has a slightly nicer
user experience.

Note that the code of this is a bit verbose because doing a naive
implementation hits dumb behaviours of browsers that result in
infinite loops.

Thanks Profpatsch for the suggestion.

Change-Id: I8945a8e722f0ad8735829807fb5e39e2101f378c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7006
Reviewed-by: j4m3s <james.landrein@gmail.com>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-16 12:26:56 +00:00
Vincent Ambo
e6d9be32a2 feat(tvixbolt): add toggle for displaying pretty-printed AST
This uses the JSON serialisation of the AST introduced earlier to
display a text box with the serialised AST to users. Useful for
debugging.

Change-Id: Ibc400eaf5ca87fa5072d5c044942505331c3bb40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7005
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-16 12:26:56 +00:00
Vincent Ambo
0abc66ad91 feat(tvix/eval): add an AST pretty-printing module
This implements serde::Serialize for the rnix AST through a wrapper
type, and exposes a function for serialising the AST into
a (pretty-printed JSON) string representation.

This can be used to debug issues with the AST, and to display an AST
reprsentation in tools like tvixbolt.

Serialize is implemented manually because we don't own any of the
structs and the way to traverse them is not easily derived
automatically, and this is quite verbose. We might be able to condense
it a little bit, but at the same time it's also fairly straightforward.

Change-Id: I922df43cfc25636f3c8baee7944c75ade516055c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6943
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-16 12:26:56 +00:00
Griffin Smith
03a3189a3d feat(tvix/eval): Initial impl of builtins.match
Implement an *initial* version of builtins.match, using the rust `regex`
crate for regular expressions. The rust regex crate definitely has
different semantics than nix's regular expressions - but we'd like to
see how far we can get before the incompatibility starts to matter.

This consciously leaves out any sort of memo for compiled regular
expressions (which upstream nix also has) for the sake of expediency -
in the future we should implement that so we don't have to compile the
same regular expression multiple times.

Change-Id: I5b718635831ec83397940e417a9047c4342b6fa1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6989
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-15 20:35:22 +00:00
Griffin Smith
5eb89be682 feat(tvix/eval): Implement builtins.fromJSON
Using `serde_json` for parsing JSON here, plus an `impl FromJSON for
Value`. The latter is primarily to stay "dependency light" for now -
likely going with an actual serde `Deserialize` impl in the future is
going to be way better as it allows saving significantly on intermediary
allocations.

Change-Id: I152a0448ff7c87cf7ebaac927c38912b99de1c18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6920
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-15 20:35:22 +00:00
Griffin Smith
273ba73754 feat(tvix/eval): Initial resolution of <...> paths
This commit implements (lazy) resolution of `<...>` paths via either the
NIX_PATH environment variable, or the -I command-line flag - both
handled via EvalOptions. As a result, EvalOptions can no longer derive
Copy, meaning we have to clone it at each line of the repl - this is
probably not a huge deal as repl performance is not exactly an inner
loop and we're not cloning very much.

Internally, this works by creating a thunk which pushes a constant
containing the string inside the brackets to the stack, then a new
opcode to resolve that path via the `NixPath`. To get that opcode to
work, we now have to pass in the NixPath when constructing the VM.

This (intentionally) leaves out proper implementation of path resolution
via `findFile` (cppnix just calls whatever identifier called findFile is
in scope!!!) as that's widely considered a bit of a misfeature, but if
we do decide to implement that down the road it likely wouldn't be more
than a few extra ops within the thunk introduced here.

Change-Id: Ibc979b7e425b65cbe88599940520239a4a10cee2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6918
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-10 20:23:41 +00:00
Vincent Ambo
07e03498f2 feat(tvix/eval): add method for emitting runtime warnings
This lets the VM emit warnings when it encounters situations that
should only be warned about at runtime.

For starters, this is used to pass through compilation warnings that
come up when `import` is used.

Change-Id: I0c4bc8c534d699999887c430d93629fadfa662c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6868
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-10-07 14:24:36 +00:00
Vincent Ambo
4b9178fa2a feat(tvix/eval): insert import into the builtins itself
Adding `import` to builtins causes causes a bootstrap cycle because
the `import` builtin needs to be initialised with the set of globals
before being inserted into the globals, which also must contain
itself.

To break out of the cycle this hack wraps the builtins passed to the
compiler in an `Rc` (probably sensible anyways, as they will end up
getting cloned a bunch), containing a RefCell which gives us mutable
access to the builtins.

This opens up a potentially dangerous footgun in which we could mutate
the builtins at runtime leading to different compiler invocations
seeing different builtins, so it'd be nice to have some kind of
"finalised" status for them or some such, but I'm not sure how to
represent that atm.

Change-Id: I25f8d4d2a7e8472d401c8ba2f4bbf9d86ab2abcb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6867
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-10-07 14:24:36 +00:00
sterni
059dea0e32 chore(3p/sources): Bump channels & overlays
* //3p/overlays: kill electrum override applied upstream

* //corp/tvixbolt,
  //users/tazjin/predlozhnik:
  update wasm-bindgen to match nixpkgs and run `cargo update`

Change-Id: If4327112832fd0c7938962dd8d3888d4730b7532
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6874
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-06 23:23:42 +00:00
Vincent Ambo
3530404a4a refactor(tvix/eval): introduce source::SourceCode type
This type hides away the lower-level handling of most codemap data
structures, especially to library consumers (see corresponding changes
in tvixbolt).

This will help with implement `import` by giving us central control
over how the codemap works.

Change-Id: Ifcea36776879725871b30c518aeb96ab5fda035a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6855
Tested-by: BuildkiteCI
Reviewed-by: wpcarro <wpcarro@gmail.com>
2022-10-05 10:29:47 +00:00
Vincent Ambo
2ff764ceb7 refactor(tvix/eval): remove unnecessary clones in compiler
There's basically nothing that needs *ownership* of an AST
node (which is just a little box full of references to other things
anyways), so we can thread this through as references all the way.

Change-Id: I35a1348a50c0e8e07d51dfc18847829379166fbf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6853
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-10-04 21:27:30 +00:00
Vincent Ambo
c46025d520 fix(tvix/eval): support string identifiers in inherits
This updates rnix-parser to a version where inherits provide an
iterator over `ast::Attr` instead of `ast::Ident`, which mirrors the
behaviour of Nix (inherits can have (statically known) strings as
their identifiers).

This actually required some fairly significant code reshuffling in the
compiler, as there was an implicit assumption in many places that we
would have an `ast::Ident` node available when dealing with variable
access (which is then explicitly only not true in this case).

Change-Id: I12f1e786c0030c85107b1aa409bd49adb5465546
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6747
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-22 23:07:14 +00:00
Vincent Ambo
fe3c4720a6 chore(corp/tvixbolt): gitignore trunk's dist folder
Change-Id: Id9df1b33f0e4f9e54b186b2040ba30d9bcd27d54
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6708
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-09-20 10:59:32 +00:00
Vincent Ambo
0e5baae7ad refactor(tvix/eval): clone the Arc<codemap::File> for the compiler
This disconnects ownership of the `File` reference in a compiler from
the calling scope, which is required for when we implement `import`.

`import` will need to carry an `Rc<RefCell<CodeMap>>` (or maybe, in
the future, Arc) to give us the ability to add new detected code
files at runtime.

Note that the choice of `Arc` over `Rc` here is not ours - it's the
codemap crate's.

Change-Id: I3aeca4ffc167acbd1701846a332d93550b56ba7d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6630
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2022-09-18 21:12:50 +00:00
Vincent Ambo
bb47baf638 feat(corp/tvixbolt): persist state in tvixbolt's URL
This makes it possible to enter something into tvixbolt and then share
the link with someone else.

Suggested by Profpatsch originally.

Change-Id: I9886e76a7b821070f13ea7005df09188821e091d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6636
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2022-09-18 16:40:23 +00:00
Vincent Ambo
993c22de59 chore(corp/tvixbolt): add yew-router dependency
This will be used to set/get query parameters for making shareable links.

Change-Id: I05ccf8cab2521564710523ccd3b25ec26f435dd5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6633
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2022-09-18 16:40:23 +00:00
Vincent Ambo
b3cc4c8c2c chore(tvix/eval): bump rnix-parser to latest HEAD
This bumps rnix-parser to a commit that should be unaffected by the
Nix >= 2.4 bug that prevents it from cloning repositories with filters.

Change-Id: Ie01da95245ec6740fa889eb710819e512202f665
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6634
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-18 16:24:40 +00:00
Vincent Ambo
a9914a79a0 feat(corp/tvixbolt): use fancy error formatting
Change-Id: I633154f3a2ae2b1789992a1137c944e897c56b2c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6535
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-09-13 14:41:57 +00:00
Vincent Ambo
e726df8dc5 feat(corp/tvixbolt): add an ad in the footer
As previously mentioned on IRC, this is why tvixbolt is under //corp.

The majority of people in our community probably block ads anyways,
but might as well ...

The ad account is linked to the TVL legal entity.

The ad is configured not to use any personalised data. In testing it's
showing me lamps and shoes. This is the same kind of ad as on my
grammar page, predlozhnik.ru

Change-Id: I172881ed5d5ceb1fdeb2298b8f822d0c2a6518a8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6558
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-09-13 12:47:18 +00:00
Vincent Ambo
118ac68800 feat(corp/tvixbolt): use fancy warning formatting
Unfortunately the codemap-diagnostic crate doesn't provide a way to
get colour control characters written to an arbitrary writer, so this
is black & white only, but we can look at this later if we introduce
something even fancier. For now it's reasonable.

Change-Id: I1c7655cc4b254f77768b5931bc95fa13b3bd7e12
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6533
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-09-13 11:12:29 +00:00
Vincent Ambo
dea6d9b73b chore(tvix/eval): add dependency on codemap-diagnostic
This is a crate for source-span based error reporting. Since all of
our spans are already codemap spans, it is a good starting point.

We have to figure out quite a bit of logic for neat error printing;
later on if we want fancier presentation we might want to look at one
of the other libraries in this space like miette.

Change-Id: I4e28886af1ed199b7112d9dbf063c9f29b612bf1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6531
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
2022-09-13 10:54:14 +00:00
Vincent Ambo
e6a6b0dccb chore(corp/tvixbolt): check in generated Cargo.lock change
Change-Id: I9f87ced716a4328b24fd8fa2eec077ce8dc4a7db
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6504
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-09-11 12:40:38 +00:00
Vincent Ambo
5eda0fbd86 feat(corp/tvixbolt): add Nix build instructions
Now that tvix-eval has almost caught up, tvixbolt can be built in
the depot tree.

Change-Id: Ib26dd98727b110ad8d668aec60db99678644a167
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6491
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-09-11 10:45:22 +00:00
Profpatsch
e646d5170c fix(corp/tvixbolt): make label really clickable
It looks like this was intended, but typoed.

Change-Id: I830d6f0488b75b859bcf4175531c35d79cd40985
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6335
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-09-05 13:16:24 +00:00
Vincent Ambo
6c40fb33b5 fix(corp/tvixbolt): move output above bytecode section
Most expressions people enter will probably have a fairly small
result. It's useful to see *that* it did the correct thing before
looking at *how* it did that.

Change-Id: I50d7d4c07e41f11b71a16c00c49b9553ae9e90a5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6334
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-09-05 02:17:04 +00:00
Vincent Ambo
01a0d5c822 feat(corp/tvixbolt): implement optional runtime tracing
Uses Tvix's new `TracingObserver` to optionally produce a runtime
trace, which the user can toggle via a checkbox.

Runtime traces can be quite long, so they're only produced if asked
for.

Change-Id: Id0f27dc8ef6e9d81a31ecf22c81757b066815320
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6331
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-09-04 18:50:06 +00:00