Commit graph

81 commits

Author SHA1 Message Date
Vincent Ambo
b8cbf3456b chore(corp): update copyright years
Change-Id: I59a93c5c41562f5c7c65c9fd6a4d4409df904beb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8607
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
2023-05-23 10:17:24 +00:00
Florian Klink
6a30eb69e0 chore(tvix/*): bump to smol_str 0.2.0
Change-Id: Ic9ac1b6fecb564eafb41b265bf317cd385fdc170
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8560
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-05-11 18:19:20 +00:00
Vincent Ambo
6daf91c9cd feat(corp/ops): add NixOS profile for Yandex Cloud machines
Sets up a virtual machine image that is bootable on Yandex Cloud.

There are some slightly wonky behaviours still, like cloud-init
apparently putting all keys into root's authorized_keys no matter what
is specified in the metadata, but it does work now.

Change-Id: I57dcb7fcfa6872a28855dc1347f73a6db3c56828
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8496
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-04-24 10:56:40 +00:00
Vincent Ambo
c7392b3c6b chore(corp/ops): move terraform config into subfolder
Change-Id: Iad5ad8d9a48c300faf2e4be7003879656817b518
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8495
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-04-24 10:56:40 +00:00
Vincent Ambo
111e1d38e5 feat(corp/ops): configure state bucket for terraform
This was a bit trickier than I anticipated, because there's no good
ways to avoid passing the credentials around manually.

What's basically happening now is that the credentials for the state
bucket are checked in (encrypted), and sourcing `creds.fish` uses the
cloud HSM to decrypt and load them into the environment.

Change-Id: I3f5ce1c9bd9d5efbf1013414f94771a09ea3a488
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8494
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-04-24 10:56:40 +00:00
Vincent Ambo
0637ab3add feat(corp/ops): add yc-cli
Change-Id: If6578693a5d5ef49d059735eeade3bebf13c4d16
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8493
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-04-24 10:56:40 +00:00
Vincent Ambo
55c9f4a803 feat(corp/ops): bootstrap separate corp terraform config
Doesn't actually contain any configuration yet, just setting up TF
with the right providers and so on.

Change-Id: Ia7128dd977b4ff69eebaa36c6cad6ac104cafcdb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8492
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-04-24 10:56:40 +00:00
Vincent Ambo
05b2f1ccb4 fix(predlozhnik): use correct link to source code after move
Change-Id: I74da72818d9afa96d6bfbfd02f0110707ef8b721
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8248
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-10 14:30:37 +00:00
Vincent Ambo
2696839770 feat(tvix/eval): introduce generators module
This module contains the request/response types for generators
requesting actions from the VM.

For most of these, an async helper function is added that will be used
inside of generator functions to make use of these requests/responses
instead of constructing them directly.

Change-Id: I1e085f88adaf784a34867957a0e82532d3a83d7c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8148
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-03-07 22:04:59 +00:00
Vincent Ambo
0a27344953 chore(3p/sources): bump nixpkgs & overlays (2023-02-07)
Included fixes:

* //3p/overlays: tdlib override no longer needed (bump has landed upstream)
* //corp/{predlozhnik,tvixbolt}: bump wasm-bindgen to match nixpkgs

Home-manager has not been bumped as it has introduced an
incompatibility with Nix 2.3

Change-Id: I96ac3462b82c73db1ba23be03d7968f10abc9b53
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8033
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-02-07 13:46:35 +00:00
Florian Klink
c13759e89f feat(corp/tvixbolt): link to tvl.su too
Make it clear that Tvixbolt is a project of TVL LLC, and link to the
community website too.

See https://b.tvl.fyi/issues/248

Change-Id: Iefefe0263fa5ef01587d49c5a130a38b78ca7981
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8019
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-02-03 19:04:21 +00:00
Florian Klink
459dadfe9d docs(corp/website): distinguish TVL LLC and TVL
See https://b.tvl.fyi/issues/248

Change-Id: Iacc10841bdcdf1dbc8eab659d82e076fa2151979
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8020
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-03 19:04:20 +00:00
Florian Klink
90f847ad32 chore(corp/tvixbolt): update Cargo.lock
Change-Id: Ie6bb2df16f79f7d977a7e95187a6b81e3ac0108f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8011
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-02-01 23:06:02 +00:00
Vincent Ambo
b1e4d66f83 docs(corp/website): update website content
Adds a multi-lingual version of the page, with the standard English
page being served at `/` and `/en`, and the new Russian version at
`/ru`.

Change-Id: I54ceea91d1442ee7b8717b59083e5d07c36ca8b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7940
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-29 10:48:20 +00:00
Vincent Ambo
d05c380504 fix(corp/data-import): rank is an integer field
Change-Id: Ifc9cd46e5b5521096db19628bd8bcf026106dcc9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7926
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-25 10:46:12 +00:00
Florian Klink
e0b05c0fa6 feat(tvix/eval): implement builtins.fromTOML
This allows parsing TOML from Tvix. We can enable the eval-okay-fromTOML
testcase from nix_tests. It uses the `toml` crate, and the serde
integration it brings with it.

Change-Id: Ic6f95aacf2aeb890116629b409752deac49dd655
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7920
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-25 07:49:44 +00:00
Florian Klink
1facd889bb feat(tvix/eval): use lexical-core to format float
Apparently our naive implementation of float formatting, which simply
used {:.5}, and trimmed trailing "0" strings not sufficient.

It wrongly trimmed numbers with zeroes but no decimal point, like
`10000` got trimmed to `1`.

Nix uses `std::to_string` on the double, which according to
https://en.cppreference.com/w/cpp/string/basic_string/to_string
is equivalent to `std::sprintf(buf, "%f", value)`.

https://en.cppreference.com/w/cpp/io/c/fprintf mentions this is treated
like this:

> Precision specifies the exact number of digits to appear after
> the decimal point character. The default precision is 6. In the
> alternative implementation decimal point character is written even if
> no digits follow it. For infinity and not-a-number conversion style
> see notes.

This doesn't seem to be the case though, and Nix uses scientific
notation in some cases.

There's a whole bunch of strategies to determine which is a more compact
notation, and which notation should be used for a given number.
https://github.com/rust-lang/rust/issues/24556 provides some pointers
into various rabbit holes for those interested.

This gist seems to be that currently a different formatting is not
exposed in rust directly, at least not for public consumption.

There is the
[lexical-core](https://github.com/Alexhuszagh/rust-lexical) crate
though, which provides a way to format floats with various strategies
and formats.

Change our implementation of `TotalDisplay` for the `Value::Float` case
to use that. We still need to do some post-processing, because Nix
always adds the sign in scientific notation (and there's no way to
configure lexical-core to do that), and lexical-core in some cases keeps
the trailing zeros.

Even with all that in place, there as a difference in `eval-okay-
fromjson.nix` (from tvix-tests), which I couldn't get to work. I updated
the fixture to a less problematic number.

With this, the testsuite passes again, and does for the upcoming CL
introducing builtins.fromTOML, and enabling the nix testsuite bits for
it, too.

Change-Id: Ie6fba5619e1d9fd7ce669a51594658b029057acc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7922
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-25 07:49:44 +00:00
Vincent Ambo
192dac5a74 feat(corp/data-import): map OR word types to sets of OC grammemes
Change-Id: I674f3a66fcd65314431a2ebd747e3830aa2dd7a1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7924
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
2023-01-24 22:41:21 +00:00
Vincent Ambo
80723b708d feat(corp/data-import): map OC lemma grammemes to OR form types
Change-Id: Ie804d185269336b0d9fe417754e5e795918e65b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7923
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-24 22:41:19 +00:00
Vincent Ambo
8d594658ab feat(corp/data-import): map OC word grammemes to OR form types
This table maps the grammemes for individual word forms (*not* for
lemmata in either corpus!) to the corresponding grammemes from the
other dataset.

These have drastically different shapes, so the mapping is not
perfect, but will help in determining which forms are intended to be
the same on both sides.

Change-Id: Ib0717e2f7a79d96bcb5e955a20f551e391fcd759
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7918
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-01-24 22:41:19 +00:00
Aaqa Ishtyaq
59c3383b10 fix(corp/tvixbolt): set output to eval value.
This CL fixes the bug where output of a nix
evaluation is not set.

Change-Id: I8ae2759a7ec26e1de2e57dd43302129347a8c302
Signed-off-by: Aaqa Ishtyaq <aaqaishtyaq@gmail.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7896
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-22 16:35:49 +00:00
Vincent Ambo
ed8dd4acd7 feat(corp/data-import): add import of OR 'translations' table
The original dataset contains translations into different languages,
but only the English ones are imported here.

Note that translations are for lemmata only.

Change-Id: Ifb9c32c25fda44c38ad899efca9d205c520c0fa3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7895
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-22 16:13:09 +00:00
Vincent Ambo
8eeb5d3bcc feat(corp/data-import): add import of OR 'words_forms' table
This is the full morphological set table for all the words from the
lemmata table, which they don't call it that.

Change-Id: I6f5be673c5f59f11e36bd8c8c935844a7d4fd170
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7894
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-21 17:49:33 +00:00
Vincent Ambo
429c0d00c4 feat(corp/data-import): add import of OpenRussian 'words' table
This is actually the lemmata table of this corpus, not the forms of
all words (they're in a separate table).

Change-Id: I89a2c2817ccce840f47406fa2a636f4ed3f49154
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7893
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-21 17:49:33 +00:00
Vincent Ambo
ee0c0ee951 chore(corp/data-import): make OR data archive available in env
Change-Id: Idacf42743051eae0cf7010f952a4f91af17ad708
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7892
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-21 17:49:33 +00:00
Vincent Ambo
0dfe460fbb docs(corp/data-import): document OpenRussian format
This is the second dataset I want to integrate as it contains some
more practically useful, but somewhat less structured, information.

Change-Id: Ib46b2597a33e76f59e030f889a0961ecc5a144eb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7873
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-18 21:58:02 +00:00
Vincent Ambo
db26825eec chore(corp/data-import): namespace tables for OpenCorpora data
I'm changing strategies to importing both OC and another dataset
before continuing to normalise the data, as it might be easier to do
in a set of table-constructing queries inside of SQLite with all raw
data in place.

Change-Id: I26b41af80586fc1bfd8e26a6be20579068a82507
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7872
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-18 21:58:02 +00:00
Vincent Ambo
c891833414 feat(corp/data-import): build morphology database in derivation
This makes the actual imported database of the ~whole Russian
language (all lemmas, grammemes, forms etc.) a Nix build target which
is built in CI.

This still needs schema normalisation (it's fairly directly mapped to
the raw data), but it's already starting to be a useful data set.

This also happens to be a pretty cool demonstration of the power of
Nix. You can do `nix-build -A corp.russian.data-import.database` and
out comes a perfectly valid SQLite database with a valid external data
import!

Change-Id: I5d6d15e67d0e4a7ff590fad06252be34f5d561fd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7866
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-18 15:44:06 +00:00
Vincent Ambo
0ed6583edc feat(corp/data-import): let users specify output path
Change-Id: I61ad021c7a5318b099f3adc8bc6aedef65500974
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7865
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-18 15:44:06 +00:00
Vincent Ambo
476e312c06 feat(corp/data-import): parse and import links
Change-Id: Iebdbc8f884f28064d7b00b8f8808b5030fa3d05c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7864
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-18 15:44:06 +00:00
Vincent Ambo
dc55ea3201 feat(corp/data-import): parse and import link types
Change-Id: Iae01d1dc6894117dc693b4690d8bc79861212ae6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7863
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-18 15:44:06 +00:00
Vincent Ambo
3f0b1d8e0b fix(corp/data-import): commit the final transaction, too
Otherwise up to 1000 elements might be missing.

Change-Id: I20d6238424eec27f0e758e7737c9c31bcb81b23d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7862
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-18 15:44:06 +00:00
Vincent Ambo
6986aa5824 feat(corp/data-import): insert OpenCorpora data into SQLite
This is an initial and kind of dumb table structure, but there's some
massaging that needs to be done before this makes more sense.

Change-Id: I441288b684ef86be507099bcc4ebf984598789c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7861
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-18 15:44:06 +00:00
Vincent Ambo
485c3cc912 feat(corp/data-import): parse lemmas from OpenCorpora dump
Change-Id: I1e4efcfc8e555f61578b563411d5e6ed9590d8e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7860
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-18 01:10:37 +00:00
Vincent Ambo
ee7616d956 feat(corp/russian/data-import): new OpenCorpora data import tool
Adds the beginning of a tool which can import OpenCorpora data into a
SQLite database. This is quite a lot of toil and there's probably a
better way to do this, but overall becoming this intimately familiar
with the data structures is quite helpful for understanding what I
can/can't do with only this dataset.

Change-Id: Ieab33a8ce07ea4ac87917b9c8132226bbc6523b1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7859
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-18 01:10:37 +00:00
Vincent Ambo
aa96e25bbc chore(tazjin/predlozhnik): move to //corp
This is currently hosted by the company, and I'm assigning my
copyright to the company, which also runs an ad placement on the page.

Note that the NixOS module for hosting it has not been moved yet.

Change-Id: Iba9e1cab9370faa79e43c3344fbfbbbabead50b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7857
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-17 18:23:52 +00:00
Vincent Ambo
d365b09226 feat(tvix/eval): implement builtins.toXML
Change-Id: I009efc53a8e98f0650ae660c4decd8216e8a06e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7835
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-16 13:43:14 +00:00
Vincent Ambo
91465dc78e refactor(tvix/eval): persistent, memory-sharing OrdMap for NixAttrs
This uses the `im::OrdMap` for `NixAttrs` to enable sharing of memory
between different iterations of a map.

This slightly speeds up eval, but not significantly. Future work might
include benchmarking whether using a `HashMap` and only ordering in
cases where order is actually required would help.

This switches to a fork of `im` that fixes some bugs with its OrdMap
implementation.

Change-Id: I2f6a5ff471b6d508c1e8a98b13f889f49c0d9537
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7676
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-12-29 17:44:56 +00:00
Vincent Ambo
5d73c06b1a refactor(tvix/eval): use im::Vector for NixList representation
This is a persistent, structurally sharing data structure which is
more efficient in some of our use-cases. I have verified the
efficiency improvement using `hyperfine` repeatedly over expressions
on nixpkgs.

Lists are not the most performance-critical structure in Nix (that
would be attribute sets), but we can already see a small (~5-10%)
improvement.

Note that there are a handful of cases where we still go via `Vec`
that need to be fixed, most notable for `builtins.sort` which can not
currently be implemented directly using `im::Vector` because of a
restrictive type bound.

Change-Id: I237cc50cbd7629a046e5a5e4601fbb40355e551d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7670
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-12-29 12:27:59 +00:00
Vincent Ambo
32dba980eb refactor(tvixbolt): use new tvix_eval API
As expected, this ends up being significantly nicer to use than the
previous API.

While doing this, I've combined the error fields into one. This is
because there would only ever be one of those anyways, and combining
them ensures that we have consistent formatting (for example,
parser errors would previously not be run through the pretty
formatter but are now).

Change-Id: I6074ec8a4a3901ea82d5d07174b76a345210967b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7547
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
2022-12-21 13:23:38 +00:00
Vincent Ambo
3aca3d3bba refactor(tvix): build Rust projects using crate2nix
Introduces granular dependency builds using crate2nix, bootstrapped
off the generated configuration from the newly introduced
workspace (see cl/7533).

This commit checks in the generated Cargo.nix file which can be
regenerated with a parameterless invocation of `crate2nix generate` in
`//tvix`. I tried generating this in IFD, but it turned out to be
harder than what seemed worthwhile for now.

In this setup, the various build targets for Rust projects end up
being attributes of the imported `Cargo.nix` file at the `tvix.crates`
attribute. These still lack configuration, however, which has been
fixed in the various `default.nix` files of individual projects.

Note that we (temporarily) lose the ability to build tvix-eval's
benchmarks in CI. I haven't figured out what magic incantation summons
them from the void again ...

The `eval-okay-readDir` tests from both test suites have been disabled
because they fail for unknown reasons when run in this new derivation.
Somebody will have to debug it!

Change-Id: I2014614ccb9c8951aedbd71df7966ca191a13695
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7538
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2022-12-15 17:26:45 +00:00
Vincent Ambo
f282984c25 fix(tvix/eval): use rnix-parser from crates.io
A few weeks ago, oberblastmeister did a release to crates.io so we can
stop importing it via GitHub.

Change-Id: I9d5fa5cd281685779c71b12fed45ed201a1db17e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7532
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2022-12-06 18:29:50 +00:00
Adam Joseph
8616f13a71 feat(tvix/eval): builtins.import without RefCell
CL/6867 added support for builtins.import, which required a cyclic
reference import->globals->builtins->import.  This was implemented
using a RefCell, which makes it possible to mutate the builtins
during evaluation.  The commit message for CL/6867 expressed a
desire to eliminate this possibility:

  This opens up a potentially dangerous footgun in which we could
  mutate the builtins at runtime leading to different compiler
  invocations seeing different builtins, so it'd be nice to have
  some kind of "finalised" status for them or some such, but I'm not
  sure how to represent that atm.

This CL replaces the RefCell with Rc::new_cyclic(), making the
globals/builtins immutable once again.  At VM runtime (once opcodes
start executing) everything is the same as before this CL, except
that the Rc<RefCell<>> introduced by CL/6867 is turned into an
rc::Weak<>.

The function passed to Rc::new_cyclic works very similarly to
overlays in nixpkgs: a function takes its own result as an argument.
However instead of laziness "breaking the cycle", Rust's
Rc::new_cyclic() instead uses an rc::Weak.  This is done to prevent
memory leaks rather than divergence.

This CL also resolves the following TODO from CL/6867:

  // TODO: encapsulate this import weirdness in builtins

The main disadvantage of this CL is the fact that the VM now must
ensure that it holds a strong reference to the globals while a
program is executing; failure to do so will cause a panic when the
weak reference in the builtins is upgrade()d.

In theory it should be possible to create strong reference cycles
the same way Rc::new_cyclic() creates weak cycles, but these cycles
would cause a permanent memory leak -- without either an rc::Weak or
RefCell there is no way to break the cycle.  At some point we will
have to implement some form of cycle collection; whatever library we
choose for that purpose is likely to provide an "immutable strong
reference cycle" primitive similar to Rc::new_cyclic(), and we
should be able to simply drop it in.

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I34bb5821628eb97e426bdb880b02e2097402adb7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7097
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-27 21:36:01 +00:00
Vincent Ambo
a8f7383fcb refactor(tvixbolt): use details elements for toggling outputs
For optional outputs (runtime trace & AST) this has a slightly nicer
user experience.

Note that the code of this is a bit verbose because doing a naive
implementation hits dumb behaviours of browsers that result in
infinite loops.

Thanks Profpatsch for the suggestion.

Change-Id: I8945a8e722f0ad8735829807fb5e39e2101f378c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7006
Reviewed-by: j4m3s <james.landrein@gmail.com>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-16 12:26:56 +00:00
Vincent Ambo
e6d9be32a2 feat(tvixbolt): add toggle for displaying pretty-printed AST
This uses the JSON serialisation of the AST introduced earlier to
display a text box with the serialised AST to users. Useful for
debugging.

Change-Id: Ibc400eaf5ca87fa5072d5c044942505331c3bb40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7005
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-16 12:26:56 +00:00
Vincent Ambo
0abc66ad91 feat(tvix/eval): add an AST pretty-printing module
This implements serde::Serialize for the rnix AST through a wrapper
type, and exposes a function for serialising the AST into
a (pretty-printed JSON) string representation.

This can be used to debug issues with the AST, and to display an AST
reprsentation in tools like tvixbolt.

Serialize is implemented manually because we don't own any of the
structs and the way to traverse them is not easily derived
automatically, and this is quite verbose. We might be able to condense
it a little bit, but at the same time it's also fairly straightforward.

Change-Id: I922df43cfc25636f3c8baee7944c75ade516055c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6943
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-16 12:26:56 +00:00
Griffin Smith
03a3189a3d feat(tvix/eval): Initial impl of builtins.match
Implement an *initial* version of builtins.match, using the rust `regex`
crate for regular expressions. The rust regex crate definitely has
different semantics than nix's regular expressions - but we'd like to
see how far we can get before the incompatibility starts to matter.

This consciously leaves out any sort of memo for compiled regular
expressions (which upstream nix also has) for the sake of expediency -
in the future we should implement that so we don't have to compile the
same regular expression multiple times.

Change-Id: I5b718635831ec83397940e417a9047c4342b6fa1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6989
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-15 20:35:22 +00:00
Griffin Smith
5eb89be682 feat(tvix/eval): Implement builtins.fromJSON
Using `serde_json` for parsing JSON here, plus an `impl FromJSON for
Value`. The latter is primarily to stay "dependency light" for now -
likely going with an actual serde `Deserialize` impl in the future is
going to be way better as it allows saving significantly on intermediary
allocations.

Change-Id: I152a0448ff7c87cf7ebaac927c38912b99de1c18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6920
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-10-15 20:35:22 +00:00
Griffin Smith
273ba73754 feat(tvix/eval): Initial resolution of <...> paths
This commit implements (lazy) resolution of `<...>` paths via either the
NIX_PATH environment variable, or the -I command-line flag - both
handled via EvalOptions. As a result, EvalOptions can no longer derive
Copy, meaning we have to clone it at each line of the repl - this is
probably not a huge deal as repl performance is not exactly an inner
loop and we're not cloning very much.

Internally, this works by creating a thunk which pushes a constant
containing the string inside the brackets to the stack, then a new
opcode to resolve that path via the `NixPath`. To get that opcode to
work, we now have to pass in the NixPath when constructing the VM.

This (intentionally) leaves out proper implementation of path resolution
via `findFile` (cppnix just calls whatever identifier called findFile is
in scope!!!) as that's widely considered a bit of a misfeature, but if
we do decide to implement that down the road it likely wouldn't be more
than a few extra ops within the thunk introduced here.

Change-Id: Ibc979b7e425b65cbe88599940520239a4a10cee2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6918
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-10 20:23:41 +00:00
Vincent Ambo
07e03498f2 feat(tvix/eval): add method for emitting runtime warnings
This lets the VM emit warnings when it encounters situations that
should only be warned about at runtime.

For starters, this is used to pass through compilation warnings that
come up when `import` is used.

Change-Id: I0c4bc8c534d699999887c430d93629fadfa662c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6868
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2022-10-07 14:24:36 +00:00