Commit graph

1163 commits

Author SHA1 Message Date
Aspen Smith
59056cf705 feat(tvix/eval): Leak strings (with flag to disable)
Default to always leaking strings, and copying strings by copying
pointers rather than cloning the underlying allocation. This (somewhat
bafflingly) doesn't seem to affect any benchmarks, but paves the way for
some tricks around string allocation that do.

Unfortunately, we can't do this (yet?) for contextful strings, for
reasons I don't currently understand but which I will address later,
when I address contextful strings more holistically.

I've left a flag in here to disable this, both to test the cloning logic
for strings for when/if we decide to bring this back, and to allow
people who care more about memory usage than perf to disable leaking.

Change-Id: Iec44bcbfe9b3d20389d2450b9a551792a79b9b26
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12045
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-05 10:49:27 +00:00
Aspen Smith
756539a596 chore(3p/sources): Bump channels & overlays (2024-07-28)
* Treewide: re-run depotfmt

* //third_party/nixpkgs:html5validator: build with Python 3.11,
  dependency openstackdocstheme doesn't support 3.12

* //users/sterni/machines/ingeborg: adapt to poorly handled fcgiwrap
  module API change: https://github.com/NixOS/nixpkgs/pull/318599

* //tvix/*-go: regenerate protobuf files

* //third_party/nixpkgs:treefmt: Remove patch for merged pull request

* //users/flokli/ipu6-softisp: rebase, drop upstreamed kernel patches

Change-Id: Ie4e0df007c287e8cd6207683a9a25838aa5bd39a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11971
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-01 10:06:33 +00:00
Matthew Tromp
038d1dd551 feat(tvix/eval): ConstantIdx expansion for more ops
Previously, OpConstant would display some detail about its
ConstantIdx: whether it's a thunk or closure, and what its address
is. This has been expanded to also show when the ConstantIdx is a
blueprint, along with the blueprint's address, and to the other
opcodes that use a ConstantIdx.

Currently, it seems like blueprint addresses don't correspond to the
address of the thunk listed in the bytecode output, but it's still
useful to see that the constant being grabbed is a blueprint, and
maybe this pointer can be made to make more sense in the future.

Change-Id: Ia212b0d52b004c87051542c093274e7106ee08e4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12044
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
Autosubmit: chickadee <matthewktromp@gmail.com>
2024-07-27 22:37:59 +00:00
Matthew Tromp
0b987a509f fix(tvix/eval): don't bubble up io errors from path_exists
path_exists was returning an error when certain common IO errors were
encountered. e.g. in the path "/dev/null/.", path_exists would throw
an error because the underlying call to Path::try_exists threw an
error because null isn't a directory. But if null isn't a directory,
then the path is invalid, so this should really be returning
false. That's what nix's behavior is and that's what makes sense.

The trait function isn't being changed because some other
implementers (e.g. tvix_store_io) have actual errors they can throw.

Fixes: b/411
Change-Id: I9e810e7a198bffe61365697c6d3d7e71f264c20b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12042
Tested-by: BuildkiteCI
Autosubmit: chickadee <matthewktromp@gmail.com>
Reviewed-by: aspen <root@gws.fyi>
2024-07-27 21:29:46 +00:00
Aspen Smith
89d204d295 feat(tvix/eval): Bump imbl to 3.0.0
Change-Id: I88053fa2df35dc486c51d015f4b2156541c07af3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11962
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
2024-07-07 15:05:27 +00:00
Aspen Smith
8821746d6c fix(tvix/repl): Share globals and sourcemap across evaluations
Now that we can bind (potentially lazy, potentially lambda-containing)
values in the REPL and then reference them in subsequent evaluations,
it's important that the values to which we construct shared references
are shared across those subsequent evaluations - otherwise, we get
panics due to unknown source map locations, or dropped weak references
to globals.

This change assigns both the globals and the source map as fields on the
Repl after the first evaluation, and then passes those in (to the
EvaluationBuilder) on subsequent evaluations.

On the EvaluationBuilder side, there's some panicking introduced - this
is intentional, as my intent is for the builder to be configured
statically enough that panicking is the best way to report errors
here (it's always a bug to misconfigure an Evaluation, and we'd never
want to handle it dynamically).

Change-Id: I37225697235c22b683ca48a17d30fa8fedd12d1b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11960
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
2024-07-07 15:04:26 +00:00
Aspen Smith
3a79f93795 refactor(tvix/eval): Construct globals in EvaluationBuilder::build
Construct the Rc<GlobalsMap> for the evaluation as part of
EvaluiationBuilder::build, rather than deferring it until we actually
compile. This changes nothing functionally, but gets us one step closer
to sharing this globals map across evaluations.

Change-Id: Id92e9fb88d974d763056d4f15ce61962ab776e84
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11957
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
2024-07-06 15:24:50 +00:00
Aspen Smith
dfe137786c refactor(tvix/eval): Builderize Evaluation
Make constructing of a new Evaluation use the builder pattern rather
than setting public mutable fields. This is currently a pure
refactor (no functionality has changed) but has a few advantages:

- We've encapsulated the internals of the fields in Evaluation, meaning
  we can change them without too much breakage of clients
- We have type safety that prevents us from ever changing the fields of
  an Evaluation after it's built (which matters more in a world where we
  reuse Evaluations).

More importantly, this paves the road for doing different things with
the construction of an Evaluation - notably, sharing certain things like
the GlobalsMap across subsequent evaluations in eg the REPL.

Fixes: b/262
Change-Id: I4a27116faac14cdd144fc7c992d14ae095a1aca4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11956
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
2024-07-06 15:03:46 +00:00
Aspen Smith
ac3d717944 feat(tvix/eval): Allow passing in an env to evaluation
Allow passing in a top-level env, a map from name to value, to
evaluation. The intent is to support bound identifiers in the REPL just
like upstream nix does.

Getting this working involves mucking around a bit with internals - most
notably, locals now only optionally have a Span (since locals don't have
an easy span we can use) - and getting that working requires propagating
some minor hacks to places where we currently *need* a span (and which
would require too much changing now to make spans optional; my guess is
that that would essentially end up making spans optional throughout the
codebase).

Also, some extra care has to be taken to close out the scope in the case
that we do pass in an env, to avoid breaking our assumptions about the
size of the stack when we return from the toplevel

Change-Id: Ie475b2d3dfc72ccbf298d2a3ea28c63ac877d653
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11953
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
2024-07-05 16:39:34 +00:00
Aspen Smith
6037888e18 refactor(tvix/eval): Drop LightSpan entirely
This was made unnecessary in c92d06271 (feat(tvix/eval): drop
LightSpan::Delayed, 2023-12-08) because it didn't improve benchmarks as
much as expected and has been vestigial since; this continues the
cleanup by just removing it altogether

Change-Id: I21ec7ae9b52a5cccd2092696a5a87f658194d672
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11949
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-05 03:53:44 +00:00
Florian Klink
afbe995aa9 chore(tvix): bump to data-encoding 2.6.0
Change-Id: I26af403bfa99e5d1cff24641a3dba908e1d06686
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11899
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-07-01 06:57:01 +00:00
Florian Klink
aa7d125c12 refactor(tvix/eval): prefix ctx iterators in NixString
NixString::iter_plain() didn't make much sense, especially without a docstring.

Rename it to iter_ctx_plain(), and copy the docstring from NixContext.

Do the same for the two other context element types too.

Change-Id: I1bbfcb967d8d9b14487d069bfe3a1f762253ef4d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11882
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
2024-06-26 14:25:20 +00:00
Florian Klink
a0993e7304 feat(tvix/eval): add ErrorKind::UnexpectedArgumentBuiltin
Change-Id: Ieb091b32aad566719fbe8604c4a589f5ccaaf6b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11877
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-26 04:51:31 +00:00
Florian Klink
970e2a045b refactor(tvix/eval): rename UnexpectedArgument{,Formals}
There's other places where unexpected arguments can be provided, like
in builtins.

Make space for that error type.

Change-Id: Ic831497a3a1dd36a3a184bedadcf1374bf0ae6db
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11876
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-06-26 04:51:31 +00:00
Florian Klink
080654aaf9 feat(tvix/eval): add file_type method
This allows peeking of the type at a given path.

It's necessary, as an open() might not fail until you try to read()
from it, and generally, stat'ing can be faster in some cases.

Change-Id: Ib002da3194a3546ca286de49aac8d1022ec5560f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11871
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-06-26 04:51:31 +00:00
Ilan Joselevich
04f04cfc27 refactor(tvix/eval): drop placeholder impls of builtins in glue
The builtins `placeholder` and `filterSource` are now implemented in
tvix-glue and so we no longer need their placeholder "implementations" in
tvix-eval.

Change-Id: Ibf161ccf1ad40103e9fbb688e9f6be58e7772af1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11867
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-22 21:50:30 +00:00
Florian Klink
5077ca70de chore(tvix/eval): move eval docs to tvix/docs
Change-Id: I75b33c43456389de6e521b4f0ad46d68bc9e98f6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11809
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-06-14 08:00:34 +00:00
binarycat
beb7f57c73 fix(tvix/eval): handle builtins.split matching the empty string
This prevents the following statements from looping endlessly:

```
builtins.split "(.*)" ""
builtins.split "([abc]*)" "abc"
builtins.split "(.*)" "abc"
builtins.split ".*" ""
```

Cover these (and some more examples) in the test suite.

Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: Ibd339f971e0f4e3e5c229816e2be5a8e3836fec9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11743
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-06-11 14:59:02 +00:00
Ben Webb
23e0973cdf docs(tvix): fix some typos across various documents
Fix some typos found while reading various documents, mostly those
relating to the castore.

Here is a summary of the edits.

- fix broken link between documents in the store and castore directories
- clarify expression in castore's data model document that indicates
  that the *name* of each child node of a directory must be unique
  across all three lists of children
- add missing closing parenthesis in castore's data model document
- replace "how" with "what" in the phrase "unclear how a ... would even
  look like" in castore's why-not-git-trees document
- remove unnecessary articles in castore's blobstore chunking document
- add missing "y" to "optionall" in eval's compilation of bindings
  document

Change-Id: I1997ea91bb4e9c40abcd81e0cde9405968580ba6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11763
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-06-08 21:17:56 +00:00
Aspen Smith
d0ab3c8d15 feat(tvix): Switch to jemalloc
Switch tvix to using jemalloc as the default global allocator on
supported (eg, non-msvc) platforms. This gives a pretty reasonable
performance boost basically for free:

int                     time:   [24.361 µs 24.386 µs 24.418 µs]
                        change: [-19.355% -18.859% -18.527%] (p = 0.00 < 0.05)
                        Performance has improved.

merge small attrs       time:   [37.201 µs 37.328 µs 37.442 µs]
                        change: [-24.609% -24.266% -23.982%] (p = 0.00 < 0.05)
                        Performance has improved.

merge large attrs with small attrs
                        time:   [20.030 ms 20.135 ms 20.251 ms]
                        change: [-4.2811% -3.2549% -2.3807%] (p = 0.00 < 0.05)
                        Performance has improved.

hello outpath           time:   [967.91 ms 974.07 ms 983.33 ms]
                        change: [-5.5008% -4.4552% -3.4099%] (p = 0.00 < 0.05)
                        Performance has improved.

Change-Id: I6c6e6e3295ec2fca01ea28dc37bcb201cd811767
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10851
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
2024-06-06 10:29:33 +00:00
Ilan Joselevich
34d93f1d96 fix(tvix): make rstest pick up new test case files
Context: https://github.com/la10736/rstest/issues/256

Cargo will now cause a rebuild whenever a new test case file is added.
Previsouly running `cargo test` after adding a new test case resulted in
that case not being picked up and ignored.

Change-Id: Ibfc420b5bfe3f3ee41d3ebd3fb9d248819fa6ed9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11751
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-05 17:52:17 +00:00
Aspen Smith
72b9a126b8 feat(tvix/glue): Implement builtins.storePath
This one's relatively simple - we just check if the store path exists,
and if it does we make a new contextful string containing the store path
as its only context element.

Automatic testing seems tricky for this (I think?) so I tested it
manually:

tvix-repl> builtins.storePath /nix/store/yn46i4xx5alh7gs6fpkxk430i34rp2q9-hello-2.12.1
=> "/nix/store/yn46i4xx5alh7gs6fpkxk430i34rp2q9-hello-2.12.1" :: string

Change-Id: I8a0d9726e4102ab872c53c2419679c2c855a5a18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11696
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
2024-06-05 17:50:15 +00:00
Ilan Joselevich
1b39d5868a feat(tvix): add CI targets for checking crate features powerset
Closes: https://b.tvl.fyi/issues/401

With this change all crate features (and their combinations) will be built and
tested in CI.

From now on, when adding/removing a Cargo feature for a crate,
you will want to add it to the features power set that gets tested in CI.
For each crate there's a default.nix with a `mkFeaturePowerset` invocation,
modify the list to include/remove the feature.
Note that you don't want to add "collection" features,
such as `fs` for tvix-[ca]store or `default`.

Change-Id: I966dde1413d057770787da3296cce9c1924570e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11717
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-03 16:35:51 +00:00
Ilan Joselevich
be624daf13 fix(tvix/eval): nix_tests.rs's eval_test requires impure flag
This fixes the following command:
'cargo test --no-default-features --features nix_tests'

Change-Id: I9883c39e1e428c72a0e7e0b75a73c8ed734abd3b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11740
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-30 21:52:16 +00:00
Florian Klink
5405ed9dc6 fix(tvix/eval): proptests require arbitrary feature
`cargo test --no-default-features` fails, if we don't conditionalize
this on the `arbitrary` feature too.

Change-Id: I81a277810119fed0cfc37c942c422f731aa14b2e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11726
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-30 16:18:30 +00:00
Florian Klink
baa39d9d09 fix(tvix/eval/tests/one_offs): test_source_builtin can be pure
`Evaluation::new_impure()` would require the test to be impure, but
there's nothing in this test specifically requiring us to make use of
impure features.

Change-Id: Idb24981195d1a94f51053ae04403eb5f0e27f3d9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11725
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-30 16:18:30 +00:00
Florian Klink
c83f2e765c fix(tvix/eval/tests/nix_oracle): tests are impure-only
This uses StdIO, which is only available when the `impure` feature is
enabled.

Change-Id: I039b1f45f6619dd099fa943e58322ff521482dfa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11724
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-30 16:18:30 +00:00
Florian Klink
2c628f6e1c refactor(tvix/eval): move nix_tests to separate module
Gate this behind the nix_tests feature flag (which wasn't applied
consistently).

Also, at least right now all of these use StdIO internally, either by
creating it on their own, or through the `eval_test` helper.

Once we have some proper classification system for tests we can probably
split this up further, but for now only run them if the impure feature
is enabled.

Change-Id: I3236cf09b3391585df99073367c4e4832c5e7898
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11723
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-30 16:18:30 +00:00
Florian Klink
543c103903 fix(tvix/eval/vm/generators): allow unused rq_{path_exists,read_dir}
These VM requests are only emitted by code gated behind the impure feature.

To prevent warning from popping up when building without default
features, allow these to be unused if `impure` is not enabled.

Change-Id: Id871a5215e9a0f09aa78edecdd111369ee7ffe34
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11722
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-05-30 16:18:30 +00:00
Florian Klink
6fb542aae6 fix(tvix/eval/nix_search_path): gate tests on impure feature
These use StdIO, which is only available if the impure feature is
enabled.

Change-Id: I18d8e191a7eba6ba5bd59f43631973eaa796c7bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11721
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2024-05-26 19:53:51 +00:00
Florian Klink
db68c104af fix(tvix/eval/io): OsStringExt and std::fs::File import are impure-only
These are only needed when building with the impure feature enabled.

This removes some warnings when building with --no-default-features.

Change-Id: I3139d9133d4846aeb1b1b5f3830c0d078d047292
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11720
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-05-26 19:52:49 +00:00
edef
edd93b1962 fix(tvix/eval): drop superfluous string context check
cl/11712 simultaneously introduced this check and made it unnecessary,
since NixString::context should never return `Some` for empty contexts
now.

Change-Id: I41a655ff33910e8326cbb7d7526eb91bd19e9585
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11713
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-25 11:33:22 +00:00
edef
49750fa1e7 fix(tvix/eval): disallow empty but allocated string contexts
Both `Some(NixContext::new())` and `None` represent empty contexts,
but the former trips up `NixString::has_context`, and seems likely
to trip up other things.

We could hide the difference in the accessors, but we don't really
*want* the distinction to exist, since heap-allocating a null value
is pretty much always a mistake.

Change-Id: Ie84d26fb0d4b59e68354891ba13bde3bae40ab6e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11712
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-25 08:42:32 +00:00
Florian Klink
649a862ae1 feat(tvix/eval): rm NixContext::join, add take_context & IntoIterator
In places where we want to extend context with that from another
NixString, use take_context() to split it off, then call .extend(),
making use of IntoIterator to avoid a bunch of clones.

Change-Id: I2460141a3ed776c64c36132b2203b6a1d710b922
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11705
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
2024-05-23 14:50:38 +00:00
Florian Klink
ec8d79f3db feat(tvix/eval): teach builtins.toXML context
XmlEmitter gains a NixContext field, and `write_typed_value` extends it
with all context elements present in the passed value.

Once all serialization is done, a into_context() function returns the
collected context, so we can construct a NixString with context.

Tests for this live in tvix-glue, as we use builtins.derivation, which
is not present in the tvix-eval crate.

Fixes b/398.

Change-Id: I85feaaa17b753885f8a017a54e419ec4e602af21
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11704
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Alyssa Ross <hi@alyssa.is>
2024-05-23 14:49:07 +00:00
Florian Klink
a4a313cdd2 feat(tvix/eval): add NixContext::extend
This is a slightly less annoying version of `join`, which does not
consume self. It's more consistent with HashSet::extend().

Change-Id: Ifd0872da36fe8e7b2aa6948674cb8e4023abe9d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11703
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2024-05-23 14:49:07 +00:00
Florian Klink
48e045299d fix(tvix/eval): NixString are bytes
This is not necessarily valid UTF-8.

Change-Id: I72f3157240772eb9c558e5699b4785e44d256fd4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11702
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2024-05-23 09:54:26 +00:00
Profpatsch
5b2ba0efa1 refactor(tvix/eval): rewrite xml emitter to be simple-stupid
In order to be compatible with the nix XML generator, it’s easier to
generate the XML directly, instead of going through a library which we
have to bend to do what we need.

Removes dependency on `xml-rs`, which came with a full XML parser that
we didn’t use. Only takes a tiny bit of code for the XML escaping,
somewhat simplified.

I add a little escaping value, to make sure we have the same behaviour
as nix proper.

Interestingly enough, we never need to escape XML attribute names,
because the `builtins.toXML` format encodes user-defined values as
attribute keys only. So we only escape attribute values.

Fixes: https://b.tvl.fyi/issues/399
Change-Id: If4d407d324864b3bb9aa3160e2ec6889f7727127
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11697
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
2024-05-22 10:32:55 +00:00
Florian Klink
d4978521b0 fix(tvix/eval): use fake values for __curPos, rather than an error
Have this return the same values as builtins.unsafeGetAttrsPos, rather
than returning a CatchableErrorKind, which crashes the CLI if it bubbles
up.

The environment we're in doesn't allow emitting a warning, as we don't
have `co` in scope, but that's probably OK as a stopgap solution.

Alternative to cl/11665.

Change-Id: I5b2c2530842547c93b6533ed9601ee9b2923b1bf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11685
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-05-20 06:52:35 +00:00
Vincent Ambo
f2f12d1556 docs(tvix/eval): add document about how binding construction works
Change-Id: I216be1b75eb9f18a58ab2164a77b3c51de8bf784
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11583
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2024-05-03 22:25:40 +00:00
Alice Carroll
8d49ff3d64 test(tvix): Fix tvix tests on macOS
Prior to this, some tests would not build
or would fail in an obscure way.

Change-Id: I68587cc7592492ebfd71ca02fc7ccc9ff7c0196f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11544
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-04-30 00:55:34 +00:00
sterni
69e4a78818 chore(3p/sources): Bump channels & overlays
- //tvix: address new clippy lints

- //users/tazjin: Satisfy gonic module's new need for a playlist folder.

- //users/aspen/games: adjust for changed location of df's default
  init.txt and d_init.txt.

Change-Id: I00a2adb506ae866206fb6f88c39c9a6af320380f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11509
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
2024-04-28 16:39:26 +00:00
Florian Klink
ff9e8743f6 fix(tvix/eval): don't impl From<NixString> for String
This caught me by accident in an earlier revision of cl/11500 -
I had a `NixString`, wanted to return it as a `String`, so I was naively
calling `s.into()`.

That unfortunately gave me the `Display` implementation of `NixString`,
which quotes strings, causing an annoying error further up the stack.

NixStrings are bytes, we can keep the impl From<NixString> for BString,
but having a `.into()` suddenly do quoting is more than unexpected.

Change-Id: I5434ba94bfe6c493d0a57e68225ecc22daa4b948
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11505
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2024-04-22 20:28:55 +00:00
Ryan Lahfa
a919efbe11 fix(tvix/eval): fix the unsafeGetAttrPos stub return value
It was returning `{ line, col, file }` instead of `{ line, column, file
}`.

Note this is not a regression in Nix, Nix always had `column` since
2013.

Change-Id: I4871a201b32324dff0432526d0449806a535828f
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11409
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-13 13:16:02 +00:00
Ryan Lahfa
863c4207cc feat(tvix/eval): contextful JSON operations
`toJSON` transform a Nix structure into a JSON string.

For each context in that Nix structure, the JSON string must possess it.

Thus, it is necessary to take the union of all contexts and attach it to
the final structure.

Unfortunately, the return type of `into_json` is a serde's JSON object,
not a string. Therefore, it is not possible to reuse `NixString`
machinery.

Context tests are reinforced as Nix does not test those behaviors.

Fixes b/393.

Change-Id: I5afdbc4e18dd70469192c1aa657d1049ba330149
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11266
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-13 10:11:18 +00:00
Florian Klink
839c971a0f docs(tvix/eval): remove link to cargo manifest reference
We don't have it in all Cargo.toml files either.

Change-Id: Ifcb333a39603a7e402ab0068e0f0f1b5af769e13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11388
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
2024-04-13 10:09:35 +00:00
Florian Klink
f8800ba189 chore(tvix): bump rstest to 0.19.0
Change-Id: Ib2f5e84fdb8be1210b3507da67d4fe84f061651e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11387
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-04-12 22:16:56 +00:00
Connor Brewster
63116d8c21 fix(tvix): Avoid buffering file into memory in builtins.hashFile
Right now `builtins.hashFile` always reads the entire file into memory
before hashing, which is not ideal for large files. This replaces
`read_to_string` with `open_file` which allows calculating the hash of
the file without buffering it entirely into memory. Other callers can
continue to buffer into memory if they choose, but they still use the
`open_file` VM request and then call `read_to_string` or `read_to_end`
on the `std::io::Reader`.

Fixes b/380

Change-Id: Ifa1c8324bcee8f751604b0b449feab875c632fda
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11236
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-09 17:31:58 +00:00
Ryan Lahfa
e7af8e0d62 feat(tvix/eval): implement appendContext
`appendContext s ctx` will just append a user-crafted context attrs to `s`.

The most important part of this builtin is to perform all the relevant invariant validations
to avoid letting the user craft invalid contexts which can never be built, e.g. invalid store paths,
inexistent derivations, etc.

This version is incomplete and full of TODOs, but passes all the Nix's
context strings tests, so we turn them on.

Change-Id: I625dc5e7c4f5b784f078b390f04b0ee5a8d65a7c
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11263
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-04 17:24:26 +00:00
Ryan Lahfa
8bccc9dca9 feat(tvix/eval): implement unsafeDiscardOutputDependency
This builtin only transforms any `NixContextElement::Derivation` into the trivial `NixContextElement::Plain`.

This is a forgetful functor on derivation-deep context strings.

The test coverage of this change is done in cl/11264.

Change-Id: Icd00778c97766be6db8a6bdabaa59e9724353ec5
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11262
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-03-31 12:40:49 +00:00