Commit graph

18149 commits

Author SHA1 Message Date
Florian Klink
f2afd38f2d docs(tvix): move mg shell command into one line
This was reflowed in a funny way. Move the whole command into its own
line, to prevent it from happening.

Change-Id: Ifba4daf418487ca4c32586820071930d29020f42
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8026
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-03 12:07:07 +00:00
Vincent Ambo
0dd07440b4 test(tvix/cli): eval nixpkgs.stdenv in CI and assert that it matches
Change-Id: If80194b5fdbf69512217bd4780e416e678045323
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8023
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-02-03 10:47:18 +00:00
Vincent Ambo
3419f63575 fix(tvix/eval): ensure all evaluated thunks are correctly memoized
This fixes a very complicated bug (b/246). Evaluation
progresses *much* further after this, leading to several less
complicated bugs likely being uncovered by this

What was the problem?
=====================

Previously, when evaluating a thunk, we had a code path that looked
like this:

    match *thunk {
        ThunkRepr::Evaluated(Value::Thunk(ref inner_thunk)) => {
            let inner_repr = inner_thunk.0.borrow().clone();
            drop(thunk);
            self.0.replace(inner_repr);
        }
        /* ... */
    }

This code path created a copy of the inner `ThunkRepr` of a nested
thunk, and moved that copy into the `ThunkRepr` of the parent.

The effect of this was that the original `ThunkRepr` (unforced!) lived
on in the original thunk, without the memoization of the subsequent
forcing applying to it.

This had the result that Tvix would repeatedly evaluate these thunks
without ever memoizing them, if they occured repeatedly as shared
inner thunks. Most notably, this would *always* occur when
builtins.import was used.

What's the solution?
====================

I have completely rewritten `Thunk::force_trampoline_self` to make all
flows that can occur in it explicit. I have also removed the outer
loop inside of that function, and resorted to more use of trampolining
instead.

The function is now well-commented and it should be possible to read
it from top-to-bottom and get a general sense of what is going on,
though the trampolining itself (which is implemented in the VM) needs
to be at least partially understood for this.

What's the new problem(s)?
==========================

One new (known) problem is that we have to construct `Error` instances
in all error types here, but we do not have spans available in some
thunk-related situations. Due to b/238 we cannot ask the VM for an
arbitrary span from the callsite leading to the force. This means that
there are now code paths where, under certain conditions, causing an
evaluation error during thunk forcing will panic.

To fix this we will need to investigate and fix b/238, and/or add a
span tracking mechanism to thunks themselves.

What other impacts does this have?
==================================

With this commit, eval of nixpkgs mostly succeeds (things like stdenv
evaluate to the same hashes for us and C++ Nix, meaning we now
construct identical derivations without eval breaking).

Due to this we progress much further into nixpkgs, which lets us
uncover more additional bugs. For example, after this commit we can
quickly see that cl/7949 introduces some kind of behavioural issue and
should not be merged as-is (this was not apparent before).

Additionally, tvix-eval is now seemingly very fast. When doing
performance analysis of a nixpkgs eval, we now mostly see the code
path for shelling out to C++ Nix to add things to the store in there.
We still need those code paths, so we can not (yet) do a performance
analysis beyond that.

Change-Id: I738525bad8bc5ede5d8c737f023b14b8f4160612
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8012
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-02-03 10:47:18 +00:00
Vincent Ambo
38e8c2e959 fix(tvix/cli): keep tracking full paths in known_paths
We need to distinguish explicitly between the paths used for the
scanner, and the paths that populate the derivation inputs. The full
paths must be accessible from the result of the refscanner to populate
drv fields correctly.

This was previously hidden by debug changes that masked actual IO
operations with no-ops.

Change-Id: I037af6e6bbe2b573034d695f8779bee1b56bc125
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8022
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-02-02 23:37:34 +00:00
Vincent Ambo
e6235e2932 feat(tvix/cli): cache imported paths in NixCompatIO
Creates a cache of imported literal files (e.g.
`./default-builder.sh`) which avoids shelling out to Nix for each
instance of the same file.

Note that a better way to tackle this is to create memoizable thunks
for these expressions in the compiler, but we are lacking a little bit
of infrastructure for that at the moment.

Change-Id: Ibc062b20d81e97dd3986e734d225a744e1779fe7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8015
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-02-02 17:50:44 +00:00
Vincent Ambo
9d6f29a72b refactor(tvix/cli): use Wu-Manber string scanning for drv references
Switch out the string-scanning algorithm used in the reference scanner.

The construction of aho-corasick automata made up the vast majority of
runtime when evaluating nixpkgs previously. While the actual scanning
with a constructed automaton is relatively fast, we almost never scan
for the same set of strings twice and the cost is not worth it.

An algorithm that better matches our needs is the Wu-Manber multiple
string match algorithm, which works efficiently on *long* and *random*
strings of the *same length*, which describes store paths (up to their
hash component).

This switches the refscanner crate to a Rust implementation[0][1] of
this algorithm.

This has several implications:

1. This crate does not provide a way to scan streams. I'm not sure if
   this is an inherent problem with the algorithm (probably not, but
   it would need buffering). Either way, related functions and
   tests (which were actually unused) have been removed.

2. All strings need to be of the same length. For this reason, we
   truncate the known paths after their hash part (they are still
   unique, of course).

3. Passing an empty set of matches, or a match that is shorter than
   the length of a store path, causes the crate to panic. We safeguard
   against this by completely skipping the refscanning if there are no
   known paths (i.e. when evaluating the first derivation of an eval),
   and by bailing out of scanning a string that is shorter than a
   store path.

On the upside, this reduces overall runtime to less 1/5 of what it was
before when evaluating `pkgs.stdenv.drvPath`.

[0]: Frankly, it's a random, research-grade MIT-licensed
     crate that I found on Github:

     https://github.com/jneem/wu-manber

[1]: We probably want to rewrite or at least fork the above crate, and
     add things like a three-byte wide scanner. Evaluating large
     portions of nixpkgs can easily lead to more than 65k derivations
     being scanned for.

Change-Id: I08926778e1e5d5a87fc9ac26e0437aed8bbd9eb0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8017
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-02-02 17:50:44 +00:00
Vincent Ambo
2c07ff0f8c docs(tvix): add more information to README
The README was very sparse before and we've actually had people email
us (as it says to contact us) just to ask what Tvix *is*. This should
answer some questions!

Change-Id: I0f248cb060eccfe086468afed1d648652b35dfd1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8018
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
2023-02-02 16:25:52 +00:00
Vincent Ambo
503e6810e7 fix(tvix/eval): unsafeDiscardStringContext is a no-op
... not just a TODO.

Most use-cases of unsafeDiscardStringContext are for cases where a
string is processed in some ways and no longer contains a "physical"
reference, but still has its context attached in C++ Nix.

We don't need to do this. This does diverge in behaviour in use-cases
related to build scheduling, but that whole behaviour will be
different in Tvix.

Change-Id: I4056d4c09f62d44d6bd52b791db03fe5556672b5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8016
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-02 14:30:32 +00:00
Vincent Ambo
75d1b72b7c refactor(tvix/eval): import_cache can be a HashMap
... instead of a BTreeMap, as we do not need ordering guarantees here.
HashMaps are noticeably faster here (especially as we've been sorting
essentially random data!).

Change-Id: Ie92d74286df9f763c04c9b226ef1066ee8484c13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8014
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-02-02 14:25:30 +00:00
Vincent Ambo
98ea614c6e chore(tvix/eval): elaborate on internal types in Value::type_of
This aids in debugging quite substantially.

Change-Id: Ic43232aa6165ae1c3db7ac2701938e1dfeeb418c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8013
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
2023-02-02 09:29:43 +00:00
Florian Klink
90f847ad32 chore(corp/tvixbolt): update Cargo.lock
Change-Id: Ie6bb2df16f79f7d977a7e95187a6b81e3ac0108f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8011
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-02-01 23:06:02 +00:00
Florian Klink
6abd1e2dae chore(tvix): update Cargo.lock
Change-Id: I08b2dc2393819c4f0d3871fe05bdbd29e89a6d75
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8010
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-01 23:06:02 +00:00
Vincent Ambo
dc1e86001c fix(tvix): add dummy target to attach extra-step to
Change-Id: I594a6652e2efe7aa6e35c7cdd84fc3097660614f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8009
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-02-01 17:34:08 +00:00
Florian Klink
30c4454056 fix(ops/buildkite): set default_branch explicitly
It looks like this needs to be set for the tvix pipeline to succeed.

It was set to `canon` for `tvl-kit` (not sure if manually, or some
autodetection previously did it for us that's not present anymore).

Anyways, this sets it to how it's set in the web interface, to hopefully
fix it.

Change-Id: Ic3eb60e3f421fa949a84dcdaa928823ff45f679a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8008
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-02-01 17:25:06 +00:00
Vincent Ambo
87de3fff52 chore(tvix): mirror exported repository
Pushes a mirror of tvix to github/tvlfyi/tvix

Change-Id: I86e7c1b1dc1fcc1012fa72234588b26ac2554056
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8007
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-02-01 17:21:05 +00:00
Florian Klink
88d86741dc feat(ops/pipelines): trigger tvix buildkite pipeline
Change-Id: I4e81694b9686f977a6590c5e1703a4ef413b0cf4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8003
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-01 17:11:50 +00:00
Florian Klink
2c4554679c feat(views/tvix): add buildkite pipeline
Change-Id: Iabfc8c64012500a39aaafb3976ceb4a249387ee8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8002
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-01 17:11:50 +00:00
Florian Klink
4a7ec1006d feat(ops/buildkite): add tvix pipeline
Change-Id: Ie701e0b77c596e07600efd1a59749d05068f0dbc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8006
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-01 17:11:50 +00:00
Florian Klink
5ac33a227f docs(tvix): update README to document usage inside tvix view
If the tvix view is cloned through josh, you don't use mg, but a
`shell.nix` is provided.

Also, add the `git clone` command, so people browsing tvix source code
in the browser know where to clone from.

Change-Id: I18483d6a52953f9f4eafd1533ea69afb0e329b04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8001
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-01 17:11:50 +00:00
Florian Klink
dbabf728b7 feat(views/tvix): add tvix josh workspace
This provides a trimmed-down version of //tvix, that allows building
tvix without all of the monorepo cloned.

We need to manually vendor in our patch for evans and ``--bytes-as-
base64`, as we don't have our depot overlays available in the josh
workspace.

Fixes https://b.tvl.fyi/issues/247.

Change-Id: I24306b8bb16ebf0df238b8f1eee0d47655a14827
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8000
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-01 17:11:50 +00:00
Vincent Ambo
8ed81cf755 feat(ops/secrets): add flokli to terraform secrets access
Change-Id: I9ede20028560f2da0fef89dfe431609c21bda51c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8005
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-02-01 16:39:32 +00:00
Florian Klink
f5ccb65f26 feat(tvix/nix-compat/derivation): Display -> to_aterm_string()
Instead of implementing `std::fmt::Display for Derivation` and relying
on the `to_string` method, introduce a `to_aterm_string()` method, which
does the same thing, but makes it clearer what we're producing, rather
than just calling `to_string()``.

Change-Id: I21823de9096a0f2c2eb6f4591e48c1aa9fd94161
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7998
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-01 16:31:56 +00:00
Florian Klink
964367044d feat(ops/secrets): add key for flokli
Change-Id: I52299b39d1d68ee1b700b631f70ef809af682e26
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8004
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-02-01 16:25:54 +00:00
Florian Klink
99d5cf822a refactor(tvix/cli): use nixhash module for output hash calculation
This covers all the weird corner cases.

Change-Id: I85637e82e8929828064ab562dc8a1c8bf161fffa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7991
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-01 15:29:48 +00:00
Florian Klink
265dd36b98 feat(tvix/nix-compat): add nixhash module
This module takes care of parsing various hashes and algorithms.

It will get used to modify derivation output hashes in the next CL.

Change-Id: Idc07c401dbb7510f49883ac02b8379b9a5d930c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7990
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-01 15:29:48 +00:00
Florian Klink
366d2c5e92 refactor(tvix/nix-compat/derivation): remove module inception
The module `src/derivation/derivation.rs` is a sign of module inception.

Move the Derivation struct definiton up into `src/derivation/mod.rs`,
and some of the helpers in a `util.rs`.

Change-Id: Ib24a5f8a27bdd45df8b1fa2b3482a79b33cab8d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7997
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-02-01 13:43:25 +00:00
Florian Klink
2551adb4f1 refactor(tvix/nix-compat): operator precedence can trip the unwary
warning: operator precedence can trip the unwary
  --> nix-compat/src/nixbase32.rs:41:23
   |
41 |                 c |= ((input[i + 1] as u16) << 8 - j as u16) as u8
   |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: consider parenthesizing your expression: `(input[i + 1] as u16) << (8 - j as u16)`
   |
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#precedence
   = note: `#[warn(clippy::precedence)]` on by default

Change-Id: I091071d649abf4ed38f5f4e39a0c5d21a0459bff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7996
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-01 13:43:24 +00:00
Florian Klink
361ffd7903 refactor(tvix/derivation): don't create deref'd immediately ref
clippy says:

> This expression creates a reference which is immediately dereferenced
> by the compiler

Change-Id: Ic2c093b043ebee9ae80912075083107e4d216cf1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7995
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-01 13:38:21 +00:00
Florian Klink
ce4d87b129 refactor(tvix/nix-compat): remove unneeded return
Change-Id: I97b9fb3ad33d2f51baf29486d4eff11f8dedcbab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7994
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-02-01 13:38:19 +00:00
Florian Klink
894d17b29b refactor(tvix/nix-compat): const is always 'static
Change-Id: If33bbf47a6cc376f2c0935ea4331a819223b2f00
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7993
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-01 13:38:18 +00:00
Florian Klink
3091ac69f4 chore(third_party/nixpkgs): drop permittedInsecurePackages
subsurface has been fixed a while ago to not pull in qtwebkit anymore,
this can be dropped.

Change-Id: I173dda71770d02ce8064d1751aff889475d12dfb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7999
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: grfn <grfn@gws.fyi>
2023-02-01 11:50:58 +00:00
Vincent Ambo
759f9dbf39 feat(tvix/cli): implement builtins.placeholder
This doesn't require any other corresponding handling *yet*, as the
actual replacements happen in the builder logic (which we delegate to
cppnix at the moment).

Change-Id: I034147c933f05ae427c7a8794647132d108d0ede
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7972
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-02-01 10:01:40 +00:00
Vincent Ambo
3caa4c4aa4 fix(ops/www): increase buffer memory size for auth.tvl.fyi
Keycloak seems to have decided today that it will now send headers
that are larger than what the nginx default configuration can handle.

The numbers are a mix of made up and taken from random nginx voodoo
posts on the internet, so they're as good a guess as anyone's.

Change-Id: If037bcba48eee371cc96304b150276c669930c75
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7992
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
2023-02-01 09:30:24 +00:00
Florian Klink
5f62cd1ad5 refactor(tvix/serde): allow dead_code in struct
This is an example, which uses the debug trait to print all field
values.

Silence the compiler warning about unused fields.

Change-Id: I5f1216c77819003302e83ba1af1ff13c924f3b38
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7971
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-01-31 15:35:46 +00:00
Florian Klink
a7e20bbef3 docs(tvix/nix-compat/store_path): fix docstrings
Change-Id: I19261233e338b406a87fe680675eb4dfd853fe11
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7970
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-01-31 15:35:45 +00:00
Vincent Ambo
ddf6e6cf47 fix(tvix/eval): allow builtins.toXML to serialise any function
This adds a fake argument name to builtins.toXML which allows toXML to
serialise any value instead of panicking on functions. We do still
have to fix the value itself, eventually, though.

Change-Id: I2e330ecddcd80442b4fac5eced64431ac86123ba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7962
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-31 15:31:30 +00:00
Florian Klink
0db73cb2bd feat(tvix/store): add write_nar function
This adds a function that consumes a [proto::node::Node] pointing to the
root of a (store) path, and writes the contents in NAR serialization to
the passed [std::io::Write].

We need this in various places:

 - tvix-store's calculate_nar() RPC method needs to render a NAR stream
   to get the nar hash, which is necessary to give things imported in
   the store a "NAR-based" store path.

 - communication with (remote) Nix (via daemon protocol) needs a NAR
   representation.

 - Things like nar-bridge, exposing a NAR/NARInfo HTTP interface need a
   NAR representation.

Change-Id: I7fb2e0bf01814a1c09094c0e35394d9d6b3e43b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7956
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 15:28:22 +00:00
Florian Klink
a23b7e17c0 refactor(tvix/nix-compat): absorb nar writer
Expose it at `nix_compat::nar::writer`.

Change-Id: I98a8acfa6277074f24608d64fb6c0082b41d10f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7969
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 15:18:39 +00:00
Florian Klink
1e28cc7ede docs(tvix/nix-compat/derivation): fix docstrings
Change-Id: I3fd02e62c8a24fa7f27461b17cdd824b060d9bdb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7968
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-01-31 15:16:40 +00:00
Florian Klink
2d24c5f260 refactor(tvix/nix-compat): absorb //tvix/derivation
Put this in its src/derivation.

Change-Id: Ic047ab1c2da555a833ee454e10ef60c77537b617
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7967
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-01-31 15:16:39 +00:00
Florian Klink
9e809e21cc feat(tvix/store): implement iteration over Directory nodes
This provides a `Directory.nodes()` function, returning an iterator over
all three node types in an ordered fashion.

Change-Id: Ib98696c03a9db8b6c613d6e2bf5587c1ae35133f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7955
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 13:29:18 +00:00
Florian Klink
1ee6bd06e3 feat(tvix/store/proto): implement get_name for node::Node, pub trait
Make the trait public, so consumers can use it.

Also, implement it for node::Node, so we can later use this to access
the name from all three if we don't care about the enum type.

Change-Id: Iae530a16b705493645e61947852c03273876cc55
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7963
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 13:29:18 +00:00
Florian Klink
c27bacd905 refactor(tvix): introduce nix-compat crate
Move nixbase32 and store_path into this.

This allows //tvix/cli to not pull in //tvix/store for now.

Change-Id: Id3a32867205d95794bc0d33b21d4cb3d9bafd02a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7964
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-31 13:26:18 +00:00
Florian Klink
8ea93bb646 refactor(tvix/cli/derivation): use if let to destructure
We only do logic here if algo and hash_mode are Some(_)
(and there's an `out` output).

The fact we don't do anything in all in other cases is a bit hidden at
the bottom. Use if let for the destructuring, and drop the other case.

Change-Id: Icc0e38e62947d52d48ef610f754749737977fca9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7966
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 13:26:11 +00:00
Florian Klink
6fa91349a9 refactor(tvix/cli): remove unneeded clone
Change-Id: I6f4cb24bdd636af8918a2ade44075af92161c97d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7965
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 13:26:11 +00:00
Vincent Ambo
4da1e64c03 feat(tvix/serde): add an example application to the project
This shows how people can use tvix_serde to deserialise configuration
structs for their programs from Nix code.

Change-Id: I71bf4e03dce19dddafe67dd729b4e4b10719a739
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7945
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-31 13:11:28 +00:00
Vincent Ambo
7702941dd9 test(tvix/eval): add tests for internal formals dependencies
Formals can depend on each other when using another formal as a
default value.

This test ensures that the compiler's declaration and initialisation
order of formals is consistent with what actually happens in the VM.

Change-Id: Ibdabe262554e8066d67fac1ebc3b5a48ef626e18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7948
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-31 10:25:02 +00:00
Florian Klink
4b3ccd205a refactor(tvix/cli): absorb construct_output_hash
This helper function only was created because
populate_output_configuration was hard to test before cl/7939.

With that out of the way, we can pull it in.

Change-Id: I64b36c0eb34343290a8d84a03b0d29392a821fc7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7961
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-31 08:52:00 +00:00
Vincent Ambo
e4bb750b3b refactor(tvix/cli): force outside of output configuration helper
Change-Id: I28357fe131cefedcef9761b08a72f675f4a10789
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7939
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 08:51:03 +00:00
Florian Klink
8ec8d03175 fix(tvix/store/nixbase32): fix encoder/decoder
Replace our data_encoding usage with the implementation taken from
https://github.com/nix-community/go-nix/tree/master/pkg/nixbase32

Also uncomment some of the unit tests, and add a regression test for a
NIXBASE32.encode with a 32 bytes sequence.

The previous implementation of NIXBASE32.encode is wrong in that case:

```
❯ nix hash to-base32 sha256-s6JN6XqP28g1uYMxaVAQMLiXcDG8tUs7OsE3QPhGqzA=
0c5b8vw40dy178xlpddw65q9gf1h2186jcc3p4swinwggbllv8mk
❯ echo -n s6JN6XqP28g1uYMxaVAQMLiXcDG8tUs7OsE3QPhGqzA= | base64 -d | hexdump -C
00000000  b3 a2 4d e9 7a 8f db c8  35 b9 83 31 69 50 10 30  |..M.z...5..1iP.0|
00000010  b8 97 70 31 bc b5 4b 3b  3a c1 37 40 f8 46 ab 30  |..p1..K;:.7@.F.0|
00000020
```

Change-Id: I0468b62bbbab390f8d7d3812e657e5d59dceed59
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7934
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-30 11:46:30 +00:00