This was written with the same intention (and reuses a little of its
code) as cl/5060 and cl/5063: We want to be able to emit dependencies
between //nix/buildkite pipeline steps, so that no agent is occupied
with waiting on locks for derivations built by a different agent.
This dependency information is already available to the Nix store
implementation (e.g. via `nix-store --query --references`) and can also
be obtained in the Nix language which is important, since the pipeline
is generated at evaluation time. (Note: For Nix 2.3, you either need a
strong convention about how derivations expose their dependencies (which
we don't) or rely on store implementation internals (drv files).
For Nix 2.6 there is a better trick, but it also relies on the existence
of drv files.)
The actual task can be formulated as follows: Given a set of
derivations, calculate the the closest derivations also in the input
each derivation depends on. (We call these (next) known dependencies.)
This is crucial because pipeline step often depend on each other only
indirectly with any number of intermediate derivations. For cl/5064 I
determined that 6 intermediate layers is quite common for dependencies
that are perceived to be “direct”.
This problem is solved as follows:
1. Calculate the dependency graph of the combined dependency closure of
all input derivations. This is quite easy and fairly quick thanks to
the C++ implementation of builtins.genericClosure. One weak point of
the current implementation is that the function to determine the
direct derivation dependencies for Nix < 2.6 is quite hacky.
2. Take the graph from 1. and calculate a dependency graph that only
connects the known derivations of the input, but retains all
connections between them (minus intermediate nodes).
In practice the dependency graph is represented as an attribute set
mapping derivation paths to a list of derivation paths it depends on.
The second step is performed by adding a second list of known derivation
paths it depends on.
The main improvements over the previous concept (cl/5060 and cl/5063):
* We only try to find the closest known dependencies in the dependency
graph whereas we would traverse emit dependencies for the entire
dependency closure.
* We immediately store the calculation of the closest known dependency
in the dependency graph, even for intermediate nodes. This avoids
recalculating the connection (which was a big drawback of the previous
approach) and makes the calculation itself cheaper.
You can run `mg build //nix/dependency-analyzer:example` to build a
visualization of the internal dependencies between `depot.ci.targets` as
discovered by dependency-analyzer.
Change-Id: If8c0cdfc8470d4b337336257d9818aaa0d51110f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6832
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This is already allowed de facto, since there seems to be a special
exception for reading from derivation outputs. What is forbidden, is
access to files imported to the store (even via builtins.toFile) and
derivation files. The latter is required for doing dependency analysis
on arbitrary derivations, unfortunately.
Access to the store allows kind of evil things, but it should
be (hopefully) hard to do this by accident, and accessing derivation
files is not impure, though it relies on store implementation internals
so to speak.
Change-Id: I33a7de83ef0ee20a7076690329d62f6caffffe5f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6835
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
This logic was duplicated between the two rnix types before, but more
crucially - it is also needed for correctly displaying the text ranges
contained in syntax errors.
Change-Id: Ifc6a521de1594d6ced9cba6274f1931df99b6634
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6870
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
This is also useful for error-handling related logic, outside of just
the compiler module.
Change-Id: I5c386e2b4c31cda0a0209b31136ca07f00e39e45
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6869
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Mostly good, but I'm not sure it works with `git diff --patch`. Not a big deal
though if true.
Change-Id: I268c52dd253f5b0f9dd462a1825206da5dd86dd2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6889
Reviewed-by: wpcarro <wpcarro@gmail.com>
Autosubmit: wpcarro <wpcarro@gmail.com>
Tested-by: BuildkiteCI
i always seem to get this wrong
Change-Id: Ib6f31523aba1d9f9a32d9af95b96b8d75e0ec16e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6863
Reviewed-by: wpcarro <wpcarro@gmail.com>
Autosubmit: wpcarro <wpcarro@gmail.com>
Tested-by: BuildkiteCI
This enables the use of string paths (and, in the future,
derivations), as long as their string values represent an absolute
path.
Change-Id: I4b198efeb70415ed52f58bd1da6fa79a24dad14c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6866
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This lets the VM emit warnings when it encounters situations that
should only be warned about at runtime.
For starters, this is used to pass through compilation warnings that
come up when `import` is used.
Change-Id: I0c4bc8c534d699999887c430d93629fadfa662c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6868
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Adding `import` to builtins causes causes a bootstrap cycle because
the `import` builtin needs to be initialised with the set of globals
before being inserted into the globals, which also must contain
itself.
To break out of the cycle this hack wraps the builtins passed to the
compiler in an `Rc` (probably sensible anyways, as they will end up
getting cloned a bunch), containing a RefCell which gives us mutable
access to the builtins.
This opens up a potentially dangerous footgun in which we could mutate
the builtins at runtime leading to different compiler invocations
seeing different builtins, so it'd be nice to have some kind of
"finalised" status for them or some such, but I'm not sure how to
represent that atm.
Change-Id: I25f8d4d2a7e8472d401c8ba2f4bbf9d86ab2abcb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6867
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Skip setting bqn-interpreter-path on 32bit – contrary to the
LanguageTool integration, bqn-mode is still useful without the binary
which doesn't compile on i686-linux.
Change-Id: If4493e3e72307ca14984c660f376952cbdcc201c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6887
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Given that the laptop I need this for is really slow, I may want to
disable even more stuff, but I'll take it step by step. This should at
least make it possible to build its system closure.
Change-Id: I50c55fa3426252e7f23f419bb2009d58a9312a98
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6876
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This adds an initial working version of builtins.import which
encapsulates the entire functionality of `import` within the builtin
itself, without requiring any changes in the compiler or VM.
The key insight that enables this is that we can simply return a Thunk
from `import` that is constructed from the output of running the
compiler and - ta-da! - no other component needs to know about it.
A couple of notes:
* builtins.import needs to capture variables like the SourceCode
structure. This means it can not currently be constructed the same
way as other builtins and has special handling, which leaks out to
`eval.rs`. I have postponed dealing with that until we have this
working a bit more.
* the `globals` are not yet passed through
* the error representation for the new variants is absolutely not done
yet, we probably want to switch to something that supports
cause-chaining now (like miette)
* there is no mechanism for emitting warnings at runtime; we need to
add that
Change-Id: I3117a7ae3ff2432bf44f5ff05ad35f47faca31d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6857
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: wpcarro <wpcarro@gmail.com>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
For some upcoming builtins (notably, import) we need to capture
arguments in the builtin's implementation.
To allow this, we can no longer use function pointers for builtins,
but must use a reference-counted closure object instead.
Unfortunately this adds an extra pointer operation to every builtin
call. We should benchmark this later against having a split builtin
representation.
Change-Id: I109d98d0e25998870542f47573eb1ec2e546f2a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6856
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
The text was a little cramped, which made the font hard to read. If
one gives it a little more breathing space, it gets easier to digest.
I couldn’t check the change locally, since `-A web.tvl` doesn’t
reference the static assets (it hardlinks to `static.tvl.su` from what
I can see). I only tested it directly in the browser css editor and
then added the values I found here.
Change-Id: Ic3cb78b2ed0f37e1c55ba70027fec2c62b43a52f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4055
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
nixpkgs has hardcoded references to Nix versions, we need to provide
it with something that looks like a Nix version while actually being a
Tvix version.
For now, we do this by stealing a trick out of the browser book and
constructing a version that looks like a Nix version to Nix, but like
a Tvix version to people who know what they are looking for.
Nevermind that we don't actually have any kind of versioning for
Tvix (yet?), other than depot revisions.
Change-Id: I7ce8079dd8164a2079891d38e707f09a45f0bbc1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6858
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
* //3p/overlays/tvl: apply more or less questionable compilation fix for
electrum with protobuf >= 4.
Change-Id: Iedbc45cb96b9e5382e304f2cd5ed9fc1c13f7884
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6850
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
The convention has changed to use `final` instead of `self` and `prev`
instead of `super`. This new convention is much easier to understand
especially for users that are new to Nix and Nixpkgs.
This change is notable in the Nixpkgs project code. I added a note
mentioning that the official documentation hasn't been update with
this information. The documentation is currently undergoing an
overhaul and that is why I believe this hasn't been reflected in the
nixpkgs documentation as of yet.
Imported-From: https://github.com/tazjin/nix-1p/pull/9/commits
Change-Id: I901df69a564969b1784b3d7ac629cfaf85093acb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6859
Reviewed-by: Profpatsch <mail@profpatsch.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This type hides away the lower-level handling of most codemap data
structures, especially to library consumers (see corresponding changes
in tvixbolt).
This will help with implement `import` by giving us central control
over how the codemap works.
Change-Id: Ifcea36776879725871b30c518aeb96ab5fda035a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6855
Tested-by: BuildkiteCI
Reviewed-by: wpcarro <wpcarro@gmail.com>
There's basically nothing that needs *ownership* of an AST
node (which is just a little box full of references to other things
anyways), so we can thread this through as references all the way.
Change-Id: I35a1348a50c0e8e07d51dfc18847829379166fbf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6853
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
There are actually two different types of observers, the ones that
observe the compiler (and emitted chunks from different kinds of
expressions), and the ones that trace runtime execution.
Use of the NoOpObserver is unchanged, it simply implements both
traits.
Change-Id: I4277b82674c259ec55238a0de3bb1cdf5e21a258
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6852
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Not sure why/how I ended-up with both dotfiles *and* configs, but I don't really
like working with `stow`, so I'm preferring dotfiles.
Change-Id: Id00566ea8206eef65b9a27dd4765c2d17f3d2317
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6865
Reviewed-by: wpcarro <wpcarro@gmail.com>
Autosubmit: wpcarro <wpcarro@gmail.com>
Tested-by: BuildkiteCI
Until we can display a chained representatino of errors in thunks, it
is most useful to forward the error code from the innermost error to
the user.
Change-Id: I8d67254d52313be40387f080e57966c001e0d51c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6854
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Returns time since epoch in seconds.
This has a slight behaviour difference from Nix, in that we don't pin
the time between REPL entries (Nix pins it for the program lifetime),
but this is probably inconsequential as long as it is pinned during an
evaluation.
Change-Id: I010c02e93097a209d8ad69e278397c7e30e54c86
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6846
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: wpcarro <wpcarro@gmail.com>
Allows impure builtins that have a different shape than a Rust
function pointer; specifically this is required for
builtins.currentTime which does not work in WASM.
Change-Id: I1362d8eeafe770ce4d1c5ebe4d119aeb0abb5c9b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6849
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: wpcarro <wpcarro@gmail.com>
Concatenates (but not flattens) a list of lists.
Change-Id: I692e0b3e7b5a5ff93d5768d3a27849b432ec5747
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6843
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This is the same code as before, just moved into a trait impl to gain
access to stuff that needs IntoIterator
Change-Id: Iff9375cd05593dd2681fa85ccc7f4554bf944a02
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6842
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This can actually legitimately be emitted by the compiler currently
when compiling formals with default values. See the scope6 test from
the Nix test suite for an example.
We should restructure this slightly to be able to reintroduce a
runtime error here in case something was compiled incorrectly.
Change-Id: Ib81f0f58ae0e850db9fbc459458b7bd0d3ac6f23
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6841
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This implements __functor calling in situations where `OpTailCall` is
used, but not yet for `OpCall`.
For some reason I have not yet figured out, this same implementation
does not work in call_value, which means that it also doesn't yet work
in builtins that apply functions.
Change-Id: I378f9065ac53d4c05166a7d0151acb1f55c91579
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6826
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Add a new derivation target to the passthru of tvix.eval that builds the
benchmark binaries, *and* copies them to the outupts of the derivation
via the (somewhat arcane) `copyBinsFilter` jq script arg to naersk. This
is a bit annoying because (as far as I can tell) the derivations
returned by naersk aren't directly overridable, so we have to explicitly
fixpoint the attrs we're passing.
Also, since this is now a separate target to build the benchmarks, we
can remove `--all-targets` from the build of `tvix-eval` itself since
that was only added to build benchmarks in CI, and make
regular (non-benchmark) builds a bit faster.
Change-Id: I136b8526790545e93b1ae666abaefb51cbbee390
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6847
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This option actually doesn't change behaviour in xss-lock, which I'm
using instead of xautolock, so this is a no-op.
Relates to b/200
Change-Id: Ia5b155bf80fcd092038ed2c95517e64d562d0219
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6840
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
The latter has been deprecated in nixpkgs.
Relates to b/200
Change-Id: I42871ce3eb54ebf092909f033b43936b9610d982
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6836
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This should take care of the chrono advisory which has finally become
actionable.
Change-Id: I0c290c10893d2b112bc17281a96c760b62dff02f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6831
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This should take care of the chrono advisory which finally has become
actionable.
Change-Id: I801330b4dc5273a76f88e3e1eae4ca5237f9a070
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6830
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
