Commit graph

18067 commits

Author SHA1 Message Date
Vincent Ambo
c98c5399b9 fix(tvix/eval): skip runtime completely on compiler errors
This branch was missing, and an assumption elsewhere just executed the
returned (broken) bytecode.

This fixes b/253.

Change-Id: I015023ba921bc08ea03882167f1f560feca25e50
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8090
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
2023-02-13 16:21:47 +00:00
Vincent Ambo
91a366af46 fix(tvix/eval): make fields of eval's Error type public
These should be inspectable by callers.

Change-Id: Ia9ef871aa63958d06066aaea61b2aecbd217369b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8089
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-02-13 16:21:46 +00:00
Alyssa Ross
3c68159b81 chore(whitby): enable zram swap
Whitby has a lot of memory, but I've still been fighting with the OOM
Killer trying to build a few big packages at the same time.  Besides,
it's generally a good idea to always have swap available even if
there's lots of memory for caching optimisation reasons[1], and zram
swap is efficient enough to basically provide bonus memory for free.

[1]: https://haydenjames.io/linux-performance-almost-always-add-swap-space/

Change-Id: I1fbe60f7975ebfa38e341e0de76848ec79b6fcf0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8065
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-11 12:14:35 +00:00
Alyssa Ross
c4ade691e1 docs(REVIEWS): update scp command
cl.tvl.fyi only supports the legacy scp protocol, not the sftp
protocol that recent versions of OpenSSH now use by default.  So we
need to pass -O to tell it to use scp.

Cc: Vincent Ambo <mail@tazj.in>
Change-Id: I2dcae77fef9126f4f8eebfa574c5a345a5c56615
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8039
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-10 15:15:36 +00:00
Alyssa Ross
08b300aaa8 chore(ops/modules): add a GECOS for my user
This way, I won't have to teach my name one at a time to every program
that wants to know my it (e.g. git).

Change-Id: I45ddd9c2343a10cd4c13bacd9a97b7470db95c14
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8038
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-02-09 09:43:48 +00:00
Florian Klink
d4cbdc2beb feat(tvix/nix-compat/nixhash): support BASE64_NOPAD SRI
Nix accepts SRI hashes that are missing their padding characters
in base64, as seen in
7e49471316/pkgs/development/libraries/kerberos/krb5.nix .

It only seems to work in the SRI case, not with `sha256` being set to a
(nopad) base64 string.

Add regression tests for this, and document why we don't want to support
*additional* characters afterwards.

Reported in https://b.tvl.fyi/issues/252

Change-Id: I9ffc2b417501b426ced1894a9cbf95ff5f0e5159
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8037
Reviewed-by: Alyssa Ross <hi@alyssa.is>
Tested-by: BuildkiteCI
2023-02-08 10:39:26 +00:00
Florian Klink
5fa35d9d49 refactor(tvix/nix-compat/nixhash): fix test function name
Change-Id: I3b089758ce8c01df544064422e56a97a8402513d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8036
Tested-by: BuildkiteCI
Reviewed-by: Alyssa Ross <hi@alyssa.is>
2023-02-08 10:39:26 +00:00
Alyssa Ross
5da1f4161c fix(tvix): fix InvalidEncodedDigestLength message
Before:
	error[E997]: invalid output hash: invalid encoded digest length '43' for algo 43

After: error[E997]: invalid output hash: invalid encoded digest length '43' for algo sha256

Change-Id: Icb533839586b47f90b4e8d22083cce397f9f3655
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8035
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-07 17:40:54 +00:00
Florian Klink
87e1934f2d docs(store/protos): update comment on blobstore Read and Put
Further emphasize Read() can be used to ask for blobs OR chunks, and
that clients usually want to stat and then request (smaller) chunks,
rather than reading whole blobs.

Also clarify that the chunking used to send BlobChunks over has nothing
to do with the chunk sizes communicated in a Stat() request.

Change-Id: Ia615d190aae570611de2655b11342a14d0b75976
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8028
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-07 14:14:35 +00:00
Vincent Ambo
0a27344953 chore(3p/sources): bump nixpkgs & overlays (2023-02-07)
Included fixes:

* //3p/overlays: tdlib override no longer needed (bump has landed upstream)
* //corp/{predlozhnik,tvixbolt}: bump wasm-bindgen to match nixpkgs

Home-manager has not been bumped as it has introduced an
incompatibility with Nix 2.3

Change-Id: I96ac3462b82c73db1ba23be03d7968f10abc9b53
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8033
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-02-07 13:46:35 +00:00
Vincent Ambo
bdf93dcefe fix(nixery): fix link to nixery logo
Change-Id: Ib78659b971696feaff579bc0a31df7d8ee24e459
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8034
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-06 17:36:54 +00:00
Vincent Ambo
720e80c1f9 fix(tvix/eval): fix the default case for path parsing
Plain paths like `foo/bar.nix` are also allowed, so we can not
determine this based on the prefix.

The upstream PR that is referenced in a comment here has a
significantly different interface than we expected, so I'm not
touching that comment yet in this CL before I've had more time to
digest it.

Change-Id: Iea33bbb35de9c00a7d7fedf64d02253c75c1cc9e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8032
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: Alyssa Ross <hi@alyssa.is>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-02-04 12:44:59 +00:00
Vincent Ambo
26b55f8cda fix(tvix/cli): use tvlfyi/wu-manber fork for refscanner
Our fork fixes a small bug (https://github.com/jneem/wu-manber/pull/1)
but it's not clear whether upstream will accept patches, so for now
lets point this directly at our fork.

Change-Id: Iccdcedae3e9a8b783241431787c952561d032694
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8031
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-04 12:44:59 +00:00
Florian Klink
c13759e89f feat(corp/tvixbolt): link to tvl.su too
Make it clear that Tvixbolt is a project of TVL LLC, and link to the
community website too.

See https://b.tvl.fyi/issues/248

Change-Id: Iefefe0263fa5ef01587d49c5a130a38b78ca7981
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8019
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-02-03 19:04:21 +00:00
Florian Klink
459dadfe9d docs(corp/website): distinguish TVL LLC and TVL
See https://b.tvl.fyi/issues/248

Change-Id: Iacc10841bdcdf1dbc8eab659d82e076fa2151979
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8020
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-03 19:04:20 +00:00
Florian Klink
d50044d570 chore(tvix/eval): only use Rc with impure feature
We only use Rc in `impl EvalIO for StdIO`, which is only included when
building with the "impure" feature.

Change-Id: Id29d647c899cbfcdda11abfb9fabd5aa7e24299f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8025
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-03 18:58:17 +00:00
Vincent Ambo
f16b0f24e2 refactor(tvix/eval): wrap Builtin type in a Box
This reduces the size of `Builtin` from 88 (!) bytes to 8, and as the
largest variant of `Value`, the size of that type from 96 to 64.

The next largest type is NixList, clocking in at 64 bytes.

This has noticeable performance impact. In an implementation without
disk I/O, evaluating nixpkgs.stdenv looks like this:

Benchmark 1: tvix -E '(import <nixpkgs> {}).stdenv.drvPath'
  Time (mean ± σ):      1.151 s ±  0.003 s    [User: 1.041 s, System: 0.109 s]
  Range (min … max):    1.147 s …  1.155 s    10 runs

After this change, it looks like this:

Benchmark 1: tvix -E '(import <nixpkgs> {}).stdenv.drvPath'
  Time (mean ± σ):      1.046 s ±  0.004 s    [User: 0.954 s, System: 0.092 s]
  Range (min … max):    1.041 s …  1.053 s    10 runs

Change-Id: I5ab7cc02a9a450c0227daf1f1f72966358311ebb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8027
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-02-03 18:47:33 +00:00
Vincent Ambo
32698766ef refactor(tvix/eval): statically resolve select from constant attrs
When resolving a select expression (`attrs.name` or `attrs.name or
default`), if the set compiles to a constant attribute set (as is most
notably the case with `builtins`) we can backtrack and replace that
attribute set directly with the compiled value.

For something like `builtins.length`, this will directly emit an
`OpConstant` that leaves the `length` builtin on the stack.

Change-Id: I639654e065a06e8cfcbcacb528c6da7ec9e513ee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7957
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-02-03 18:47:33 +00:00
Florian Klink
f2afd38f2d docs(tvix): move mg shell command into one line
This was reflowed in a funny way. Move the whole command into its own
line, to prevent it from happening.

Change-Id: Ifba4daf418487ca4c32586820071930d29020f42
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8026
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-03 12:07:07 +00:00
Vincent Ambo
0dd07440b4 test(tvix/cli): eval nixpkgs.stdenv in CI and assert that it matches
Change-Id: If80194b5fdbf69512217bd4780e416e678045323
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8023
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-02-03 10:47:18 +00:00
Vincent Ambo
3419f63575 fix(tvix/eval): ensure all evaluated thunks are correctly memoized
This fixes a very complicated bug (b/246). Evaluation
progresses *much* further after this, leading to several less
complicated bugs likely being uncovered by this

What was the problem?
=====================

Previously, when evaluating a thunk, we had a code path that looked
like this:

    match *thunk {
        ThunkRepr::Evaluated(Value::Thunk(ref inner_thunk)) => {
            let inner_repr = inner_thunk.0.borrow().clone();
            drop(thunk);
            self.0.replace(inner_repr);
        }
        /* ... */
    }

This code path created a copy of the inner `ThunkRepr` of a nested
thunk, and moved that copy into the `ThunkRepr` of the parent.

The effect of this was that the original `ThunkRepr` (unforced!) lived
on in the original thunk, without the memoization of the subsequent
forcing applying to it.

This had the result that Tvix would repeatedly evaluate these thunks
without ever memoizing them, if they occured repeatedly as shared
inner thunks. Most notably, this would *always* occur when
builtins.import was used.

What's the solution?
====================

I have completely rewritten `Thunk::force_trampoline_self` to make all
flows that can occur in it explicit. I have also removed the outer
loop inside of that function, and resorted to more use of trampolining
instead.

The function is now well-commented and it should be possible to read
it from top-to-bottom and get a general sense of what is going on,
though the trampolining itself (which is implemented in the VM) needs
to be at least partially understood for this.

What's the new problem(s)?
==========================

One new (known) problem is that we have to construct `Error` instances
in all error types here, but we do not have spans available in some
thunk-related situations. Due to b/238 we cannot ask the VM for an
arbitrary span from the callsite leading to the force. This means that
there are now code paths where, under certain conditions, causing an
evaluation error during thunk forcing will panic.

To fix this we will need to investigate and fix b/238, and/or add a
span tracking mechanism to thunks themselves.

What other impacts does this have?
==================================

With this commit, eval of nixpkgs mostly succeeds (things like stdenv
evaluate to the same hashes for us and C++ Nix, meaning we now
construct identical derivations without eval breaking).

Due to this we progress much further into nixpkgs, which lets us
uncover more additional bugs. For example, after this commit we can
quickly see that cl/7949 introduces some kind of behavioural issue and
should not be merged as-is (this was not apparent before).

Additionally, tvix-eval is now seemingly very fast. When doing
performance analysis of a nixpkgs eval, we now mostly see the code
path for shelling out to C++ Nix to add things to the store in there.
We still need those code paths, so we can not (yet) do a performance
analysis beyond that.

Change-Id: I738525bad8bc5ede5d8c737f023b14b8f4160612
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8012
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-02-03 10:47:18 +00:00
Vincent Ambo
38e8c2e959 fix(tvix/cli): keep tracking full paths in known_paths
We need to distinguish explicitly between the paths used for the
scanner, and the paths that populate the derivation inputs. The full
paths must be accessible from the result of the refscanner to populate
drv fields correctly.

This was previously hidden by debug changes that masked actual IO
operations with no-ops.

Change-Id: I037af6e6bbe2b573034d695f8779bee1b56bc125
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8022
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-02-02 23:37:34 +00:00
Vincent Ambo
e6235e2932 feat(tvix/cli): cache imported paths in NixCompatIO
Creates a cache of imported literal files (e.g.
`./default-builder.sh`) which avoids shelling out to Nix for each
instance of the same file.

Note that a better way to tackle this is to create memoizable thunks
for these expressions in the compiler, but we are lacking a little bit
of infrastructure for that at the moment.

Change-Id: Ibc062b20d81e97dd3986e734d225a744e1779fe7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8015
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-02-02 17:50:44 +00:00
Vincent Ambo
9d6f29a72b refactor(tvix/cli): use Wu-Manber string scanning for drv references
Switch out the string-scanning algorithm used in the reference scanner.

The construction of aho-corasick automata made up the vast majority of
runtime when evaluating nixpkgs previously. While the actual scanning
with a constructed automaton is relatively fast, we almost never scan
for the same set of strings twice and the cost is not worth it.

An algorithm that better matches our needs is the Wu-Manber multiple
string match algorithm, which works efficiently on *long* and *random*
strings of the *same length*, which describes store paths (up to their
hash component).

This switches the refscanner crate to a Rust implementation[0][1] of
this algorithm.

This has several implications:

1. This crate does not provide a way to scan streams. I'm not sure if
   this is an inherent problem with the algorithm (probably not, but
   it would need buffering). Either way, related functions and
   tests (which were actually unused) have been removed.

2. All strings need to be of the same length. For this reason, we
   truncate the known paths after their hash part (they are still
   unique, of course).

3. Passing an empty set of matches, or a match that is shorter than
   the length of a store path, causes the crate to panic. We safeguard
   against this by completely skipping the refscanning if there are no
   known paths (i.e. when evaluating the first derivation of an eval),
   and by bailing out of scanning a string that is shorter than a
   store path.

On the upside, this reduces overall runtime to less 1/5 of what it was
before when evaluating `pkgs.stdenv.drvPath`.

[0]: Frankly, it's a random, research-grade MIT-licensed
     crate that I found on Github:

     https://github.com/jneem/wu-manber

[1]: We probably want to rewrite or at least fork the above crate, and
     add things like a three-byte wide scanner. Evaluating large
     portions of nixpkgs can easily lead to more than 65k derivations
     being scanned for.

Change-Id: I08926778e1e5d5a87fc9ac26e0437aed8bbd9eb0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8017
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-02-02 17:50:44 +00:00
Vincent Ambo
2c07ff0f8c docs(tvix): add more information to README
The README was very sparse before and we've actually had people email
us (as it says to contact us) just to ask what Tvix *is*. This should
answer some questions!

Change-Id: I0f248cb060eccfe086468afed1d648652b35dfd1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8018
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
2023-02-02 16:25:52 +00:00
Vincent Ambo
503e6810e7 fix(tvix/eval): unsafeDiscardStringContext is a no-op
... not just a TODO.

Most use-cases of unsafeDiscardStringContext are for cases where a
string is processed in some ways and no longer contains a "physical"
reference, but still has its context attached in C++ Nix.

We don't need to do this. This does diverge in behaviour in use-cases
related to build scheduling, but that whole behaviour will be
different in Tvix.

Change-Id: I4056d4c09f62d44d6bd52b791db03fe5556672b5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8016
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-02 14:30:32 +00:00
Vincent Ambo
75d1b72b7c refactor(tvix/eval): import_cache can be a HashMap
... instead of a BTreeMap, as we do not need ordering guarantees here.
HashMaps are noticeably faster here (especially as we've been sorting
essentially random data!).

Change-Id: Ie92d74286df9f763c04c9b226ef1066ee8484c13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8014
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-02-02 14:25:30 +00:00
Vincent Ambo
98ea614c6e chore(tvix/eval): elaborate on internal types in Value::type_of
This aids in debugging quite substantially.

Change-Id: Ic43232aa6165ae1c3db7ac2701938e1dfeeb418c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8013
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
2023-02-02 09:29:43 +00:00
Florian Klink
90f847ad32 chore(corp/tvixbolt): update Cargo.lock
Change-Id: Ie6bb2df16f79f7d977a7e95187a6b81e3ac0108f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8011
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-02-01 23:06:02 +00:00
Florian Klink
6abd1e2dae chore(tvix): update Cargo.lock
Change-Id: I08b2dc2393819c4f0d3871fe05bdbd29e89a6d75
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8010
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-01 23:06:02 +00:00
Vincent Ambo
dc1e86001c fix(tvix): add dummy target to attach extra-step to
Change-Id: I594a6652e2efe7aa6e35c7cdd84fc3097660614f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8009
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-02-01 17:34:08 +00:00
Florian Klink
30c4454056 fix(ops/buildkite): set default_branch explicitly
It looks like this needs to be set for the tvix pipeline to succeed.

It was set to `canon` for `tvl-kit` (not sure if manually, or some
autodetection previously did it for us that's not present anymore).

Anyways, this sets it to how it's set in the web interface, to hopefully
fix it.

Change-Id: Ic3eb60e3f421fa949a84dcdaa928823ff45f679a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8008
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-02-01 17:25:06 +00:00
Vincent Ambo
87de3fff52 chore(tvix): mirror exported repository
Pushes a mirror of tvix to github/tvlfyi/tvix

Change-Id: I86e7c1b1dc1fcc1012fa72234588b26ac2554056
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8007
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-02-01 17:21:05 +00:00
Florian Klink
88d86741dc feat(ops/pipelines): trigger tvix buildkite pipeline
Change-Id: I4e81694b9686f977a6590c5e1703a4ef413b0cf4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8003
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-01 17:11:50 +00:00
Florian Klink
2c4554679c feat(views/tvix): add buildkite pipeline
Change-Id: Iabfc8c64012500a39aaafb3976ceb4a249387ee8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8002
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-01 17:11:50 +00:00
Florian Klink
4a7ec1006d feat(ops/buildkite): add tvix pipeline
Change-Id: Ie701e0b77c596e07600efd1a59749d05068f0dbc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8006
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-01 17:11:50 +00:00
Florian Klink
5ac33a227f docs(tvix): update README to document usage inside tvix view
If the tvix view is cloned through josh, you don't use mg, but a
`shell.nix` is provided.

Also, add the `git clone` command, so people browsing tvix source code
in the browser know where to clone from.

Change-Id: I18483d6a52953f9f4eafd1533ea69afb0e329b04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8001
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-01 17:11:50 +00:00
Florian Klink
dbabf728b7 feat(views/tvix): add tvix josh workspace
This provides a trimmed-down version of //tvix, that allows building
tvix without all of the monorepo cloned.

We need to manually vendor in our patch for evans and ``--bytes-as-
base64`, as we don't have our depot overlays available in the josh
workspace.

Fixes https://b.tvl.fyi/issues/247.

Change-Id: I24306b8bb16ebf0df238b8f1eee0d47655a14827
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8000
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-01 17:11:50 +00:00
Vincent Ambo
8ed81cf755 feat(ops/secrets): add flokli to terraform secrets access
Change-Id: I9ede20028560f2da0fef89dfe431609c21bda51c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8005
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-02-01 16:39:32 +00:00
Florian Klink
f5ccb65f26 feat(tvix/nix-compat/derivation): Display -> to_aterm_string()
Instead of implementing `std::fmt::Display for Derivation` and relying
on the `to_string` method, introduce a `to_aterm_string()` method, which
does the same thing, but makes it clearer what we're producing, rather
than just calling `to_string()``.

Change-Id: I21823de9096a0f2c2eb6f4591e48c1aa9fd94161
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7998
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-01 16:31:56 +00:00
Florian Klink
964367044d feat(ops/secrets): add key for flokli
Change-Id: I52299b39d1d68ee1b700b631f70ef809af682e26
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8004
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-02-01 16:25:54 +00:00
Florian Klink
99d5cf822a refactor(tvix/cli): use nixhash module for output hash calculation
This covers all the weird corner cases.

Change-Id: I85637e82e8929828064ab562dc8a1c8bf161fffa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7991
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-01 15:29:48 +00:00
Florian Klink
265dd36b98 feat(tvix/nix-compat): add nixhash module
This module takes care of parsing various hashes and algorithms.

It will get used to modify derivation output hashes in the next CL.

Change-Id: Idc07c401dbb7510f49883ac02b8379b9a5d930c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7990
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-01 15:29:48 +00:00
Florian Klink
366d2c5e92 refactor(tvix/nix-compat/derivation): remove module inception
The module `src/derivation/derivation.rs` is a sign of module inception.

Move the Derivation struct definiton up into `src/derivation/mod.rs`,
and some of the helpers in a `util.rs`.

Change-Id: Ib24a5f8a27bdd45df8b1fa2b3482a79b33cab8d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7997
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-02-01 13:43:25 +00:00
Florian Klink
2551adb4f1 refactor(tvix/nix-compat): operator precedence can trip the unwary
warning: operator precedence can trip the unwary
  --> nix-compat/src/nixbase32.rs:41:23
   |
41 |                 c |= ((input[i + 1] as u16) << 8 - j as u16) as u8
   |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: consider parenthesizing your expression: `(input[i + 1] as u16) << (8 - j as u16)`
   |
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#precedence
   = note: `#[warn(clippy::precedence)]` on by default

Change-Id: I091071d649abf4ed38f5f4e39a0c5d21a0459bff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7996
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-01 13:43:24 +00:00
Florian Klink
361ffd7903 refactor(tvix/derivation): don't create deref'd immediately ref
clippy says:

> This expression creates a reference which is immediately dereferenced
> by the compiler

Change-Id: Ic2c093b043ebee9ae80912075083107e4d216cf1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7995
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-01 13:38:21 +00:00
Florian Klink
ce4d87b129 refactor(tvix/nix-compat): remove unneeded return
Change-Id: I97b9fb3ad33d2f51baf29486d4eff11f8dedcbab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7994
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-02-01 13:38:19 +00:00
Florian Klink
894d17b29b refactor(tvix/nix-compat): const is always 'static
Change-Id: If33bbf47a6cc376f2c0935ea4331a819223b2f00
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7993
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-01 13:38:18 +00:00
Florian Klink
3091ac69f4 chore(third_party/nixpkgs): drop permittedInsecurePackages
subsurface has been fixed a while ago to not pull in qtwebkit anymore,
this can be dropped.

Change-Id: I173dda71770d02ce8064d1751aff889475d12dfb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7999
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: grfn <grfn@gws.fyi>
2023-02-01 11:50:58 +00:00
Vincent Ambo
759f9dbf39 feat(tvix/cli): implement builtins.placeholder
This doesn't require any other corresponding handling *yet*, as the
actual replacements happen in the builder logic (which we delegate to
cppnix at the moment).

Change-Id: I034147c933f05ae427c7a8794647132d108d0ede
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7972
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-02-01 10:01:40 +00:00