Commit graph

952 commits

Author SHA1 Message Date
Vincent Ambo
a5d8f5273c fix(3p/nix): Fix dependency inclusion and reenable CI
Change-Id: Ic2b7db4abe636b73b4c71e6a2f1501d3b4b8b90e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2922
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-10 21:46:02 +00:00
Vincent Ambo
473604f567 refactor: Move nixpkgs attribute to third_party.nixpkgs
Please read b/108 to make sense of this.

This gets rid of the explicit list of exposed packages from nixpkgs,
and instead makes the entire package set available at
`third_party.nixpkgs`.

To accommodate this, a LOT of things have to be very slightly shuffled
around. Some of this was done in already submitted CLs, but this
change is unfortunately still quite noisy.

Pay extra attention to:

* overlay-like functionality that was partially moved to actual
  overlays (partially as in, the minimum required to get a green
  build)

* modified uses of the package set path, esp. in NixOS systems

Special notes:

* xanthous has been disabled in CI because of issues with the Haskell
  overlay
* //third_party/nix has been disabled because of other unclear
  dependency issues

Both of these will be tackled in a followup CL.

Change-Id: I2f9c60a4d275fdb5209264be0addfd7e06c53118
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2910
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2021-04-10 21:18:55 +00:00
Vincent Ambo
a83abc9024 chore(tazjin/emacs): Use telega from nixpkgs
adisbladis fixed the tdlib/telega versioning issues in nixpkgs at some
point, so this isn't required anymore.

Change-Id: Ib98e73d0e4394765f08f5f3741f70adab459c22f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2909
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-04-10 20:04:40 +00:00
Vincent Ambo
e6d1e68f4a refactor(3p/overlays): Move haskell overlay to new tree location
nixpkgs_exposed is going away, and the haskell overlay is independent
from that.

See also b/108, cl/2910

Change-Id: I3aea6dfc427a914f3f88146fd0b45d60dfd45a1a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2918
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-10 15:23:29 +00:00
Vincent Ambo
6489d94ad6 refactor(3p/dhall): Move easy-dhall-nix import into separate folder
This is part of paving the way for clearly distinguishing between
packages from nixpkgs and //third_party.

See also: b/108, cl/2910

Change-Id: I28b5abd1f0f9fa3c4478c9f255b2025f4a4139f1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2917
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Reviewed-by: tazjin <mail@tazj.in>
2021-04-10 14:32:02 +00:00
Vincent Ambo
f2443911cc refactor(fun): Consistent use of depot.third_party vs. pkgs
In preparation for the solution of b/108, we need to consistently use
`depot.third_party` for packages that are only packed in the TVL depot
and `pkgs` for things that come from nixpkgs.

This commit cleans up a huge chunk of these uses in //fun

Change-Id: I45a7b392a9749fa7859ff5100dcea415bda807c3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2914
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-10 12:04:30 +00:00
Vincent Ambo
8361b82d0a refactor(third_party): Consistent use of depot.third_party vs. pkgs
In preparation for the solution of b/108, we need to consistently use
`depot.third_party` for packages that are only packed in the TVL depot
and `pkgs` for things that come from nixpkgs.

This commit cleans up a huge chunk of these uses in //third_party

Change-Id: Ic382c0cdea7330a84d5f0b7d109c824ddceb94e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2912
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-10 11:48:55 +00:00
Griffin Smith
0419aa1f63 feat(3p/buzz): Init
Package buzz, a desktop email notifier that uses passive (push-based)
IMAP connections to get email notifications quickly.

Change-Id: Idbd175db882cdab975b0928cd6920954500954a1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2902
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-04-07 15:27:43 +00:00
Luke Granger-Brown
79aa142788 feat(3p/gerrit): expose the Gerrit API in the output
Writing Gerrit plugins that don't use the in-tree build system is more
convenient if the API is actually exposed in the derivation's output.

Change-Id: I3408d35498ca879576d532b005e36fde8ff2ea61
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2871
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-04-06 18:43:04 +00:00
Luke Granger-Brown
0dba3ce133 chore(3p/gerrit): update gerrit and plugins
This reverts commit 9551b628d0 (i.e. this is a rollfoward for https://cl.tvl.fyi/c/depot/+/2817)

Change-Id: Iaffcf1cdbe119d26ecb09cc88f9a56436b374c08
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2870
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2021-04-06 18:43:04 +00:00
sterni
c28d9710d6 fix(3p/cgit): fix max-width for #summary
Having a space between the number and the unit is not valid CSS.
I was aware of this problem, but apparently forgot to amend the fix.

Change-Id: I74936db515799763038669d0a11da53f28f722be
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2867
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-04-06 09:22:57 +00:00
Profpatsch
25ba2ef5f6 feat(third_party/rust-crates): default to 2018 rust edition
We override the default `buildRustCrate` with our default options.

Kinda amazing how many crates still default to the 2015 edition;
probably to be backwards compatible with older compilers?

Change-Id: Ic571f527b1575a03b8b58e6b75bcf12c4b9b7d9c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2842
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2021-04-04 14:26:23 +00:00
lukegb
9551b628d0 revert(3p/gerrit): chore(3p/gerrit): update gerrit and plugins
This reverts commit f59c6214c4.

Reason for revert: new gerrit's JS appears to not have compiled correctly; rolling back until I can figure out why

Change-Id: If16fe341aad25bef30ed7be8c6ac49cadf2a732c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2821
Reviewed-by: lukegb <lukegb@tvl.fyi>
Tested-by: BuildkiteCI
2021-04-04 10:30:19 +00:00
Luke Granger-Brown
f59c6214c4 chore(3p/gerrit): update gerrit and plugins
Yeet.

Change-Id: I4de6453419f520a44af19d561a36d242195a64e9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2817
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-04-04 10:11:25 +00:00
sterni
e5b8bd2842 refactor(3p/rust-crates): use imported buildRustCrate everywhere
A bit less noisy in the definitions and the nix parser can already
detect it being misspelled.

Change-Id: I979da11471187e36cde5c015aaf654f925757a8b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2814
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-04-03 22:46:29 +00:00
sterni
abe1de4302 feat(3p/rust-crates): get dependencies from the depot fix point
I think it is good practice to always get dependencies from the depot
fix point if they are exposed. The reasoning for this is that if we
improve the support for overriding in depot, say by introducing a
depot.extend functions or even full blown overlay support, this will
already work as expected.

Change-Id: Ibb8dffcf32e8f46817a2db2da26139fabdce55bc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2770
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2021-04-03 22:46:15 +00:00
sterni
0133fdc737 chore: move all 3p buildRustCrate derivations to //third_party
Profpatsch and me are basically the only users of
depot.users.Profpatsch.writers.rustSimple*. To pull in the odd
dependency we usually use buildRustCrate which is rather convenient.
However we've picked up the bad habit of inlining these in a let
somewhere instead of managing them in a more central location although
there has been an (unsuccesful) attempt at this in
//users/Profpatsch/rust-crates.nix.

This CL moves all buildRustCrate based derivations into
third_party.rust-crates and deletes any duplicate derivations we have
accumulated in the tree.

Change-Id: I8f68b95ebd546708e9af07dca36d72dba9ca8c77
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2769
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: Profpatsch <mail@profpatsch.de>
2021-04-03 22:16:35 +00:00
Luke Granger-Brown
3e7073f106 feat(3p/apereo-cas): add the LDAP DN to the list of attributes
Panettone currently uses the LDAP DN as the user key, so we collect it
here so that we can later make sure its exposed to Panettone.

Change-Id: Ia2048cb479a2afe6fe9f47181115ae7ec13dedf3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2811
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
2021-04-03 21:51:58 +00:00
Luke Granger-Brown
85900e150e chore(3p/apereo-cas): move TVL config
There's some non-secret config that made its way into the secrets file.
This CL moves it into git so we can track it properly.

Change-Id: I3f5bf5e1f7addabb199997fb7b1f805b9157fbbe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2810
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-04-03 21:51:58 +00:00
Luke Granger-Brown
81a7bd4765 chore(3p/gerrit_plugins): init oauth
Add the OAuth gerrit plugin to our mini collection of Gerrit plugins.

This includes a patch to make the plugin work correctly with CAS 6.x,
which has changed the attributes into a JSON object with the attributes
nested inside, instead of a JSON list.

Change-Id: I4741f137cca9c8eb45b9ea660fb4cbf6962be9a4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2782
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-04-03 19:11:18 +00:00
Griffin Smith
e908ccdbbb feat(3p/lisp): Add cl-smtp
Change-Id: Idbf63e346b696fb6704390d7a76a2f2b2d3bc190
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2802
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-04-03 18:54:39 +00:00
Luke Granger-Brown
268cb4cb11 chore(3p/gerrit_plugins): attempt to stabilise hashes a bit
I'm dropping the leaveDotGit and deepClone bits; they were set like that
purely to try to make the build stamping work. In practice, not only
does the build stamping not work, but it also means we hit some
inconveniently-different hashes from time to time when gitiles does...
something??? on its backend.

I'm also putting some gcroots for these on whitby, which should also
help a bit, although it's a bit of a hack.

Change-Id: Ie6082248393e62795c18b1971fc2d16f4e8cc81d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2781
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-04-03 18:39:17 +00:00
sterni
db80c2d368 feat(3p/cgit): limit width of div#summary, i. e. /about/**
Decreased text width for the /about pages should increase readability
considerably as jumping back to the beginning is hard for longer
lines. The result is still not perfect as the font size for the /about
pages is rather small and many lines thus get broken somewhat awkwardly.

We could probably migitate that using a larger font size.

The implementation choice of adding a tvl-extra.css which we inject into
cgit.css in preBuild is for simplicity: We don't need to worry about
routing an extra CSS file and loading it from the right location via
extra cgit head entries and serving it at the correct location using
either nginx or thttpd.

A drawback of this is however that iteration is slowed down by cgit's
compilation time.

Additionally, this should be the basis for implementing a bubblegum
themed cgit for Profpatsch.

Change-Id: I18060f735167acd623cef7a17c83408978461249
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2756
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-04-02 10:24:41 +00:00
sterni
1f40a52280 feat(3p/cgit): enableParallelBuilding
This gives a decent compilation speedup even on slow machines, so seems
worth it. Let's hope the cgit build process is not racy.

Change-Id: Ic4ae72789da2ccae16fd48e46aec624244b25035
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2755
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-04-02 09:17:05 +00:00
sterni
93a746aaaa feat(web/bubblegum): nix CGI programming framework
So here is what has been keeping me up at night: At some point I
realized that nix actually made a somewhat passable language for CGI
programming:

* That `builtins.getEnv` exists as one of the impurities of Nix is
  perfect as environment variables are the main way of communication
  from the web server to the CGI application.

* We can actually read from the filesystem via builtins.readDir and
  builtins.readFile with bearable overhead if we avoid importing the
  used paths into the nix store.

* Templating and routing are convenient to implement via indented strings
  and attribute sets respectively.

Of course there are obvious limitation:

* The overhead of derivations is probably much to great for them to be
  useful via IfD.

* Even without derivations, nix evaluation is very slow to the point
  were a trivial application takes between 100ms and 400ms to produce a
  response.

* We can't really cause effects other than producing a response which
  makes it not viable for a lot of applications. There are some ways
  around this:

  * With a custom interpreter we could have streaming and multiplexed
    I/O (using lazy lists emulated via attrsets) to cause such effects,
    but it would probably perform terribly.

  * We can use builtins.fetchurl to call other HTTP-based microservices,
    but only in very limited constraints, i. e. only GET, no headers,
    and only if the tarball ttl is set to 0 in the global nix.conf.

* Terrible error handling capabilities because builtins.tryEval actually
  doesn't catch a lot of errors.

To prove that it actually works, there are some demo applications,
which I invite you to run and potentially break horribly:

    nix-build -A web.bubblegum.examples && ./result
    # navigate to http://localhost:9000

The setup uses thttpd and executes the nix CGI scripts using
users.sterni.nint which automatically passed `depot`, so they can
import the cgi library.

Change-Id: I3a22a749612211627e5f8301c31ec2e7a872812c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2746
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-04-01 18:50:36 +00:00
Florian Klink
ab7e752b4a chore(third_party/cgit): use lib instead of stdenv.lib
Getting `lib` from `stdenv.lib` is deprecated, and throws warnings.

Change-Id: Ic925818c1b5a67d15d6d40ad784554328cd603e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2751
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2021-04-01 17:30:20 +00:00
Profpatsch
28b2c79c14 feat(third_party): init hii at 1.0.0
`hii` is a derived rewrite of suckless’s `ii`.
It is not backwards compatible.

Change-Id: Ife2a43863b5b6ba38333d7ae5f2cb9bd4787b5fc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2706
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-01 11:29:51 +00:00
Profpatsch
b89d4c1578 feat(third_party): add dhall packages from easy-dhall-nix
About to do some dhalllll! \o/

Change-Id: Ie58c335d80f4a5abeb8296ece5a24377f07e6369
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2585
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-04-01 11:27:39 +00:00
adisbladis
58cca2faaa chore(tvix): Add doc build infrastructure
Change-Id: I2cf67df085d0c008b5ff5efff2235a670207024a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2711
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <mail@tazj.in>
2021-03-31 21:51:59 +00:00
Griffin Smith
8d094be9ce feat(gws.fyi): Add some very light CSS
web-brutalism is so 3 years ago, bro

but seriously, I'd like to start putting some actual stuff here, so
let's make it look halfway-decent.

Change-Id: Ic78d725b3755c2307c7ea155af8d0f90e287830c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2697
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2021-03-29 00:00:30 +00:00
Griffin Smith
4a17fe5922 fix(3p/lisp/puri): Use fetchgit for source, update version
Something changed in the upstream we fetch this source from that's
causing the fetch to fail - I can only assume it's a yanked rev, but
I'm not really sure. fetchgit from nixpkgs appears to be a little bit
more robust than builtins.fetchGit, so let's switch to that, and also
upgrade to a rev that we know is present.

Fixes: b/96
Change-Id: I8983c2df11ab4fa20f60915f950c6a7378efd2fd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2691
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-03-27 16:10:20 +00:00
sterni
3a64d087f2 chore(3p): use haskell-language-server from nixpkgs
Upstream haskellPackages has a newer haskell-language-server than we had
and it is always in sync with the default GHC version of that package
set which we incidentally use to build all haskell derivations in depot.

I hope this will not cause us more trouble in the future, but I've
gotten the impression that maralorn makes an effort to prevent
haskell-language-server from being broken in haskellPackages, so ideally
we'll never have to worry about hls again. If we do have to, we may need
to switch to easy-hls-nix.

Note that I haven't had the time yet to verify that the shells actually
work since it's kinda late now and the rebuilds are many.

Change-Id: I74c192d57355904cfa45bb76d70346792ba05af5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2662
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2021-03-26 10:20:25 +00:00
sterni
3a49e4f4bf chore(3p): bump NixOS channels to 2021-03-25
OpenSSL released an update which fixes two severity high security
issues:

* https://mta.openssl.org/pipermail/openssl-announce/2021-March/000197.html
* https://mta.openssl.org/pipermail/openssl-announce/2021-March/000198.html

Update to the nixpkgs (currently still master) commits updating OpenSSL.

Other changes:

* Use GHC 8.8.4 for haskell-language-server as GHC 8.8.3 got removed
  from nixpkgs last friday.

Change-Id: Ic1b2f49284e78193a4330da4bb4b718a797f5ab1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2653
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
2021-03-26 01:43:32 +00:00
Vincent Ambo
d01d6bb63d chore(3p): Clean up unused third-party packages
Removes the following currently unused packages:

* terraform-gcp
* cpp.googleapis
* nixery

Change-Id: I02c38f0e6d23b1b9008706ca5a3b9db51cc9b689
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2650
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: glittershark <grfn@gws.fyi>
2021-03-25 18:37:33 +00:00
Vincent Ambo
b59f712535 fix(gemma): Fix build by backporting Elm 0.18
Backports an old version of the Elm language (before the release where
they made everyone rewrite their programs), from the same old nixpkgs
commit as was previously used in overrides (see CL/2646).

Change-Id: I0ae4cc611aa40269b290651ab982c1db93518d8a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2649
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-03-24 00:54:43 +00:00
Vincent Ambo
3842c184b6 chore(3p/gopkgs): 'depot.buildGo' -> 'depot.nix.buildGo'
This makes it possible to deprecate the previous alias.

Change-Id: I6efe73a21dc771cfa28094bbac434a188aebfd45
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2647
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-03-23 23:44:14 +00:00
Vincent Ambo
2b42afa800 chore(3p/gerrit_plugins): Build plugins separately in CI
Change-Id: I76aeaa879e43b1fd08440f60a4f320bb7b48c6ab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2630
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2021-03-21 13:01:51 +00:00
Vincent Ambo
39c400fa09 chore(3p/gerrit): Update pinned dependency hash
Change-Id: I1807ac1771d23b00e3a78cc36fb2f82a12900993
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2629
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-03-21 13:01:51 +00:00
Vincent Ambo
c477311071 feat(3p): Import nix-community/impermanence module
Change-Id: I5b9386b1db74dafe921d7957bc726cfba403c155
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2618
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-03-21 00:55:58 +00:00
Griffin Smith
b1c4b84dba chore(gs/achilles): Integrate with the depot build
Get achilles building in Nix as part of the depot's build tree. This
involved making it work with stable rust, since the depot only exposes
stable rust to sub-packages, which turned out to be fairly
straightforward.

Also adds libffi as a new top-level expose, since it's required to build achilles

Change-Id: I5f6dedb26c0b81ec258aedde1973e74903c07ece
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2612
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2021-03-20 20:20:20 +00:00
Griffin Smith
d031ec0234 chore(3p/exposed): Add glittershark to OWNERS
Change-Id: I17474fbee75666d825b4c4db1af37fff1e214818
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2611
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-03-20 20:20:20 +00:00
sterni
1799ddf7f0 fix(gs/xanthous): update xanthous.cabal, make CI catch such errors
Since xanthous has a checked-in package.yaml and cabal file, the
haskellPackages build infrastructure will use the package.yaml file for
all builds. The resulting problem is that our CI won't actually catch build
failures that would be observable with cabal or when building from the
sdist.

We fix that by filtering out the package.yaml file in pkg.nix
additionally to the filters specified in .gitignore. For this we need
gitignoreFilter from gitignore.nix which we expose as part of a functor
set from third_party.gitignoreSource to maintain interface
compatibility.

Change-Id: I337185f484d2027341f38031dcd78898706904eb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2609
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2021-03-19 23:21:54 +00:00
sterni
4cf7dced54 chore(gs/xanthous): fix for hgeometry 0.12.0.1
The breaking removal of triangulationEdges was not that big of a deal
after all: It was just renamed to edgesAsPoints apparently, so the fix
is easy enough and we can save one override.

hgeometry-combinatorial's doctests seem to trigger some kind of GHC
dynamic linking bug (https://github.com/noinia/hgeometry/issues/132) so
we disable the tests.

Change-Id: Iba2a64cade4d1a55fa4b81846e1116f282d4590a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2608
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2021-03-19 23:16:30 +00:00
sterni
a01fe5c06f fix(gs/xanthous): fix build failures caused by dependency updates
The following changes in dependencies of xanthous broke the build and
have been fixed in this CL. Thus we can reenable CI for xanthous.

* random 1.2.0 removed the Read instance for StdGen, so we need use
  System.Random.Internal to un-newtype StdGen into an SMGen in the
  appropriate places as that type still has a Show and Read instance.
  Requires a new direct dependency on splitmix as well.

* witherable 4.0 renamed Data.Witherable into Witherable and no longer
  exports Filter.

* random 1.2.0 probably also broke the Function instance for GameState
  which contains a StdGen. I'm not exactly sure which change exactly
  triggered this, but the fix is easy enough: We implement a Function
  instance for SMGen using functionShow allowing us to write a Function
  instance for StdGen using functionMap. I've put these instances into
  Xanthous.Orphans.

* hgeometry 0.12.0.0 removes the triangulationEdges function (which is
  also not mentioned in the changelog, so I'm not sure if there's a
  replacement yet). Fix by pinning to 0.11.0.0 for now.

* hedgehog-classes: relax bounds on semirings

Change-Id: I3617d8916d753b386c9fa80062be6bcbdfee0131
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2607
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2021-03-19 16:49:44 +00:00
sterni
531fc2b80b chore(3p/haskell_overlay): remove obsolete overrides
I actually wanted to check up on regex-tdfa-text in owothia, but
realized it was actually in a dependency. When porting the patch for
chatter to nixpkgs, I wondered if we could get rid of other overrides
or if we need to fix anything else in upstream.

* aeson, attoparsec, cassava, psqueues, hedgehog: jailbreaks are
  no longer necessary

* fgl, fgl-arbitrary: upstream has the versions pinned by now

* hgeometry, hgeometry-combinatoral: upstream has moved past the
  pinned versions, but we don't need to keep them downgraded as
  xanthous's build is not broken by them.

* random-source: the upstream compiler shouldn't crash anymore,
  additionally upstream has the version pinned here currently

* semialign: upstream also has 1.1.0.1 by now

* splitmix: splitmix has been fixed upstream and haskellPackages
  has moved past 0.1

* hspec-core: test suite passes or upstream has disabled it as well

* QuickCheck: upstream advanced to the same version

* vinyl: upstream moved past the pinned version, causes no build failures

* comonad-extras: has been fixed upstream

Change-Id: I34eff81ceaac005f2ad90dd9c1d3e623b8da91c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2606
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2021-03-19 16:49:44 +00:00
sterni
90fdd7f23d chore(3p): bump NixOS channels to 2021-03-18
Making this a monthly service apparently.

Necessary changes:

* 3p: expose emacs27 instead of emacs26 which got removed
  users/tazjin/{camden, frog}: switch from emacs26 to emacs27

* 3p/lieer: google_api_python_client got renamed to
  google-api-python-client

Change-Id: I1011665d10eebc99990addbef6a8a6b000b93896
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2605
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2021-03-19 14:00:50 +00:00
Vincent Ambo
b4e87f8254 feat(ops/dns): Import tvl.fyi DNS zone into depot
Imports the current state of the tvl.fyi zone and configures simple CI
checks on the file format.

No deployment automation exists for this (yet?).

Change-Id: Ia7d72e02b9f6d3adef994c5dc1898cc0df9dfcfb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2600
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-03-15 21:23:35 +00:00
sterni
2cd2b58a04 feat(users/sterni/htmlman): static site generator for manual pages
htmlman is a very simple nix based static site generator which is
intended for rendering HTML representations for man pages plus an index
page listing all available pages. For the sake of simplicity (and unlike
previous iterations of this piece of code) other documentation artifacts
and formats are not supported.

Usually web services like GitHub and depot's web interface are pretty
good at displaying "normal" documentation artifacts like markdown files,
but man pages are usually not rendered — with the additional problem
that it's source is virtually unreadable. htmlman should provide a
simple static site generator which can be plugged into GitHub actions or
the like to automatically generate rendered version of man pages tracked
in version control.

Change-Id: Ib53292964b3ff84c32d70c5fde257a2edb8c2122
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2596
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-03-11 13:12:31 +00:00
Vincent Ambo
0f1a497361 chore(3p): Remove git-bug
We stopped using this in favour of //web/panettone quite a while ago,
so lets clean it up.

Change-Id: I8aa8d86288933d470ab3962ffbb60294eaddd27b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2540
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-02-19 16:40:45 +00:00
sterni
e628862e97 chore(3p): Bump NixOS channels to 2021-02-18
Main motivation for this is to get the openldap update that fixes
10 CVEs: CVE-2020-36221 to including CVE-2020-36230. See also this
issue which lists them all: https://github.com/NixOS/nixpkgs/issues/113490

Someone should also redeploy whitby as soon as this lands in canon and
all build failures have been fixed.

Things done to resolve upstream breakages:

* grpc no longer takes abseil-cpp as an input, it has also been removed
  in the override.

* Upgrade glittershark's kernel to 5.11 since the linuxPackages_5_9
  attribute has been removed by upstream and the patch used by them is
  available for 5.11 as well.

* The fixed output hash for third_patry.apereo-cas changed for some reason.

* Remove the pin of haskellPackages.vector from the haskell overlay. It
  broke as the most recent version of vector in nixos-unstable no longer
  depends on semigroups. This effectively updates vector from 0.12.1.2
  to 0.12.2.0.

* Align two comments in tvix/libstore/worker-protocol.hh because the
  updated clang-format now demands that.

Change-Id: I2ecf10a98de935e9222acf1feaea447d4c11ed2d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2538
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-02-19 14:45:43 +00:00
sterni
4f6f4e6588 docs(tvix): fix path in build instructions, add note about tests
Change-Id: If9dcd8e39f416c11460e66454df9f53096691699
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2484
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2021-02-02 17:59:44 +00:00
sterni
8709153973 fix(tvix/libexpr): backport fix for functionArgs failing on primops
Nix internally differentiates between lambdas and primops, but their
type in the nix expression language is the same (lambda). The
implementation of builtins.functionArgs only checks if the given
expression is of type tLambda and fails if the type is tPrimop or
tPrimopApp which are also functions. This most notably breaks
lib.generators.toPretty when called on a builtin making for example
yants fail if a primop is typechecked and an error message is
generated.

This fix generates an empty set for primops like for plain lambdas
and is based upstream commit b2748c6e99239ff6803ba0da76c362790c8be192.

Additionally we add to two tests:

* eval-okay-functionargs now includes a few test cases checking that
  builtins.functionArgs always returns an empty set for builtins and
  also works as expected for normal functions.
* eval-okay-types now also checks if builtins are functions.

Future work would be to make builtins.functionArgs work as users would
expect for builtins like builtins.fetchurl, builtins.fetchGit etc. which
take a set as an argument. These currently don't register as formal
arguments, but it would be an usability improvement at least if they
did.

See also https://github.com/NixOS/nix/pull/3626#issuecomment-698546704

Change-Id: I2bf4cb80d44a4b72ade13d3e0dbd7dfb1d049f32
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2477
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Reviewed-by: glittershark <grfn@gws.fyi>
2021-02-02 10:52:33 +00:00
sterni
c6b243a7a5 chore(3p/gerrit_plugins): fix fixed-output sha256 of check
Once again the sha256 of the fetchgit fixed output derivation for check
changed which was brought to light by the recent GC on whitby.

Change-Id: Ib3c3b5b489717ac6d73631282f27e4363d4ac5c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2481
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-02-01 23:14:49 +00:00
Vincent Ambo
8f57ca92bd chore(3p|nix): Remove typed Go
Nobody has actually done any experimentation with typed Go, so we're
getting rid of it for now - it's causing annoying IFD during build
graph generation.

Change-Id: Ibac3dea98ebed1b3ee08acda184d24c500cf695d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2458
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: Profpatsch <mail@profpatsch.de>
2021-01-30 08:20:45 +00:00
Vincent Ambo
83465be599 feat(3p/telega): Pin tdlib version used for telega
Recent channel updates have caused some issues for telega.el, mostly
because the version of tdlib (the C++ library for Telegram) and the
Emacs package are out of sync.

This overrides the version used in the Emacs package to a "known good"
commit. It would be useful to change the tdlib derivation in nixpkgs
to make this version mismatch a hard build error.

Change-Id: I9c994f783e1cc17e933432507cd13b65697efd4a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2445
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2021-01-26 13:25:43 +00:00
Vincent Ambo
8ab02db6c5 chore(3p): Bump NixOS channels to 2021-01-19
Change-Id: I07b0d4991a07e15b1ab1ec8f371542868874eda5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2421
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: Profpatsch <mail@profpatsch.de>
2021-01-19 12:43:10 +00:00
V
29db630a39 chore: Remove banned user
Change-Id: Icd61f7c567a327c74a4f381168e94737b2b30702
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2422
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: tazjin <mail@tazj.in>
2021-01-19 10:30:19 +00:00
Profpatsch
80e1ece329 feat(users/Profpatsch): set up a file watcher for tree sitter
Uses inotify to watch a file and print when it is modified, so we can
update the parser and display the sexp on the terminal.

Now the setup is good enough to start experiementing with queries on
the syntax tree.

Change-Id: I091587fc495ff627c79a69a52915aaaa8c51fcd2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2411
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2021-01-17 11:37:39 +00:00
Vincent Ambo
b4e420d4ff chore(3p): Bump NixOS channels to 2021-01-09
Your regularly scheduled channel update, but slightly more regular
than before.

Included fixes:

* 3p/emacs: Pick telega.el from stable channel, unstable is broken.
* glittershark/fprintd: Compile with gcc9, since build fails with the
  new default of gcc10
* glittershark/fprintd: Use a global overlay for the fprintd package
  until https://github.com/NixOS/nixpkgs/pull/108962 lands in
  nixos-unstable
* glittershark/home: Don't install rr, as it's not building with gcc10

Co-Author: Griffin Smith <grfn@gws.fyi>
Change-Id: Ia715fef64a405a220049fc540017356fa7370e0b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2341
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Tested-by: BuildkiteCI
2021-01-13 16:53:20 +00:00
Profpatsch
6843016aae chore(third_party): move nixkpgs-exposed to subdir to fix OWNERS
The owners plugin should in theory be able to match on subdirs (at
least according to its documentation, but it doesn’t and nobody has
any idea how to debug it.

We already know that subdirectories work just fine, so let’s go the
path of least resistance because frankly, I couldn’t care any less.

The haskell overlay also moves to the subdir, this way both can be
changed in the same go by the same people.

Change-Id: I7d98f48afa649ad2c58e38e674e1c4df09039c1c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2347
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-01-11 19:44:38 +00:00
Profpatsch
04f91b5957 feat(users/Profpatsch): add script for randomly evaling nixpkgs
Running this after a codified refactor acts as a good smoke test,
if a big subset of packages is broken or any central packages are
broken, this should find them quite quickly, thanks to randomness™.

Just let it run for a few minutes and check the errors that pop up.

Change-Id: I1505dd31ca25b29254474a15cd6cb71d9743038a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2346
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-01-11 08:58:57 +00:00
Vincent Ambo
bf697b4b9d refactor(3p/exwm): Inline build instructions instead of overriding
Change-Id: If16a57e8dd905c8f709bee70dac575509dd22b02
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2343
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2021-01-10 13:32:47 +00:00
Vincent Ambo
51a80c2e9d fix(3p/exwm): Work around Chrome input focus issue
In recent Chrome versions, EXWM has some issue around handing focus
back to the application. There is a Github issue about this and this
commit implements the suggested workaround, which I've verified
locally.

Change-Id: Ib451e8d8b34921665c3015853850d12e04612929
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2342
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-01-09 15:30:52 +00:00
Vincent Ambo
88bf43878f chore(3p): Bump NixOS channels to 2020-12-28
Changes:

* ops/nixos/tvl-slapd: The NixOS module for OpenLDAP has removed the
  ability to configure OpenLDAP directly and now forces users to use
  some kind of weird Nix->OLC mapping that is mostly undocumented.

  This moves the config we need to the new format in a way that may or
  may not work and does the other arbitrary dance steps that someone
  decided to impose on us. Note that this now throws lots of warnings,
  but I can't be bothered to fix them.

* 3p: Random package removals accomodated

* users/glittershark: Pin grfn's kernel to 5.9, because the CK patch
  is not yet updated for 5.10

* users/glittershark: Update vendor hash for pg-dump-upsert, I suspect
  this changed because of something in the Go build machinery in
  nixpkgs. The deleteVendor flag also has no effect anymore and has been
  removed.

* users/glittershark: agda build is broken, commenting out development
  home-manager environment until it can be fixed

* third_party/haskell_overlay: updating random needs upper boundarles
  of a few dependencies relaxed (curse them)

* third_party/gerrit_plugins: for some cursed reason the fixed-output
  hash of the gerrit owners plugin fetchgit changed, updated.
  Same for the checks plugin.

Change-Id: Ica37995fe8039d3ba80eab643867f98795c56734
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2295
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2021-01-09 13:21:00 +00:00
Profpatsch
68c4730365 feat(third_party): separate nixpkgs whitelist to allow more owners
The exposed package list has to be changed/amended quite frequently,
every time somebody wants to use a package not yet in that list and
thus has to whitelist it here.

This effectively requires a superowner review every single time, which
is an unreasonable blocker for many CLs.

I thus propose moving the list into a separate file (I called it
`nixpkgs-whitelist.nix` which is more descriptive than `exposed.nix`
and letting anybody add themselves to the OWNERS on that file.

Change-Id: Ied8bac066e4b9a91ddd642db805fe33dc37872c9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2323
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-01-08 12:56:26 +00:00
Profpatsch
1261616bff feat(Profpatsch/writers): add rustSimple writers
A bunch of writer functions wrapping the `buildRustCrate`
functionality of nixpkgs. Can be used to write inline rust code, or
rust code read from files with `builtins.readFile`.

Change-Id: I9d74e9381b858b485925e4dc3fbb7fc392877c0a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2318
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2021-01-03 16:29:57 +00:00
Profpatsch
533e365c12 feat(users/Profpatsch): add python3Lib writer
Smol writer to create a python lib directly from a nix string.
The resulting library can be consumed by the writePython3 writer.

Change-Id: Id3d793564d230b38a08f65140bda4287285e1a72
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2310
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-01-03 16:12:33 +00:00
Profpatsch
806c281b34 feat(users/Profpatsch): moving around via the tree-sitter parse tree
Has a little setup to get the cursor position and map it onto a tree
sitter node. The current node is saved in a cursor variable, and a
highlight overlay marks the range of the current node in the buffer.

Change-Id: I0af56115f928732e993fbefe978a246ca7c757ee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2258
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2021-01-01 19:03:30 +00:00
Vincent Ambo
73e9265f95 chore(3p/immer): Remove vendored copy of immer
We ended up dropping the use of this library again.

Change-Id: I2c44cd22a6128d23f87a582402bf5fb84991d608
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2292
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2020-12-23 12:44:52 +00:00
Vincent Ambo
f7ea650142 refactor(tvix/libexpr): Remove Bindings::SortedByKeys()
Since we don't have a Bindings implementation with unstable order this
function is not required, as its callers can just iterate over the
attributes instead.

Change-Id: I01b35277b5a2dde69d684bc881dbd7c0701bcbb3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2291
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-12-23 11:33:54 +00:00
Luke Granger-Brown
e115e58f9c fix(3p/gerrit): fix left padding on account chip
We don't have an avatar provider at the moment, so it's wonky. I'll make
this a CL upstream as well once my work laptop has charged.

Change-Id: I79754560b2de6981508ba7e10faf6b50cb9c3f8f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2266
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-12-20 16:10:40 +00:00
Luke Granger-Brown
bbfcc7bbf0 chore(3p/gerrit): bump to latest version deployed on gerrit-review
Change-Id: I0aee39b21346f378964bab69bd903e8f43b7e7e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2265
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-12-20 16:10:40 +00:00
Vincent Ambo
a9014203b6 chore(3p/lieer): Upgrade to lieer 1.3
* rebases the send scope patch
* removes the API key override patch, lieer now has a configuration
  option for this

Change-Id: I198e8b61855f6cdb2b1439a1c8f2d9d69261c1b5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2242
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-12-14 19:41:30 +00:00
Vincent Ambo
316a4fa1e0 fix(3p/notmuch): Rebase dottime patch on notmuch 0.31
Change-Id: I43e8f59cd7f58b9c8b501983edbe55a3cb2f20d0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2241
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-12-14 19:41:30 +00:00
Andreas Rammhold
150b2fa799 chore(3p): move quassel to the right inherit statement
As the comment above the inherit statement says it is supposed to only
contain packages required for //third_party/nix. As much as I like IRC I
can't find a usage of Quassel within Nix itself. Perhaps a plugin to
report build status at some point?

Until that plugin mainfests we might as well move it to the correct
location.

Change-Id: I14fd6b698bb779b80829e7bebd5daa69cacdde78
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2177
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Tested-by: BuildkiteCI
2020-11-30 21:43:15 +00:00
Kane York
844c84a38a chore(3p): upgrade llvm to version 11
Additionally, anchor clang-tools to match the current llvmPackages version.

Change-Id: I27a199cadb55907c3e2deabfd1d14a5b50fc541b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2166
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-11-29 03:02:14 +00:00
Luke Granger-Brown
77be22549a chore(3p/gerrit): update to latest upstream HEAD
Change-Id: Ia7b17ca83ace164881a4dd8b0288741489bc467c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2164
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-11-29 02:26:46 +00:00
Kane York
516c046ed9 refactor(tvix): remove all 'using namespace' from nix command
clang-tidy: google-build-using-namespace
Change-Id: I07ea10b03a6d9582c0508747698038f7106e8f63
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2181
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-11-28 00:59:25 +00:00
Griffin Smith
363dbeae95 refactor(tvix): Centralize arbitrary definitions
To make it easier for future tests to use the arbitrary specialisations
we've defined for some nix types, centralize them all in a single
arbitrary.hh header file.

Change-Id: I767f27949cfe7ec55c79901f7d7aa538d2f98c6b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2182
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-11-28 00:47:48 +00:00
Griffin Smith
6f38ac6657 fix(tvix): Add missing break;s to BuildResult::FromProto
Add missing break statements to the big switch block in
BuildResult::FromProto, and cover the whole thing with a rapidcheck
round-trip test.

Change-Id: I7a07ca398cc5f02ca4fd8e6256fd563c6f3aea9d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2178
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-11-28 00:47:48 +00:00
Griffin Smith
49024be056 feat(tvix): Thread a log sink through calls to buildDerivation
Similarly to how we did for buildPaths, add a std::ostream& log_sink
parameter to the build_derivation method on Store, and pass it std::cerr
when called at the top level by nix commands - most notably, the
build-remote hook binary, so that we get build logs when using tvix as a
remote builder.

Change-Id: I0f8f729ba8429d4838a0a135a5c2ac1e1a95d575
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2176
Tested-by: BuildkiteCI
Reviewed-by: andi <andi@notmuch.email>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-11-28 00:47:48 +00:00
Griffin Smith
1d22874ab2 fix(tvix): Don't call assertStorePath on BasicDerivation.builder
Upstream doesn't, and for good reason - this might not be a store path,
for example if it's /bin/sh

Fixes: b/73
Change-Id: If9aa96de2cd8ab941c098a9f122b8b302a92ec38
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2175
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-11-27 21:28:40 +00:00
multi
c885bd0274 fix(tvix): globally reintroduce --quiet and --verbose flags.
The --quiet and -v/--verbose flags were removed during the glog
conversion in d0c44425e1, which has
previously broken programs like e.g. home-manager, which passes --quiet
to nix-build.

A nix-build-specific workaround was added in
24f9354d5b, which manipulates the
FLAGS_stderrthreshold global variable from glog. This commit moves the
--quiet logic back into the argument handling code in libmain, and adds
corresponding handling for -v/--verbose.

Change-Id: I13d860ebbb78541d9f1236691a1efe8bd2163c67
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2170
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-11-27 21:03:45 +00:00
Griffin Smith
5c3d58bb60 refactor(tvix): Prefer absl::StrFormat/StrAppend
This fmt call was particularly egregious

Change-Id: I2a3b1006c285170ab3374d1c8d81fc53e82a7b05
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2174
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-11-27 20:14:43 +00:00
Griffin Smith
bbfc47e96d feat(tvix): Write build logs to LOG(INFO) in buildDerivation
This was referencing a nonexistent note in buildPaths, for one, but for
another let's get log outputs when calling this RPC.

Change-Id: Ic9d17834b356ea84d69692ccc0249d09777e833b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2173
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-11-27 20:10:35 +00:00
Griffin Smith
39f96c4d9f fix(txix): Override overflow in BuildLogStreambuf
Subclasses of std::streambuf are expected to override overflow in
addition to xsputn, as it's called in certain cases by the non-virtual
methods. In our case, this was preventing endlines from getting sent
over the log stream.

Change-Id: I70d00f0c7cb8f8cf2f744f58974c21e7a70a715b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2172
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-11-27 20:08:45 +00:00
Griffin Smith
afdf08ead5 fix(tvix): Remove more superfluous log output
- A couple of log messages had double-newlines still
- Remove the [ drvPath ] bit from the "building path" output, since the
  derivation path is already present in the message itself.

Change-Id: Ib4b514a18fcb63c7191f4d7cf0a9b857aaa3f3fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2171
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-11-27 20:06:38 +00:00
Michael Reilly
db52903581 feat(tvix): Bump nlohmann-json version to 3.9.1
Backporting from upstream at d860295e116f9aa0f37e2637b5ec4b9c86fdf4d3

Change-Id: Iefde47ff6f24d21e9b37c9b52a615bd8c9f2fdb2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2141
Tested-by: BuildkiteCI
Reviewed-by: andi <andi@notmuch.email>
2020-11-27 19:59:51 +00:00
Eelco Dolstra
a30e616efb refactor(tvix): JSONSax: Use a RootValue
More #3377.

Backported from upstream at 9f46f54de4e55267df492456fc0393f74616366

Change-Id: I11bfca4ec56bd4e45291ce3f98a60f198dff0196
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2140
Tested-by: BuildkiteCI
Reviewed-by: andi <andi@notmuch.email>
2020-11-27 19:59:51 +00:00
Yorick van Pelt
89f1489916 refactor(tvix): json-to-value: use unique_ptr instead of raw pointers
Backported from upstream at a350d0beb0e13d0f58698510bd6a96d894cd06fd

Change-Id: Ib486bc1b36ef65041fab1a4634a0a82e13036fd4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2139
Tested-by: BuildkiteCI
Reviewed-by: andi <andi@notmuch.email>
2020-11-27 19:59:51 +00:00
Andreas Rammhold
1ee3a7b2bf feat(tvix): run the test-vm with ncurses support
This allows users to use C-A C to access the QEMU console while within
the VM terminal. From there you can control the process (e.g. kill it).

Change-Id: If2eec2e385fd27fc4e20921dcad78be7390f9d2b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2167
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-11-27 19:09:21 +00:00
Andreas Rammhold
9bf6eb7f82 chore(tvix): avoid unnesscary rebuilds of protos and tvix
For the build of tvix we can safely ignore all the files in src/protos
as well as all the nix files in the root of the repository. By passing
the input sources through builtins.filterSource we can make a per-file
decision whether or not we want to copy it into the store (in other
words: if we consider it relevant for the build).

This enables much faster development on the actual build expression.

Change-Id: Ib4c01dbe9cbfa9770922f6257af2a7259814ce0d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2159
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-11-27 19:09:07 +00:00
Griffin Smith
e772f38a5e fix(tvix): Don't double-up on endlines from log messages
Change-Id: I676dd4e9d8b1a7ff5389323e0462ba6681d62cf0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2168
Reviewed-by: andi <andi@notmuch.email>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-11-27 19:06:06 +00:00
Kane York
0c8c1227f1 refactor(tvix): remove signedness conversions by using std::optional
The different signedness of level and withLevel was causing implicit conversions. Use a nullopt instead of a -1 sentinel value.

third_party/nix/src/libexpr/nixexpr.cc:242:21: warning: narrowing conversion from 'unsigned int' to signed type 'int' is implementation-defined [bugprone-narrowing-conversions]

Change-Id: I7c2cadb6fd6bbff6c5b84028651ad4ebba423297
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2157
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-11-27 18:34:20 +00:00
Kane York
5fb58e23de fix(tvix): make unsigned conversion explicit
Implicit signedness conversion was causing warnings. Perform a static_cast; this value should never be out of range.

third_party/nix/src/libmain/shared.cc:310:12: warning: narrowing conversion from 'unsigned int' to signed type 'int' is implementation-defined [bugprone-narrowing-conversions]

Change-Id: I34fe53d186ae33f970147135bbe3894fb39a0ec7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2156
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-11-27 18:34:20 +00:00
Kane York
35bd980bd1 fix(tvix): sqlite bind argument number is signed
third_party/nix/src/libstore/sqlite.cc:89:33: warning: narrowing conversion from 'unsigned int' to signed type 'int' is implementation-defined [bugprone-narrowing-conversions]

Change-Id: I1dbc01acaecc4f382f63e7108429cd814ff05f4f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2155
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-11-27 18:34:20 +00:00
Yorick van Pelt
b595553f63 feat(tvix): builtins.fromJSON: use nlohmann/json parser instead of custom parser
backported from upstream at f1fac0b5c3b75efab781949fdff2b67ffdda2cb3

Change-Id: I788e3a9b930351118a5f248b356c351afd7f5391
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2138
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-11-27 17:02:46 +00:00
Griffin Smith
24f9354d5b feat(tvix): Support a --quiet argument to nix-build
This is passed by some third-party tools, notably home-manager. At some
point we should fully revamp argument parsing and log verbosity setting,
but for now this gets us to more of a working state

Fixes: b/78
Change-Id: I4134b17315e30145fd43d3659f407d7121ce8876
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2136
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-11-22 23:17:23 +00:00
Griffin Smith
9e3f541546 fix(tvix): Add endlines to build log messages
Change-Id: Id6e871a2b7fbb24fb5c8e84e1638bfe94e363b57
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2135
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-11-22 23:17:23 +00:00
Griffin Smith
9945bd1746 feat(tvix): Convert some DLOGs to VLOGs
It seems like the amount of logging is causing things to move a little
slower - even if that's not really the case, it gets in the way of
debugging things.

Refs: b/76
Change-Id: I9ea99a3b16e3307a0b0371bad22d03b0e2175af6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2134
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-11-22 23:17:23 +00:00
Griffin Smith
9a24294b8a feat(tvix): Add a pos field to more expr classes
To aid in both debugging and (eventually) printing stacktraces, add a
Pos member to a few more Expr variants.

Change-Id: Ic1d2a056fc7e6c07bc3e79fa38845cb4a5da5ca5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2133
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-11-22 21:59:58 +00:00
Vincent Ambo
5a00e58904 chore(3p): Bump nixpkgs to nixos-unstable from 2020-11-21
Included fixes for random breakage:

* 3p/awscli: pick from the stable channel; it is broken on unstable
* 3p/googletest: bumped version & removed patches that nixpkgs applies
* 3p/lisp/cffi: bumped library version for SBCL compat
* 3p/nix: fix libsystemd attribute
* 3p/nix: reformatted (clang-format handling of ternaries changed)
* glittershark/home: Use home-manager from nixkpgs
* glittershark/kernel: bumped linux-ck patch hash
* glittershark/kernel: removed "patch patch"
* multi/whitby: Use home-manager from nixpkgs
* tazjin/frog: drop Sourcetrail (it doesn't build currently)

Note that in addition to these changes, some previous CLs updated the
versions of git and cgit which was necessary for this channel bump,
but which could not be done in the same commit due to the nature of
the subtree merges.

Change-Id: If2563e8a68e2750c4b913a976ff7b93b42e8b7f3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2110
Tested-by: BuildkiteCI
Reviewed-by: multi <depot@in-addr.xyz>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-11-21 23:18:27 +00:00
Vincent Ambo
cd652b3b22 fix(3p/cgit): Rename remaining 'sha1' references to 'oid'
This changed upstream & upstream-upstream, but was still used in our
patches.

Change-Id: Id33f868a4f188deb3348425276bc5b4c4b36aff9
2020-11-21 20:09:42 +01:00
Vincent Ambo
1e81ce6622 merge(3p/cgit): subtree merge at adcc4f82
Change-Id: I16941629a17a2f39b9324cb85dc0cec6b104dfe4
2020-11-21 20:09:33 +01:00
Vincent Ambo
f4609b896f merge(3p/git): Merge git subtree at v2.29.2
This also bumps the stable nixpkgs to 20.09 as of 2020-11-21, because
there is some breakage in the git build related to the netrc
credentials helper which someone has taken care of in nixpkgs.

The stable channel is not used for anything other than git, so this
should be fine.

Change-Id: I3575a19dab09e1e9556cf8231d717de9890484fb
2020-11-21 19:45:56 +01:00
Vincent Ambo
082c006c04 merge(3p/absl): subtree merge of Abseil up to e19260f
... notably, this includes Abseil's own StatusOr type, which
conflicted with our implementation (that was taken from TensorFlow).

Change-Id: Ie7d6764b64055caaeb8dc7b6b9d066291e6b538f
2020-11-21 15:48:57 +01:00
Vincent Ambo
cc27324d02 chore(tvix): Use StatusOr API available in Abseil's version
The Abseil version of `StatusOr` does not come with the status macros
or the `Consume*` family of functions.

This change modifies the existing code to use the common denominator
of the API that is available between Abseil's own implementation of
`StatusOr` and the one from Tensorflow that we are currently using.

Change-Id: I5c37f68636a1fd54d153f95d7303ab8644abb774
2020-11-21 15:26:02 +01:00
Griffin Smith
8d24a975f1 fix(tvix): Use copy constructor to add strings to protos
Passing a string directly to add_paths like this causes the proto class
to take ownership over the string, meaning when it is destructed it
will *explicitly* free the string. When the string's actual owner (the
derivation struct) then goes out of scope it'll get freed again, causing
a double-free. This fixes that to instead use the copy constructor to
assign to a pointer to a new path, and covers the whole to_proto method
with a rapidcheck test.

Fixes: b/64
Change-Id: I84235bed9104ff430a0acf686d4a96f1e2e9a897
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2106
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-11-19 00:29:42 +00:00
Griffin Smith
20e206a3f6 fix(tvix): Use mutable field for assigning proto fields
This was accidentally using the proto arena API to assign the derivation
field of a BuildDerivationRequest. We *thought* this was causing a
double free, but even with this change that's still happening. That
said, this change is probably still a good idea since it's using the
proto API as intended.

References: b/64
Change-Id: I950a4eafb214e9113639ea54d2dfd4659b7be931
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2104
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-11-19 00:29:42 +00:00
Vincent Ambo
8a6d00aceb feat(3p/irccat): init at '06a985y4'
This is to be used for forwarding messages to our IRC channels.

Change-Id: I6362c6f50a22f504588a7161d41a170f4e7a6edc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2073
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-11-08 18:38:08 +00:00
Vincent Ambo
1d71c29919 chore(3p/exwm): Update derivation to build local sources
... this should also update my system EXWM.

Change-Id: Idfbbda67613ac678dc2d5f82533e1c6176ab4a28
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2072
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-11-08 01:26:53 +00:00
Vincent Ambo
6104f6514f chore(3p/emacs/exwm): Import from commit '0368127976bda29d35eed788edfe74644ecd3845'
Imported from 0368127976

git-subtree-dir: third_party/emacs/exwm
git-subtree-mainline: e84f9ef0ad
git-subtree-split: 0368127976
Change-Id: Id3af5610254180f42947d71265aad89def7c6a3d
2020-11-08 01:01:28 +01:00
Vincent Ambo
4f7bb40a31 chore(3p/rr): Upgrade to latest master with Zen workarounds
* bumps to HEAD of master
* include the zen_workaround script in the drv output

Change-Id: I8f5f33a61b9395295c02f3ea5e930a81b3ffa7ac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1990
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-09-23 15:23:25 +00:00
Griffin Smith
96dbf4d686 fix(tvix): Convert INVALID_ARGUMENT to InvalidPath
The code that calls queryPathInfoUncached explicitly catches the
InvalidPath exception and translates it into a null result - but the RPC
code was throwing a regular old Error for invalid paths. At some point
we should get rid of all the exception-driven control flow in this whole
thing, but in the meantime this gets us back to functional.

Change-Id: I2a38790ee0c691ab0c8394c7738d7693fa42aa10
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1980
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-09-16 20:17:01 +00:00
Kane York
fedbe693b9 refactor(3p/tvix): make Value.primOp a std::shared_ptr
Change-Id: I04cc42ae62d2ff71b07fde8e3710dcc9b4363a05
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1973
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-09-15 21:52:56 +00:00
Kane York
d9c7f32a9c fix(3p/tvix): properly zero the largest member of Value
The static_assert is present to force us to change the constructor when a member grows.

Change-Id: Ifa4f4a03eb7ce13cff109cf26ecf0032045905e9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1972
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-09-15 21:42:54 +00:00
Kane York
2c19bd6662 refactor(3p/tvix): convert NixList usage to shared_ptr
Starting to reclaim memory. Fairly simple mechanical replacement.

Change-Id: I6b4c5c5596729470d1a049eba61e69e8097decf4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1971
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-09-15 21:42:54 +00:00
Griffin Smith
381ce8a666 refactor(tvix): Make static strings constexpr string_views
Make all static std::strings constexpr std::string_views, and replace
concatenation with absl::StrCat where necessary.

Technically all of these are constant, so they really don't need to be
top-level statics - and since I'm trying to get rid of as much global
state as possible in preparation for making the nix daemon properly
multithreaded I figured I'd knock these out while I was at it.

Change-Id: Ibd3ad9ef68f0a0eacb135541b39fdb13dae042e1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1939
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-09-14 21:38:30 +00:00
Griffin Smith
65967bbbb1 refactor(tvix): Convert refLength to constexpr
This is constant, so let's make it a constexpr.

Change-Id: I6a9eb9f740979740f730ebe142d2cbf23dbcfd70
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1961
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-09-10 17:03:45 +00:00
Griffin Smith
5dcf64c128 fix(tvix): Make working flag thread_local
Since the daemon is running in threads now rather than forking a process
per connection (thanks to grpc) this static flag to prevent accidentally
initializing a new worker during a build is getting stepped on by
multiple threads. This converts it to thread-local, and also adds an
actual message to the check so that if it hits in the future we know
what's going on.

Fixes: b/58
Change-Id: I07a2f1582e56709c104f79935e5405fa24888f59
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1940
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: V <v@anomalous.eu>
2020-09-10 17:03:45 +00:00
Griffin Smith
2ca20da43f chore(tvix): dontStrip tvix build
This is mostly to help in tracing through the core dumps for b/58, but
is also generally a good idea while we're still in hardcore
debugging-mode.

Fixes: b/63
Change-Id: Ib816a2a98bf3b8afa6e8b255a74ccb93af6508c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1937
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-09-06 22:21:12 +00:00
Griffin Smith
8f4d89e429 feat(tvix): Implement EnsurePath RPC handler
Missed this one

Fixes: b/62
Change-Id: I6f3bbcd3e2a7db5786283761b57466da33a497f7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1936
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-09-06 21:41:39 +00:00
Griffin Smith
053a98a7d7 revert(tvix): "Disable sandboxing by default until its time comes"
This reverts commit a41c3dedb1.

Reason for revert: I believe its time has come.

Change-Id: I4133dc8a3b57b40b280646d1622ed7a0409c5246
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1941
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-09-06 21:12:11 +00:00
Griffin Smith
8b7dcfaeba feat(tvix): Write even more build logs to client
Write *all* build logs (essentially everything that wasn't a DLOG), not
just the stdout of the builder, to the client-bound log_sink_ that's
threaded through build.cc. These logs are significantly more useful on
the client, both when debugging tvix itself and (more importantly) when
building things with it.

To make this all work, this commit also refactors the passing-around of
the log sink in build.cc slightly - the main reference is now owned by
the Worker class, which already stores pointers to itself on the Goals
it creates, so those Goals can then load the log sink from the Worker.
This change was originally made to allow for logging to the log_sink
from instance methods of the worker, but actually feels significantly
more natural than storing the sink on the goal anyway.

Change-Id: I461685bc7f5c0a326054940ac5971b568dd43f8e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1925
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-09-06 21:12:11 +00:00
Griffin Smith
c4136447b3 refactor(tvix): Remove default buildMode arguments from libstore
These were not used anyway, and *may* have been causing a weird series
of problems leading to things sometimes aborting. Or maybe it has
nothing to do with it - the only correlation we've found is from
disassembling the output of a core dump. Regardless, this feels
stylistically like a good idea.

Change-Id: I288b50945e74ea90c118a21767fa4387c9f47f7d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1921
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-09-01 22:05:21 +00:00
Griffin Smith
adebfbbf24 feat(tvix): Print "Tvix" in the version string
Rather than printing "Nix" in the version string, print "Tvix". This is
cosmetically nice, but also is very useful when making sure I actually
have installed tvix on a system successfully.

Change-Id: Idd1a9954bd66509327f52b7457cfba2f090ab30c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1924
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-09-01 03:30:12 +00:00
Eelco Dolstra
785cb3a754 refactor(tvix): getEnv(): Return std::optional
This allows distinguishing between an empty value and no value.

Patch ported from upstream at
ba87b08f85

Change-Id: I061cc8e16b1a7a0341adfc3b0edca1c0c51d5c97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1884
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-09-01 02:21:53 +00:00
Griffin Smith
c5f3b12f04 feat(tvix): Support systemd socket activation
When the nix daemon starts up, first check (using sd_listen_fds) whether
we have been systemd socket-activated. If so, instead of passing the nix
daemon socket path to grpc, start a manual accept(2) loop, passing the
client file descriptors to grpc via AddInsecureChannelFromFd. There's an
open grpc issue at https://github.com/grpc/grpc/issues/19133 for
building support into grpc to do this automatically, but as of right now
this appears to be the only way to make this happen.

Making this happen, by the way, was a bit of a journey - at one point I
attempted to ServerBuilder's experimental AddExternalConnectionAcceptor
API, and that didn't work either - it appears that the final missing
piece to getting this working was explicitly fcntl(2)ing the client file
descriptors to set O_NONBLOCK before passing them into gRPC. With that
set, this all works inside of the test vm.

Fixes: b/56
Change-Id: I5d2ab2b5b02eb570249b30a9674e115c61b0ab0e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1882
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-09-01 01:07:11 +00:00
Griffin Smith
e472aa016e feat(tvix): Expose depot in the test vm
Expose depot to the test vm via NIX_PATH, so we can test the build of
depot packages from inside the test vm.

Change-Id: I26314634895ab1339f5715c0b0b261028eefdaa3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1923
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-09-01 01:07:11 +00:00
Griffin Smith
2bc564bd0d chore(tvix): Add riking to owners
Change-Id: Icefafe6967cb83a43c9e417a29ce8cc97fb3432d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1892
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <mail@tazj.in>
2020-08-31 23:01:44 +00:00
Griffin Smith
2279419ede refactor(tvix): Drop meson.build files
These are unused, and I'm not sure they ever actually worked.

Change-Id: I6bb6b4257b7815932581ded7929437dba80961c9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1885
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <mail@tazj.in>
2020-08-31 15:38:16 +00:00
Griffin Smith
de84e491cf fix(tvix): localstatedir should be /nix/var
The various bits (including the unix socket that we listen on) append
/nix to this - so it isn't expected to be in there.

Change-Id: I0d95f2cc5641f610c1bdb03b036fac61dac9842d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1890
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-08-30 16:39:25 +00:00
Griffin Smith
1c64930e9f feat(tvix): Install profile scripts into the output
The nix package is expected to install /etc/profile.d/nix.sh and
/etc/profile.d/nix-daemon.sh into its output - these set the environment
variables expected by the client-side nix user commands, eg NIX_DAEMON.

Fixes: b/52
Change-Id: I5fb964c076f092d0ef7e1d49aca37875c866fb90
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1889
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-08-30 16:39:25 +00:00
Griffin Smith
74a8c3d359 fix(tvix): Chunk the AddTextToStore request
Rather than sending the entire AddTextToStore request along in a single
message, send it in a stream of chunks using the same metadata-first
approach we've been using for the other store gRPC requests. This fixes
a bug where certain builds could send more data than the maximum gRPC
request size (4194304 bytes, it would appear), resulting in a
RESOURCE_EXHAUSTED error.

The initial chunk size, which is currently constant but should be made
dynamic at some point in the future, has been chosen based on the IPC
bandwidth delay product for tazjin's desktop, rounded up.

Change-Id: I6f0232cdbc98653484816b39855126873fc59a03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1835
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-29 14:29:45 +00:00
Griffin Smith
059d90dd6d feat(tvix): Add scripts for local nix daemon testing
These are the scripts I use to test the nix daemon interaction with a
non-/nix store directory during development, copied almost verbatim from
my cmake build directory. As such, there's likely a *lot* of cleanup and
deduplication to be done here, but I'm committing these as is in the
hope that others can benefit from them somehow.

Change-Id: I42a10a85e6731fa2014c7ea9738224d678a8376b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1881
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-08-29 01:52:29 +00:00
Griffin Smith
a7cd92770d revert(glittershark): "Temporarily skip grfn's subtree"
This reverts commit 475d41f698. I'd like
my derivations back, please.

Changes necessary to get this working:

- Don't depend on `nixpkgs` being in the NIX_PATH for my website - it's
  not necessary anyway since emacs 27 is mainline now
- .skip-subtrees on things that shouldn't be evaluated anyway
- Get rid of system/pkgs, and move the one thing in there that *wasn't*
  already in third_party (alsi) to third_party
- Drop notifymuch for now - it's not working, and I'll probably get it
  landed in nixpkgs before I manage to get it working
- Add __readTree = true to my systems so they get built.
- explicitly disable ci for xanthous, which is failing to build and had
  been omitted previously

Change-Id: I20f5e81d6eb7ffe040091a08d75d0cb15304f707
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1864
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-28 21:24:36 +00:00
Griffin Smith
fb0528c5da fix(tvix): Pass all args when auto-calling a function with an ellipsis
The command line options --arg and --argstr that are used by a bunch of
CLI commands to pass arguments to top-level functions in files go
through the same code-path as auto-calling top-level functions with
their default arguments - this, however, was only passing the arguments
that were *explicitly* mentioned in the formals of the function - in the
case of an as-pattern with an ellipsis (eg args @ { ... }) extra passed
arguments would get omitted. This fixes that to instead pass *all*
specified auto args in the case that our function has an ellipsis.

Submitted upstream at https://github.com/NixOS/nix/pull/3965

Fixes: #46
Change-Id: I32b7ee0e5bacf75b2bc43a3f0796f533f4bd5959
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1863
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-08-28 01:18:19 +00:00
Griffin Smith
3bada1d41e refactor(tvix): Clean up MixEvalArgs internals
Previously, MixEvalArgs (a generic data type used to handle --arg,
--argstr, and -I arguments to `nix-build`, `nix eval`, etc.) was storing
the difference between --arg and --argstr by prepending a single
character (either 'E' or 'S') to the value of the arg. This is messy and
un-type-safe, so this commit refactors that to use a proper enum and a
std::pair, which allows us to add a switch and get totality checking.
yay, types!

Change-Id: I883f8db3136d05dda190fac0a1b494386c2ff87b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1862
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-28 01:18:19 +00:00
Vincent Ambo
eb3c4a3c11 fix(3p/haskell_overlay): Fix imports and remove unused file
Change-Id: I1973bc1b11442dc8150c3b07bd06a6f94c4c4380
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1851
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-26 23:49:32 +00:00
Vincent Ambo
7e6b3e1b42 chore: Add meta.ci=false to various broken projects
These projects, which are not currently included in CI runs, don't
build at the moment.

Upcoming logic changes would mean that we would start including them
in CI, which is undesirable until they're fixed - but I'm not going to
be doing that now.

Change-Id: I7c337e098be8bff00db6d99fc7236a695f5a85f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1850
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-26 23:49:32 +00:00
Vincent Ambo
8acbde4e15 chore(third_party): Skip subtrees of large external projects
Change-Id: Icef3900f9d1e55fd15c5286a794293368c33492c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1846
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-26 23:49:32 +00:00
Vincent Ambo
1cf11317ca refactor(tvix/libutil): Mark single-argument constructors explicit
This is the clang-tidy lint 'google-explicit-constructor'.

There's a whole bunch of breakage that was introduced by this, and we
had to opt out a few types of this (esp. the string formatting crap).

In some cases minor other changes have been done to keep the code
working, instead of converting between types (e.g. an explicit
comparison operator implementation for nix::Pid).

Change-Id: I12e1ca51a6bc2c882dba81a2526b9729d26988e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1832
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-23 11:58:44 +00:00
Vincent Ambo
1443298657 style(tvix): Add missing braces in expressions
The previous clang-tidy invocation missed some header files, which has
now been rectified.

Change-Id: I31547754fbf52f439dc7aeefb08ab90bd50c4156
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1831
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-21 03:55:56 +00:00
Vincent Ambo
7edbe59c6c style(tvix): Replace remaining typedefs with using statements
Change (mostly) created via:

  fd . -e hh -e cc | parallel clang-tidy -p ~/builds/nix --fix "'--checks=-*,modernize-use-using'"

Change-Id: I90ad6e64a5399f2005dfb9956074ec51c2c78cea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1830
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-21 03:55:56 +00:00
Vincent Ambo
ec72ca1961 fix(tvix): Add newlines to build logs
.. turns out producing an entire build log on a single line is not
useful.

This does not use `<< std::endl` because we have run into buffering
issues with the implementation of the logs->gRPC sink, but intend to
replace this in the future using a structured sink for BuildEvent
protos rather than a raw stream.

Change-Id: Ia9b05fa804391d389e2ef53ab4436c0ec5cc452e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1828
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-21 01:35:08 +00:00
Vincent Ambo
06681c35a9 refactor(tvix): Replace several DiscardLogsSink() with std::cerr
Introduces the actual log sink in several places where we actually
want the build logs to thread through correctly.

This should cover *most* build paths.

Change-Id: I735dff8a79f7e35a5874eb89b4abb980f9703dc2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1827
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-21 01:35:08 +00:00
Vincent Ambo
262af89f81 feat(tvix): Write build logs into the build log sink
This *should* wire up the builder's logs all the way back through the
gRPC client, where they are then conveniently discarded.

Change-Id: I65f22526d0b5a8b8d90f28665bc1b4bc7f7c802a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1825
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-21 00:57:18 +00:00
Vincent Ambo
e08f36c32f chore(tvix): Thread std::ostream through builder goals
This passes an output stream for build logs to almost all relevant
functions inside of build.cc by threading it through the
`Goal`-abstraction.

Store calls that create goals but don't have a sink available use the
DiscardLogsSink().

Change-Id: I2c0cb1aec1f9150f33113f4752055cea518ede8b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1824
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-21 00:37:16 +00:00
Vincent Ambo
8893c30114 chore(tvix): Add TODO comment for std::ostream in buildDerivation
Change-Id: I1e3b6abd4606d86f00ddaa8e2a1809f51e27126d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1823
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-21 00:37:16 +00:00
Vincent Ambo
375c1c2a4e style(tvix): Move a comment to fix a formatting issue
Change-Id: Idd490b978c07d1b1113ee97b5d19719a27436fba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1822
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-21 00:37:16 +00:00
Vincent Ambo
33e8b0f975 chore(tvix): Thread a std::ostream through Store::buildPaths
This part of the store API needs to carry a handle to the log sink
from now on, so that it can be passed in as appropriate from the gRPC
handlers.

In all places where there is no such handler available at the moment,
the discarding log sink has been inserted. This can be used as a
convenient grep target in the future.

Change-Id: I26628e30b4c6437dccdf8f722ca2e8ed827dfc19
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1797
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-20 11:48:40 +00:00
Vincent Ambo
f7fa77f14d style(tvix): typedef -> using in build.cc/store-api.hh
this shuts up a few lints ...

Change-Id: I51ee9418767e97247f98b170cdad137e5b12b44d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1796
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-20 11:48:40 +00:00
Vincent Ambo
a75c0278db refactor(tvix): Remove some default values from virtual methods
This is not actually legal code, but it kind of ... works. There are
more of these around, these were just the ones Griffin stumbled upon
while working on the build logs.

Change-Id: Iff9821d8fe145dd426648a8ff4510a73f67c9b7d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1795
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-20 11:48:40 +00:00
Griffin Smith
e09a6262d5 feat(tvix): Implement std::streambuf for a build log -> gRPC sink
Introduces a class which implements std::streambuf by sending build
log lines to the provided gRPC stream writer as individual messages.

This can be used in the implementations of calls which trigger builds
to forward logs back to the clients.

Change-Id: I3cecba2219cc24d56692056079c7d7e4e0fc1e2c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1794
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-20 11:48:40 +00:00
Vincent Ambo
19e874a985 feat(tvix): Introduce build event streams in worker protocol
Introduces a new `BuildEvent` proto type which is streamed in response
to calls that trigger builds of derivations.

This type can currently supply build statuses, log lines and
information about builds starting.

This is in preparation for threading build logs through the processes.

Since we have nowhere to send the logs (yet), a null sink is used
instead.

Co-authored-by: Griffin Smith <grfn@gws.fyi>
Change-Id: If7332337b89506c7e404cd20174acdaa1a3be4e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1793
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-20 11:48:40 +00:00
Vincent Ambo
883de9b8d7 feat(tvix): Add a no-op stream buffer for discarding build logs
In some cases we don't have anywhere for the build logs to go. Until
we understand those cases fully and can get rid of them, this null
sink implementation can be used.

Change-Id: Ib93c43caf268e2c01c43d59737a829e8c43d223e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1792
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-20 11:48:40 +00:00
edef
50b200f21a chore(3p/loxy): add OWNERS
Change-Id: I054711b07307ce9c0b3ce0f85319e58616db3639
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1790
Tested-by: BuildkiteCI
Reviewed-by: V <v@anomalous.eu>
Reviewed-by: tazjin <mail@tazj.in>
2020-08-19 13:20:04 +00:00
Kane York
92d4554b62 feat(tools/tvlc): init project
tvlc is a tool for managing sparse git worktrees of the TVL depot.
It is still in development; near-term tasks include a setup script,
Nix dependency resolution, worktree removal, and the dispatch
script.

See cs.tvl.fyi/depot/docs/designs/SPARSE_CHECKOUTS.md for more info.

Change-Id: Iad96656f0206178980fe7dcadd3dffe70d690f8f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1760
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-19 01:23:54 +00:00
kanepyork
32d9e6c8d0 revert(3p/loxy): Add link to source repository
This reverts commit 4d113ae9f9.

Reason for revert: Source repository link has been added to the homepage

Change-Id: I21667fab928621ba8973c8598d01920ecbf512ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1801
Reviewed-by: lukegb <lukegb@tvl.fyi>
Tested-by: BuildkiteCI
2020-08-18 22:36:18 +00:00
Vincent Ambo
4ed096b7d7 chore(3p/emacs): Enable cairo for Emacs 27.1
... this in turn enables support for Harfbuzz, which enables support
for ligatures - maybe this will make grfn happy!

Change-Id: I331c5171a829800fe532ca0b0d4910478e63ef1b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1783
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-18 22:06:16 +00:00
Kane York
290df663af fix(tvix/config): properly handle nonexistent config files
If the global nix config, or any available user nix config location, does not exist, then the loadConfFile() function will throw and not finish initalizing the Nix configuration.

Change-Id: I6db83bca54d9b66699356d107720603476e32c23
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1657
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-18 12:05:41 +00:00
Kane York
b76cd7253a feat(tvix): Re-enable language tests that needed a store
Now that we have access to a store in tests, we can enable the tests that needed a store.

Additionally, move the expected output files for disabled tests into the disabled folder.

Change-Id: I2492d49d43b93c7c9b0463e4d3d2855a5a51365d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1758
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-18 03:58:13 +00:00
Kane York
fea4df560b chore(tvix/tests): port upstream add.sh test
Change-Id: I5151d142d6b2b7f1df37b170b0160b8f77a89120
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1755
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-18 03:58:13 +00:00
Kane York
010dba3495 chore(tvix/tests): Add OpenTemporaryStore() helper
Change-Id: Ia655175c255f9cf5a47e4e7a20373a4ba4315ed9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1753
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-18 03:58:13 +00:00
Vincent Ambo
c344e03705 chore(3p): Provide Emacs 27.1
This is now available in the upstream channel, and I need it on one
machine for $reasons.

Normally the Emacs attribute is versioned to make it clear which
version we're dealing with, but not in this case. I've added an assert
to check that it is indeed 27.

Change-Id: I827df0d36350066bf4e177150d32af0c8d123b9b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1764
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-17 18:14:40 +00:00
Vincent Ambo
6f5659c88b chore(3p): Bump nixos-unstable channel
This bumps the channel to a commit that includes fixes for an nginx
issue we have been seeing:

https://github.com/NixOS/nixpkgs/pull/95264

Includes the following compatibility fixes:

- tests disabled in third_party.bufbuild: These were enabled
  unexpectedly by the update, but don't run in the sandbox because
  they want to download things from github

Change-Id: I98a3b5de57f62f1fd3a37701fa1896eddeedff85
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1759
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-08-17 09:54:18 +00:00
Vincent Ambo
4d113ae9f9 chore(3p/loxy): Add link to source repository
This is not obvious otherwise.

Change-Id: Ic07993a58e545345dae4529d281d6fdae35c19aa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1761
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-17 09:54:18 +00:00
Kane York
1fc9ba4885 refactor(tvix): always pass Bindings by ptr, use shared/unique_ptr
Value now carries a shared_ptr<Bindings>, and all Bindings constructors return a unique_ptr<Bindings>.

The test that wanted to compare two Bindings by putting them into Values has been modified to use the new Equal() method on Bindings (extracted from EvalState).

Change-Id: I8dfb60e65fdabb717e3b3e5d56d5b3fc82f70883
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1744
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-08-17 02:23:49 +00:00
Kane York
38f2ea34f4 refactor(tvix): add explicit copy/move constructors for Value
This is in preparation for making some of Value's members into refcounted ('smart') pointers.

Change-Id: Ibc54e23ac35766a2fd4e14871c9a7c936a603778
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1743
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-17 02:23:49 +00:00
Vincent Ambo
cbebc75d94 chore(3p): Remove telega derivation
The upstream telega.el derivation (under emacsPackages) now includes
the server component.

Change-Id: I49d85ea16d2fce10a505e2459fcd396706e39923
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1748
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-15 12:11:36 +00:00
Vincent Ambo
1a48f39078 fix(tvix): Ensure prim_sort actually uses the right outlist
Previously the outlist would not be used, and it would sort god knows
what in the out value.

This was probably introduced by the std::vector refactoring, and the
language test for builtins.sort was disabled. Whatever reason there
was for disabling it seems to be gone, so we're re-enabling it.

Change-Id: I98941c2cad78df58ff7bea1ece3aaa4133e94bf8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1757
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-08-15 01:58:44 +00:00
Kane York
e458e5255a feat(tvix/tests): add gtest matchers for absl::Status
This allows you to write EXPECT_OK(statusor), as well as
EXPECT_THAT(status, IsStatusCode(StatusCode::kInvalidArgument).

Change-Id: I53bed694d812c501eb305ed4ddb358e1f9a68277
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1704
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-15 00:52:54 +00:00
Vincent Ambo
d44565d17e chore(3p): Pick Emacs & friends from the default channel
I had previously pinned Emacs to the (older) stable channel, because
of a heretofore undiagnosed issue in newer versions.

It turns out that the older Emacs is linking against X11-related
libraries, namely <unintelligible>, which cause even *weirder*
undiagnosed issues if paired with the rest of the recent packages.

This commit removes the exception for Emacs and just picks it from the
same channel as everything else.

Change-Id: Ifce21c99f0204d4cf131bacb366dd9617fc301b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1751
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-15 00:26:42 +00:00
Griffin Smith
d1c38d9597 refactor(tvix): Make Store::buildPaths return a Status
Make Store::buildPaths return a Status with [[nodiscard]] rather than
throwing exceptions to signal failure. This is the beginning of a long
road to refactor the entire store API to be status/statusor based
instead of using exceptions.

Change-Id: I2e32371c95a25b87ad129987c217d49c6d6e0c85
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1745
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-14 03:06:46 +00:00
Kane York
0124691fdd fix(tvix): use correct value of SANDBOX_SHELL
Change-Id: I8e471c0f0a30337681042beeff633821eee67346
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1746
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-14 02:56:04 +00:00
Kane York
72e61aa584 refactor(tvix): completely remove boehm gc
We have decided that leaking memory is a better fate than random,
non-debuggable memory corruption. Future CLs will begin changing
various fields to std::unique_ptr and std::shared_ptr.

It turns out that disabling the GC does not have disasterous impact.
The Nix evaluator only runs on the client CLI, never in any long-
running process. Even the REPL does not leak too badly under this
change, because it uses one EvalState for the duration of the REPL.

Building an explicitly tracing garbage collector is likely in the
future of this project, but that giant amount of work cannot be
done under a nix evaluator that is constantly crashing. We need to
restore development velocity here, and this is the best way we've
figured out to do it.

Change-Id: I2fcda8fcee853c15a9a5e22eca7c5a784bc2bf76
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1720
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-08-14 00:35:18 +00:00
Vincent Ambo
e38015f7b5 refactor(tvix): Very minor lint fixes
Change-Id: I4aca504d98c79f931b7faadff86f40d27ed5a2c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1719
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-13 01:39:03 +00:00
Kane York
906f5c1d2d fix(3p/nix): inherit Expr from gc, make parser state traceable
The parser contained vectors, and the primary parser state, that
were not participating in GC tracing.

Change-Id: Ie198592cd7acffd390e3e2ae9595138b56416838
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1706
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-10 22:16:42 +00:00
Vincent Ambo
42bdaacca6 style(tvix): Sort out minor formatting errors
Invocations of the MakeError macro that were not followed by a
semicolon messed up indentation in the next lines.

Change-Id: I03d7d1443f062a38af2c7da3da8928e0ed05e274
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1708
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-10 17:40:18 +00:00
Vincent Ambo
a1965e9182 fix(tvix): Print invalid store path when encountered
... instead of silently swallowing it.

Change-Id: I084c51d633d632bd4e3f1ca42987dd6068fbb2a9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1707
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2020-08-10 16:49:46 +00:00
Griffin Smith
d1653533a6 test(tvix): Cover scanForReferences in a test
Aded a few test cases covering the scanForReferences function, which had
been accidentally broken in 976a36c (which is now partially-reverted).
As part of this, since the test needed to generate hashes for store
paths, the logic in MakeStorePath to compress a sha256 hash down to 20
bytes and convert it to base32 has been extracted to a member function
on the Hash class.

Fixes: #34
Change-Id: Ie2d914688a80f42d0234d351a7cc0714fd15709e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1698
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-09 17:26:19 +00:00
Griffin Smith
ec7d368546 fix(tvix): Make nix value structs inherit from GC
All of the miscellanious structs that make up a nix Value should inherit
from gc, as they contain pointers to GC'ed things as members

Fixes: #42
Change-Id: I057d8c9f7dafbee7de7644ff152c9cba1aa6bc03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1696
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-09 02:22:44 +00:00
Griffin Smith
059468ba15 refactor(tvix): Remove UDSRemoteStore
Now that we've fully implemented the RPC-based store client, we can get
rid of the UDSRemoteStore, whose only use was connecting to the locally
running nix daemon. The RemoteStore still needs to be around to connect
to remote upstream nix stores over SSH.

Change-Id: I0699819803cbfe966b9a46786f2c927d8e4bf1a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1693
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-09 02:22:44 +00:00
Griffin Smith
e440f60b6c feat(tvix): Implement all remaining RPC calls
Implement all remaining RPC calls on the RpcSstore client, remove a few
stub methods we had added that weren't actually present in the old
RemoteStore implementation, and add one more RPC call for getBuildLog
that is present in the store API, but that we hadn't added as a
stub *or* to the proto.

Fixes: #29
Change-Id: Id827f51a393ece4bc7bbecaf38aee9eb4b329770
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1692
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-08-09 02:22:44 +00:00
Griffin Smith
747dc65154 refactor(tvix): Factor-out proto utilities
Factor out the shared utilities for interacting with protobufs/grpc from
libstore to a proto.hh header in libproto.

Change-Id: I1cb8d94867d5d4b63a9994be0b53f8f612eb8e3a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1691
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-09 01:52:59 +00:00
Griffin Smith
ef7894273b feat(tvix): Pass function name to SuccessOrThrow
To aid in debugging RPC call failures, pass the name of the function
being called to the SuccessOrThrow util funcion in the RpcStore

Change-Id: I523dacfab896b85a3dbe6050c07ee6bd5906fa44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1690
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-09 01:17:09 +00:00
Kane York
1de00e6c42 chore(3p/nix): apply google-readability-casting
Command run: jq <compile_commands.json -r 'map(.file)|.[]' | grep -v '/generated/' | parallel clang-tidy -p compile_commands.json -checks=-*,google-readability-casting --fix

Manual fixes applied in src/nix-env/nix-env.cc, src/libstore/store-api.cc

Change-Id: I406b4be9368c557ca59329bf6f7002704e955f8d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1557
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-08-08 22:16:02 +00:00
Griffin Smith
053a138002 fix(tvix): Wrap remaining RPCs in HandleExceptions
Wrap the BuildPaths and AddTextToStore RPC handlers in HandleExceptions.
These were missed in the original pass due to a merge.

Change-Id: Ie5be45e6098fba7a2b6b1c1be81578cb742c2880
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1689
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-08 20:03:59 +00:00
edef
3049f31d28 chore(3p/nix): don't optional-wrap EvalState::file_access_trace_fn further
std::function has a natural null we can't eliminate anyway, so this was
effectively std::optional<std::optional<non_nullable_function>>.

Change-Id: If99f139146021edb25d133dad7f0f6e125ef53df
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1688
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-08 18:01:11 +00:00
Griffin Smith
7db734afad feat(tvix): Implement AddToStoreNar
Implement both the client and server sides of AddToStoreNar, using a
templated generalization of the sources and sinks we were using for
AddToStore on both ends.

Change-Id: I73d0ed34118c711b125851dff99a7518ced4af35
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1686
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-08 17:51:13 +00:00
Kane York
a5dae62e85 fix(3p/nix): Use SkipEmpty in all calls to absl::StrSplit
The behavior to return a list containing a single empty string when
provided an empty string is a behavior that absl inherited from legacy
code. However, the behavior expected by legacy code in Nix is the
behavior provided by the SkipEmpty option. Switch all calls to use
SkipEmpty, except for the call already using SkipWhitespace.

See also commit 26a59482d2, with the
partly-prophetic message: "there may be other places we need to
fix this as well."

Change-Id: I6e94856a12cfb1b7e4a3b4e221769ed446648861
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1687
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-06 20:50:47 +00:00
Vincent Ambo
6a97206ceb refactor(tvix): Use absl::btree_map for BasicDerivation's env
Change-Id: I111a9a268debea322f23fdced3bed9ff3e8ed3b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1685
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-06 02:49:58 +00:00
Vincent Ambo
3ec3661678 refactor(tvix): Use absl::btree_map for DerivationInputs
Change-Id: If160ab1b09161969d9080d5d0d6f82e44a953e3c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1684
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-06 02:30:28 +00:00
Vincent Ambo
91bd7ce73a refactor(tvix): Use absl::btree_map for DerivationOutputs
This container implementation is much faster than std::map. We have
stuck to an ordered container because it's unclear whether the
accesses of this field (of which there are *many*) are actually
ordering dependent.

Also includes an Arbitrary implementation for absl::btree_map (for any
K, V that are also Arbitrary).

Change-Id: I04f58ca0ce32b9ae1759313b01508b0e44bae793
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1683
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-06 02:20:36 +00:00
Vincent Ambo
a41c3dedb1 chore(tvix): Disable sandboxing by default until its time comes
Change-Id: Ifcc776841924abcb43fb115f3de569fe500756c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1682
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-06 02:20:36 +00:00
Vincent Ambo
d42c3dd72f chore(3p): Bump nixpkgs channels
Bumps both nixos-unstable and nixos-20.03 to today's versions, as per
status.nixos.org

Contains minor fixes to things that broke because of the update:

* tazjin/frog: hardware.u2f is a deprecated setting
* glittershark/system: modSha256 in Go modules is now vendorSha256
* glittershark/owothia: removed version constraint on relude

Change-Id: Ib3e9612b1b06ed547b90e4f8b0ffe5ed7fe0a5c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1642
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-06 00:15:33 +00:00
Kane York
68b5306c56 feat(3p/nix): add --trace-file-access to nix-instantiate
This builds on edef's work with depot-scan by adding a dedicated
flag to the command. We piggyback on upstream's restricted-mode
implementation, the checkSourcePath function.

Change-Id: I52bb613549f40dbca1e8caa036635910c1a3d6d0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1654
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 22:49:22 +00:00
Griffin Smith
ea488b5705 feat(3p/nix): Implement FindRoots, CollectGarbage
Implement the RPC client and server handlers for the FindRoots and
CollectGarbage RPC calls

Change-Id: Ifa5d582c6a33bd1e7661ac2fc860505ef404dad0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1656
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-05 22:28:34 +00:00
Kane York
f9df9b4733 fix(3p/nix): fix usage error of absl::Base64Unescape
The source and destination strings cannot be the same string - absl
will write to the destination in a streaming manner, causing the
source to become invalid.

Change-Id: I3578cf1f8789a51d85e0950f7987c398f0a00953
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1659
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 18:29:22 +00:00
Kane York
f10d60a454 fix(3p/nix/store): fix race condition in downloader
In certain circumstances, the decompression thread could race ahead of the downloader thread and process the same chunk twice. Clear the data buffer while the lock is held to prevent this kind of incident.

Change-Id: I19a84a0c5768d1228c6c18a7664a7b8893ef96de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1658
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 18:29:22 +00:00
Vincent Ambo
3f923b2aa0 refactor(tvix): Use SANDBOX_SHELL from environment in CMake
This makes the configuration work out-of-the-box using CMake in a Nix
shell, rather than having to pass the additional variable to CMake on
the command line.

Change-Id: I04d3cc4f2e5ecf47bf2ee459d5e48588b84ae4dd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1643
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 11:37:20 +00:00
Kane York
ffa8e7a998 feat(3p/nix): add tests for BinaryCacheStore
These tests are in preparation for factoring the Store away from libutil's Callback to absl::StatusOr.
They use the newly added MockBinaryCacheStore.

Updates: #25

Change-Id: I30c207589ec38254806ebc9a983f35668e353ae9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1595
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 04:11:57 +00:00
Kane York
3c3527e16b feat(3p/nix): add MockBinaryCacheStore
This adds an implementation of BinaryCacheStore to be used by tests exercising both the logic inherent to BinaryCacheStore and more general Store tests.

A new library target, nixstoremock, is created to indicate that this file is intended only for use in tests and not in user-facing code.

Change-Id: Ib68f777238843a4f3a2303db8a69735fbc22d161
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1645
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 04:11:57 +00:00
Griffin Smith
28c7e926ea feat(3p/nix): Implement AddToStore rpc client
Implement the AddToStore RPC client method, which uses an
AddToStorePathWriterSink (the dual of the AddToStorePathReaderSource on
the server side) to hook into the dumpPath machinery (which we should
refactor not to use sinks or sources, but not yet) and write dumped
paths as binary data over gRPC. With this commit and sandboxing
disabled, the following derivation builds cleanly:

    derivation {
      name = "test";
      builder = ./build.sh;
      system = "x86_64-linux";
    }

where build.sh has chmod +x and contains:

    #!/bin/sh
    echo 1 > $out

Change-Id: I94cab86f0825a3a9993262a9807130645c13bf44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1638
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-08-05 03:24:36 +00:00
Kane York
b15b447fcb fix(3p/nix): fix clang-tidy for int-string appends
Change-Id: I276de7a5fd1c705c87d35dd616e5980c747190aa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1597
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 02:54:47 +00:00
Griffin Smith
5feed69260 feat(tvix): Implement AddIndirectRoot, SyncWithGC
Implement RPC handlers and client methods for the AddWithRoot and
SyncWithGC calls. With this and sandboxing disabled, the following
derivation is now building successfully:

    derivation {
      name = "test";
      builder = "/bin/sh";
      args = [ "-c" "echo 1 > $out" ];
      system = "x86_64-linux";
    }

Co-authored-by: Vincent Ambo <mail@tazj.in>
Change-Id: I245dca0dfca882e2eae5d93364287a8b79fead71
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1637
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-08-04 22:49:56 +00:00
Griffin Smith
d8386d6f7e chore(tvix): Make not implemented errors nix::Unsupported
Without this, these errors print as "uncaught exception of type
std::string", but with this they print using nix's built-in error
printer, which will be nice to have as we bang out the remaining store
API methods.

Co-authored-by: Vincent Ambo <mail@tazj.in>
Change-Id: I9bd4f77f65b27c8f97b727b98ef62e2061999e71
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1636
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-08-04 22:49:56 +00:00
Griffin Smith
e634e27577 fix(tvix): Partially revert "eliminate exposed global variable"
This reverts part of commit 976a36c2e4,
which was causing scanForReferences to hang indefinitely. I'm not sure
what the original intent of the commit was since there's not really much
of a commit message on it, but with this I am able to fully realise
derivations.

Change-Id: I620e892e2fe2244f066512286b086b3dd8f8f6c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1635
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-04 22:49:56 +00:00
edef
94387dbc25 chore: simplify buildGo expressions
This removes redundant `map (p: p.gopkg)` expressions.

Change-Id: If25d603c27d63ae011f396caf5d26cdfdb8c28d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1625
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-08-04 07:43:25 +00:00
edef
cf35ad4fa9 fix(3p/gopkgs/cloud.google.com/go): add missing depot arg
Change-Id: If08a43e18a41abd3e2ae7c08ceb6f48844d20df1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1626
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-04 07:42:53 +00:00
Griffin Smith
0aff4c256c fix(3p/nix): Don't assert on argc count for build-remote
This executable doesn't appear to ever actually look at argv, so
throwing an error if argc is wrong seems kinda silly.

Change-Id: Iabc9dad785d4c99be9e8fd4ecd954d38b876cecd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1633
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-04 02:15:10 +00:00
Griffin Smith
09cb41b7ac fix(3p/nix): Properly configure SANDBOX_SHELL
point the SANDBOX_SHELL macro at the actual path to busybox on the build
machine, or allow it to be configured at build-time with a cmake option.

Change-Id: I044a1315ba9baa3bc9ceddf29f36d14f9f9ccd96
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1632
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-04 02:15:10 +00:00
Griffin Smith
26a59482d2 fix(3p/nix): pass SkipEmpty to StrSplit("", ...)
When tokenizeString was changed to absl::StrSplit, there was a behavior
change because tokenizeString on an empty string returned an empty
vector - which the derivation builder (and likely a bunch of other
stuff) was depending on. The canonical way of fixing this is by passing
absl::SkipEmpty() to the function - there may be other places we need to
fix this as well.

This commit also includes some opportunistic absl::StrFormats and
StrCats, because I was here anyway, but those have no semantic
difference.

Change-Id: Ibf9bb602284f793fa55728481f63b838fb7a41db
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1631
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-08-04 00:43:02 +00:00
Kane York
2344f8e528 feat(3p/nix/daemon): catch-all explicit Error-Status conversion
We wrap every server-side proto handler with a macro that catches
exceptions and turns them into proper grpc error codes. For the
time being, most exceptions map to INTERNAL, the existing mapping.

Change-Id: Id6ed6a279b198ad185d32562f39000ccc15eadbf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1599
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-04 00:26:01 +00:00
Vincent Ambo
ae31f51dc8 fix(3p/nix): Initialise glog when starting nix-daemon
Change-Id: Ic9b5bd2f66fb54675498d8328edecfd31a03997a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1630
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-03 23:18:58 +00:00
Griffin Smith
31b9578ad0 fix(3p/nix): Fix parseDrvPathWithOutputs
At some point the behavior of this function got changed as part of our
cleanup - this fixes it to behave the way the rest of the codebase
expects (and how it is documented in the header) and covers it with a
few tests.

Change-Id: Id4c91232968e73489cd866fb4a2a84bcf20d875e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1629
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-03 23:17:41 +00:00
Kane York
661353f200 chore(3p/nix/daemon): replace all assertStorePath with a macro
Two exceptions: IsValidPath needs to return success for invalid paths, and QueryAllValidPaths shouldn't need to check the paths it gets from itself.

Change-Id: I4d9d4125d34e8de42f30070aec607f8a902eded7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1598
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-03 18:46:45 +00:00
Vincent Ambo
1f12544179 refactor(3p/nix): Build nix-daemon as a separate binary
Skips over all the monobinary stuff and moves to a separate binary for
nix-daemon.

This also replaces the flag parsing logic with absl::flags. This
causes a behaviour change for --help, which no longer tries to display
a man page but instead shows the actual command-line help.

Note: This binary no longer links to the Boehm GC.

Change-Id: Ib852e994b82f2d56e91262878c10650e656427a9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1622
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-03 18:26:44 +00:00
Vincent Ambo
229c1ed820 chore(3p/nix): Move legacy Nix daemon implementation
The filename this is squatting is too good.

Change-Id: I694132b94d24786df84b6b5a698679bdce818b2e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1621
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-03 18:18:54 +00:00
Griffin Smith
209489e348 feat(3p/nix): Implement two more RPC calls
Implement AddTextToStore and BuildPaths both on the client and the
server

Refs: #29
Change-Id: I45294c3e1c1a7489e42099d36425b7acc04e0427
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1560
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-08-03 17:32:27 +00:00
Kane York
8a1c7da357 feat(3p/nix/build-shell): add run_clang_tidy script
This makes it easy to quickly run clang-tidy on tvix without seeing errors from the generated files.

Change-Id: I0e25089c5626aebdb5c016629a68da9ccd26c124
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1556
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-03 17:06:41 +00:00
edef
c17be2364e chore(3p/loxy): Move //users/edef/loxy to //third_party
Change-Id: Ia451d190d4e415ef875c51ebeca12c7c5dc72342
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1590
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-08-03 10:32:39 +00:00
Alyssa Ross
6b181d314f fix(3p/cgit): transpose html_option parameters
I got these the wrong way round, fixed them in the patch in my Nix
configuration, and then neglected to fix them in the version for
depot.

Fixes: be98295dc2
Change-Id: Ib7d4e433887f474ae9675e9948ab38f640181095
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1383
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-08-03 10:28:44 +00:00
Vincent Ambo
46c78aa0f9 refactor(3p/nix): Only initialise garbage-collector where needed
Only libexpr depends on the garbage collector, specifically only
instantiations of EvalState actually require the GC to be initialised.

Rather than always starting it for the whole program, even if it is
not needed, this change moves the GC initialisation into libexpr,
guarded by absl::call_once.

This should make it possible to run the nix daemon without the garbage
collector interfering, granted that things are correcty separated and
the daemon does not actually invoke the evaluator.

Based on my investigation so far, the daemon logic itself does not
require libexpr to be present at all - so I think it is safe - but the
current monobinary might have some tricks up its sleeve that will
cause problems for us. We can deal with those if they arise.

Relates to https://b.tvl.fyi/issues/30

Change-Id: I61c745f96420c02e089bd3c362ac3ccb117d3073
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1584
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-03 08:38:57 +00:00
edef
8862f3fbaf feat(3p): Whitelist unzip
Change-Id: Iae7e661dc45f8bc8668c1aa4fe82b88e8868c22f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1585
Tested-by: BuildkiteCI
Reviewed-by: ericvolp12 <ericvolp12@gmail.com>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-08-03 08:17:51 +00:00
Griffin Smith
cc82d6e360 test(3p/nix): Add test for derivation parse/serialize
Add a rapidcheck test covering roundtrip parse and serialize for Nix
derivations. This covers a bug we discovered in ef54f5d which broke this
roundtrip.

Change-Id: I72d140334b5f24f79e82e34f98609c695dbfbf93
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1582
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-02 20:00:08 +00:00
Griffin Smith
e0d25449e3 fix(3p/nix): Don't reuse ClientContext
It turns out these aren't supposed to be reused after the first request
- reusing it was causing things to blow up.

Change-Id: I935f1ee82ed4e7f26528ae105c8fa2501fa81b97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1559
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-08-02 02:12:14 +00:00
Griffin Smith
2646e57aae fix(3p/nix): Prepend unix:// to daemon socket
Prepend the unix:// URI scheme to the daemon socket so that grpc knows
we want to connect to a unix socket rather than another type of URI. As
part of debugging this I made the failure message for the RPCStore
include the URI, which I'm leaving in since it'll be nice to have.

Change-Id: I6e70596895117b9a0d53fe2a61d8542ceb64c940
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1543
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-08-02 02:03:24 +00:00
Luke Granger-Brown
173f2c8f76 chore(gerrit): Update to latest HEAD.
Change-Id: I650777bbbd24a1922f26967fbbd7da06d14b6786
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1516
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-02 01:01:08 +00:00
Kane York
d2eff4153d chore(3p/nix): disable modernize-use-trailing-return-type
Change-Id: I47cba88de5c82923a57804432bdd4f787b78ac8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1552
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-01 23:52:45 +00:00
Kane York
72fc2fd27e fix(3p/nix): revert "apply all clang-tidy fixes"
This reverts commit ef54f5da9f.

Resolved conflicts:
	third_party/nix/src/libexpr/eval.cc
	third_party/nix/src/libstore/builtins/fetchurl.cc
	third_party/nix/src/libstore/references.cc
	third_party/nix/src/libutil/hash.cc
	third_party/nix/src/nix-daemon/nix-daemon.cc

Change-Id: Ib9cf6e96a79a23bde3983579ced3f92e530cb011
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1547
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-01 22:45:15 +00:00
Kane York
f7a85c5060 fix(3p/nix/expr): use traceable_allocator for btree_map
The prior use of gc_allocator meant that the btree nodes themselves were being collected. Additionally, have Attr (contains a Value) and Bindings explicitly inherit from gc, even though Bindings is always allocated under `new (GC)`.

Detected by running under GC_ENABLE_INCREMENTAL=1.

Change-Id: Iacf13b34b5aa12e417ea87c9b46e2bf9199fdb26
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1544
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-01 21:16:29 +00:00
Griffin Smith
3fdce7c6be feat(3p/nix): Implement a few more RPC calls
Implement the RPC client calls for QueryPathFromHashPart,
QuerySubstitutablePaths, and QuerySubstitutablePathInfos, and the
handler for QuerySubstitutablePathInfos.

Refs: #29
Change-Id: Idf383b771f159f267d8f65367bc4af3d239e32b7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1515
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-01 19:00:08 +00:00
Kane York
ee48e830e6 fix(3p/nix/libexpr): fix GC tracing in valueSize
Change-Id: I2f6bef7b090d44f50bd27fbd19b50f9cf100b238
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1506
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-08-01 18:54:50 +00:00
Kane York
64f6bb6951 feat(3p/nix): remove External values feature
External values are only useful when using the plugin framework, which we are not interested in carrying forward.

Reverts commit 320659b0cd

Change-Id: Ib4929c349bbb33f16224fc674e94c7b7d5953c6a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1505
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-08-01 18:54:50 +00:00
Kane York
770034042a fix(3p/nix): Use a proper pointer in Env to carry with-attrs
This eliminates the value-smuggling that would trip up the GC.

Change-Id: I8057df78cf0bf6bea9faf1b44233aa9820ae44f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1504
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-08-01 18:54:50 +00:00
Kane York
dc4c0bad65 chore(3p/nix/libexpr): Cleanups and notes in eval.cc
Add two more garbage-collection flags. Annotate how terrible tExternal is. Prepare to fix the smuggle casting in ExprWith. Add a static_cast.

Change-Id: I20f980abc8cb192e094f539185900a6df5457c29
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1503
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-08-01 18:54:50 +00:00
Luke Granger-Brown
f7d5d5c542 fix(gerrit): Don't do a deep fetch
I don't know what I was smoking.

Change-Id: I650777bbbd24a1922f26967fbbd7da06d14b6781
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1514
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-01 15:02:05 +00:00
Griffin Smith
05e44c121d feat(3p/nix): Implement AddToStore proto handler
Implement the proto handler for AddToStore, which adds a nix path to the
store. This is implemented by adding a new (probably
soon-to-be-generalized) Source concretion that wraps a grpc ServerReader
for the stream of data we're receiving from the client - this is less
than ideal, as it's perpetuating the source/sink thing that's going on
and storing entire nars in memory, but is at the very worst an
incremental step towards a functioning nix that we can refactor in the
future.

Paired-With: Perry Lorier <isomer@tvl.fyi>
Paired-With: Vincent Ambo <mail@tazj.in>
Change-Id: I48db734e7460a47aee4a85dd5137b690980859e3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1441
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <mail@tazj.in>
2020-08-01 14:11:41 +00:00
Kane York
674dbade27 fix(3p/nix/hash): param of Unknown allows any hash type
Fixes a crash in the self-hosting instantiate test:

NIX_REMOTE="$(mktemp -d)" nix-instantiate -E 'let depot = import ./default.nix {}; in depot.third_party.nix.outPath'

Change-Id: If99494aa07ec248d3894d4709ab0fde7fa81aff3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1508
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-08-01 01:15:29 +00:00
Kane York
1cbffe21f3 chore(3p/nix/hash): prefer StatusOr over throwing constructor
The use of `unwrap_throw` can be used as a later grep target.

Change-Id: I8c54ed90c4289f07aecb8a1393dd10204c8bce4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1493
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-08-01 01:15:29 +00:00
Kane York
2a292c71f4 fix(3p/nix/hash): initialize HashSink.ctx
Fixup for CL 1492 (addcba11b0)

Additionally, add a test to verify functionality of HashSink.

Change-Id: I2a74b925a1b93ed4d3add29021d759c93e813424
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1507
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-01 00:09:23 +00:00
Kane York
addcba11b0 fix(3p/nix/hash): smart pointers in HashSink
Change-Id: Ib2aaf42c8b234ee343c4653eb03f328c113dea86
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1492
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-29 06:48:47 +00:00
Kane York
31f9ee58d0 fix(3p/nix/hash): provide a Status-returning constructor
Additionally, add IsValidBase16() to restore the behavior of rejecting invalid base16, which absl's HexStringToBytes does not do.

Change-Id: I777a36f5dc787aa54a2aa316d6728f68da129768
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1484
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-28 02:04:42 +00:00
Kane York
976a36c2e4 chore(3p/nix/hash): eliminate exposed global variable
Change-Id: I3b34e3e17a751e225831ae599c6c6bb782a25679
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1486
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-28 02:00:23 +00:00
Kane York
ec349328e0 chore(3p/nix/hash): add decoding tests
Change-Id: Ifbeaf1822fa920f929482510ee79a5b24d7976ae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1485
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-28 00:38:29 +00:00
Kane York
ef54f5da9f fix(3p/nix): apply all clang-tidy fixes
Change-Id: I265e763393422ee1881653527c91024458060825
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1432
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-27 21:16:39 +00:00
Kane York
80ff83e698 fix(3p/nix): convert local-store asserts into throws
This fixes a clang-tidy DeadStore warning by turning debug asserts into production checks.

Updates: #11
Change-Id: Ia6e5a4cb1b56594c9844c6bbd3d152f84b426d09
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1409
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-27 20:50:57 +00:00
Griffin Smith
d98f2ea68f feat(3p/lisp): Add checkl
Change-Id: Ib73ed9637b7e22050727ac5ec117241e18d3cc45
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1472
Tested-by: BuildkiteCI
Reviewed-by: eta <eta@theta.eu.org>
2020-07-27 14:18:06 +00:00
Luke Granger-Brown
a8bc8357d8 feat(3p/gerrit_plugins): init Gerrit checks plugin
The Gerrit Checks plugin adds a new tab to the Gerrit UI, which is
intended for display of status of automated checks which are being run.
We can use this for e.g. reporting the run status of our CI builds/other
stuff.

Change-Id: Ib0d9a8ae68061a76191a56d467d915100b766e1b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1462
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-27 00:00:48 +00:00