Commit graph

400 commits

Author SHA1 Message Date
sterni
a2ecd53f58 chore: clean up some obsolete TODOs
* //3p/nix: probably not worth investing time into this anymore

* //users/sterni/emacs: The emoji problem disappeared by itself with a
  newer emacs version, however a different one remains…

* //web/panettone: If we ever want to change the behavior, we should
  just decide the behavior statically instead of using conditions and
  restarts, as we only call it in one place, so making different
  decisions depending on call sites is not really a use case we have.

Change-Id: Iff9d439ce356db41ce34d690fb7b6a01822022fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5223
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
2022-02-04 15:55:19 +00:00
Vincent Ambo
aa122cbae7 style: format entire depot with nixpkgs-fmt
This CL can be used to compare the style of nixpkgs-fmt against other
formatters (nixpkgs, alejandra).

Change-Id: I87c6abff6bcb546b02ead15ad0405f81e01b6d9e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4397
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: wpcarro <wpcarro@gmail.com>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: cynthia <cynthia@tvl.fyi>
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: eta <tvl@eta.st>
Reviewed-by: grfn <grfn@gws.fyi>
2022-01-31 16:11:53 +00:00
zseri
358a4ac6d4 feat(3p/tvix): run formatting check in parallel to rest
Change-Id: Ia5b3a5e76672143df361fec375903e3582694112
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4632
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: zseri <zseri.devel@ytrizja.de>
Autosubmit: zseri <zseri.devel@ytrizja.de>
2021-12-27 04:10:32 +00:00
sterni
04f7cc3880 fix(3p): resolve linker issues with clang-compiled C++ packages
The point where this broke is likely when llvmPackages_11.stdenv started
respecting stdenv.hostPlatform.linker and thus using GNU binutils'
ld.bfd for linking. That linker can't deal with clang's C++ ABI.

To fix this we introduce a modified version of llvmPackages_11.stdenv
which uses llvmPackages_11.bintools for linking purposes (I also aim to
upstream this in some form, but am not sure about the details atm, so
we'll just do this in the tvl overlay for now). This is the precondition
for getting our C++ packages in //third_party to work
again. Additionally the following fixes were necessary:

* abseil-cpp needed to be updated (by overriding the version from
  nixpkgs for now, since I can't update the subtree on my own) to
  fulfill grpc's requirements (grpc 1.41.0 needs abseil-cpp LTS
  20210324).

* gtest needs a patch from nixpkgs which fixes the path to its include
  directory for CMake.

* We need to build re2 with clang as well, otherwise linking will fail.

Fixes b/132.

Change-Id: I7b64579fe96451547babe070fd35db398581e49d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3701
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2021-10-10 10:03:03 +00:00
Vincent Ambo
65be8f20e0 chore(nixpkgs): Bump channels to 2021-05-25
* users/grfn/system/home/yeren: remove obsolete awscli2 overrides

* ops: make new isSystemUser || isNormalUser assertion happy

* users/grfn/system/system/mugwump: make buildkite agents system users

* users/tazjin/nixos/camden: set isSystemUser = true for git

* users/tazjin/emacs: Remove missing & broken packages

* third_party/openldap: remove, as the argon2 module is now enabled upstream

* third_party/gerrit_plugins: Pinned new unstable hashes

* third_party/nix, third_party/grpc: Disabled CI as these are broken

* third_party/overlays/emacs: Bumped version to stay in sync with channel

* third_party/buzz: Update LIBCLANG_PATH to reference libclang.lib,
  since libclang's default output no longer contains libclang.so

* users/grfn/system/home: Install julia-stable instead of julia (which
  aliases to julia-lts), as the latter depends on an insecure version of
  libgit

Change-Id: Iff33b0ecb0ef07a82d1de35e23c40d2f4bf0f8ed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3001
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
2021-05-25 17:09:28 +00:00
Vincent Ambo
3c7d4c74be chore(3p/overlays): Remove LLVM11 bump from overlay
* This was mostly for //third_party/nix and its dependencies which now
  have been set to use llvmPackages_11 manually.

* For //users/grfn/achilles we also manually select the newer LLVM version.

* //tools/cheddar doesn't seem to need llvm anymore.

* //third_party/buzz also compiles with clang 7.1.0

* replace clang-tools everywhere with new attribute clang-tools_11

For the future we may want to have something similar again, but it may
not be necessary to invest too much time into it: nixpkgs is set to
upgrade their default llvmPackages to LLVM 11 as well at some point in
the near future.

Co-Authored-By: sterni <sternenseemann@systemli.org>
Change-Id: Id83868dbc476a6c776b59518b856c933f30ea79d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3135
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
2021-05-24 18:15:05 +00:00
Vincent Ambo
f520bd40ca refactor: Replace 'depotPath' with 'depot.path'
Instead of having two ways of accessing the path to the depot (one of
which was stuttering, depot.depotPath) we settle on only one:
depot.path.

This was mostly used for NixOS module imports.

Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: I2c0db23383fc34f6ca76baaad4cc4af2d9dfae15
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2962
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-12 21:55:07 +00:00
Griffin Smith
6266c5d32f refactor(users/glittershark): Rename to grfn
Rename my //users directory and all places that refer to glittershark to
grfn, including nix references and documentation.

This may require some extra attention inside of gerrit's database after
it lands to allow me to actually push things.

Change-Id: I4728b7ec2c60024392c1c1fa6e0d4a59b3e266fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2933
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
2021-04-12 14:45:51 +00:00
Vincent Ambo
7deabb8c8d chore(3p): Bump channels to 2021-04-09
Fixes included:

* exposed gtest in the package set, required for protobuf
* pinned SBCL to version 2.0.8: The channel moved it to >2.1, and a
  bunch of warnings seemed to be killing our builds - we should
  investigate this later.
* removed kernel patches from //users/tazjin/frog: this machine is
  currently out of service anyways, not worth fixing while it's offline
* removed steam & lutris from frog (they're currently broken)
* removed Haskell overrides for hedgehog-classes & hgeometry-combinatorial
* use gRPC sources from upstream and inject Abseil via Nix instead
* fix for renamed grpc import in //third_party/nix
* use libfprint-tod from upstream nixpkgs in glittershark/yeren and
  delete glittershark/pkgs/fprintd entirely, since all of the patches used
  there are available and working from upstream now (and stopped working
  here after the bump)

Change-Id: Ia90e6f774f7b88bc9e60d28351b900ca43ee2695
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2901
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Tested-by: BuildkiteCI
2021-04-11 22:13:32 +00:00
Vincent Ambo
a5d8f5273c fix(3p/nix): Fix dependency inclusion and reenable CI
Change-Id: Ic2b7db4abe636b73b4c71e6a2f1501d3b4b8b90e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2922
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-10 21:46:02 +00:00
Vincent Ambo
473604f567 refactor: Move nixpkgs attribute to third_party.nixpkgs
Please read b/108 to make sense of this.

This gets rid of the explicit list of exposed packages from nixpkgs,
and instead makes the entire package set available at
`third_party.nixpkgs`.

To accommodate this, a LOT of things have to be very slightly shuffled
around. Some of this was done in already submitted CLs, but this
change is unfortunately still quite noisy.

Pay extra attention to:

* overlay-like functionality that was partially moved to actual
  overlays (partially as in, the minimum required to get a green
  build)

* modified uses of the package set path, esp. in NixOS systems

Special notes:

* xanthous has been disabled in CI because of issues with the Haskell
  overlay
* //third_party/nix has been disabled because of other unclear
  dependency issues

Both of these will be tackled in a followup CL.

Change-Id: I2f9c60a4d275fdb5209264be0addfd7e06c53118
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2910
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2021-04-10 21:18:55 +00:00
sterni
e628862e97 chore(3p): Bump NixOS channels to 2021-02-18
Main motivation for this is to get the openldap update that fixes
10 CVEs: CVE-2020-36221 to including CVE-2020-36230. See also this
issue which lists them all: https://github.com/NixOS/nixpkgs/issues/113490

Someone should also redeploy whitby as soon as this lands in canon and
all build failures have been fixed.

Things done to resolve upstream breakages:

* grpc no longer takes abseil-cpp as an input, it has also been removed
  in the override.

* Upgrade glittershark's kernel to 5.11 since the linuxPackages_5_9
  attribute has been removed by upstream and the patch used by them is
  available for 5.11 as well.

* The fixed output hash for third_patry.apereo-cas changed for some reason.

* Remove the pin of haskellPackages.vector from the haskell overlay. It
  broke as the most recent version of vector in nixos-unstable no longer
  depends on semigroups. This effectively updates vector from 0.12.1.2
  to 0.12.2.0.

* Align two comments in tvix/libstore/worker-protocol.hh because the
  updated clang-format now demands that.

Change-Id: I2ecf10a98de935e9222acf1feaea447d4c11ed2d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2538
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-02-19 14:45:43 +00:00
sterni
4f6f4e6588 docs(tvix): fix path in build instructions, add note about tests
Change-Id: If9dcd8e39f416c11460e66454df9f53096691699
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2484
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2021-02-02 17:59:44 +00:00
sterni
8709153973 fix(tvix/libexpr): backport fix for functionArgs failing on primops
Nix internally differentiates between lambdas and primops, but their
type in the nix expression language is the same (lambda). The
implementation of builtins.functionArgs only checks if the given
expression is of type tLambda and fails if the type is tPrimop or
tPrimopApp which are also functions. This most notably breaks
lib.generators.toPretty when called on a builtin making for example
yants fail if a primop is typechecked and an error message is
generated.

This fix generates an empty set for primops like for plain lambdas
and is based upstream commit b2748c6e99239ff6803ba0da76c362790c8be192.

Additionally we add to two tests:

* eval-okay-functionargs now includes a few test cases checking that
  builtins.functionArgs always returns an empty set for builtins and
  also works as expected for normal functions.
* eval-okay-types now also checks if builtins are functions.

Future work would be to make builtins.functionArgs work as users would
expect for builtins like builtins.fetchurl, builtins.fetchGit etc. which
take a set as an argument. These currently don't register as formal
arguments, but it would be an usability improvement at least if they
did.

See also https://github.com/NixOS/nix/pull/3626#issuecomment-698546704

Change-Id: I2bf4cb80d44a4b72ade13d3e0dbd7dfb1d049f32
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2477
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Reviewed-by: glittershark <grfn@gws.fyi>
2021-02-02 10:52:33 +00:00
Vincent Ambo
f7ea650142 refactor(tvix/libexpr): Remove Bindings::SortedByKeys()
Since we don't have a Bindings implementation with unstable order this
function is not required, as its callers can just iterate over the
attributes instead.

Change-Id: I01b35277b5a2dde69d684bc881dbd7c0701bcbb3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2291
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-12-23 11:33:54 +00:00
Kane York
516c046ed9 refactor(tvix): remove all 'using namespace' from nix command
clang-tidy: google-build-using-namespace
Change-Id: I07ea10b03a6d9582c0508747698038f7106e8f63
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2181
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-11-28 00:59:25 +00:00
Griffin Smith
363dbeae95 refactor(tvix): Centralize arbitrary definitions
To make it easier for future tests to use the arbitrary specialisations
we've defined for some nix types, centralize them all in a single
arbitrary.hh header file.

Change-Id: I767f27949cfe7ec55c79901f7d7aa538d2f98c6b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2182
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-11-28 00:47:48 +00:00
Griffin Smith
6f38ac6657 fix(tvix): Add missing break;s to BuildResult::FromProto
Add missing break statements to the big switch block in
BuildResult::FromProto, and cover the whole thing with a rapidcheck
round-trip test.

Change-Id: I7a07ca398cc5f02ca4fd8e6256fd563c6f3aea9d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2178
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-11-28 00:47:48 +00:00
Griffin Smith
49024be056 feat(tvix): Thread a log sink through calls to buildDerivation
Similarly to how we did for buildPaths, add a std::ostream& log_sink
parameter to the build_derivation method on Store, and pass it std::cerr
when called at the top level by nix commands - most notably, the
build-remote hook binary, so that we get build logs when using tvix as a
remote builder.

Change-Id: I0f8f729ba8429d4838a0a135a5c2ac1e1a95d575
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2176
Tested-by: BuildkiteCI
Reviewed-by: andi <andi@notmuch.email>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-11-28 00:47:48 +00:00
Griffin Smith
1d22874ab2 fix(tvix): Don't call assertStorePath on BasicDerivation.builder
Upstream doesn't, and for good reason - this might not be a store path,
for example if it's /bin/sh

Fixes: b/73
Change-Id: If9aa96de2cd8ab941c098a9f122b8b302a92ec38
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2175
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-11-27 21:28:40 +00:00
multi
c885bd0274 fix(tvix): globally reintroduce --quiet and --verbose flags.
The --quiet and -v/--verbose flags were removed during the glog
conversion in d0c44425e1, which has
previously broken programs like e.g. home-manager, which passes --quiet
to nix-build.

A nix-build-specific workaround was added in
24f9354d5b, which manipulates the
FLAGS_stderrthreshold global variable from glog. This commit moves the
--quiet logic back into the argument handling code in libmain, and adds
corresponding handling for -v/--verbose.

Change-Id: I13d860ebbb78541d9f1236691a1efe8bd2163c67
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2170
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-11-27 21:03:45 +00:00
Griffin Smith
5c3d58bb60 refactor(tvix): Prefer absl::StrFormat/StrAppend
This fmt call was particularly egregious

Change-Id: I2a3b1006c285170ab3374d1c8d81fc53e82a7b05
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2174
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-11-27 20:14:43 +00:00
Griffin Smith
bbfc47e96d feat(tvix): Write build logs to LOG(INFO) in buildDerivation
This was referencing a nonexistent note in buildPaths, for one, but for
another let's get log outputs when calling this RPC.

Change-Id: Ic9d17834b356ea84d69692ccc0249d09777e833b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2173
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-11-27 20:10:35 +00:00
Griffin Smith
39f96c4d9f fix(txix): Override overflow in BuildLogStreambuf
Subclasses of std::streambuf are expected to override overflow in
addition to xsputn, as it's called in certain cases by the non-virtual
methods. In our case, this was preventing endlines from getting sent
over the log stream.

Change-Id: I70d00f0c7cb8f8cf2f744f58974c21e7a70a715b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2172
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-11-27 20:08:45 +00:00
Griffin Smith
afdf08ead5 fix(tvix): Remove more superfluous log output
- A couple of log messages had double-newlines still
- Remove the [ drvPath ] bit from the "building path" output, since the
  derivation path is already present in the message itself.

Change-Id: Ib4b514a18fcb63c7191f4d7cf0a9b857aaa3f3fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2171
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-11-27 20:06:38 +00:00
Michael Reilly
db52903581 feat(tvix): Bump nlohmann-json version to 3.9.1
Backporting from upstream at d860295e116f9aa0f37e2637b5ec4b9c86fdf4d3

Change-Id: Iefde47ff6f24d21e9b37c9b52a615bd8c9f2fdb2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2141
Tested-by: BuildkiteCI
Reviewed-by: andi <andi@notmuch.email>
2020-11-27 19:59:51 +00:00
Eelco Dolstra
a30e616efb refactor(tvix): JSONSax: Use a RootValue
More #3377.

Backported from upstream at 9f46f54de4e55267df492456fc0393f74616366

Change-Id: I11bfca4ec56bd4e45291ce3f98a60f198dff0196
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2140
Tested-by: BuildkiteCI
Reviewed-by: andi <andi@notmuch.email>
2020-11-27 19:59:51 +00:00
Yorick van Pelt
89f1489916 refactor(tvix): json-to-value: use unique_ptr instead of raw pointers
Backported from upstream at a350d0beb0e13d0f58698510bd6a96d894cd06fd

Change-Id: Ib486bc1b36ef65041fab1a4634a0a82e13036fd4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2139
Tested-by: BuildkiteCI
Reviewed-by: andi <andi@notmuch.email>
2020-11-27 19:59:51 +00:00
Andreas Rammhold
1ee3a7b2bf feat(tvix): run the test-vm with ncurses support
This allows users to use C-A C to access the QEMU console while within
the VM terminal. From there you can control the process (e.g. kill it).

Change-Id: If2eec2e385fd27fc4e20921dcad78be7390f9d2b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2167
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-11-27 19:09:21 +00:00
Andreas Rammhold
9bf6eb7f82 chore(tvix): avoid unnesscary rebuilds of protos and tvix
For the build of tvix we can safely ignore all the files in src/protos
as well as all the nix files in the root of the repository. By passing
the input sources through builtins.filterSource we can make a per-file
decision whether or not we want to copy it into the store (in other
words: if we consider it relevant for the build).

This enables much faster development on the actual build expression.

Change-Id: Ib4c01dbe9cbfa9770922f6257af2a7259814ce0d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2159
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-11-27 19:09:07 +00:00
Griffin Smith
e772f38a5e fix(tvix): Don't double-up on endlines from log messages
Change-Id: I676dd4e9d8b1a7ff5389323e0462ba6681d62cf0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2168
Reviewed-by: andi <andi@notmuch.email>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-11-27 19:06:06 +00:00
Kane York
0c8c1227f1 refactor(tvix): remove signedness conversions by using std::optional
The different signedness of level and withLevel was causing implicit conversions. Use a nullopt instead of a -1 sentinel value.

third_party/nix/src/libexpr/nixexpr.cc:242:21: warning: narrowing conversion from 'unsigned int' to signed type 'int' is implementation-defined [bugprone-narrowing-conversions]

Change-Id: I7c2cadb6fd6bbff6c5b84028651ad4ebba423297
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2157
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-11-27 18:34:20 +00:00
Kane York
5fb58e23de fix(tvix): make unsigned conversion explicit
Implicit signedness conversion was causing warnings. Perform a static_cast; this value should never be out of range.

third_party/nix/src/libmain/shared.cc:310:12: warning: narrowing conversion from 'unsigned int' to signed type 'int' is implementation-defined [bugprone-narrowing-conversions]

Change-Id: I34fe53d186ae33f970147135bbe3894fb39a0ec7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2156
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-11-27 18:34:20 +00:00
Kane York
35bd980bd1 fix(tvix): sqlite bind argument number is signed
third_party/nix/src/libstore/sqlite.cc:89:33: warning: narrowing conversion from 'unsigned int' to signed type 'int' is implementation-defined [bugprone-narrowing-conversions]

Change-Id: I1dbc01acaecc4f382f63e7108429cd814ff05f4f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2155
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-11-27 18:34:20 +00:00
Yorick van Pelt
b595553f63 feat(tvix): builtins.fromJSON: use nlohmann/json parser instead of custom parser
backported from upstream at f1fac0b5c3b75efab781949fdff2b67ffdda2cb3

Change-Id: I788e3a9b930351118a5f248b356c351afd7f5391
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2138
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-11-27 17:02:46 +00:00
Griffin Smith
24f9354d5b feat(tvix): Support a --quiet argument to nix-build
This is passed by some third-party tools, notably home-manager. At some
point we should fully revamp argument parsing and log verbosity setting,
but for now this gets us to more of a working state

Fixes: b/78
Change-Id: I4134b17315e30145fd43d3659f407d7121ce8876
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2136
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-11-22 23:17:23 +00:00
Griffin Smith
9e3f541546 fix(tvix): Add endlines to build log messages
Change-Id: Id6e871a2b7fbb24fb5c8e84e1638bfe94e363b57
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2135
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-11-22 23:17:23 +00:00
Griffin Smith
9945bd1746 feat(tvix): Convert some DLOGs to VLOGs
It seems like the amount of logging is causing things to move a little
slower - even if that's not really the case, it gets in the way of
debugging things.

Refs: b/76
Change-Id: I9ea99a3b16e3307a0b0371bad22d03b0e2175af6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2134
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-11-22 23:17:23 +00:00
Griffin Smith
9a24294b8a feat(tvix): Add a pos field to more expr classes
To aid in both debugging and (eventually) printing stacktraces, add a
Pos member to a few more Expr variants.

Change-Id: Ic1d2a056fc7e6c07bc3e79fa38845cb4a5da5ca5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2133
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-11-22 21:59:58 +00:00
Vincent Ambo
5a00e58904 chore(3p): Bump nixpkgs to nixos-unstable from 2020-11-21
Included fixes for random breakage:

* 3p/awscli: pick from the stable channel; it is broken on unstable
* 3p/googletest: bumped version & removed patches that nixpkgs applies
* 3p/lisp/cffi: bumped library version for SBCL compat
* 3p/nix: fix libsystemd attribute
* 3p/nix: reformatted (clang-format handling of ternaries changed)
* glittershark/home: Use home-manager from nixkpgs
* glittershark/kernel: bumped linux-ck patch hash
* glittershark/kernel: removed "patch patch"
* multi/whitby: Use home-manager from nixpkgs
* tazjin/frog: drop Sourcetrail (it doesn't build currently)

Note that in addition to these changes, some previous CLs updated the
versions of git and cgit which was necessary for this channel bump,
but which could not be done in the same commit due to the nature of
the subtree merges.

Change-Id: If2563e8a68e2750c4b913a976ff7b93b42e8b7f3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2110
Tested-by: BuildkiteCI
Reviewed-by: multi <depot@in-addr.xyz>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-11-21 23:18:27 +00:00
Vincent Ambo
cc27324d02 chore(tvix): Use StatusOr API available in Abseil's version
The Abseil version of `StatusOr` does not come with the status macros
or the `Consume*` family of functions.

This change modifies the existing code to use the common denominator
of the API that is available between Abseil's own implementation of
`StatusOr` and the one from Tensorflow that we are currently using.

Change-Id: I5c37f68636a1fd54d153f95d7303ab8644abb774
2020-11-21 15:26:02 +01:00
Griffin Smith
8d24a975f1 fix(tvix): Use copy constructor to add strings to protos
Passing a string directly to add_paths like this causes the proto class
to take ownership over the string, meaning when it is destructed it
will *explicitly* free the string. When the string's actual owner (the
derivation struct) then goes out of scope it'll get freed again, causing
a double-free. This fixes that to instead use the copy constructor to
assign to a pointer to a new path, and covers the whole to_proto method
with a rapidcheck test.

Fixes: b/64
Change-Id: I84235bed9104ff430a0acf686d4a96f1e2e9a897
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2106
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-11-19 00:29:42 +00:00
Griffin Smith
20e206a3f6 fix(tvix): Use mutable field for assigning proto fields
This was accidentally using the proto arena API to assign the derivation
field of a BuildDerivationRequest. We *thought* this was causing a
double free, but even with this change that's still happening. That
said, this change is probably still a good idea since it's using the
proto API as intended.

References: b/64
Change-Id: I950a4eafb214e9113639ea54d2dfd4659b7be931
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2104
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-11-19 00:29:42 +00:00
Griffin Smith
96dbf4d686 fix(tvix): Convert INVALID_ARGUMENT to InvalidPath
The code that calls queryPathInfoUncached explicitly catches the
InvalidPath exception and translates it into a null result - but the RPC
code was throwing a regular old Error for invalid paths. At some point
we should get rid of all the exception-driven control flow in this whole
thing, but in the meantime this gets us back to functional.

Change-Id: I2a38790ee0c691ab0c8394c7738d7693fa42aa10
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1980
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-09-16 20:17:01 +00:00
Kane York
fedbe693b9 refactor(3p/tvix): make Value.primOp a std::shared_ptr
Change-Id: I04cc42ae62d2ff71b07fde8e3710dcc9b4363a05
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1973
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-09-15 21:52:56 +00:00
Kane York
d9c7f32a9c fix(3p/tvix): properly zero the largest member of Value
The static_assert is present to force us to change the constructor when a member grows.

Change-Id: Ifa4f4a03eb7ce13cff109cf26ecf0032045905e9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1972
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-09-15 21:42:54 +00:00
Kane York
2c19bd6662 refactor(3p/tvix): convert NixList usage to shared_ptr
Starting to reclaim memory. Fairly simple mechanical replacement.

Change-Id: I6b4c5c5596729470d1a049eba61e69e8097decf4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1971
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-09-15 21:42:54 +00:00
Griffin Smith
381ce8a666 refactor(tvix): Make static strings constexpr string_views
Make all static std::strings constexpr std::string_views, and replace
concatenation with absl::StrCat where necessary.

Technically all of these are constant, so they really don't need to be
top-level statics - and since I'm trying to get rid of as much global
state as possible in preparation for making the nix daemon properly
multithreaded I figured I'd knock these out while I was at it.

Change-Id: Ibd3ad9ef68f0a0eacb135541b39fdb13dae042e1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1939
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-09-14 21:38:30 +00:00
Griffin Smith
65967bbbb1 refactor(tvix): Convert refLength to constexpr
This is constant, so let's make it a constexpr.

Change-Id: I6a9eb9f740979740f730ebe142d2cbf23dbcfd70
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1961
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-09-10 17:03:45 +00:00
Griffin Smith
5dcf64c128 fix(tvix): Make working flag thread_local
Since the daemon is running in threads now rather than forking a process
per connection (thanks to grpc) this static flag to prevent accidentally
initializing a new worker during a build is getting stepped on by
multiple threads. This converts it to thread-local, and also adds an
actual message to the check so that if it hits in the future we know
what's going on.

Fixes: b/58
Change-Id: I07a2f1582e56709c104f79935e5405fa24888f59
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1940
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: V <v@anomalous.eu>
2020-09-10 17:03:45 +00:00