Commit graph

733 commits

Author SHA1 Message Date
Vincent Ambo
cbebc75d94 chore(3p): Remove telega derivation
The upstream telega.el derivation (under emacsPackages) now includes
the server component.

Change-Id: I49d85ea16d2fce10a505e2459fcd396706e39923
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1748
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-15 12:11:36 +00:00
Vincent Ambo
1a48f39078 fix(tvix): Ensure prim_sort actually uses the right outlist
Previously the outlist would not be used, and it would sort god knows
what in the out value.

This was probably introduced by the std::vector refactoring, and the
language test for builtins.sort was disabled. Whatever reason there
was for disabling it seems to be gone, so we're re-enabling it.

Change-Id: I98941c2cad78df58ff7bea1ece3aaa4133e94bf8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1757
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-08-15 01:58:44 +00:00
Kane York
e458e5255a feat(tvix/tests): add gtest matchers for absl::Status
This allows you to write EXPECT_OK(statusor), as well as
EXPECT_THAT(status, IsStatusCode(StatusCode::kInvalidArgument).

Change-Id: I53bed694d812c501eb305ed4ddb358e1f9a68277
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1704
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-15 00:52:54 +00:00
Vincent Ambo
d44565d17e chore(3p): Pick Emacs & friends from the default channel
I had previously pinned Emacs to the (older) stable channel, because
of a heretofore undiagnosed issue in newer versions.

It turns out that the older Emacs is linking against X11-related
libraries, namely <unintelligible>, which cause even *weirder*
undiagnosed issues if paired with the rest of the recent packages.

This commit removes the exception for Emacs and just picks it from the
same channel as everything else.

Change-Id: Ifce21c99f0204d4cf131bacb366dd9617fc301b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1751
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-15 00:26:42 +00:00
Griffin Smith
d1c38d9597 refactor(tvix): Make Store::buildPaths return a Status
Make Store::buildPaths return a Status with [[nodiscard]] rather than
throwing exceptions to signal failure. This is the beginning of a long
road to refactor the entire store API to be status/statusor based
instead of using exceptions.

Change-Id: I2e32371c95a25b87ad129987c217d49c6d6e0c85
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1745
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-14 03:06:46 +00:00
Kane York
0124691fdd fix(tvix): use correct value of SANDBOX_SHELL
Change-Id: I8e471c0f0a30337681042beeff633821eee67346
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1746
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-14 02:56:04 +00:00
Kane York
72e61aa584 refactor(tvix): completely remove boehm gc
We have decided that leaking memory is a better fate than random,
non-debuggable memory corruption. Future CLs will begin changing
various fields to std::unique_ptr and std::shared_ptr.

It turns out that disabling the GC does not have disasterous impact.
The Nix evaluator only runs on the client CLI, never in any long-
running process. Even the REPL does not leak too badly under this
change, because it uses one EvalState for the duration of the REPL.

Building an explicitly tracing garbage collector is likely in the
future of this project, but that giant amount of work cannot be
done under a nix evaluator that is constantly crashing. We need to
restore development velocity here, and this is the best way we've
figured out to do it.

Change-Id: I2fcda8fcee853c15a9a5e22eca7c5a784bc2bf76
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1720
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-08-14 00:35:18 +00:00
Vincent Ambo
e38015f7b5 refactor(tvix): Very minor lint fixes
Change-Id: I4aca504d98c79f931b7faadff86f40d27ed5a2c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1719
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-13 01:39:03 +00:00
Kane York
906f5c1d2d fix(3p/nix): inherit Expr from gc, make parser state traceable
The parser contained vectors, and the primary parser state, that
were not participating in GC tracing.

Change-Id: Ie198592cd7acffd390e3e2ae9595138b56416838
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1706
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-10 22:16:42 +00:00
Vincent Ambo
42bdaacca6 style(tvix): Sort out minor formatting errors
Invocations of the MakeError macro that were not followed by a
semicolon messed up indentation in the next lines.

Change-Id: I03d7d1443f062a38af2c7da3da8928e0ed05e274
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1708
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-10 17:40:18 +00:00
Vincent Ambo
a1965e9182 fix(tvix): Print invalid store path when encountered
... instead of silently swallowing it.

Change-Id: I084c51d633d632bd4e3f1ca42987dd6068fbb2a9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1707
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2020-08-10 16:49:46 +00:00
Griffin Smith
d1653533a6 test(tvix): Cover scanForReferences in a test
Aded a few test cases covering the scanForReferences function, which had
been accidentally broken in 976a36c (which is now partially-reverted).
As part of this, since the test needed to generate hashes for store
paths, the logic in MakeStorePath to compress a sha256 hash down to 20
bytes and convert it to base32 has been extracted to a member function
on the Hash class.

Fixes: #34
Change-Id: Ie2d914688a80f42d0234d351a7cc0714fd15709e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1698
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-09 17:26:19 +00:00
Griffin Smith
ec7d368546 fix(tvix): Make nix value structs inherit from GC
All of the miscellanious structs that make up a nix Value should inherit
from gc, as they contain pointers to GC'ed things as members

Fixes: #42
Change-Id: I057d8c9f7dafbee7de7644ff152c9cba1aa6bc03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1696
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-09 02:22:44 +00:00
Griffin Smith
059468ba15 refactor(tvix): Remove UDSRemoteStore
Now that we've fully implemented the RPC-based store client, we can get
rid of the UDSRemoteStore, whose only use was connecting to the locally
running nix daemon. The RemoteStore still needs to be around to connect
to remote upstream nix stores over SSH.

Change-Id: I0699819803cbfe966b9a46786f2c927d8e4bf1a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1693
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-09 02:22:44 +00:00
Griffin Smith
e440f60b6c feat(tvix): Implement all remaining RPC calls
Implement all remaining RPC calls on the RpcSstore client, remove a few
stub methods we had added that weren't actually present in the old
RemoteStore implementation, and add one more RPC call for getBuildLog
that is present in the store API, but that we hadn't added as a
stub *or* to the proto.

Fixes: #29
Change-Id: Id827f51a393ece4bc7bbecaf38aee9eb4b329770
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1692
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-08-09 02:22:44 +00:00
Griffin Smith
747dc65154 refactor(tvix): Factor-out proto utilities
Factor out the shared utilities for interacting with protobufs/grpc from
libstore to a proto.hh header in libproto.

Change-Id: I1cb8d94867d5d4b63a9994be0b53f8f612eb8e3a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1691
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-09 01:52:59 +00:00
Griffin Smith
ef7894273b feat(tvix): Pass function name to SuccessOrThrow
To aid in debugging RPC call failures, pass the name of the function
being called to the SuccessOrThrow util funcion in the RpcStore

Change-Id: I523dacfab896b85a3dbe6050c07ee6bd5906fa44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1690
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-09 01:17:09 +00:00
Kane York
1de00e6c42 chore(3p/nix): apply google-readability-casting
Command run: jq <compile_commands.json -r 'map(.file)|.[]' | grep -v '/generated/' | parallel clang-tidy -p compile_commands.json -checks=-*,google-readability-casting --fix

Manual fixes applied in src/nix-env/nix-env.cc, src/libstore/store-api.cc

Change-Id: I406b4be9368c557ca59329bf6f7002704e955f8d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1557
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-08-08 22:16:02 +00:00
Griffin Smith
053a138002 fix(tvix): Wrap remaining RPCs in HandleExceptions
Wrap the BuildPaths and AddTextToStore RPC handlers in HandleExceptions.
These were missed in the original pass due to a merge.

Change-Id: Ie5be45e6098fba7a2b6b1c1be81578cb742c2880
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1689
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-08 20:03:59 +00:00
edef
3049f31d28 chore(3p/nix): don't optional-wrap EvalState::file_access_trace_fn further
std::function has a natural null we can't eliminate anyway, so this was
effectively std::optional<std::optional<non_nullable_function>>.

Change-Id: If99f139146021edb25d133dad7f0f6e125ef53df
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1688
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-08 18:01:11 +00:00
Griffin Smith
7db734afad feat(tvix): Implement AddToStoreNar
Implement both the client and server sides of AddToStoreNar, using a
templated generalization of the sources and sinks we were using for
AddToStore on both ends.

Change-Id: I73d0ed34118c711b125851dff99a7518ced4af35
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1686
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-08 17:51:13 +00:00
Kane York
a5dae62e85 fix(3p/nix): Use SkipEmpty in all calls to absl::StrSplit
The behavior to return a list containing a single empty string when
provided an empty string is a behavior that absl inherited from legacy
code. However, the behavior expected by legacy code in Nix is the
behavior provided by the SkipEmpty option. Switch all calls to use
SkipEmpty, except for the call already using SkipWhitespace.

See also commit 26a59482d2, with the
partly-prophetic message: "there may be other places we need to
fix this as well."

Change-Id: I6e94856a12cfb1b7e4a3b4e221769ed446648861
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1687
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-06 20:50:47 +00:00
Vincent Ambo
6a97206ceb refactor(tvix): Use absl::btree_map for BasicDerivation's env
Change-Id: I111a9a268debea322f23fdced3bed9ff3e8ed3b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1685
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-06 02:49:58 +00:00
Vincent Ambo
3ec3661678 refactor(tvix): Use absl::btree_map for DerivationInputs
Change-Id: If160ab1b09161969d9080d5d0d6f82e44a953e3c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1684
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-06 02:30:28 +00:00
Vincent Ambo
91bd7ce73a refactor(tvix): Use absl::btree_map for DerivationOutputs
This container implementation is much faster than std::map. We have
stuck to an ordered container because it's unclear whether the
accesses of this field (of which there are *many*) are actually
ordering dependent.

Also includes an Arbitrary implementation for absl::btree_map (for any
K, V that are also Arbitrary).

Change-Id: I04f58ca0ce32b9ae1759313b01508b0e44bae793
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1683
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-06 02:20:36 +00:00
Vincent Ambo
a41c3dedb1 chore(tvix): Disable sandboxing by default until its time comes
Change-Id: Ifcc776841924abcb43fb115f3de569fe500756c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1682
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-06 02:20:36 +00:00
Vincent Ambo
d42c3dd72f chore(3p): Bump nixpkgs channels
Bumps both nixos-unstable and nixos-20.03 to today's versions, as per
status.nixos.org

Contains minor fixes to things that broke because of the update:

* tazjin/frog: hardware.u2f is a deprecated setting
* glittershark/system: modSha256 in Go modules is now vendorSha256
* glittershark/owothia: removed version constraint on relude

Change-Id: Ib3e9612b1b06ed547b90e4f8b0ffe5ed7fe0a5c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1642
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-06 00:15:33 +00:00
Kane York
68b5306c56 feat(3p/nix): add --trace-file-access to nix-instantiate
This builds on edef's work with depot-scan by adding a dedicated
flag to the command. We piggyback on upstream's restricted-mode
implementation, the checkSourcePath function.

Change-Id: I52bb613549f40dbca1e8caa036635910c1a3d6d0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1654
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 22:49:22 +00:00
Griffin Smith
ea488b5705 feat(3p/nix): Implement FindRoots, CollectGarbage
Implement the RPC client and server handlers for the FindRoots and
CollectGarbage RPC calls

Change-Id: Ifa5d582c6a33bd1e7661ac2fc860505ef404dad0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1656
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-05 22:28:34 +00:00
Kane York
f9df9b4733 fix(3p/nix): fix usage error of absl::Base64Unescape
The source and destination strings cannot be the same string - absl
will write to the destination in a streaming manner, causing the
source to become invalid.

Change-Id: I3578cf1f8789a51d85e0950f7987c398f0a00953
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1659
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 18:29:22 +00:00
Kane York
f10d60a454 fix(3p/nix/store): fix race condition in downloader
In certain circumstances, the decompression thread could race ahead of the downloader thread and process the same chunk twice. Clear the data buffer while the lock is held to prevent this kind of incident.

Change-Id: I19a84a0c5768d1228c6c18a7664a7b8893ef96de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1658
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 18:29:22 +00:00
Vincent Ambo
3f923b2aa0 refactor(tvix): Use SANDBOX_SHELL from environment in CMake
This makes the configuration work out-of-the-box using CMake in a Nix
shell, rather than having to pass the additional variable to CMake on
the command line.

Change-Id: I04d3cc4f2e5ecf47bf2ee459d5e48588b84ae4dd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1643
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 11:37:20 +00:00
Kane York
ffa8e7a998 feat(3p/nix): add tests for BinaryCacheStore
These tests are in preparation for factoring the Store away from libutil's Callback to absl::StatusOr.
They use the newly added MockBinaryCacheStore.

Updates: #25

Change-Id: I30c207589ec38254806ebc9a983f35668e353ae9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1595
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 04:11:57 +00:00
Kane York
3c3527e16b feat(3p/nix): add MockBinaryCacheStore
This adds an implementation of BinaryCacheStore to be used by tests exercising both the logic inherent to BinaryCacheStore and more general Store tests.

A new library target, nixstoremock, is created to indicate that this file is intended only for use in tests and not in user-facing code.

Change-Id: Ib68f777238843a4f3a2303db8a69735fbc22d161
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1645
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 04:11:57 +00:00
Griffin Smith
28c7e926ea feat(3p/nix): Implement AddToStore rpc client
Implement the AddToStore RPC client method, which uses an
AddToStorePathWriterSink (the dual of the AddToStorePathReaderSource on
the server side) to hook into the dumpPath machinery (which we should
refactor not to use sinks or sources, but not yet) and write dumped
paths as binary data over gRPC. With this commit and sandboxing
disabled, the following derivation builds cleanly:

    derivation {
      name = "test";
      builder = ./build.sh;
      system = "x86_64-linux";
    }

where build.sh has chmod +x and contains:

    #!/bin/sh
    echo 1 > $out

Change-Id: I94cab86f0825a3a9993262a9807130645c13bf44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1638
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-08-05 03:24:36 +00:00
Kane York
b15b447fcb fix(3p/nix): fix clang-tidy for int-string appends
Change-Id: I276de7a5fd1c705c87d35dd616e5980c747190aa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1597
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 02:54:47 +00:00
Griffin Smith
5feed69260 feat(tvix): Implement AddIndirectRoot, SyncWithGC
Implement RPC handlers and client methods for the AddWithRoot and
SyncWithGC calls. With this and sandboxing disabled, the following
derivation is now building successfully:

    derivation {
      name = "test";
      builder = "/bin/sh";
      args = [ "-c" "echo 1 > $out" ];
      system = "x86_64-linux";
    }

Co-authored-by: Vincent Ambo <mail@tazj.in>
Change-Id: I245dca0dfca882e2eae5d93364287a8b79fead71
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1637
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-08-04 22:49:56 +00:00
Griffin Smith
d8386d6f7e chore(tvix): Make not implemented errors nix::Unsupported
Without this, these errors print as "uncaught exception of type
std::string", but with this they print using nix's built-in error
printer, which will be nice to have as we bang out the remaining store
API methods.

Co-authored-by: Vincent Ambo <mail@tazj.in>
Change-Id: I9bd4f77f65b27c8f97b727b98ef62e2061999e71
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1636
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-08-04 22:49:56 +00:00
Griffin Smith
e634e27577 fix(tvix): Partially revert "eliminate exposed global variable"
This reverts part of commit 976a36c2e4,
which was causing scanForReferences to hang indefinitely. I'm not sure
what the original intent of the commit was since there's not really much
of a commit message on it, but with this I am able to fully realise
derivations.

Change-Id: I620e892e2fe2244f066512286b086b3dd8f8f6c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1635
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-04 22:49:56 +00:00
edef
94387dbc25 chore: simplify buildGo expressions
This removes redundant `map (p: p.gopkg)` expressions.

Change-Id: If25d603c27d63ae011f396caf5d26cdfdb8c28d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1625
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-08-04 07:43:25 +00:00
edef
cf35ad4fa9 fix(3p/gopkgs/cloud.google.com/go): add missing depot arg
Change-Id: If08a43e18a41abd3e2ae7c08ceb6f48844d20df1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1626
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-04 07:42:53 +00:00
Griffin Smith
0aff4c256c fix(3p/nix): Don't assert on argc count for build-remote
This executable doesn't appear to ever actually look at argv, so
throwing an error if argc is wrong seems kinda silly.

Change-Id: Iabc9dad785d4c99be9e8fd4ecd954d38b876cecd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1633
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-04 02:15:10 +00:00
Griffin Smith
09cb41b7ac fix(3p/nix): Properly configure SANDBOX_SHELL
point the SANDBOX_SHELL macro at the actual path to busybox on the build
machine, or allow it to be configured at build-time with a cmake option.

Change-Id: I044a1315ba9baa3bc9ceddf29f36d14f9f9ccd96
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1632
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-04 02:15:10 +00:00
Griffin Smith
26a59482d2 fix(3p/nix): pass SkipEmpty to StrSplit("", ...)
When tokenizeString was changed to absl::StrSplit, there was a behavior
change because tokenizeString on an empty string returned an empty
vector - which the derivation builder (and likely a bunch of other
stuff) was depending on. The canonical way of fixing this is by passing
absl::SkipEmpty() to the function - there may be other places we need to
fix this as well.

This commit also includes some opportunistic absl::StrFormats and
StrCats, because I was here anyway, but those have no semantic
difference.

Change-Id: Ibf9bb602284f793fa55728481f63b838fb7a41db
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1631
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-08-04 00:43:02 +00:00
Kane York
2344f8e528 feat(3p/nix/daemon): catch-all explicit Error-Status conversion
We wrap every server-side proto handler with a macro that catches
exceptions and turns them into proper grpc error codes. For the
time being, most exceptions map to INTERNAL, the existing mapping.

Change-Id: Id6ed6a279b198ad185d32562f39000ccc15eadbf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1599
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-04 00:26:01 +00:00
Vincent Ambo
ae31f51dc8 fix(3p/nix): Initialise glog when starting nix-daemon
Change-Id: Ic9b5bd2f66fb54675498d8328edecfd31a03997a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1630
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-03 23:18:58 +00:00
Griffin Smith
31b9578ad0 fix(3p/nix): Fix parseDrvPathWithOutputs
At some point the behavior of this function got changed as part of our
cleanup - this fixes it to behave the way the rest of the codebase
expects (and how it is documented in the header) and covers it with a
few tests.

Change-Id: Id4c91232968e73489cd866fb4a2a84bcf20d875e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1629
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-03 23:17:41 +00:00
Kane York
661353f200 chore(3p/nix/daemon): replace all assertStorePath with a macro
Two exceptions: IsValidPath needs to return success for invalid paths, and QueryAllValidPaths shouldn't need to check the paths it gets from itself.

Change-Id: I4d9d4125d34e8de42f30070aec607f8a902eded7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1598
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-03 18:46:45 +00:00
Vincent Ambo
1f12544179 refactor(3p/nix): Build nix-daemon as a separate binary
Skips over all the monobinary stuff and moves to a separate binary for
nix-daemon.

This also replaces the flag parsing logic with absl::flags. This
causes a behaviour change for --help, which no longer tries to display
a man page but instead shows the actual command-line help.

Note: This binary no longer links to the Boehm GC.

Change-Id: Ib852e994b82f2d56e91262878c10650e656427a9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1622
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-03 18:26:44 +00:00
Vincent Ambo
229c1ed820 chore(3p/nix): Move legacy Nix daemon implementation
The filename this is squatting is too good.

Change-Id: I694132b94d24786df84b6b5a698679bdce818b2e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1621
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-03 18:18:54 +00:00
Griffin Smith
209489e348 feat(3p/nix): Implement two more RPC calls
Implement AddTextToStore and BuildPaths both on the client and the
server

Refs: #29
Change-Id: I45294c3e1c1a7489e42099d36425b7acc04e0427
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1560
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-08-03 17:32:27 +00:00
Kane York
8a1c7da357 feat(3p/nix/build-shell): add run_clang_tidy script
This makes it easy to quickly run clang-tidy on tvix without seeing errors from the generated files.

Change-Id: I0e25089c5626aebdb5c016629a68da9ccd26c124
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1556
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-03 17:06:41 +00:00
edef
c17be2364e chore(3p/loxy): Move //users/edef/loxy to //third_party
Change-Id: Ia451d190d4e415ef875c51ebeca12c7c5dc72342
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1590
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-08-03 10:32:39 +00:00
Alyssa Ross
6b181d314f fix(3p/cgit): transpose html_option parameters
I got these the wrong way round, fixed them in the patch in my Nix
configuration, and then neglected to fix them in the version for
depot.

Fixes: be98295dc2
Change-Id: Ib7d4e433887f474ae9675e9948ab38f640181095
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1383
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-08-03 10:28:44 +00:00
Vincent Ambo
46c78aa0f9 refactor(3p/nix): Only initialise garbage-collector where needed
Only libexpr depends on the garbage collector, specifically only
instantiations of EvalState actually require the GC to be initialised.

Rather than always starting it for the whole program, even if it is
not needed, this change moves the GC initialisation into libexpr,
guarded by absl::call_once.

This should make it possible to run the nix daemon without the garbage
collector interfering, granted that things are correcty separated and
the daemon does not actually invoke the evaluator.

Based on my investigation so far, the daemon logic itself does not
require libexpr to be present at all - so I think it is safe - but the
current monobinary might have some tricks up its sleeve that will
cause problems for us. We can deal with those if they arise.

Relates to https://b.tvl.fyi/issues/30

Change-Id: I61c745f96420c02e089bd3c362ac3ccb117d3073
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1584
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-03 08:38:57 +00:00
edef
8862f3fbaf feat(3p): Whitelist unzip
Change-Id: Iae7e661dc45f8bc8668c1aa4fe82b88e8868c22f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1585
Tested-by: BuildkiteCI
Reviewed-by: ericvolp12 <ericvolp12@gmail.com>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-08-03 08:17:51 +00:00
Griffin Smith
cc82d6e360 test(3p/nix): Add test for derivation parse/serialize
Add a rapidcheck test covering roundtrip parse and serialize for Nix
derivations. This covers a bug we discovered in ef54f5d which broke this
roundtrip.

Change-Id: I72d140334b5f24f79e82e34f98609c695dbfbf93
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1582
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-02 20:00:08 +00:00
Griffin Smith
e0d25449e3 fix(3p/nix): Don't reuse ClientContext
It turns out these aren't supposed to be reused after the first request
- reusing it was causing things to blow up.

Change-Id: I935f1ee82ed4e7f26528ae105c8fa2501fa81b97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1559
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-08-02 02:12:14 +00:00
Griffin Smith
2646e57aae fix(3p/nix): Prepend unix:// to daemon socket
Prepend the unix:// URI scheme to the daemon socket so that grpc knows
we want to connect to a unix socket rather than another type of URI. As
part of debugging this I made the failure message for the RPCStore
include the URI, which I'm leaving in since it'll be nice to have.

Change-Id: I6e70596895117b9a0d53fe2a61d8542ceb64c940
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1543
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-08-02 02:03:24 +00:00
Luke Granger-Brown
173f2c8f76 chore(gerrit): Update to latest HEAD.
Change-Id: I650777bbbd24a1922f26967fbbd7da06d14b6786
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1516
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-02 01:01:08 +00:00
Kane York
d2eff4153d chore(3p/nix): disable modernize-use-trailing-return-type
Change-Id: I47cba88de5c82923a57804432bdd4f787b78ac8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1552
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-01 23:52:45 +00:00
Kane York
72fc2fd27e fix(3p/nix): revert "apply all clang-tidy fixes"
This reverts commit ef54f5da9f.

Resolved conflicts:
	third_party/nix/src/libexpr/eval.cc
	third_party/nix/src/libstore/builtins/fetchurl.cc
	third_party/nix/src/libstore/references.cc
	third_party/nix/src/libutil/hash.cc
	third_party/nix/src/nix-daemon/nix-daemon.cc

Change-Id: Ib9cf6e96a79a23bde3983579ced3f92e530cb011
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1547
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-01 22:45:15 +00:00
Kane York
f7a85c5060 fix(3p/nix/expr): use traceable_allocator for btree_map
The prior use of gc_allocator meant that the btree nodes themselves were being collected. Additionally, have Attr (contains a Value) and Bindings explicitly inherit from gc, even though Bindings is always allocated under `new (GC)`.

Detected by running under GC_ENABLE_INCREMENTAL=1.

Change-Id: Iacf13b34b5aa12e417ea87c9b46e2bf9199fdb26
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1544
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-01 21:16:29 +00:00
Griffin Smith
3fdce7c6be feat(3p/nix): Implement a few more RPC calls
Implement the RPC client calls for QueryPathFromHashPart,
QuerySubstitutablePaths, and QuerySubstitutablePathInfos, and the
handler for QuerySubstitutablePathInfos.

Refs: #29
Change-Id: Idf383b771f159f267d8f65367bc4af3d239e32b7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1515
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-01 19:00:08 +00:00
Kane York
ee48e830e6 fix(3p/nix/libexpr): fix GC tracing in valueSize
Change-Id: I2f6bef7b090d44f50bd27fbd19b50f9cf100b238
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1506
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-08-01 18:54:50 +00:00
Kane York
64f6bb6951 feat(3p/nix): remove External values feature
External values are only useful when using the plugin framework, which we are not interested in carrying forward.

Reverts commit 320659b0cd

Change-Id: Ib4929c349bbb33f16224fc674e94c7b7d5953c6a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1505
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-08-01 18:54:50 +00:00
Kane York
770034042a fix(3p/nix): Use a proper pointer in Env to carry with-attrs
This eliminates the value-smuggling that would trip up the GC.

Change-Id: I8057df78cf0bf6bea9faf1b44233aa9820ae44f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1504
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-08-01 18:54:50 +00:00
Kane York
dc4c0bad65 chore(3p/nix/libexpr): Cleanups and notes in eval.cc
Add two more garbage-collection flags. Annotate how terrible tExternal is. Prepare to fix the smuggle casting in ExprWith. Add a static_cast.

Change-Id: I20f980abc8cb192e094f539185900a6df5457c29
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1503
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-08-01 18:54:50 +00:00
Luke Granger-Brown
f7d5d5c542 fix(gerrit): Don't do a deep fetch
I don't know what I was smoking.

Change-Id: I650777bbbd24a1922f26967fbbd7da06d14b6781
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1514
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-01 15:02:05 +00:00
Griffin Smith
05e44c121d feat(3p/nix): Implement AddToStore proto handler
Implement the proto handler for AddToStore, which adds a nix path to the
store. This is implemented by adding a new (probably
soon-to-be-generalized) Source concretion that wraps a grpc ServerReader
for the stream of data we're receiving from the client - this is less
than ideal, as it's perpetuating the source/sink thing that's going on
and storing entire nars in memory, but is at the very worst an
incremental step towards a functioning nix that we can refactor in the
future.

Paired-With: Perry Lorier <isomer@tvl.fyi>
Paired-With: Vincent Ambo <mail@tazj.in>
Change-Id: I48db734e7460a47aee4a85dd5137b690980859e3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1441
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <mail@tazj.in>
2020-08-01 14:11:41 +00:00
Kane York
674dbade27 fix(3p/nix/hash): param of Unknown allows any hash type
Fixes a crash in the self-hosting instantiate test:

NIX_REMOTE="$(mktemp -d)" nix-instantiate -E 'let depot = import ./default.nix {}; in depot.third_party.nix.outPath'

Change-Id: If99494aa07ec248d3894d4709ab0fde7fa81aff3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1508
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-08-01 01:15:29 +00:00
Kane York
1cbffe21f3 chore(3p/nix/hash): prefer StatusOr over throwing constructor
The use of `unwrap_throw` can be used as a later grep target.

Change-Id: I8c54ed90c4289f07aecb8a1393dd10204c8bce4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1493
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-08-01 01:15:29 +00:00
Kane York
2a292c71f4 fix(3p/nix/hash): initialize HashSink.ctx
Fixup for CL 1492 (addcba11b0)

Additionally, add a test to verify functionality of HashSink.

Change-Id: I2a74b925a1b93ed4d3add29021d759c93e813424
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1507
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-01 00:09:23 +00:00
Kane York
addcba11b0 fix(3p/nix/hash): smart pointers in HashSink
Change-Id: Ib2aaf42c8b234ee343c4653eb03f328c113dea86
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1492
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-29 06:48:47 +00:00
Kane York
31f9ee58d0 fix(3p/nix/hash): provide a Status-returning constructor
Additionally, add IsValidBase16() to restore the behavior of rejecting invalid base16, which absl's HexStringToBytes does not do.

Change-Id: I777a36f5dc787aa54a2aa316d6728f68da129768
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1484
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-28 02:04:42 +00:00
Kane York
976a36c2e4 chore(3p/nix/hash): eliminate exposed global variable
Change-Id: I3b34e3e17a751e225831ae599c6c6bb782a25679
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1486
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-28 02:00:23 +00:00
Kane York
ec349328e0 chore(3p/nix/hash): add decoding tests
Change-Id: Ifbeaf1822fa920f929482510ee79a5b24d7976ae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1485
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-28 00:38:29 +00:00
Kane York
ef54f5da9f fix(3p/nix): apply all clang-tidy fixes
Change-Id: I265e763393422ee1881653527c91024458060825
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1432
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-27 21:16:39 +00:00
Kane York
80ff83e698 fix(3p/nix): convert local-store asserts into throws
This fixes a clang-tidy DeadStore warning by turning debug asserts into production checks.

Updates: #11
Change-Id: Ia6e5a4cb1b56594c9844c6bbd3d152f84b426d09
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1409
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-27 20:50:57 +00:00
Griffin Smith
d98f2ea68f feat(3p/lisp): Add checkl
Change-Id: Ib73ed9637b7e22050727ac5ec117241e18d3cc45
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1472
Tested-by: BuildkiteCI
Reviewed-by: eta <eta@theta.eu.org>
2020-07-27 14:18:06 +00:00
Luke Granger-Brown
a8bc8357d8 feat(3p/gerrit_plugins): init Gerrit checks plugin
The Gerrit Checks plugin adds a new tab to the Gerrit UI, which is
intended for display of status of automated checks which are being run.
We can use this for e.g. reporting the run status of our CI builds/other
stuff.

Change-Id: Ib0d9a8ae68061a76191a56d467d915100b766e1b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1462
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-27 00:00:48 +00:00
Luke Granger-Brown
e780435d09 chore(3p/gerrit_plugins): add machinery for compiling Gerrit plugins from source
This looks particularly obnoxious for the owners plugin, because it's
actually two plugins with a common library in the same repo. Other
plugins are much cleaner to deal with (hence the default for
overlayPluginCmd).

Change-Id: Ibb9588c8a29b63e8509436fcbb70054e89349712
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1461
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-27 00:00:48 +00:00
Griffin Smith
696924b5ef feat(web/panettone): Add dev helpers for postgres db
Add a docker-compose file and lorri-based direnv for aiding in
running and connecting to a postgres database during development of
panettone.

Change-Id: I319eee52b52cd48e1f3d2e32c558989768dc19d8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1465
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: eta <eta@theta.eu.org>
2020-07-26 21:55:41 +00:00
Griffin Smith
da8bc7635e feat(3p/lisp): Add postmodern
Change-Id: If6ffd9a2344dc98e95312ddcce14ba5c2519d004
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1420
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-26 21:55:41 +00:00
Griffin Smith
e231493ff4 feat(3p/lisp): Add ironclad
Add ironclad, a common lisp library for cryptography. This is a huge
library with a lot of moving parts - probably most notable here is that
I've had to turn off compiling with `:ironclad-assembly`, as it was
causing an infinite loop in the compiler due to
https://github.com/sharplispers/ironclad/blob/master/src/opt/sbcl/cpu-features.lisp#L9-L10,
a mutually self-recursive function that looks like:

    (defun aes-ni-support-p ()
      (aes-ni-support-p))

Without knowing much about how sbcl handles native-compiled assembly, it
seems like this definition should actually be skipped entirely, due to
it being defined as a `defknown` in `fndb.lisp`:

    (defknown ironclad::aes-ni-support-p
      ()
      (boolean)
      (any)
      :overwrite-fndb-silently t)

But something about how we're compiling things was causing that not to
happen, and the infinite recursion caused the compiler to hang. This
should be fixed at some point, but given I only need this library as a
transitive dependency down a level I'm not going to attempt to do so now.

Change-Id: Id768717991404f959b003c7e2f28f1f4d532b94b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1333
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-07-26 21:21:57 +00:00
Vincent Ambo
b0e80d9465 chore(3p/lisp): Add eta & glittershark to OWNERS
Change-Id: I372d84b2fa979059bda14264b23f9c11adaf4222
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1470
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-26 21:21:30 +00:00
Luke Granger-Brown
a5c7e03dbd fix(gerrit): fix hovercards under Firefox
Firefox doesn't implement the IE6 fromElement/toElement, and it's
not in the MouseEvent spec (at the moment).

Replace with the worse-named but better-specified target and
relatedTarget attributes instead.

Upstream change: https://gerrit-review.googlesource.com/q/I9eeb26c032a38de9d7185749373c7982c796acb2

Change-Id: I9f9a1eb9342bc80b91b5b364a04cc5fa9a7ccaeb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1442
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-26 11:29:28 +00:00
Kane York
c3608fdf87 fix(3p/nix): fix broken clang-tidy config
LineFilter doesn't actually exist in this version of clang-tidy. It was only working because the config was ignored.

Change-Id: Ice5ddb5d1031dfc2cc4fee24674464f965323d8b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1431
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-26 01:04:10 +00:00
Kane York
dafb571818 chore(3p/nix): build in RelWithDebInfo mode
This eases debugging of live crashes.

Change-Id: Ie15a7f8fb3f091cae0fbe012e58862d416a42891
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1433
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-25 23:37:46 +00:00
Luke Granger-Brown
e010efb040 chore(gerrit): update to 3.2.2-1002-gea26b5ee7a
This is the version currently (2020-07-25) deployed on https://gerrit-review.googlesource.com/,
and includes features such as The Attention Set.

Change-Id: Idf29f96c38d7737efb0d64c4cd294dab46fe5412
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1437
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-25 20:27:13 +00:00
Griffin Smith
b10970a66f feat(3p/nix): Start implementing RPC store client
Add a stub class for wrapping a gRPC client to the new, proto-backed nix
store protocol, along with several methods implemented but several left
throwing a not implemented exception.

Paired-With: Vincent Ambo <mail@tazj.in>
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: Id943d4f6d75084b8498786d580e6c9f7c92c104d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1436
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-07-25 20:18:29 +00:00
Griffin Smith
dcaba9de64 feat(3p/nix): Add main function for grpc daemon
Implement the main function for the new, proto-based nix daemon. This
mostly replicates the behavior of the previous nix daemon - it starts a
grpc server listening on the same nix socket path and supports the same
set of options - with the exception of --stdio, which has been renamed
to --pipe and documented in the man page.

Change-Id: Ib729283c1d5d35c0e1b0a968bc1f052f5527f2d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1356
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-07-25 20:11:41 +00:00
Griffin Smith
7c56fccdad feat(3p/nix): Template systemd unit files in build
Template the systemd unit file templates as part of the nix
installPhase, putting them in the same place that the upstream nix
derivation does.

Change-Id: I3ceabfc0c837564e33b9ae7f9eeb7185d6fbe907
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1429
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-07-25 14:12:04 +00:00
Kane York
bbbb97e142 chore(3p/nix): disable clang-tidy in cmake
Change-Id: I9ff1ae10afe5af066aaff377424522a19977dc3d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1428
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-25 03:06:10 +00:00
Griffin Smith
0b74688e9c chore(3p/nix): Remove obsolete upload-release.pl script
This hardcodes eelco's home directory, among other things we no longer
care about.

Change-Id: Ide894107c091c37e30bba9daf473fc7dfc8b8563
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1427
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-25 02:56:05 +00:00
Griffin Smith
2f85dfe50c fix(3p/nix): Don't include depotPath in the main drv
Including depotPath in the shellHook of the main derivation for tvix
was, unsurprisingly, causing spurious rebuilds. It's still useful when
developing locally, though, so I've extracted it to a `build-shell`
derivation as a passthru attribute and updated the documentation.

Fixes: #20
Change-Id: Ibc686b9f06ec68e79759ca2c989414bd5fbce696
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1426
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-07-25 02:29:14 +00:00
Griffin Smith
4dca023385 feat(3p/nix): Add build for VM for testing tvix
Add an expression, based on the nixos qemu virtualisation framework, for
a basic system whose nix is tvix, to be used for testing tvix

Change-Id: I3c7422bb10d3ce05a3094671cb770f1f745d814c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1423
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: Alyssa Ross <hi@alyssa.is>
Tested-by: BuildkiteCI
2020-07-25 02:29:14 +00:00
Vincent Ambo
a12b0efc9e chore(3p/nix): Reduce number of active clang-tidy checks
The full clang-tidy check suite is *very* slow. This reduced list is
some kind of middle-ground between running all checks, and having a
useful developer experience.

Crucially most of the static analyzer checks (except for the ones
related to security issues) have been disabled.

We should look into running the full suite in CI only.

Change-Id: I02b96ad3b4d1a43bd6aa90ffdcba800dad966714
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1422
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-25 01:02:41 +00:00
Griffin Smith
2fcf2d0d20 refactor(3p/nix): Remove custom base64 implementation
Replace the custom, rather questionable base64 implementation with
absl::Base64{Une,E}scape. To make sure that the custom implementation
was doing the same thing I've also added a test covering
nix::Hash::to_string, which was one function that used it - the test
passed prior to the replacement, and continued to pass afterwards.

The previous base64Decode function threw an exception on failure - to
avoid going too far down the rabbit hole I've replicated that
functionality at all call sites, but this should be replaced with more
sensible error handling such as StatusOr eventually.

Also, before this change:

    ❯ nix eval -f . users.tazjin.emacs.outPath
    "/nix/store/g6ri2q8nra96ix20bcsc734r1yyaylb1-tazjins-emacs"

And after:

    ❯ ./result/bin/nix eval -f . users.tazjin.emacs.outPath
    "/nix/store/g6ri2q8nra96ix20bcsc734r1yyaylb1-tazjins-emacs"

Change-Id: Id292ffbb82fe808f3f1b34670afbe7b8c13ad615
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1385
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-07-25 00:04:43 +00:00
Kane York
989563a75b chore(3p/nix): log whether or not clang-tidy is enabled
CMake likes to remember that we overrode it with an empty string and not use the new env var.

Change-Id: I16587f27750c9ebd4f65349bb59b37e0f8117f18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1406
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-24 22:18:40 +00:00
Kane York
6039417431 fix(3p/nix): fix all UninitializedObject warnings
The cpptoml occurrences are ignored in CL 1419.

Updates: #11
Change-Id: Ifb21c93b09c67e6b6a4c9e7089abffe7616569fd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1407
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-24 22:18:23 +00:00
Kane York
6a0be19edd chore(3p/nix): ignore all unfixable clang-tidy warnings
Change-Id: I641c447fc57df4fed40ed4e89d79a88342b67bc8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1419
Reviewed-by: Alyssa Ross <hi@alyssa.is>
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-24 22:16:48 +00:00
Kane York
bd77090703 fix(3p/nix): do not call vfork
The use of vfork() in Nix is entirely illegal. Quote:

If the process created by vfork() returns from the function in which vfork() was
called, or calls any other function before successfully calling _exit() or
one of the exec*() family of functions, the behavior is undefined.

-- Linux man-pages, release 5.05

Add a TODO to use the higher-performance variants of clone() on Linux when it
is available.

Change-Id: I42370e1568ad6e2d00d70d0b66c8aded8f1288bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1418
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: Alyssa Ross <hi@alyssa.is>
2020-07-24 22:16:08 +00:00
Kane York
388b5f1abe fix(3p/nix): fix null pointer dereferences
Change-Id: I4c37febcd4978f17720b2526c267c6bf96e84bcc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1417
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-24 22:15:19 +00:00
Kane York
3ce4a6cc18 fix(3p/nix): fix more clang-tidy warnings
Change-Id: I1e8a4852ffa450c552a4cdfc709554799854934c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1416
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-24 02:40:11 +00:00
Griffin Smith
280ec4a46c fix(3p/nix): Link nixutil to its dependencies
nixutil depends on bzip2, lzma, boost::context, brotli{enc,dec}, and
openssl, but wasn't directly linking to them. This was causing linker
errors in a test that only depended on nixutil.

Change-Id: I60e77ea7b18b08e2946fcf9176ae0f355cd71844
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1384
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-07-24 00:56:07 +00:00
Kane York
9a85694b86 fix(3p/nix): remove usage of strcpy
Change-Id: I86125609f433469a8722c780fd758234211d677e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1381
Tested-by: BuildkiteCI
Reviewed-by: Alyssa Ross <hi@alyssa.is>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-23 22:00:08 +00:00
Kane York
ec46a594df docs(3p/nix): Document fast build by disabling clang-tidy
Change-Id: I5a3a4a4013c35a3a377ad0b6068a1a40bf6d77e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1386
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-23 21:58:58 +00:00
Kane York
b8e94002d6 feat(3p/nix): run clang-tidy during builds
We need to -isystem the libcxx header files in order for clang-tidy to ignore them, as the Nix clang toolchain isn't doing that automatically.

Change-Id: I05b9e9bd522de4c0e2ad543214f6bf6ab66a306b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1359
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-23 21:46:24 +00:00
Alyssa Ross
be98295dc2 feat(3p/cgit): add "this commit" option to switch form
Branches are grouped into their own section to make the "this commit"
option visually distinct.

Adding this option will result in two options being marked as selected
if a branch has the same name as a commit oid.  But that would cause
all sorts of other problems anyway (attempting to switch to the branch
would actually give you the commit, etc.), so let's not worry about
that.

A "permalink" link on the blob view next to the "plain" link would
probably be more discoverable, but that would only work for the blob
view.  The switch UI is visible everywhere.

This patch is in use already at <https://git.qyliss.net/> and
<https://spectrum-os.org/git/>.

Signed-off-by: Alyssa Ross <hi@alyssa.is>
Message-Id: <20200723204820.16776-1-hi@alyssa.is>
Cc: Profpatsch <mail@profpatsch.de>
Change-Id: I7e88d1231dd402e0ad764e16b28e9a51964c6293
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1382
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-23 20:53:04 +00:00
Alyssa Ross
231a36e203 chore(3p/nix): fix extra space in usage error
Change-Id: Ic8f8820ba8967166354951cf13af5fa134a1b6b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1353
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-23 18:55:20 +00:00
Alyssa Ross
5183e92e4f feat(3p/nix): remove allow-unsafe-native-code-during-evaluation
This formerly controlled access to builtins.exec and
builtins.importNative, but both of those are gone now, so there's no
need for this option any more.

Change-Id: I6850cbd6be264fbfb1b209a60026dadbd0ba1232
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1341
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-23 18:55:12 +00:00
Alyssa Ross
855995325e feat(3p/nix): remove builtins.importNative
This is the shared object equivalent of builtins.exec, or a plugins
equivalent accessible from the Nix language.  Either way, since we
don't have builtins.exec or plugins any more, I think it makes sense
to remove this builtin.

This will also allow us to drop the
allow-unsafe-native-code-during-evaluation option, which formerly
controlled whether builtins.exec and builtins.importNative were
enabled.

Cc: Griffin Smith <grfn@gws.fyi>
Cc: Profpatsch <mail@profpatsch.de>
Change-Id: I8993a8a79d559c102647308a2684c089bbc06713
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1340
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2020-07-23 04:53:23 +00:00
Griffin Smith
b1c0866037 feat(3p/lisp): Add LASS
Add LASS, a CSS library

Change-Id: I225959b12210895d0e5baf8a41ac09b0351f0aed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1349
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-23 04:28:13 +00:00
Griffin Smith
08397fd757 feat(3p/lisp): Add many packages
Add the following packages along with their transitive dependencies:
- trivial-ldap
- defclass-std
- easy-routes

Change-Id: Ib24517f18d14d57540362132c9a24ff2a5a4bd9d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1336
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-07-23 00:08:29 +00:00
Vincent Ambo
950ba404be fix(3p/nix): Fix string escaping issue in schema include
The SQL schemas are included as string constants which are
concatenated into a header file. In the previous Makefiles, this was
done with envsubst or something - we moved it to CMake.

There was a missing quote around the string to be interpolated, which
meant that CMake interpreted the semicolons as part of its language
syntax and did not emit them.

Change-Id: Ibb4512788b26b53f297db3535094dc0194614446
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1342
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-22 01:28:32 +00:00
Griffin Smith
3b02fcb0a7 feat(3p/lisp): Add cl-who
Change-Id: I35fcb65966be2bc63ef534ab0d575aae8beb3cdf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1327
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-07-21 22:58:17 +00:00
Vincent Ambo
38a262872a chore(3p/nix): Compile under -Wall -Werror
Change-Id: Ia44c68678a0b62e14228fafec88e7591ce3f4cee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1295
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-20 16:47:39 +00:00
Griffin Smith
17ca547ed3 feat(3p/nix/nix-daemon): add factory function for WorkerServiceImpl
Add a forward-declared factory function for constructing and returning a
WorkerServiceImpl, for eventual use in the main function for the nix
daemon

Change-Id: I9032d69b6ee3bc3b1f39f3d5d55f951cffad8145
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1293
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-20 14:45:43 +00:00
Griffin Smith
2ef1060361 chore(3p/nix): Remove support for plugins
Plugins seem to not really be used anywhere (I can find one plugin
that's actually defined, and it doesn't seem very useful, especially
since we got rid of builtins.exec) and their presence is adding
additional complexity and potential sources of bugs to an already
unsteady refactor. At some point we may want to bring back
something *like* plugins, but their design will likely be different and
it will definitely be after we have a functioning Nix again.

Change-Id: I3bc40e55917f70bf260fbc208c1705e2e6a7c626
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1291
Tested-by: BuildkiteCI
Reviewed-by: Alyssa Ross <hi@alyssa.is>
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-20 14:45:43 +00:00
Vincent Ambo
15afa8472e fix(3p/nix): Fix all remaining compiler warnings
This compiles under `-Wall -Werror`.

The largest chunk of this change is `final` qualifiers for the various
Nix CLI command structs, which inherit from a Command class that has
more virtual functions than are implemented by each command.

Change-Id: I0925e6e1a39013f026773db5816e4a77d50f3b4a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1294
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-07-19 22:00:59 +00:00
Vincent Ambo
ca50fb66d2 chore(3p/nix): Update deprecated 'pure-parser' setting
This setting was renamed to api.pure about 300 years ago and caused
warnings to be issued.

Change-Id: If883b0667c3afe67ae3d2a9950a796688cfbea7a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1290
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-19 19:02:06 +00:00
Vincent Ambo
7a8880a0df refactor(3p/nix): Remove remaining prefork-compat type
Removes the verbosity enum, which is no longer actively used anywhere
other than a daemon protocol implementation bit that doesn't actually
work.

Since the verbosity was marked deprecated, this removes one of the
last remaining warnings.

Change-Id: Iaee9d1d6c14b30daac83bb44bcacff32a0e07fb0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1289
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-19 19:02:06 +00:00
Vincent Ambo
324d385b29 refactor(3p/nix): Rename & undeprecate Bindings::lexicographicOrder
The function is renamed to `SortedByKeys`, which is more descriptive,
and annotated with a comment about what it is used for.

The deprecation warning has been removed because this function is
currently functionally required.

Change-Id: I0ee3a76deff05f366feca9ddac8f38ab34bffbd0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1288
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-19 19:02:06 +00:00
Vincent Ambo
02066a4bab chore(3p/nix): Enable clang-format check in derivation build
This is the easiest way to get the checks up and running for now, but
we will probably want to separate out things like this into a separate
build step in the future.

Change-Id: I8e1a1095aef09b1eee97abad5b6240bc64d14b8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1287
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-19 16:54:52 +00:00
Vincent Ambo
61a7b97c39 test(3p/nix): Enable output comparison for evaluator success tests
Enables loading of the expected output of evaluator tests from the
corresponding .exp files, and checks that the output matches.

This again leaves some tests behind in the disabled folder, but we now
have almost the entire suite up and running so I can get around to
cleaning up the disabled ones.

Other note: Some tests had XML output, despite not being related to
XML testing at all - I'm not sure why they chose to do this, but have
converted those test outputs to normal Nix instead.

We have a separate test suite for JSON & XML serialisation already,
which was contributed by andi-.

Change-Id: Id7c42c836edfec4c22db9d893e35489f3e6dd559
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1285
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-19 16:33:34 +00:00
Vincent Ambo
97e6499225 test(3p/nix): Partially enable evaluator success tests
Enables the `eval-okay-` test suite, with some caveats:

* The output is not yet checked against the expected value, so the
  tests only check that pure evaluation succeeds
* A handful of tests have been disabled as they are causing failures
  that may be related to the DummyStore implementation.

Both of these will be addressed in followup commits, but there is
already some value in having the pure evaluation tests pass.

Change-Id: I62236c95ebffb06dc64a45455a8ee04344c400b7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1284
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-19 16:33:34 +00:00
Griffin Smith
54a5750770 test(3p/nix): Add property tests for attribute set
Add a set of property tests for the attribute set (Bindings) class
checking that the Merge operation satisfies the monoid laws. This
will hopefully become useful to make sure we're not breaking the
language semantics as we work towards optimizing or replacing the
implementation, but also serves as a test bed for adding
rapidcheck-based property tests to the codebase.

Change-Id: I1b4b7b6503d08d80c1c5a8f9408fd4b787d00e8e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1283
Reviewed-by: isomer <isomer@tvl.fyi>
Tested-by: BuildkiteCI
2020-07-19 16:26:52 +00:00
Griffin Smith
6ff0d7992f feat(3p): Add rapidcheck
Pulled from the commit at current master, overridden to build with clang
and enable gtest and gmock integration.

Change-Id: I10008e8c591bd0c7cc26566b2a050ef2a55bb346
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1282
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-19 16:26:52 +00:00
Kane York
f1080c022a fix(3p/nix): convert all holders of EvalState into gc
Change-Id: Ia0d1fdd5c73941d02da0ca7152600f966cee5ccb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1280
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-19 06:40:35 +00:00
Vincent Ambo
105ad71015 refactor(3p/nix/libexpr): Move some code out of lexer.l
Moves a function that is not dependent on the generated code over to
parser.hh. This function also looks like it could be improved, but
that is left as an exercise for the reader.

Code that remains in lexer.l has been reformatted, while we're here.

Change-Id: I9c26bb4eed0772a720d0715029e8bc10ab16ac38
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1279
Tested-by: BuildkiteCI
Reviewed-by: Kane York <rikingcoding@gmail.com>
2020-07-19 02:59:41 +00:00
Kane York
9f5f71616a feat(3p/absl): import StatusOr from tensorflow
Additionally, add tests for the macros. A future CL will enable the tests in CI.

Change-Id: Id4445a1aa65bf6751b87606f37654f3fc6d20efc
Tested-By: kanepyork <rikingcoding@gmail.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1274
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-19 00:51:00 +00:00
Vincent Ambo
bd02cae032 test(3p/nix): Port language test suite to Googletest
This moves the language test suite into Googletest by constructing
parameterised tests out of the same language snippets used for the
previous lang.sh evaluation.

So far this includes support for about 3/4 of all tests, specifically:

* all parser success/failure tests
* all evaluator failure tests

The evaluator success tests will be implemented in a subsequent commit,
because the output comparison contains a whole bunch of additional
logic that I did not want to cram in here.

Change-Id: Icec9f368366cdbaa53b4c7e4472b8b6e8dd72eba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1278
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: isomer <isomer@tvl.fyi>
Tested-by: BuildkiteCI
2020-07-19 00:49:29 +00:00
Vincent Ambo
a802b4b44d fix(3p/nix/tests): Fail DummyStore::addToStore on non-existent srcPath
Unfortunately, to guarantee correct behaviour of some evaluation (!)
tests, addToStore needs to actually check whether passed in source
files exist and fail appropriately.

There is a chance that the dependency on this behaviour is actually a
bug in the upstream test suite, but my attempts at finding out more
about this from, say, the git history have so far been unsuccessful.

Change-Id: I311999ea28fcedf5da13a4e627b1c1c8e4e59cbd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1276
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-19 00:49:29 +00:00
Vincent Ambo
989537f972 fix(3p/nix/libexpr): Make EvalState internals visible to GC
Change-Id: Ia7de49acb859040429fdd8ab143d485589ac02d4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1277
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: Kane York <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-07-19 00:49:29 +00:00
Vincent Ambo
70633a3058 chore(3p/nix/tests): Move language test files to src/tests
These files will be integrated into the evaluator unit tests instead
of running separately via a shell script.

Change-Id: I1d229e73b1d862777f5108c86891689900edefbe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1275
Tested-by: BuildkiteCI
Reviewed-by: Kane York <rikingcoding@gmail.com>
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-19 00:49:29 +00:00
Alyssa Ross
7010d4dc86 fix(3p/nix): nix-build: don't accept --(im)pure
These options only apply to nix-shell.  The fact that nix-build
previously accepted them was an accident that resulted from the two
programs sharing an implementation.

Change-Id: I0047c98e2096010797316bff3ea4faf722fab86a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1273
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-18 21:40:36 +00:00
Griffin Smith
a724158512 fix(3p/nix/nix-daemon): Add assertStorePath to all proto impls
I missed calling store_->assertStorePath on paths in several daemon
proto handlers, which the previous implementation did.

Change-Id: Ifad6eeb03b5a5babec7b4bcf7aca060813f15bb7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1272
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-18 19:52:23 +00:00
Griffin Smith
ecf288e70a feat(3p/nix/nix-daemon): Implement Worker::AddSignatures handler
I got the message proto wrong on this one as well - it needs both a path
and a signatures.

Change-Id: I9a489b1285bda61c15b2a3b47d9cfc3b50e387da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1270
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-18 19:52:23 +00:00
Griffin Smith
a79df261b4 feat(3p/nix/nix-daemon): Implement Worker::BuildDerivation handler
Implement the proto handler on the server side for
Worker::BuildDerivation. This includes several additions to the proto
which I had missed on the first pass, including the actual proto
definition for the Derivation itself and a few sequence number
reorderings which are fine because this is all provisional and not
deployed yet.

A couple things to note

- I implemented a couple constructors for nix classes that initialize
  themselves based on their proto variants, which felt nice and didn't
  end up causing any issues.
- I've made the conversions between the enum types in nix and in proto
  explicit via switch statements rather than using a static_cast, out of
  an abundance of caution that the error would get mismatched in the
  future and we'd convert the wrong thing to the wrong thing - this is
  verbose, but exceptionally future proof.

Change-Id: Iecf6b88e76bc37e49efa05fd65d6cd0cb0deffed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1249
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: Kane York <rikingcoding@gmail.com>
2020-07-18 19:52:23 +00:00
Griffin Smith
3f4e5050cd chore(3p/nix): Add missing override to UDSRemoteStore::sameMachine
This silences a -Winconsistent-missing-override

Change-Id: I2492a8ab3c0b0a33a794f012bfee6bbd0c2f0a8e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1271
Reviewed-by: Kane York <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-07-18 19:22:46 +00:00
Alyssa Ross
cbea59c5f6 chore(3p/nix): don't rebuild nix-proto-srcs as much
Interpolating a path into a string will copy the referenced path into
the Nix store, so this got a dependency on all of src.  By first
constructing a path to the src/proto directory using the + operator,
and then interpolating it, we limit what is copied to the store, and
therefore what code triggers a nix-proto-srcs rebuild.

Change-Id: I8dd750f6bc5902b74ffb56470bc8a5f2c01c8cf1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1263
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-07-18 18:19:38 +00:00
Vincent Ambo
4b1818cb01 refactor(3p/nix/tests): Move DummyStore into a separate file
This store implementation is required in all unit tests that use the
evaluator.

Change-Id: I1cfe8cecab8722cd66dc803747821a2be2b2619f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1269
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-18 18:08:42 +00:00
Vincent Ambo
cdc1a0bb6e refactor(3p/nix/libexpr): Back Nix lists with std::vector
This change does away with the previous special-casing of lists of
certain element sizes, and the use of raw C-style arrays.

Lists are now backed by a std::vector of nix::Value*, which uses the
traceable GC allocator.

This change is unfortunately quite noisy because the accessor methods
were updated/removed accordingly, so all callsites of Nix-related
lists have changed.

For some operations in primops.cc where keeping the previous code
structure would have been more difficult with a "proper" vector, the
implementation has been replaced with std::vector methods. For
example, list concatenation now uses appropriate range inserts.

Anecdotally the performance of this is about equal, to even slightly
better, than the previous implementation.

All language tests pass and the depot paths I've used for testing
still evaluate.

Change-Id: Ib5eca6c0207429cb323a330c838c3a2200b2c693
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1266
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: Kane York <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-18 18:08:42 +00:00
Vincent Ambo
56614c75e4 refactor(3p/nix/libexpr): Store nix::Env values in a std::vector
This has several advantages:

* we can ensure that the vector is traced by the GC
* we don't need to unsafely allocate memory to make an Env

Note that there was previously a check about the size of the
environment, but it's unclear why this was the case (git history
yielded nothing interesting) and it seems to have no effect.

Change-Id: I4998b879a728a6fb68e1bd187c521e2304e5047e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1265
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: Kane York <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-18 18:08:42 +00:00
Vincent Ambo
c9ba75cc95 chore(3p/immer): Remove .gitmodules
This breaks tooling for some people, and we aren't actually using the
git submodules.

Change-Id: I5b4dfd4ad76bf72e9dbc7de29f17f28ebf6b8ba0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1255
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-18 00:56:17 +00:00
Griffin Smith
be6fe93dd4 feat(3p/nix): Add NIX_DATA_DIR to shell hook
all of the executables that get built during regular development depend
on this being set to a directory that contains the nix directory -
previously we had been doing it manually every time, this automates it.

Change-Id: I4c957c0abf0a92ca7122a47d3b141a8ede280e13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1258
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-18 00:03:24 +00:00
Vincent Ambo
f4f72bcf21 test(3p/nix): Set up scaffolding & fix up tests for value-to-{json|xml}
Configures the CMake build to load & run the GoogleTest tests.

I (grfn) also updated this to get the tests running as part of the nix
derivation, which required defining our own manual configurePhase and
installCheckPhase, rather than depending on the one provided by stdenv.
Not doing this would cause cmake to attempt to *run* the tests as part
of the buildPhase, which wouldn't work because the dynamic libraries
hadn't been put into a place where the test executables knew where to
find them. We're not sure *why* this fixes it, and for some reason
fixing this also breaks the automatic behavior of nixpkgs of passing
-j$NIX_BUILD_CORES -l$NIX_BUILD_CORES to make, but that's eaasy enough
to fix manually in a preBuild

Paired-With: Griffin Smith <grfn@gws.fyi>
Change-Id: I79d61854a3ff47301cdce8a40c76820a97bdf901
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1240
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-18 00:01:53 +00:00
Andreas Rammhold
e4b7d6d592 test(3p/nix/libexpr): add json/xml tests
Backported from:
c1371b6fbb

Change-Id: Ida326cf1567081a56d747c4a72f036f55cfdea37
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1239
Tested-by: BuildkiteCI
Reviewed-by: Kane York <rikingcoding@gmail.com>
2020-07-18 00:00:51 +00:00
Griffin Smith
b32274120c feat(3p/nix/nix-daemon): Implement Worker::VerifyStore handler
Also adds the missing check_contents field to the VerifyStoreRequest
proto message, since it was missed in the original pass. This is done
using a renumbering, which is fine in this case since the proto hasn't
been deployed yet

Change-Id: I92bf4e48a71a25ae02ae02b3deaf6e7c71fe5da7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1237
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: Kane York <rikingcoding@gmail.com>
2020-07-17 20:37:34 +00:00
Griffin Smith
d18246f8b2 feat(3p/nix/nix-daemon): Implement Worker::OptimiseStore handler
Change-Id: I8b1d84799a608a516d0b4980022a7edd545a1ca1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1235
Tested-by: BuildkiteCI
Reviewed-by: Kane York <rikingcoding@gmail.com>
2020-07-17 20:37:34 +00:00
Griffin Smith
1b07e588c8 feat(3p/nix/nix-daemon): Implement Worker::QuerySubstitutablePaths handler
Change-Id: I243d67b0bce29d54c7d6e08f5eee70bd395cf9a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1234
Tested-by: BuildkiteCI
Reviewed-by: Kane York <rikingcoding@gmail.com>
2020-07-17 20:37:34 +00:00
Griffin Smith
d36ee69061 feat(3p/nix/nix-daemon): Implement Worker::QueryValidPaths handler
Change-Id: I221dcd2a3fa53ead3ea9ac47830d968bb587c6e5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1233
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-17 20:37:34 +00:00
Griffin Smith
1933e4f97f feat(3p/nix/nix-daemon): Implement Worker::QueryPathFromHashPart
Change-Id: Id351a0c82f4d6e79a8f272674d8d8e86e12a9400
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1232
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-17 20:37:34 +00:00
Griffin Smith
a67db573e5 feat(3p/nix/nix-daemon): Implement Worker::QueryDerivationOutputNames handler
Change-Id: Ie17e37322dd0e241cd35b6d8e69bd6c968fd6ff2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1231
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-17 20:37:34 +00:00
Griffin Smith
eb4c8717cd feat(3p/nix/nix-daemon): Implement Worker::QueryPathInfo handler
Change-Id: I580bd29356f7bcd0cc2050afda11d2e115d44c94
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1230
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-17 20:37:34 +00:00
Eelco Dolstra
80eaa1eaf9 feat(3p/nix): Add function to allocate a Value in traceable memory
Backported from:
b3e5eea4a9
fcd048a526

Intentionally skipped because we have not backported the JSON changes:
9f46f54de4

Did not apply changes ni primops.cc, because those look suspect and
are also based on something that we don't have in our tree.

Change-Id: I837787ce9f2c90267bc39fce15177980d209d4e9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1253
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-17 20:04:14 +00:00
Eelco Dolstra
014436eb4a fix(3p/nix): ValueMap, VectorVector: Use traceable_allocator
We want to *trace* the 'Value *' arrays, not garbage-collect them!
Otherwise the vectors/maps can end up pointing to nowhere.

Backported from:
10e17eaa58

Change-Id: I30dc94caa80c9d982e7a14bc67ba2d065e8203aa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1252
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-17 20:04:14 +00:00
Andreas Rammhold
5218c83b9f fix(3p/nix): allocate the vSourceExpr via uncollectable memory
Previously the memory would occasionally be collected during eval since
the GC doesn't consider the member variable as alive / doesn't scan the
region of memory where the pointer lives.

By using the traceable_allocator<T> allocator provided by Boehm GC we
can ensure the memory isn't collected. It should be properly freed when
SourceExprCommand goes out of scope.

Backported from:
d2c371927e

Change-Id: I1f7c745dbc66c7164bee50f4d9b0d437dbc7dd51
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1251
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-17 20:04:14 +00:00
Kane York
af47a89d6f chore(3p/nix/proto): Fix style violations in new proto defs
This is a blocker to enabling the linter.

Change-Id: I9f8d3cc19c7539086f53474a505362230fc56c04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1245
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-17 20:01:29 +00:00
Kane York
f6a9706b3b feat(3p/bufbuild): Import buf
Change-Id: Ibb35980ce3a398dd7af1c14df9eba84f0fa414b7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1243
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-17 20:01:29 +00:00
Vincent Ambo
8d5857abbc chore(3p/nix): Remove builtins.exec
This is a bad idea, it shouldn't exist, nixpkgs doesn't use it.

Change-Id: Ic4d1b936d8f059d5c40f0567af165b02427d7e36
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1241
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-17 19:03:59 +00:00
Vincent Ambo
a2870d05a3 feat(3p/gtest): Bump version & override stdenv to libcxx
Change-Id: I19a34e80f51add5a7c6cde2650aae60c1919ad43
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1238
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-17 02:59:42 +00:00
Kane York
a556913079 chore(3p/nix): add glittershark to OWNERS
Change-Id: I71228383c8c7810ff8032e67bde0f45339809934
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1236
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-17 01:24:38 +00:00
Griffin Smith
d1b3514910 feat(3p/nix/nix-daemon): Implement Worker::QueryAllValidPaths handler
Change-Id: If9d2144ae872183abf47d84c94cc5c5fa23ced87
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1229
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 23:57:49 +00:00
Griffin Smith
ec859afe95 feat(3p/nix/nix-daemon): Implement Worker::QueryMissing handler
Change-Id: I004a0c7969e1fe27b844adfed2d3ba3e6c5279bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1227
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 23:25:25 +00:00
Vincent Ambo
5c9af8faee feat(3p/nix): Implement similar group of Worker::Query* handlers
These are the queries that are handled in the confusing case statement
in the old daemon implementation, because they have very similar
structure.

Change-Id: Ie7143354f66cef4336dff8072ede9a56271a7e89
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1228
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 23:20:15 +00:00
Vincent Ambo
32735eebdd feat(3p/nix/nix-daemon): Implement Worker::HasSubstitutes handler
Change-Id: I0b01b554eee4b4e31416aabb498ea3d8b8a8092b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1226
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 22:48:09 +00:00
Vincent Ambo
bb66262068 feat(3p/nix/nix-daemon): Implement Worker::IsValidPath handler
Change-Id: I741c2b9b58f234a21850640e2b0c071ff4441234
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1223
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 22:48:09 +00:00
Vincent Ambo
ba11727ce0 feat(3p/nix/proto): Add QueryMissing RPC call
Change-Id: I6a29c56a0f945afdef7434da539b4b91b2e4d2e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1222
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-16 22:10:22 +00:00
Vincent Ambo
afe99225b9 feat(3p/nix/proto): Add AddToStoreNar RPC call
Change-Id: I335630644205adb280d83e006baf9d497ac1ac95
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1221
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 22:10:22 +00:00
Griffin Smith
cf6c48ef47 feat(3p/nix): Add NarFromPath
What it does I will never know. It takes a path and returns a path.
Something happens in the middle.

Change-Id: I499a9df700e5b954c9aaf6d694753ff34e773dfd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1210
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 21:41:40 +00:00
Griffin Smith
c3e96bcfa2 feat(3p/nix): Add AddSignatures
Change-Id: Ic37409c7987c261dc3eb91382d40f458fae6ea0c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1209
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-07-16 21:41:40 +00:00
Griffin Smith
f4baaf79e2 feat(3p/nix): Add BuildDerivation proto
Change-Id: Ib66b4ff2da72677e74b6a713186c72f66d9f6f88
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1208
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 21:41:40 +00:00
Griffin Smith
db4ac8ea7c chore(3p/nix/worker-protocol): Add missing includes
These files were always implicitly depended upon by worker-protocol.hh, but just so happened also always be included in places where the other ones were already present.

We are likely getting rid of this file sooner rather than later, but in
the meantime this will cause clangd to shut up while I'm editing the
file.

Change-Id: I0d765f8b86828b6612e5483417e452221ea3c5b1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1207
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-07-16 21:40:58 +00:00
Griffin Smith
4468e411fd feat(3p/nix): Add VerifyStore
Change-Id: I6a0596e986867e94857da461c746a80750830f7b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1206
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 21:40:58 +00:00
Griffin Smith
3999439fbb feat(3p/nix): Add OptimiseStore
Change-Id: I0d254504801283e018eef41bd5ea9850b17c0d80
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1205
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 21:40:58 +00:00
Griffin Smith
b6cb681c0b feat(3p/nix): Add QueryValidDerivers
Change-Id: I52965bdbf143ae11201e1fafed1c25b3d8f1bee9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1204
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 21:40:58 +00:00
Griffin Smith
143ef52592 feat(3p/nix): Add path query ops
Add QueryValidPaths and QuerySubstitutablePaths, both of which filter a
list of paths based on a set of criteria.

Change-Id: I6aa4647efe82b82dc9582a311643d5f9b6d521d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1203
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 21:40:58 +00:00
Griffin Smith
863a77dd2a feat(3p/nix): Add QueryPathFromHashPart
Change-Id: I0a31557f4cf585d3e539e29ef1a07b443aeec9fc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1202
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 21:40:58 +00:00
Vincent Ambo
d470ec0d29 refactor(3p/nix/libexpr): Use Abseil collection types for parser state
Replaces the previous uses of the (ordered!) std::map and std::set
with absl::flat_hash_{map|set}.

After some careful reading it seems that there is actually no
ordering dependency on these types, and the (drop-in) replacements
perform slightly better.

Overall this is not fixing a bottleneck, just a driveby thing.

Change-Id: Ided695dc75676bd58515aa9382df0be0a09c565e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1220
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: isomer <isomer@tvl.fyi>
Tested-by: BuildkiteCI
2020-07-16 21:03:52 +00:00
Vincent Ambo
e05eb20102 refactor(3p/nix/libexpr): Move EvalState:* implementations out of parser.y
Several definitions of functions declared in eval.hh were previously
implemented in parser.y, this moves them over to parser.cc.

While this still isn't a reasonable place to keep them, the long-term
fix is more likely to be that eval.hh needs to be split up.

Before we get to that point however, this already gives us the ability
to use tooling with this code.

Change-Id: If06fb655325fe281564047ffab0a0a640428a0ee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1219
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-16 19:53:51 +00:00
Vincent Ambo
5ae2a2b847 refactor(3p/nix/libexpr): Extract static helpers out of parser.y
Moves several of the static helper functions into a new parser.cc
file.

Once the rest of the code is usefully extracted, these will be moved
to a private namespace.

Change-Id: I0d7b53dcefe31bb5c6bad3ad7f5fcb48276bf799
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1218
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-16 19:53:51 +00:00
Vincent Ambo
d579ceb783 refactor(3p/nix/libexpr): Extract parser header out of Yacc file
First step (of many?) towards extracting all the inline code from the
Yacc file and keeping it somewhere more accessible instead.

Note that none of this code has previously been touched by a linter or
formatter, pretty much ever, so as it is extracted it also undergoes
similar changes to the whole codebase after the initial fork.

Change-Id: If3b7181f22e3b3fd8c58dfa9befa7ee2896ea06d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1217
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-16 19:53:51 +00:00
Vincent Ambo
cb3d967508 refactor(3p/nix/libexpr): Use range insertion to merge nix::Bindings
Instead of manually iterating over the two bindings to be combined,
this adds a new static method on the Bindings class which merges two
attribute sets by calling the range insertion operator over them.

In some anecdotal tests, this can lead to a ~10% speed bump -
depending on the specific operation.

Change-Id: I5dea03b0589a83a789d3a8a0fc81d0d9e6598371
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1216
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 18:51:47 +00:00
Vincent Ambo
9a4a4058a8 style(3p/nix): Rename the project to tvix
Changes the derivation name & README overview to say "Tvix" instead of
"tazjix".

The previous name was mostly intended as a joke, and a way for me to
distinguish output paths. It's certainly not the intention to have a
portmanteau with my name here, especially now that several people are
contributing to the fork.

Change-Id: Icface5484d52355111eca23b2f6bd3b9e5567275
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1212
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-16 15:28:52 +00:00
Griffin Smith
14e9e161c9 feat(3p/nix/proto): Add QueryDerivationOutputNames
Change-Id: I5452d0f76441a61d70031f567c9cb75249569c2a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1201
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-07-16 01:54:05 +00:00
Griffin Smith
17c0962802 feat(3p/nix): Add proto for QueryPathInfo
Change-Id: I10ec338ef2d5360954abdc5bb3d4789f34b031fc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1200
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-07-16 01:54:05 +00:00
Griffin Smith
0386fad78f chore(3p/nix): {query,clear}FailedPaths are obsolete
They're not mentioned anywhere on the client or the server.

Change-Id: Ia78c8bde49326f6bf69b4aeded083cfd3235131f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1188
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 01:51:19 +00:00
Griffin Smith
150e2bc908 feat(3p/nix/worker.proto): Add QueryAllValidPaths
takes nothing, returns paths

Change-Id: I4087b9b5a25d9edc76b97d0a0f553259693cca3a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1187
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 01:51:19 +00:00
Vincent Ambo
622250bff9 feat(3p/nix/proto): Add additional worker-protocol operations
On the chopping block this time:

* requests/responses that return one or more store paths, and contain
  nothing else, have been changed to use the same types (StorePath and
  StorePaths, respectively)
* QuerySubstitutablePathInfos has been added. It should be noted that
  legacy Nix has two versions of this call, one that only queries a
  single info (deprecated) and one that queries multiple. We have only
  implemented the latter.
* QueryDerivationOutputs has been added.

Change-Id: Iccc9041e7064e141cf593467ecdcc327581c4056
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1186
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 01:19:38 +00:00
Vincent Ambo
eff165425a refactor(3p/libstore): Optimise nix::printString
Slight performance optimisation of nix::printString by copying chunks
of the input string which do not need escaping as contiguous blocks.

Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: I48bad90c8f2831ae4524c814a12b1982989922f9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1184
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 00:37:29 +00:00
Vincent Ambo
4303cedde4 style(3p/nix): Minor style fix in DrvInfo::setMeta
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: I0655ecc675239b3d90e5adc305c3f37c1a904cf5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1181
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 00:37:29 +00:00
Vincent Ambo
d8ee0afda9 refactor(3p/nix): Compare attribute sets by pointer equality
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: I418e9127c5d9d31559c59e461f17726ddbc051c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1180
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 00:37:29 +00:00
Vincent Ambo
5cd7cf93fc refactor(3p/nix): Use a static empty Bindings for 0-element attrs
A significant fraction of all created attribute sets are empty; hence
this is an easy optimisation to make.

Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: I0884194d04c1ee95b2b239a253515f2152bc0856
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1179
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 00:37:29 +00:00
Griffin Smith
04ae293360 feat(tools): Add script to flamegraph a command
Add tools.perf-flamegraph, which collects the base case execution of
perf piped through stackcollapse-perf and flamegraph to flamegraph the
execution of an external command via perf.

Change-Id: I671fe254dc374b6cd7deca2d3bdea266164de025
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1176
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-07-15 20:21:34 +00:00
Vincent Ambo
3fcebfdaa1 chore(3p/immer): Amend derivation to work in depot
Change-Id: If580b76b7b7170ce7840d3a97c3a547c29405a6f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1174
Tested-by: BuildkiteCI
Reviewed-by: Kane York <rikingcoding@gmail.com>
2020-07-15 18:39:02 +00:00
Vincent Ambo
1213b086a1 merge(3p/immer): Subtree merge at 'ad3e3556d' as 'third_party/immer'
Change-Id: I9636a41ad44b4218293833fd3e9456d9b07c731b
2020-07-15 08:23:32 +01:00
Vincent Ambo
1390827b9e refactor(3p/nix): Revert VectorBindings implementation
This reverts parts of the CLs splitting the backing implementation for
Bindings and moves back to only the BTreeMap-backed implementation.

Our evaluation has indicated that the Vector-backed implementation
does not match the performance of the plain array used upstream, and
in my view the complexity introduced by it is not worth the relatively
small (single-digit percentage) performance increase with a
pivot-point close to the number of attributes yielded by
stdenv.mkDerivation.

Going forward we will trial implementations of attribute sets backed
by HAMTs, and investigate other mechanisms of speeding up the language.

Some changes from the previous CLs are retained, for example the
removal of insert_or_assign and the passing of capacity.

Change-Id: I6eb4b075b453949583360755055c21a29d7ff642
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1172
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-15 00:16:06 +00:00
Vincent Ambo
c5de35ee62 refactor(3p/nix/libexpr): Backport upstream VectorBindings merge sort
Since one of the two implementations essentially uses the same shape
as the upstream Bindings, we backport their merge sort implementation
to ensure that we're doing the same thing semantically.

Paired-With: Luke Granger-Brown <git@lukegb.com>
Paired-With: Vincent Ambo <mail@tazj.in>
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: I0d865897991eec0c4dd84d9bd0415cd1ca437792
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1162
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-13 23:51:29 +00:00