Write *all* build logs (essentially everything that wasn't a DLOG), not
just the stdout of the builder, to the client-bound log_sink_ that's
threaded through build.cc. These logs are significantly more useful on
the client, both when debugging tvix itself and (more importantly) when
building things with it.
To make this all work, this commit also refactors the passing-around of
the log sink in build.cc slightly - the main reference is now owned by
the Worker class, which already stores pointers to itself on the Goals
it creates, so those Goals can then load the log sink from the Worker.
This change was originally made to allow for logging to the log_sink
from instance methods of the worker, but actually feels significantly
more natural than storing the sink on the goal anyway.
Change-Id: I461685bc7f5c0a326054940ac5971b568dd43f8e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1925
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
These were not used anyway, and *may* have been causing a weird series
of problems leading to things sometimes aborting. Or maybe it has
nothing to do with it - the only correlation we've found is from
disassembling the output of a core dump. Regardless, this feels
stylistically like a good idea.
Change-Id: I288b50945e74ea90c118a21767fa4387c9f47f7d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1921
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Rather than printing "Nix" in the version string, print "Tvix". This is
cosmetically nice, but also is very useful when making sure I actually
have installed tvix on a system successfully.
Change-Id: Idd1a9954bd66509327f52b7457cfba2f090ab30c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1924
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
This allows distinguishing between an empty value and no value.
Patch ported from upstream at
ba87b08f85
Change-Id: I061cc8e16b1a7a0341adfc3b0edca1c0c51d5c97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1884
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
When the nix daemon starts up, first check (using sd_listen_fds) whether
we have been systemd socket-activated. If so, instead of passing the nix
daemon socket path to grpc, start a manual accept(2) loop, passing the
client file descriptors to grpc via AddInsecureChannelFromFd. There's an
open grpc issue at https://github.com/grpc/grpc/issues/19133 for
building support into grpc to do this automatically, but as of right now
this appears to be the only way to make this happen.
Making this happen, by the way, was a bit of a journey - at one point I
attempted to ServerBuilder's experimental AddExternalConnectionAcceptor
API, and that didn't work either - it appears that the final missing
piece to getting this working was explicitly fcntl(2)ing the client file
descriptors to set O_NONBLOCK before passing them into gRPC. With that
set, this all works inside of the test vm.
Fixes: b/56
Change-Id: I5d2ab2b5b02eb570249b30a9674e115c61b0ab0e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1882
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
Expose depot to the test vm via NIX_PATH, so we can test the build of
depot packages from inside the test vm.
Change-Id: I26314634895ab1339f5715c0b0b261028eefdaa3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1923
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
These are unused, and I'm not sure they ever actually worked.
Change-Id: I6bb6b4257b7815932581ded7929437dba80961c9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1885
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <mail@tazj.in>
The various bits (including the unix socket that we listen on) append
/nix to this - so it isn't expected to be in there.
Change-Id: I0d95f2cc5641f610c1bdb03b036fac61dac9842d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1890
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
The nix package is expected to install /etc/profile.d/nix.sh and
/etc/profile.d/nix-daemon.sh into its output - these set the environment
variables expected by the client-side nix user commands, eg NIX_DAEMON.
Fixes: b/52
Change-Id: I5fb964c076f092d0ef7e1d49aca37875c866fb90
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1889
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
Rather than sending the entire AddTextToStore request along in a single
message, send it in a stream of chunks using the same metadata-first
approach we've been using for the other store gRPC requests. This fixes
a bug where certain builds could send more data than the maximum gRPC
request size (4194304 bytes, it would appear), resulting in a
RESOURCE_EXHAUSTED error.
The initial chunk size, which is currently constant but should be made
dynamic at some point in the future, has been chosen based on the IPC
bandwidth delay product for tazjin's desktop, rounded up.
Change-Id: I6f0232cdbc98653484816b39855126873fc59a03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1835
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
These are the scripts I use to test the nix daemon interaction with a
non-/nix store directory during development, copied almost verbatim from
my cmake build directory. As such, there's likely a *lot* of cleanup and
deduplication to be done here, but I'm committing these as is in the
hope that others can benefit from them somehow.
Change-Id: I42a10a85e6731fa2014c7ea9738224d678a8376b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1881
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
The command line options --arg and --argstr that are used by a bunch of
CLI commands to pass arguments to top-level functions in files go
through the same code-path as auto-calling top-level functions with
their default arguments - this, however, was only passing the arguments
that were *explicitly* mentioned in the formals of the function - in the
case of an as-pattern with an ellipsis (eg args @ { ... }) extra passed
arguments would get omitted. This fixes that to instead pass *all*
specified auto args in the case that our function has an ellipsis.
Submitted upstream at https://github.com/NixOS/nix/pull/3965Fixes: #46
Change-Id: I32b7ee0e5bacf75b2bc43a3f0796f533f4bd5959
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1863
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
Previously, MixEvalArgs (a generic data type used to handle --arg,
--argstr, and -I arguments to `nix-build`, `nix eval`, etc.) was storing
the difference between --arg and --argstr by prepending a single
character (either 'E' or 'S') to the value of the arg. This is messy and
un-type-safe, so this commit refactors that to use a proper enum and a
std::pair, which allows us to add a switch and get totality checking.
yay, types!
Change-Id: I883f8db3136d05dda190fac0a1b494386c2ff87b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1862
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
This is the clang-tidy lint 'google-explicit-constructor'.
There's a whole bunch of breakage that was introduced by this, and we
had to opt out a few types of this (esp. the string formatting crap).
In some cases minor other changes have been done to keep the code
working, instead of converting between types (e.g. an explicit
comparison operator implementation for nix::Pid).
Change-Id: I12e1ca51a6bc2c882dba81a2526b9729d26988e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1832
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
The previous clang-tidy invocation missed some header files, which has
now been rectified.
Change-Id: I31547754fbf52f439dc7aeefb08ab90bd50c4156
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1831
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
.. turns out producing an entire build log on a single line is not
useful.
This does not use `<< std::endl` because we have run into buffering
issues with the implementation of the logs->gRPC sink, but intend to
replace this in the future using a structured sink for BuildEvent
protos rather than a raw stream.
Change-Id: Ia9b05fa804391d389e2ef53ab4436c0ec5cc452e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1828
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
Introduces the actual log sink in several places where we actually
want the build logs to thread through correctly.
This should cover *most* build paths.
Change-Id: I735dff8a79f7e35a5874eb89b4abb980f9703dc2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1827
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
This *should* wire up the builder's logs all the way back through the
gRPC client, where they are then conveniently discarded.
Change-Id: I65f22526d0b5a8b8d90f28665bc1b4bc7f7c802a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1825
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
This passes an output stream for build logs to almost all relevant
functions inside of build.cc by threading it through the
`Goal`-abstraction.
Store calls that create goals but don't have a sink available use the
DiscardLogsSink().
Change-Id: I2c0cb1aec1f9150f33113f4752055cea518ede8b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1824
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
This part of the store API needs to carry a handle to the log sink
from now on, so that it can be passed in as appropriate from the gRPC
handlers.
In all places where there is no such handler available at the moment,
the discarding log sink has been inserted. This can be used as a
convenient grep target in the future.
Change-Id: I26628e30b4c6437dccdf8f722ca2e8ed827dfc19
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1797
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
this shuts up a few lints ...
Change-Id: I51ee9418767e97247f98b170cdad137e5b12b44d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1796
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
This is not actually legal code, but it kind of ... works. There are
more of these around, these were just the ones Griffin stumbled upon
while working on the build logs.
Change-Id: Iff9821d8fe145dd426648a8ff4510a73f67c9b7d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1795
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
Introduces a class which implements std::streambuf by sending build
log lines to the provided gRPC stream writer as individual messages.
This can be used in the implementations of calls which trigger builds
to forward logs back to the clients.
Change-Id: I3cecba2219cc24d56692056079c7d7e4e0fc1e2c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1794
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
Introduces a new `BuildEvent` proto type which is streamed in response
to calls that trigger builds of derivations.
This type can currently supply build statuses, log lines and
information about builds starting.
This is in preparation for threading build logs through the processes.
Since we have nowhere to send the logs (yet), a null sink is used
instead.
Co-authored-by: Griffin Smith <grfn@gws.fyi>
Change-Id: If7332337b89506c7e404cd20174acdaa1a3be4e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1793
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
In some cases we don't have anywhere for the build logs to go. Until
we understand those cases fully and can get rid of them, this null
sink implementation can be used.
Change-Id: Ib93c43caf268e2c01c43d59737a829e8c43d223e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1792
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
If the global nix config, or any available user nix config location, does not exist, then the loadConfFile() function will throw and not finish initalizing the Nix configuration.
Change-Id: I6db83bca54d9b66699356d107720603476e32c23
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1657
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Now that we have access to a store in tests, we can enable the tests that needed a store.
Additionally, move the expected output files for disabled tests into the disabled folder.
Change-Id: I2492d49d43b93c7c9b0463e4d3d2855a5a51365d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1758
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Value now carries a shared_ptr<Bindings>, and all Bindings constructors return a unique_ptr<Bindings>.
The test that wanted to compare two Bindings by putting them into Values has been modified to use the new Equal() method on Bindings (extracted from EvalState).
Change-Id: I8dfb60e65fdabb717e3b3e5d56d5b3fc82f70883
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1744
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
This is in preparation for making some of Value's members into refcounted ('smart') pointers.
Change-Id: Ibc54e23ac35766a2fd4e14871c9a7c936a603778
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1743
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Previously the outlist would not be used, and it would sort god knows
what in the out value.
This was probably introduced by the std::vector refactoring, and the
language test for builtins.sort was disabled. Whatever reason there
was for disabling it seems to be gone, so we're re-enabling it.
Change-Id: I98941c2cad78df58ff7bea1ece3aaa4133e94bf8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1757
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
This allows you to write EXPECT_OK(statusor), as well as
EXPECT_THAT(status, IsStatusCode(StatusCode::kInvalidArgument).
Change-Id: I53bed694d812c501eb305ed4ddb358e1f9a68277
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1704
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Make Store::buildPaths return a Status with [[nodiscard]] rather than
throwing exceptions to signal failure. This is the beginning of a long
road to refactor the entire store API to be status/statusor based
instead of using exceptions.
Change-Id: I2e32371c95a25b87ad129987c217d49c6d6e0c85
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1745
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
We have decided that leaking memory is a better fate than random,
non-debuggable memory corruption. Future CLs will begin changing
various fields to std::unique_ptr and std::shared_ptr.
It turns out that disabling the GC does not have disasterous impact.
The Nix evaluator only runs on the client CLI, never in any long-
running process. Even the REPL does not leak too badly under this
change, because it uses one EvalState for the duration of the REPL.
Building an explicitly tracing garbage collector is likely in the
future of this project, but that giant amount of work cannot be
done under a nix evaluator that is constantly crashing. We need to
restore development velocity here, and this is the best way we've
figured out to do it.
Change-Id: I2fcda8fcee853c15a9a5e22eca7c5a784bc2bf76
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1720
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
The parser contained vectors, and the primary parser state, that
were not participating in GC tracing.
Change-Id: Ie198592cd7acffd390e3e2ae9595138b56416838
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1706
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
Invocations of the MakeError macro that were not followed by a
semicolon messed up indentation in the next lines.
Change-Id: I03d7d1443f062a38af2c7da3da8928e0ed05e274
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1708
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Aded a few test cases covering the scanForReferences function, which had
been accidentally broken in 976a36c (which is now partially-reverted).
As part of this, since the test needed to generate hashes for store
paths, the logic in MakeStorePath to compress a sha256 hash down to 20
bytes and convert it to base32 has been extracted to a member function
on the Hash class.
Fixes: #34
Change-Id: Ie2d914688a80f42d0234d351a7cc0714fd15709e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1698
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
All of the miscellanious structs that make up a nix Value should inherit
from gc, as they contain pointers to GC'ed things as members
Fixes: #42
Change-Id: I057d8c9f7dafbee7de7644ff152c9cba1aa6bc03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1696
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Now that we've fully implemented the RPC-based store client, we can get
rid of the UDSRemoteStore, whose only use was connecting to the locally
running nix daemon. The RemoteStore still needs to be around to connect
to remote upstream nix stores over SSH.
Change-Id: I0699819803cbfe966b9a46786f2c927d8e4bf1a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1693
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Implement all remaining RPC calls on the RpcSstore client, remove a few
stub methods we had added that weren't actually present in the old
RemoteStore implementation, and add one more RPC call for getBuildLog
that is present in the store API, but that we hadn't added as a
stub *or* to the proto.
Fixes: #29
Change-Id: Id827f51a393ece4bc7bbecaf38aee9eb4b329770
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1692
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
Factor out the shared utilities for interacting with protobufs/grpc from
libstore to a proto.hh header in libproto.
Change-Id: I1cb8d94867d5d4b63a9994be0b53f8f612eb8e3a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1691
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
To aid in debugging RPC call failures, pass the name of the function
being called to the SuccessOrThrow util funcion in the RpcStore
Change-Id: I523dacfab896b85a3dbe6050c07ee6bd5906fa44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1690
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Wrap the BuildPaths and AddTextToStore RPC handlers in HandleExceptions.
These were missed in the original pass due to a merge.
Change-Id: Ie5be45e6098fba7a2b6b1c1be81578cb742c2880
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1689
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
std::function has a natural null we can't eliminate anyway, so this was
effectively std::optional<std::optional<non_nullable_function>>.
Change-Id: If99f139146021edb25d133dad7f0f6e125ef53df
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1688
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
Implement both the client and server sides of AddToStoreNar, using a
templated generalization of the sources and sinks we were using for
AddToStore on both ends.
Change-Id: I73d0ed34118c711b125851dff99a7518ced4af35
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1686
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
The behavior to return a list containing a single empty string when
provided an empty string is a behavior that absl inherited from legacy
code. However, the behavior expected by legacy code in Nix is the
behavior provided by the SkipEmpty option. Switch all calls to use
SkipEmpty, except for the call already using SkipWhitespace.
See also commit 26a59482d2, with the
partly-prophetic message: "there may be other places we need to
fix this as well."
Change-Id: I6e94856a12cfb1b7e4a3b4e221769ed446648861
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1687
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
This container implementation is much faster than std::map. We have
stuck to an ordered container because it's unclear whether the
accesses of this field (of which there are *many*) are actually
ordering dependent.
Also includes an Arbitrary implementation for absl::btree_map (for any
K, V that are also Arbitrary).
Change-Id: I04f58ca0ce32b9ae1759313b01508b0e44bae793
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1683
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
Bumps both nixos-unstable and nixos-20.03 to today's versions, as per
status.nixos.org
Contains minor fixes to things that broke because of the update:
* tazjin/frog: hardware.u2f is a deprecated setting
* glittershark/system: modSha256 in Go modules is now vendorSha256
* glittershark/owothia: removed version constraint on relude
Change-Id: Ib3e9612b1b06ed547b90e4f8b0ffe5ed7fe0a5c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1642
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
This builds on edef's work with depot-scan by adding a dedicated
flag to the command. We piggyback on upstream's restricted-mode
implementation, the checkSourcePath function.
Change-Id: I52bb613549f40dbca1e8caa036635910c1a3d6d0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1654
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Implement the RPC client and server handlers for the FindRoots and
CollectGarbage RPC calls
Change-Id: Ifa5d582c6a33bd1e7661ac2fc860505ef404dad0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1656
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
The source and destination strings cannot be the same string - absl
will write to the destination in a streaming manner, causing the
source to become invalid.
Change-Id: I3578cf1f8789a51d85e0950f7987c398f0a00953
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1659
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
In certain circumstances, the decompression thread could race ahead of the downloader thread and process the same chunk twice. Clear the data buffer while the lock is held to prevent this kind of incident.
Change-Id: I19a84a0c5768d1228c6c18a7664a7b8893ef96de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1658
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
This makes the configuration work out-of-the-box using CMake in a Nix
shell, rather than having to pass the additional variable to CMake on
the command line.
Change-Id: I04d3cc4f2e5ecf47bf2ee459d5e48588b84ae4dd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1643
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
These tests are in preparation for factoring the Store away from libutil's Callback to absl::StatusOr.
They use the newly added MockBinaryCacheStore.
Updates: #25
Change-Id: I30c207589ec38254806ebc9a983f35668e353ae9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1595
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
This adds an implementation of BinaryCacheStore to be used by tests exercising both the logic inherent to BinaryCacheStore and more general Store tests.
A new library target, nixstoremock, is created to indicate that this file is intended only for use in tests and not in user-facing code.
Change-Id: Ib68f777238843a4f3a2303db8a69735fbc22d161
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1645
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Implement the AddToStore RPC client method, which uses an
AddToStorePathWriterSink (the dual of the AddToStorePathReaderSource on
the server side) to hook into the dumpPath machinery (which we should
refactor not to use sinks or sources, but not yet) and write dumped
paths as binary data over gRPC. With this commit and sandboxing
disabled, the following derivation builds cleanly:
derivation {
name = "test";
builder = ./build.sh;
system = "x86_64-linux";
}
where build.sh has chmod +x and contains:
#!/bin/sh
echo 1 > $out
Change-Id: I94cab86f0825a3a9993262a9807130645c13bf44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1638
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
Implement RPC handlers and client methods for the AddWithRoot and
SyncWithGC calls. With this and sandboxing disabled, the following
derivation is now building successfully:
derivation {
name = "test";
builder = "/bin/sh";
args = [ "-c" "echo 1 > $out" ];
system = "x86_64-linux";
}
Co-authored-by: Vincent Ambo <mail@tazj.in>
Change-Id: I245dca0dfca882e2eae5d93364287a8b79fead71
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1637
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
Without this, these errors print as "uncaught exception of type
std::string", but with this they print using nix's built-in error
printer, which will be nice to have as we bang out the remaining store
API methods.
Co-authored-by: Vincent Ambo <mail@tazj.in>
Change-Id: I9bd4f77f65b27c8f97b727b98ef62e2061999e71
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1636
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
This reverts part of commit 976a36c2e4,
which was causing scanForReferences to hang indefinitely. I'm not sure
what the original intent of the commit was since there's not really much
of a commit message on it, but with this I am able to fully realise
derivations.
Change-Id: I620e892e2fe2244f066512286b086b3dd8f8f6c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1635
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
This executable doesn't appear to ever actually look at argv, so
throwing an error if argc is wrong seems kinda silly.
Change-Id: Iabc9dad785d4c99be9e8fd4ecd954d38b876cecd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1633
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
point the SANDBOX_SHELL macro at the actual path to busybox on the build
machine, or allow it to be configured at build-time with a cmake option.
Change-Id: I044a1315ba9baa3bc9ceddf29f36d14f9f9ccd96
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1632
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
When tokenizeString was changed to absl::StrSplit, there was a behavior
change because tokenizeString on an empty string returned an empty
vector - which the derivation builder (and likely a bunch of other
stuff) was depending on. The canonical way of fixing this is by passing
absl::SkipEmpty() to the function - there may be other places we need to
fix this as well.
This commit also includes some opportunistic absl::StrFormats and
StrCats, because I was here anyway, but those have no semantic
difference.
Change-Id: Ibf9bb602284f793fa55728481f63b838fb7a41db
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1631
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
We wrap every server-side proto handler with a macro that catches
exceptions and turns them into proper grpc error codes. For the
time being, most exceptions map to INTERNAL, the existing mapping.
Change-Id: Id6ed6a279b198ad185d32562f39000ccc15eadbf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1599
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
At some point the behavior of this function got changed as part of our
cleanup - this fixes it to behave the way the rest of the codebase
expects (and how it is documented in the header) and covers it with a
few tests.
Change-Id: Id4c91232968e73489cd866fb4a2a84bcf20d875e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1629
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Two exceptions: IsValidPath needs to return success for invalid paths, and QueryAllValidPaths shouldn't need to check the paths it gets from itself.
Change-Id: I4d9d4125d34e8de42f30070aec607f8a902eded7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1598
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Skips over all the monobinary stuff and moves to a separate binary for
nix-daemon.
This also replaces the flag parsing logic with absl::flags. This
causes a behaviour change for --help, which no longer tries to display
a man page but instead shows the actual command-line help.
Note: This binary no longer links to the Boehm GC.
Change-Id: Ib852e994b82f2d56e91262878c10650e656427a9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1622
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
The filename this is squatting is too good.
Change-Id: I694132b94d24786df84b6b5a698679bdce818b2e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1621
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Implement AddTextToStore and BuildPaths both on the client and the
server
Refs: #29
Change-Id: I45294c3e1c1a7489e42099d36425b7acc04e0427
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1560
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
This makes it easy to quickly run clang-tidy on tvix without seeing errors from the generated files.
Change-Id: I0e25089c5626aebdb5c016629a68da9ccd26c124
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1556
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Only libexpr depends on the garbage collector, specifically only
instantiations of EvalState actually require the GC to be initialised.
Rather than always starting it for the whole program, even if it is
not needed, this change moves the GC initialisation into libexpr,
guarded by absl::call_once.
This should make it possible to run the nix daemon without the garbage
collector interfering, granted that things are correcty separated and
the daemon does not actually invoke the evaluator.
Based on my investigation so far, the daemon logic itself does not
require libexpr to be present at all - so I think it is safe - but the
current monobinary might have some tricks up its sleeve that will
cause problems for us. We can deal with those if they arise.
Relates to https://b.tvl.fyi/issues/30
Change-Id: I61c745f96420c02e089bd3c362ac3ccb117d3073
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1584
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Add a rapidcheck test covering roundtrip parse and serialize for Nix
derivations. This covers a bug we discovered in ef54f5d which broke this
roundtrip.
Change-Id: I72d140334b5f24f79e82e34f98609c695dbfbf93
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1582
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
It turns out these aren't supposed to be reused after the first request
- reusing it was causing things to blow up.
Change-Id: I935f1ee82ed4e7f26528ae105c8fa2501fa81b97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1559
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
Prepend the unix:// URI scheme to the daemon socket so that grpc knows
we want to connect to a unix socket rather than another type of URI. As
part of debugging this I made the failure message for the RPCStore
include the URI, which I'm leaving in since it'll be nice to have.
Change-Id: I6e70596895117b9a0d53fe2a61d8542ceb64c940
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1543
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
The prior use of gc_allocator meant that the btree nodes themselves were being collected. Additionally, have Attr (contains a Value) and Bindings explicitly inherit from gc, even though Bindings is always allocated under `new (GC)`.
Detected by running under GC_ENABLE_INCREMENTAL=1.
Change-Id: Iacf13b34b5aa12e417ea87c9b46e2bf9199fdb26
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1544
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
Implement the RPC client calls for QueryPathFromHashPart,
QuerySubstitutablePaths, and QuerySubstitutablePathInfos, and the
handler for QuerySubstitutablePathInfos.
Refs: #29
Change-Id: Idf383b771f159f267d8f65367bc4af3d239e32b7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1515
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
External values are only useful when using the plugin framework, which we are not interested in carrying forward.
Reverts commit 320659b0cd
Change-Id: Ib4929c349bbb33f16224fc674e94c7b7d5953c6a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1505
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
This eliminates the value-smuggling that would trip up the GC.
Change-Id: I8057df78cf0bf6bea9faf1b44233aa9820ae44f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1504
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
Add two more garbage-collection flags. Annotate how terrible tExternal is. Prepare to fix the smuggle casting in ExprWith. Add a static_cast.
Change-Id: I20f980abc8cb192e094f539185900a6df5457c29
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1503
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
Implement the proto handler for AddToStore, which adds a nix path to the
store. This is implemented by adding a new (probably
soon-to-be-generalized) Source concretion that wraps a grpc ServerReader
for the stream of data we're receiving from the client - this is less
than ideal, as it's perpetuating the source/sink thing that's going on
and storing entire nars in memory, but is at the very worst an
incremental step towards a functioning nix that we can refactor in the
future.
Paired-With: Perry Lorier <isomer@tvl.fyi>
Paired-With: Vincent Ambo <mail@tazj.in>
Change-Id: I48db734e7460a47aee4a85dd5137b690980859e3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1441
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <mail@tazj.in>
The use of `unwrap_throw` can be used as a later grep target.
Change-Id: I8c54ed90c4289f07aecb8a1393dd10204c8bce4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1493
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
Fixup for CL 1492 (addcba11b0)
Additionally, add a test to verify functionality of HashSink.
Change-Id: I2a74b925a1b93ed4d3add29021d759c93e813424
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1507
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
