Implement RPC handlers and client methods for the AddWithRoot and
SyncWithGC calls. With this and sandboxing disabled, the following
derivation is now building successfully:
derivation {
name = "test";
builder = "/bin/sh";
args = [ "-c" "echo 1 > $out" ];
system = "x86_64-linux";
}
Co-authored-by: Vincent Ambo <mail@tazj.in>
Change-Id: I245dca0dfca882e2eae5d93364287a8b79fead71
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1637
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
Without this, these errors print as "uncaught exception of type
std::string", but with this they print using nix's built-in error
printer, which will be nice to have as we bang out the remaining store
API methods.
Co-authored-by: Vincent Ambo <mail@tazj.in>
Change-Id: I9bd4f77f65b27c8f97b727b98ef62e2061999e71
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1636
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
This reverts part of commit 976a36c2e4,
which was causing scanForReferences to hang indefinitely. I'm not sure
what the original intent of the commit was since there's not really much
of a commit message on it, but with this I am able to fully realise
derivations.
Change-Id: I620e892e2fe2244f066512286b086b3dd8f8f6c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1635
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
This executable doesn't appear to ever actually look at argv, so
throwing an error if argc is wrong seems kinda silly.
Change-Id: Iabc9dad785d4c99be9e8fd4ecd954d38b876cecd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1633
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
When tokenizeString was changed to absl::StrSplit, there was a behavior
change because tokenizeString on an empty string returned an empty
vector - which the derivation builder (and likely a bunch of other
stuff) was depending on. The canonical way of fixing this is by passing
absl::SkipEmpty() to the function - there may be other places we need to
fix this as well.
This commit also includes some opportunistic absl::StrFormats and
StrCats, because I was here anyway, but those have no semantic
difference.
Change-Id: Ibf9bb602284f793fa55728481f63b838fb7a41db
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1631
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
We wrap every server-side proto handler with a macro that catches
exceptions and turns them into proper grpc error codes. For the
time being, most exceptions map to INTERNAL, the existing mapping.
Change-Id: Id6ed6a279b198ad185d32562f39000ccc15eadbf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1599
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
At some point the behavior of this function got changed as part of our
cleanup - this fixes it to behave the way the rest of the codebase
expects (and how it is documented in the header) and covers it with a
few tests.
Change-Id: Id4c91232968e73489cd866fb4a2a84bcf20d875e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1629
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Two exceptions: IsValidPath needs to return success for invalid paths, and QueryAllValidPaths shouldn't need to check the paths it gets from itself.
Change-Id: I4d9d4125d34e8de42f30070aec607f8a902eded7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1598
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Skips over all the monobinary stuff and moves to a separate binary for
nix-daemon.
This also replaces the flag parsing logic with absl::flags. This
causes a behaviour change for --help, which no longer tries to display
a man page but instead shows the actual command-line help.
Note: This binary no longer links to the Boehm GC.
Change-Id: Ib852e994b82f2d56e91262878c10650e656427a9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1622
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
The filename this is squatting is too good.
Change-Id: I694132b94d24786df84b6b5a698679bdce818b2e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1621
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Implement AddTextToStore and BuildPaths both on the client and the
server
Refs: #29
Change-Id: I45294c3e1c1a7489e42099d36425b7acc04e0427
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1560
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
Only libexpr depends on the garbage collector, specifically only
instantiations of EvalState actually require the GC to be initialised.
Rather than always starting it for the whole program, even if it is
not needed, this change moves the GC initialisation into libexpr,
guarded by absl::call_once.
This should make it possible to run the nix daemon without the garbage
collector interfering, granted that things are correcty separated and
the daemon does not actually invoke the evaluator.
Based on my investigation so far, the daemon logic itself does not
require libexpr to be present at all - so I think it is safe - but the
current monobinary might have some tricks up its sleeve that will
cause problems for us. We can deal with those if they arise.
Relates to https://b.tvl.fyi/issues/30
Change-Id: I61c745f96420c02e089bd3c362ac3ccb117d3073
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1584
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Add a rapidcheck test covering roundtrip parse and serialize for Nix
derivations. This covers a bug we discovered in ef54f5d which broke this
roundtrip.
Change-Id: I72d140334b5f24f79e82e34f98609c695dbfbf93
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1582
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
It turns out these aren't supposed to be reused after the first request
- reusing it was causing things to blow up.
Change-Id: I935f1ee82ed4e7f26528ae105c8fa2501fa81b97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1559
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
Prepend the unix:// URI scheme to the daemon socket so that grpc knows
we want to connect to a unix socket rather than another type of URI. As
part of debugging this I made the failure message for the RPCStore
include the URI, which I'm leaving in since it'll be nice to have.
Change-Id: I6e70596895117b9a0d53fe2a61d8542ceb64c940
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1543
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
The prior use of gc_allocator meant that the btree nodes themselves were being collected. Additionally, have Attr (contains a Value) and Bindings explicitly inherit from gc, even though Bindings is always allocated under `new (GC)`.
Detected by running under GC_ENABLE_INCREMENTAL=1.
Change-Id: Iacf13b34b5aa12e417ea87c9b46e2bf9199fdb26
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1544
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
Implement the RPC client calls for QueryPathFromHashPart,
QuerySubstitutablePaths, and QuerySubstitutablePathInfos, and the
handler for QuerySubstitutablePathInfos.
Refs: #29
Change-Id: Idf383b771f159f267d8f65367bc4af3d239e32b7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1515
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
External values are only useful when using the plugin framework, which we are not interested in carrying forward.
Reverts commit 320659b0cd
Change-Id: Ib4929c349bbb33f16224fc674e94c7b7d5953c6a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1505
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
This eliminates the value-smuggling that would trip up the GC.
Change-Id: I8057df78cf0bf6bea9faf1b44233aa9820ae44f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1504
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
Add two more garbage-collection flags. Annotate how terrible tExternal is. Prepare to fix the smuggle casting in ExprWith. Add a static_cast.
Change-Id: I20f980abc8cb192e094f539185900a6df5457c29
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1503
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
Implement the proto handler for AddToStore, which adds a nix path to the
store. This is implemented by adding a new (probably
soon-to-be-generalized) Source concretion that wraps a grpc ServerReader
for the stream of data we're receiving from the client - this is less
than ideal, as it's perpetuating the source/sink thing that's going on
and storing entire nars in memory, but is at the very worst an
incremental step towards a functioning nix that we can refactor in the
future.
Paired-With: Perry Lorier <isomer@tvl.fyi>
Paired-With: Vincent Ambo <mail@tazj.in>
Change-Id: I48db734e7460a47aee4a85dd5137b690980859e3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1441
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <mail@tazj.in>
The use of `unwrap_throw` can be used as a later grep target.
Change-Id: I8c54ed90c4289f07aecb8a1393dd10204c8bce4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1493
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
Fixup for CL 1492 (addcba11b0)
Additionally, add a test to verify functionality of HashSink.
Change-Id: I2a74b925a1b93ed4d3add29021d759c93e813424
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1507
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Additionally, add IsValidBase16() to restore the behavior of rejecting invalid base16, which absl's HexStringToBytes does not do.
Change-Id: I777a36f5dc787aa54a2aa316d6728f68da129768
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1484
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Add a stub class for wrapping a gRPC client to the new, proto-backed nix
store protocol, along with several methods implemented but several left
throwing a not implemented exception.
Paired-With: Vincent Ambo <mail@tazj.in>
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: Id943d4f6d75084b8498786d580e6c9f7c92c104d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1436
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Implement the main function for the new, proto-based nix daemon. This
mostly replicates the behavior of the previous nix daemon - it starts a
grpc server listening on the same nix socket path and supports the same
set of options - with the exception of --stdio, which has been renamed
to --pipe and documented in the man page.
Change-Id: Ib729283c1d5d35c0e1b0a968bc1f052f5527f2d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1356
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Replace the custom, rather questionable base64 implementation with
absl::Base64{Une,E}scape. To make sure that the custom implementation
was doing the same thing I've also added a test covering
nix::Hash::to_string, which was one function that used it - the test
passed prior to the replacement, and continued to pass afterwards.
The previous base64Decode function threw an exception on failure - to
avoid going too far down the rabbit hole I've replicated that
functionality at all call sites, but this should be replaced with more
sensible error handling such as StatusOr eventually.
Also, before this change:
❯ nix eval -f . users.tazjin.emacs.outPath
"/nix/store/g6ri2q8nra96ix20bcsc734r1yyaylb1-tazjins-emacs"
And after:
❯ ./result/bin/nix eval -f . users.tazjin.emacs.outPath
"/nix/store/g6ri2q8nra96ix20bcsc734r1yyaylb1-tazjins-emacs"
Change-Id: Id292ffbb82fe808f3f1b34670afbe7b8c13ad615
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1385
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
The use of vfork() in Nix is entirely illegal. Quote:
If the process created by vfork() returns from the function in which vfork() was
called, or calls any other function before successfully calling _exit() or
one of the exec*() family of functions, the behavior is undefined.
-- Linux man-pages, release 5.05
Add a TODO to use the higher-performance variants of clone() on Linux when it
is available.
Change-Id: I42370e1568ad6e2d00d70d0b66c8aded8f1288bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1418
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: Alyssa Ross <hi@alyssa.is>
nixutil depends on bzip2, lzma, boost::context, brotli{enc,dec}, and
openssl, but wasn't directly linking to them. This was causing linker
errors in a test that only depended on nixutil.
Change-Id: I60e77ea7b18b08e2946fcf9176ae0f355cd71844
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1384
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
This formerly controlled access to builtins.exec and
builtins.importNative, but both of those are gone now, so there's no
need for this option any more.
Change-Id: I6850cbd6be264fbfb1b209a60026dadbd0ba1232
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1341
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
This is the shared object equivalent of builtins.exec, or a plugins
equivalent accessible from the Nix language. Either way, since we
don't have builtins.exec or plugins any more, I think it makes sense
to remove this builtin.
This will also allow us to drop the
allow-unsafe-native-code-during-evaluation option, which formerly
controlled whether builtins.exec and builtins.importNative were
enabled.
Cc: Griffin Smith <grfn@gws.fyi>
Cc: Profpatsch <mail@profpatsch.de>
Change-Id: I8993a8a79d559c102647308a2684c089bbc06713
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1340
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
The SQL schemas are included as string constants which are
concatenated into a header file. In the previous Makefiles, this was
done with envsubst or something - we moved it to CMake.
There was a missing quote around the string to be interpolated, which
meant that CMake interpreted the semicolons as part of its language
syntax and did not emit them.
Change-Id: Ibb4512788b26b53f297db3535094dc0194614446
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1342
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
Add a forward-declared factory function for constructing and returning a
WorkerServiceImpl, for eventual use in the main function for the nix
daemon
Change-Id: I9032d69b6ee3bc3b1f39f3d5d55f951cffad8145
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1293
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Plugins seem to not really be used anywhere (I can find one plugin
that's actually defined, and it doesn't seem very useful, especially
since we got rid of builtins.exec) and their presence is adding
additional complexity and potential sources of bugs to an already
unsteady refactor. At some point we may want to bring back
something *like* plugins, but their design will likely be different and
it will definitely be after we have a functioning Nix again.
Change-Id: I3bc40e55917f70bf260fbc208c1705e2e6a7c626
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1291
Tested-by: BuildkiteCI
Reviewed-by: Alyssa Ross <hi@alyssa.is>
Reviewed-by: isomer <isomer@tvl.fyi>
This compiles under `-Wall -Werror`.
The largest chunk of this change is `final` qualifiers for the various
Nix CLI command structs, which inherit from a Command class that has
more virtual functions than are implemented by each command.
Change-Id: I0925e6e1a39013f026773db5816e4a77d50f3b4a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1294
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
This setting was renamed to api.pure about 300 years ago and caused
warnings to be issued.
Change-Id: If883b0667c3afe67ae3d2a9950a796688cfbea7a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1290
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
