Commit graph

398 commits

Author SHA1 Message Date
Kane York
2a292c71f4 fix(3p/nix/hash): initialize HashSink.ctx
Fixup for CL 1492 (addcba11b0)

Additionally, add a test to verify functionality of HashSink.

Change-Id: I2a74b925a1b93ed4d3add29021d759c93e813424
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1507
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-01 00:09:23 +00:00
Kane York
addcba11b0 fix(3p/nix/hash): smart pointers in HashSink
Change-Id: Ib2aaf42c8b234ee343c4653eb03f328c113dea86
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1492
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-29 06:48:47 +00:00
Kane York
31f9ee58d0 fix(3p/nix/hash): provide a Status-returning constructor
Additionally, add IsValidBase16() to restore the behavior of rejecting invalid base16, which absl's HexStringToBytes does not do.

Change-Id: I777a36f5dc787aa54a2aa316d6728f68da129768
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1484
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-28 02:04:42 +00:00
Kane York
976a36c2e4 chore(3p/nix/hash): eliminate exposed global variable
Change-Id: I3b34e3e17a751e225831ae599c6c6bb782a25679
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1486
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-28 02:00:23 +00:00
Kane York
ec349328e0 chore(3p/nix/hash): add decoding tests
Change-Id: Ifbeaf1822fa920f929482510ee79a5b24d7976ae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1485
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-28 00:38:29 +00:00
Kane York
ef54f5da9f fix(3p/nix): apply all clang-tidy fixes
Change-Id: I265e763393422ee1881653527c91024458060825
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1432
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-27 21:16:39 +00:00
Kane York
80ff83e698 fix(3p/nix): convert local-store asserts into throws
This fixes a clang-tidy DeadStore warning by turning debug asserts into production checks.

Updates: #11
Change-Id: Ia6e5a4cb1b56594c9844c6bbd3d152f84b426d09
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1409
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-27 20:50:57 +00:00
Kane York
c3608fdf87 fix(3p/nix): fix broken clang-tidy config
LineFilter doesn't actually exist in this version of clang-tidy. It was only working because the config was ignored.

Change-Id: Ice5ddb5d1031dfc2cc4fee24674464f965323d8b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1431
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-26 01:04:10 +00:00
Kane York
dafb571818 chore(3p/nix): build in RelWithDebInfo mode
This eases debugging of live crashes.

Change-Id: Ie15a7f8fb3f091cae0fbe012e58862d416a42891
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1433
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-25 23:37:46 +00:00
Griffin Smith
b10970a66f feat(3p/nix): Start implementing RPC store client
Add a stub class for wrapping a gRPC client to the new, proto-backed nix
store protocol, along with several methods implemented but several left
throwing a not implemented exception.

Paired-With: Vincent Ambo <mail@tazj.in>
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: Id943d4f6d75084b8498786d580e6c9f7c92c104d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1436
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-07-25 20:18:29 +00:00
Griffin Smith
dcaba9de64 feat(3p/nix): Add main function for grpc daemon
Implement the main function for the new, proto-based nix daemon. This
mostly replicates the behavior of the previous nix daemon - it starts a
grpc server listening on the same nix socket path and supports the same
set of options - with the exception of --stdio, which has been renamed
to --pipe and documented in the man page.

Change-Id: Ib729283c1d5d35c0e1b0a968bc1f052f5527f2d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1356
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-07-25 20:11:41 +00:00
Griffin Smith
7c56fccdad feat(3p/nix): Template systemd unit files in build
Template the systemd unit file templates as part of the nix
installPhase, putting them in the same place that the upstream nix
derivation does.

Change-Id: I3ceabfc0c837564e33b9ae7f9eeb7185d6fbe907
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1429
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-07-25 14:12:04 +00:00
Kane York
bbbb97e142 chore(3p/nix): disable clang-tidy in cmake
Change-Id: I9ff1ae10afe5af066aaff377424522a19977dc3d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1428
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-25 03:06:10 +00:00
Griffin Smith
0b74688e9c chore(3p/nix): Remove obsolete upload-release.pl script
This hardcodes eelco's home directory, among other things we no longer
care about.

Change-Id: Ide894107c091c37e30bba9daf473fc7dfc8b8563
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1427
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-25 02:56:05 +00:00
Griffin Smith
2f85dfe50c fix(3p/nix): Don't include depotPath in the main drv
Including depotPath in the shellHook of the main derivation for tvix
was, unsurprisingly, causing spurious rebuilds. It's still useful when
developing locally, though, so I've extracted it to a `build-shell`
derivation as a passthru attribute and updated the documentation.

Fixes: #20
Change-Id: Ibc686b9f06ec68e79759ca2c989414bd5fbce696
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1426
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-07-25 02:29:14 +00:00
Griffin Smith
4dca023385 feat(3p/nix): Add build for VM for testing tvix
Add an expression, based on the nixos qemu virtualisation framework, for
a basic system whose nix is tvix, to be used for testing tvix

Change-Id: I3c7422bb10d3ce05a3094671cb770f1f745d814c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1423
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: Alyssa Ross <hi@alyssa.is>
Tested-by: BuildkiteCI
2020-07-25 02:29:14 +00:00
Vincent Ambo
a12b0efc9e chore(3p/nix): Reduce number of active clang-tidy checks
The full clang-tidy check suite is *very* slow. This reduced list is
some kind of middle-ground between running all checks, and having a
useful developer experience.

Crucially most of the static analyzer checks (except for the ones
related to security issues) have been disabled.

We should look into running the full suite in CI only.

Change-Id: I02b96ad3b4d1a43bd6aa90ffdcba800dad966714
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1422
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-25 01:02:41 +00:00
Griffin Smith
2fcf2d0d20 refactor(3p/nix): Remove custom base64 implementation
Replace the custom, rather questionable base64 implementation with
absl::Base64{Une,E}scape. To make sure that the custom implementation
was doing the same thing I've also added a test covering
nix::Hash::to_string, which was one function that used it - the test
passed prior to the replacement, and continued to pass afterwards.

The previous base64Decode function threw an exception on failure - to
avoid going too far down the rabbit hole I've replicated that
functionality at all call sites, but this should be replaced with more
sensible error handling such as StatusOr eventually.

Also, before this change:

    ❯ nix eval -f . users.tazjin.emacs.outPath
    "/nix/store/g6ri2q8nra96ix20bcsc734r1yyaylb1-tazjins-emacs"

And after:

    ❯ ./result/bin/nix eval -f . users.tazjin.emacs.outPath
    "/nix/store/g6ri2q8nra96ix20bcsc734r1yyaylb1-tazjins-emacs"

Change-Id: Id292ffbb82fe808f3f1b34670afbe7b8c13ad615
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1385
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-07-25 00:04:43 +00:00
Kane York
989563a75b chore(3p/nix): log whether or not clang-tidy is enabled
CMake likes to remember that we overrode it with an empty string and not use the new env var.

Change-Id: I16587f27750c9ebd4f65349bb59b37e0f8117f18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1406
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-24 22:18:40 +00:00
Kane York
6039417431 fix(3p/nix): fix all UninitializedObject warnings
The cpptoml occurrences are ignored in CL 1419.

Updates: #11
Change-Id: Ifb21c93b09c67e6b6a4c9e7089abffe7616569fd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1407
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-24 22:18:23 +00:00
Kane York
6a0be19edd chore(3p/nix): ignore all unfixable clang-tidy warnings
Change-Id: I641c447fc57df4fed40ed4e89d79a88342b67bc8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1419
Reviewed-by: Alyssa Ross <hi@alyssa.is>
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-24 22:16:48 +00:00
Kane York
bd77090703 fix(3p/nix): do not call vfork
The use of vfork() in Nix is entirely illegal. Quote:

If the process created by vfork() returns from the function in which vfork() was
called, or calls any other function before successfully calling _exit() or
one of the exec*() family of functions, the behavior is undefined.

-- Linux man-pages, release 5.05

Add a TODO to use the higher-performance variants of clone() on Linux when it
is available.

Change-Id: I42370e1568ad6e2d00d70d0b66c8aded8f1288bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1418
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: Alyssa Ross <hi@alyssa.is>
2020-07-24 22:16:08 +00:00
Kane York
388b5f1abe fix(3p/nix): fix null pointer dereferences
Change-Id: I4c37febcd4978f17720b2526c267c6bf96e84bcc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1417
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-24 22:15:19 +00:00
Kane York
3ce4a6cc18 fix(3p/nix): fix more clang-tidy warnings
Change-Id: I1e8a4852ffa450c552a4cdfc709554799854934c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1416
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-24 02:40:11 +00:00
Griffin Smith
280ec4a46c fix(3p/nix): Link nixutil to its dependencies
nixutil depends on bzip2, lzma, boost::context, brotli{enc,dec}, and
openssl, but wasn't directly linking to them. This was causing linker
errors in a test that only depended on nixutil.

Change-Id: I60e77ea7b18b08e2946fcf9176ae0f355cd71844
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1384
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-07-24 00:56:07 +00:00
Kane York
9a85694b86 fix(3p/nix): remove usage of strcpy
Change-Id: I86125609f433469a8722c780fd758234211d677e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1381
Tested-by: BuildkiteCI
Reviewed-by: Alyssa Ross <hi@alyssa.is>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-23 22:00:08 +00:00
Kane York
ec46a594df docs(3p/nix): Document fast build by disabling clang-tidy
Change-Id: I5a3a4a4013c35a3a377ad0b6068a1a40bf6d77e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1386
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-23 21:58:58 +00:00
Kane York
b8e94002d6 feat(3p/nix): run clang-tidy during builds
We need to -isystem the libcxx header files in order for clang-tidy to ignore them, as the Nix clang toolchain isn't doing that automatically.

Change-Id: I05b9e9bd522de4c0e2ad543214f6bf6ab66a306b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1359
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-23 21:46:24 +00:00
Alyssa Ross
231a36e203 chore(3p/nix): fix extra space in usage error
Change-Id: Ic8f8820ba8967166354951cf13af5fa134a1b6b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1353
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-23 18:55:20 +00:00
Alyssa Ross
5183e92e4f feat(3p/nix): remove allow-unsafe-native-code-during-evaluation
This formerly controlled access to builtins.exec and
builtins.importNative, but both of those are gone now, so there's no
need for this option any more.

Change-Id: I6850cbd6be264fbfb1b209a60026dadbd0ba1232
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1341
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-23 18:55:12 +00:00
Alyssa Ross
855995325e feat(3p/nix): remove builtins.importNative
This is the shared object equivalent of builtins.exec, or a plugins
equivalent accessible from the Nix language.  Either way, since we
don't have builtins.exec or plugins any more, I think it makes sense
to remove this builtin.

This will also allow us to drop the
allow-unsafe-native-code-during-evaluation option, which formerly
controlled whether builtins.exec and builtins.importNative were
enabled.

Cc: Griffin Smith <grfn@gws.fyi>
Cc: Profpatsch <mail@profpatsch.de>
Change-Id: I8993a8a79d559c102647308a2684c089bbc06713
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1340
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2020-07-23 04:53:23 +00:00
Vincent Ambo
950ba404be fix(3p/nix): Fix string escaping issue in schema include
The SQL schemas are included as string constants which are
concatenated into a header file. In the previous Makefiles, this was
done with envsubst or something - we moved it to CMake.

There was a missing quote around the string to be interpolated, which
meant that CMake interpreted the semicolons as part of its language
syntax and did not emit them.

Change-Id: Ibb4512788b26b53f297db3535094dc0194614446
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1342
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-22 01:28:32 +00:00
Vincent Ambo
38a262872a chore(3p/nix): Compile under -Wall -Werror
Change-Id: Ia44c68678a0b62e14228fafec88e7591ce3f4cee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1295
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-20 16:47:39 +00:00
Griffin Smith
17ca547ed3 feat(3p/nix/nix-daemon): add factory function for WorkerServiceImpl
Add a forward-declared factory function for constructing and returning a
WorkerServiceImpl, for eventual use in the main function for the nix
daemon

Change-Id: I9032d69b6ee3bc3b1f39f3d5d55f951cffad8145
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1293
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-20 14:45:43 +00:00
Griffin Smith
2ef1060361 chore(3p/nix): Remove support for plugins
Plugins seem to not really be used anywhere (I can find one plugin
that's actually defined, and it doesn't seem very useful, especially
since we got rid of builtins.exec) and their presence is adding
additional complexity and potential sources of bugs to an already
unsteady refactor. At some point we may want to bring back
something *like* plugins, but their design will likely be different and
it will definitely be after we have a functioning Nix again.

Change-Id: I3bc40e55917f70bf260fbc208c1705e2e6a7c626
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1291
Tested-by: BuildkiteCI
Reviewed-by: Alyssa Ross <hi@alyssa.is>
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-20 14:45:43 +00:00
Vincent Ambo
15afa8472e fix(3p/nix): Fix all remaining compiler warnings
This compiles under `-Wall -Werror`.

The largest chunk of this change is `final` qualifiers for the various
Nix CLI command structs, which inherit from a Command class that has
more virtual functions than are implemented by each command.

Change-Id: I0925e6e1a39013f026773db5816e4a77d50f3b4a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1294
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-07-19 22:00:59 +00:00
Vincent Ambo
ca50fb66d2 chore(3p/nix): Update deprecated 'pure-parser' setting
This setting was renamed to api.pure about 300 years ago and caused
warnings to be issued.

Change-Id: If883b0667c3afe67ae3d2a9950a796688cfbea7a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1290
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-19 19:02:06 +00:00
Vincent Ambo
7a8880a0df refactor(3p/nix): Remove remaining prefork-compat type
Removes the verbosity enum, which is no longer actively used anywhere
other than a daemon protocol implementation bit that doesn't actually
work.

Since the verbosity was marked deprecated, this removes one of the
last remaining warnings.

Change-Id: Iaee9d1d6c14b30daac83bb44bcacff32a0e07fb0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1289
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-19 19:02:06 +00:00
Vincent Ambo
324d385b29 refactor(3p/nix): Rename & undeprecate Bindings::lexicographicOrder
The function is renamed to `SortedByKeys`, which is more descriptive,
and annotated with a comment about what it is used for.

The deprecation warning has been removed because this function is
currently functionally required.

Change-Id: I0ee3a76deff05f366feca9ddac8f38ab34bffbd0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1288
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-19 19:02:06 +00:00
Vincent Ambo
02066a4bab chore(3p/nix): Enable clang-format check in derivation build
This is the easiest way to get the checks up and running for now, but
we will probably want to separate out things like this into a separate
build step in the future.

Change-Id: I8e1a1095aef09b1eee97abad5b6240bc64d14b8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1287
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-19 16:54:52 +00:00
Vincent Ambo
61a7b97c39 test(3p/nix): Enable output comparison for evaluator success tests
Enables loading of the expected output of evaluator tests from the
corresponding .exp files, and checks that the output matches.

This again leaves some tests behind in the disabled folder, but we now
have almost the entire suite up and running so I can get around to
cleaning up the disabled ones.

Other note: Some tests had XML output, despite not being related to
XML testing at all - I'm not sure why they chose to do this, but have
converted those test outputs to normal Nix instead.

We have a separate test suite for JSON & XML serialisation already,
which was contributed by andi-.

Change-Id: Id7c42c836edfec4c22db9d893e35489f3e6dd559
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1285
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-19 16:33:34 +00:00
Vincent Ambo
97e6499225 test(3p/nix): Partially enable evaluator success tests
Enables the `eval-okay-` test suite, with some caveats:

* The output is not yet checked against the expected value, so the
  tests only check that pure evaluation succeeds
* A handful of tests have been disabled as they are causing failures
  that may be related to the DummyStore implementation.

Both of these will be addressed in followup commits, but there is
already some value in having the pure evaluation tests pass.

Change-Id: I62236c95ebffb06dc64a45455a8ee04344c400b7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1284
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-19 16:33:34 +00:00
Griffin Smith
54a5750770 test(3p/nix): Add property tests for attribute set
Add a set of property tests for the attribute set (Bindings) class
checking that the Merge operation satisfies the monoid laws. This
will hopefully become useful to make sure we're not breaking the
language semantics as we work towards optimizing or replacing the
implementation, but also serves as a test bed for adding
rapidcheck-based property tests to the codebase.

Change-Id: I1b4b7b6503d08d80c1c5a8f9408fd4b787d00e8e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1283
Reviewed-by: isomer <isomer@tvl.fyi>
Tested-by: BuildkiteCI
2020-07-19 16:26:52 +00:00
Kane York
f1080c022a fix(3p/nix): convert all holders of EvalState into gc
Change-Id: Ia0d1fdd5c73941d02da0ca7152600f966cee5ccb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1280
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-19 06:40:35 +00:00
Vincent Ambo
105ad71015 refactor(3p/nix/libexpr): Move some code out of lexer.l
Moves a function that is not dependent on the generated code over to
parser.hh. This function also looks like it could be improved, but
that is left as an exercise for the reader.

Code that remains in lexer.l has been reformatted, while we're here.

Change-Id: I9c26bb4eed0772a720d0715029e8bc10ab16ac38
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1279
Tested-by: BuildkiteCI
Reviewed-by: Kane York <rikingcoding@gmail.com>
2020-07-19 02:59:41 +00:00
Vincent Ambo
bd02cae032 test(3p/nix): Port language test suite to Googletest
This moves the language test suite into Googletest by constructing
parameterised tests out of the same language snippets used for the
previous lang.sh evaluation.

So far this includes support for about 3/4 of all tests, specifically:

* all parser success/failure tests
* all evaluator failure tests

The evaluator success tests will be implemented in a subsequent commit,
because the output comparison contains a whole bunch of additional
logic that I did not want to cram in here.

Change-Id: Icec9f368366cdbaa53b4c7e4472b8b6e8dd72eba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1278
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: isomer <isomer@tvl.fyi>
Tested-by: BuildkiteCI
2020-07-19 00:49:29 +00:00
Vincent Ambo
a802b4b44d fix(3p/nix/tests): Fail DummyStore::addToStore on non-existent srcPath
Unfortunately, to guarantee correct behaviour of some evaluation (!)
tests, addToStore needs to actually check whether passed in source
files exist and fail appropriately.

There is a chance that the dependency on this behaviour is actually a
bug in the upstream test suite, but my attempts at finding out more
about this from, say, the git history have so far been unsuccessful.

Change-Id: I311999ea28fcedf5da13a4e627b1c1c8e4e59cbd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1276
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-19 00:49:29 +00:00
Vincent Ambo
989537f972 fix(3p/nix/libexpr): Make EvalState internals visible to GC
Change-Id: Ia7de49acb859040429fdd8ab143d485589ac02d4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1277
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: Kane York <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-07-19 00:49:29 +00:00
Vincent Ambo
70633a3058 chore(3p/nix/tests): Move language test files to src/tests
These files will be integrated into the evaluator unit tests instead
of running separately via a shell script.

Change-Id: I1d229e73b1d862777f5108c86891689900edefbe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1275
Tested-by: BuildkiteCI
Reviewed-by: Kane York <rikingcoding@gmail.com>
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-19 00:49:29 +00:00
Alyssa Ross
7010d4dc86 fix(3p/nix): nix-build: don't accept --(im)pure
These options only apply to nix-shell.  The fact that nix-build
previously accepted them was an accident that resulted from the two
programs sharing an implementation.

Change-Id: I0047c98e2096010797316bff3ea4faf722fab86a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1273
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-18 21:40:36 +00:00
Griffin Smith
a724158512 fix(3p/nix/nix-daemon): Add assertStorePath to all proto impls
I missed calling store_->assertStorePath on paths in several daemon
proto handlers, which the previous implementation did.

Change-Id: Ifad6eeb03b5a5babec7b4bcf7aca060813f15bb7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1272
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-18 19:52:23 +00:00
Griffin Smith
ecf288e70a feat(3p/nix/nix-daemon): Implement Worker::AddSignatures handler
I got the message proto wrong on this one as well - it needs both a path
and a signatures.

Change-Id: I9a489b1285bda61c15b2a3b47d9cfc3b50e387da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1270
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-18 19:52:23 +00:00
Griffin Smith
a79df261b4 feat(3p/nix/nix-daemon): Implement Worker::BuildDerivation handler
Implement the proto handler on the server side for
Worker::BuildDerivation. This includes several additions to the proto
which I had missed on the first pass, including the actual proto
definition for the Derivation itself and a few sequence number
reorderings which are fine because this is all provisional and not
deployed yet.

A couple things to note

- I implemented a couple constructors for nix classes that initialize
  themselves based on their proto variants, which felt nice and didn't
  end up causing any issues.
- I've made the conversions between the enum types in nix and in proto
  explicit via switch statements rather than using a static_cast, out of
  an abundance of caution that the error would get mismatched in the
  future and we'd convert the wrong thing to the wrong thing - this is
  verbose, but exceptionally future proof.

Change-Id: Iecf6b88e76bc37e49efa05fd65d6cd0cb0deffed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1249
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: Kane York <rikingcoding@gmail.com>
2020-07-18 19:52:23 +00:00
Griffin Smith
3f4e5050cd chore(3p/nix): Add missing override to UDSRemoteStore::sameMachine
This silences a -Winconsistent-missing-override

Change-Id: I2492a8ab3c0b0a33a794f012bfee6bbd0c2f0a8e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1271
Reviewed-by: Kane York <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-07-18 19:22:46 +00:00
Alyssa Ross
cbea59c5f6 chore(3p/nix): don't rebuild nix-proto-srcs as much
Interpolating a path into a string will copy the referenced path into
the Nix store, so this got a dependency on all of src.  By first
constructing a path to the src/proto directory using the + operator,
and then interpolating it, we limit what is copied to the store, and
therefore what code triggers a nix-proto-srcs rebuild.

Change-Id: I8dd750f6bc5902b74ffb56470bc8a5f2c01c8cf1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1263
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-07-18 18:19:38 +00:00
Vincent Ambo
4b1818cb01 refactor(3p/nix/tests): Move DummyStore into a separate file
This store implementation is required in all unit tests that use the
evaluator.

Change-Id: I1cfe8cecab8722cd66dc803747821a2be2b2619f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1269
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-18 18:08:42 +00:00
Vincent Ambo
cdc1a0bb6e refactor(3p/nix/libexpr): Back Nix lists with std::vector
This change does away with the previous special-casing of lists of
certain element sizes, and the use of raw C-style arrays.

Lists are now backed by a std::vector of nix::Value*, which uses the
traceable GC allocator.

This change is unfortunately quite noisy because the accessor methods
were updated/removed accordingly, so all callsites of Nix-related
lists have changed.

For some operations in primops.cc where keeping the previous code
structure would have been more difficult with a "proper" vector, the
implementation has been replaced with std::vector methods. For
example, list concatenation now uses appropriate range inserts.

Anecdotally the performance of this is about equal, to even slightly
better, than the previous implementation.

All language tests pass and the depot paths I've used for testing
still evaluate.

Change-Id: Ib5eca6c0207429cb323a330c838c3a2200b2c693
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1266
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: Kane York <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-18 18:08:42 +00:00
Vincent Ambo
56614c75e4 refactor(3p/nix/libexpr): Store nix::Env values in a std::vector
This has several advantages:

* we can ensure that the vector is traced by the GC
* we don't need to unsafely allocate memory to make an Env

Note that there was previously a check about the size of the
environment, but it's unclear why this was the case (git history
yielded nothing interesting) and it seems to have no effect.

Change-Id: I4998b879a728a6fb68e1bd187c521e2304e5047e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1265
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: Kane York <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-18 18:08:42 +00:00
Griffin Smith
be6fe93dd4 feat(3p/nix): Add NIX_DATA_DIR to shell hook
all of the executables that get built during regular development depend
on this being set to a directory that contains the nix directory -
previously we had been doing it manually every time, this automates it.

Change-Id: I4c957c0abf0a92ca7122a47d3b141a8ede280e13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1258
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-18 00:03:24 +00:00
Vincent Ambo
f4f72bcf21 test(3p/nix): Set up scaffolding & fix up tests for value-to-{json|xml}
Configures the CMake build to load & run the GoogleTest tests.

I (grfn) also updated this to get the tests running as part of the nix
derivation, which required defining our own manual configurePhase and
installCheckPhase, rather than depending on the one provided by stdenv.
Not doing this would cause cmake to attempt to *run* the tests as part
of the buildPhase, which wouldn't work because the dynamic libraries
hadn't been put into a place where the test executables knew where to
find them. We're not sure *why* this fixes it, and for some reason
fixing this also breaks the automatic behavior of nixpkgs of passing
-j$NIX_BUILD_CORES -l$NIX_BUILD_CORES to make, but that's eaasy enough
to fix manually in a preBuild

Paired-With: Griffin Smith <grfn@gws.fyi>
Change-Id: I79d61854a3ff47301cdce8a40c76820a97bdf901
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1240
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-18 00:01:53 +00:00
Andreas Rammhold
e4b7d6d592 test(3p/nix/libexpr): add json/xml tests
Backported from:
c1371b6fbb

Change-Id: Ida326cf1567081a56d747c4a72f036f55cfdea37
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1239
Tested-by: BuildkiteCI
Reviewed-by: Kane York <rikingcoding@gmail.com>
2020-07-18 00:00:51 +00:00
Griffin Smith
b32274120c feat(3p/nix/nix-daemon): Implement Worker::VerifyStore handler
Also adds the missing check_contents field to the VerifyStoreRequest
proto message, since it was missed in the original pass. This is done
using a renumbering, which is fine in this case since the proto hasn't
been deployed yet

Change-Id: I92bf4e48a71a25ae02ae02b3deaf6e7c71fe5da7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1237
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: Kane York <rikingcoding@gmail.com>
2020-07-17 20:37:34 +00:00
Griffin Smith
d18246f8b2 feat(3p/nix/nix-daemon): Implement Worker::OptimiseStore handler
Change-Id: I8b1d84799a608a516d0b4980022a7edd545a1ca1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1235
Tested-by: BuildkiteCI
Reviewed-by: Kane York <rikingcoding@gmail.com>
2020-07-17 20:37:34 +00:00
Griffin Smith
1b07e588c8 feat(3p/nix/nix-daemon): Implement Worker::QuerySubstitutablePaths handler
Change-Id: I243d67b0bce29d54c7d6e08f5eee70bd395cf9a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1234
Tested-by: BuildkiteCI
Reviewed-by: Kane York <rikingcoding@gmail.com>
2020-07-17 20:37:34 +00:00
Griffin Smith
d36ee69061 feat(3p/nix/nix-daemon): Implement Worker::QueryValidPaths handler
Change-Id: I221dcd2a3fa53ead3ea9ac47830d968bb587c6e5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1233
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-17 20:37:34 +00:00
Griffin Smith
1933e4f97f feat(3p/nix/nix-daemon): Implement Worker::QueryPathFromHashPart
Change-Id: Id351a0c82f4d6e79a8f272674d8d8e86e12a9400
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1232
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-17 20:37:34 +00:00
Griffin Smith
a67db573e5 feat(3p/nix/nix-daemon): Implement Worker::QueryDerivationOutputNames handler
Change-Id: Ie17e37322dd0e241cd35b6d8e69bd6c968fd6ff2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1231
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-17 20:37:34 +00:00
Griffin Smith
eb4c8717cd feat(3p/nix/nix-daemon): Implement Worker::QueryPathInfo handler
Change-Id: I580bd29356f7bcd0cc2050afda11d2e115d44c94
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1230
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-17 20:37:34 +00:00
Eelco Dolstra
80eaa1eaf9 feat(3p/nix): Add function to allocate a Value in traceable memory
Backported from:
b3e5eea4a9
fcd048a526

Intentionally skipped because we have not backported the JSON changes:
9f46f54de4

Did not apply changes ni primops.cc, because those look suspect and
are also based on something that we don't have in our tree.

Change-Id: I837787ce9f2c90267bc39fce15177980d209d4e9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1253
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-17 20:04:14 +00:00
Eelco Dolstra
014436eb4a fix(3p/nix): ValueMap, VectorVector: Use traceable_allocator
We want to *trace* the 'Value *' arrays, not garbage-collect them!
Otherwise the vectors/maps can end up pointing to nowhere.

Backported from:
10e17eaa58

Change-Id: I30dc94caa80c9d982e7a14bc67ba2d065e8203aa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1252
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-17 20:04:14 +00:00
Andreas Rammhold
5218c83b9f fix(3p/nix): allocate the vSourceExpr via uncollectable memory
Previously the memory would occasionally be collected during eval since
the GC doesn't consider the member variable as alive / doesn't scan the
region of memory where the pointer lives.

By using the traceable_allocator<T> allocator provided by Boehm GC we
can ensure the memory isn't collected. It should be properly freed when
SourceExprCommand goes out of scope.

Backported from:
d2c371927e

Change-Id: I1f7c745dbc66c7164bee50f4d9b0d437dbc7dd51
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1251
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-17 20:04:14 +00:00
Kane York
af47a89d6f chore(3p/nix/proto): Fix style violations in new proto defs
This is a blocker to enabling the linter.

Change-Id: I9f8d3cc19c7539086f53474a505362230fc56c04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1245
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-17 20:01:29 +00:00
Vincent Ambo
8d5857abbc chore(3p/nix): Remove builtins.exec
This is a bad idea, it shouldn't exist, nixpkgs doesn't use it.

Change-Id: Ic4d1b936d8f059d5c40f0567af165b02427d7e36
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1241
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-17 19:03:59 +00:00
Kane York
a556913079 chore(3p/nix): add glittershark to OWNERS
Change-Id: I71228383c8c7810ff8032e67bde0f45339809934
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1236
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-17 01:24:38 +00:00
Griffin Smith
d1b3514910 feat(3p/nix/nix-daemon): Implement Worker::QueryAllValidPaths handler
Change-Id: If9d2144ae872183abf47d84c94cc5c5fa23ced87
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1229
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 23:57:49 +00:00
Griffin Smith
ec859afe95 feat(3p/nix/nix-daemon): Implement Worker::QueryMissing handler
Change-Id: I004a0c7969e1fe27b844adfed2d3ba3e6c5279bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1227
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 23:25:25 +00:00
Vincent Ambo
5c9af8faee feat(3p/nix): Implement similar group of Worker::Query* handlers
These are the queries that are handled in the confusing case statement
in the old daemon implementation, because they have very similar
structure.

Change-Id: Ie7143354f66cef4336dff8072ede9a56271a7e89
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1228
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 23:20:15 +00:00
Vincent Ambo
32735eebdd feat(3p/nix/nix-daemon): Implement Worker::HasSubstitutes handler
Change-Id: I0b01b554eee4b4e31416aabb498ea3d8b8a8092b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1226
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 22:48:09 +00:00
Vincent Ambo
bb66262068 feat(3p/nix/nix-daemon): Implement Worker::IsValidPath handler
Change-Id: I741c2b9b58f234a21850640e2b0c071ff4441234
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1223
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 22:48:09 +00:00
Vincent Ambo
ba11727ce0 feat(3p/nix/proto): Add QueryMissing RPC call
Change-Id: I6a29c56a0f945afdef7434da539b4b91b2e4d2e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1222
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-16 22:10:22 +00:00
Vincent Ambo
afe99225b9 feat(3p/nix/proto): Add AddToStoreNar RPC call
Change-Id: I335630644205adb280d83e006baf9d497ac1ac95
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1221
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 22:10:22 +00:00
Griffin Smith
cf6c48ef47 feat(3p/nix): Add NarFromPath
What it does I will never know. It takes a path and returns a path.
Something happens in the middle.

Change-Id: I499a9df700e5b954c9aaf6d694753ff34e773dfd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1210
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 21:41:40 +00:00
Griffin Smith
c3e96bcfa2 feat(3p/nix): Add AddSignatures
Change-Id: Ic37409c7987c261dc3eb91382d40f458fae6ea0c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1209
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-07-16 21:41:40 +00:00
Griffin Smith
f4baaf79e2 feat(3p/nix): Add BuildDerivation proto
Change-Id: Ib66b4ff2da72677e74b6a713186c72f66d9f6f88
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1208
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 21:41:40 +00:00
Griffin Smith
db4ac8ea7c chore(3p/nix/worker-protocol): Add missing includes
These files were always implicitly depended upon by worker-protocol.hh, but just so happened also always be included in places where the other ones were already present.

We are likely getting rid of this file sooner rather than later, but in
the meantime this will cause clangd to shut up while I'm editing the
file.

Change-Id: I0d765f8b86828b6612e5483417e452221ea3c5b1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1207
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-07-16 21:40:58 +00:00
Griffin Smith
4468e411fd feat(3p/nix): Add VerifyStore
Change-Id: I6a0596e986867e94857da461c746a80750830f7b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1206
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 21:40:58 +00:00
Griffin Smith
3999439fbb feat(3p/nix): Add OptimiseStore
Change-Id: I0d254504801283e018eef41bd5ea9850b17c0d80
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1205
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 21:40:58 +00:00
Griffin Smith
b6cb681c0b feat(3p/nix): Add QueryValidDerivers
Change-Id: I52965bdbf143ae11201e1fafed1c25b3d8f1bee9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1204
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 21:40:58 +00:00
Griffin Smith
143ef52592 feat(3p/nix): Add path query ops
Add QueryValidPaths and QuerySubstitutablePaths, both of which filter a
list of paths based on a set of criteria.

Change-Id: I6aa4647efe82b82dc9582a311643d5f9b6d521d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1203
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 21:40:58 +00:00
Griffin Smith
863a77dd2a feat(3p/nix): Add QueryPathFromHashPart
Change-Id: I0a31557f4cf585d3e539e29ef1a07b443aeec9fc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1202
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 21:40:58 +00:00
Vincent Ambo
d470ec0d29 refactor(3p/nix/libexpr): Use Abseil collection types for parser state
Replaces the previous uses of the (ordered!) std::map and std::set
with absl::flat_hash_{map|set}.

After some careful reading it seems that there is actually no
ordering dependency on these types, and the (drop-in) replacements
perform slightly better.

Overall this is not fixing a bottleneck, just a driveby thing.

Change-Id: Ided695dc75676bd58515aa9382df0be0a09c565e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1220
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: isomer <isomer@tvl.fyi>
Tested-by: BuildkiteCI
2020-07-16 21:03:52 +00:00
Vincent Ambo
e05eb20102 refactor(3p/nix/libexpr): Move EvalState:* implementations out of parser.y
Several definitions of functions declared in eval.hh were previously
implemented in parser.y, this moves them over to parser.cc.

While this still isn't a reasonable place to keep them, the long-term
fix is more likely to be that eval.hh needs to be split up.

Before we get to that point however, this already gives us the ability
to use tooling with this code.

Change-Id: If06fb655325fe281564047ffab0a0a640428a0ee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1219
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-16 19:53:51 +00:00
Vincent Ambo
5ae2a2b847 refactor(3p/nix/libexpr): Extract static helpers out of parser.y
Moves several of the static helper functions into a new parser.cc
file.

Once the rest of the code is usefully extracted, these will be moved
to a private namespace.

Change-Id: I0d7b53dcefe31bb5c6bad3ad7f5fcb48276bf799
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1218
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-16 19:53:51 +00:00
Vincent Ambo
d579ceb783 refactor(3p/nix/libexpr): Extract parser header out of Yacc file
First step (of many?) towards extracting all the inline code from the
Yacc file and keeping it somewhere more accessible instead.

Note that none of this code has previously been touched by a linter or
formatter, pretty much ever, so as it is extracted it also undergoes
similar changes to the whole codebase after the initial fork.

Change-Id: If3b7181f22e3b3fd8c58dfa9befa7ee2896ea06d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1217
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-16 19:53:51 +00:00
Vincent Ambo
cb3d967508 refactor(3p/nix/libexpr): Use range insertion to merge nix::Bindings
Instead of manually iterating over the two bindings to be combined,
this adds a new static method on the Bindings class which merges two
attribute sets by calling the range insertion operator over them.

In some anecdotal tests, this can lead to a ~10% speed bump -
depending on the specific operation.

Change-Id: I5dea03b0589a83a789d3a8a0fc81d0d9e6598371
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1216
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 18:51:47 +00:00
Vincent Ambo
9a4a4058a8 style(3p/nix): Rename the project to tvix
Changes the derivation name & README overview to say "Tvix" instead of
"tazjix".

The previous name was mostly intended as a joke, and a way for me to
distinguish output paths. It's certainly not the intention to have a
portmanteau with my name here, especially now that several people are
contributing to the fork.

Change-Id: Icface5484d52355111eca23b2f6bd3b9e5567275
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1212
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-16 15:28:52 +00:00
Griffin Smith
14e9e161c9 feat(3p/nix/proto): Add QueryDerivationOutputNames
Change-Id: I5452d0f76441a61d70031f567c9cb75249569c2a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1201
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-07-16 01:54:05 +00:00
Griffin Smith
17c0962802 feat(3p/nix): Add proto for QueryPathInfo
Change-Id: I10ec338ef2d5360954abdc5bb3d4789f34b031fc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1200
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-07-16 01:54:05 +00:00
Griffin Smith
0386fad78f chore(3p/nix): {query,clear}FailedPaths are obsolete
They're not mentioned anywhere on the client or the server.

Change-Id: Ia78c8bde49326f6bf69b4aeded083cfd3235131f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1188
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 01:51:19 +00:00
Griffin Smith
150e2bc908 feat(3p/nix/worker.proto): Add QueryAllValidPaths
takes nothing, returns paths

Change-Id: I4087b9b5a25d9edc76b97d0a0f553259693cca3a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1187
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 01:51:19 +00:00
Vincent Ambo
622250bff9 feat(3p/nix/proto): Add additional worker-protocol operations
On the chopping block this time:

* requests/responses that return one or more store paths, and contain
  nothing else, have been changed to use the same types (StorePath and
  StorePaths, respectively)
* QuerySubstitutablePathInfos has been added. It should be noted that
  legacy Nix has two versions of this call, one that only queries a
  single info (deprecated) and one that queries multiple. We have only
  implemented the latter.
* QueryDerivationOutputs has been added.

Change-Id: Iccc9041e7064e141cf593467ecdcc327581c4056
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1186
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 01:19:38 +00:00
Vincent Ambo
eff165425a refactor(3p/libstore): Optimise nix::printString
Slight performance optimisation of nix::printString by copying chunks
of the input string which do not need escaping as contiguous blocks.

Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: I48bad90c8f2831ae4524c814a12b1982989922f9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1184
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 00:37:29 +00:00
Vincent Ambo
4303cedde4 style(3p/nix): Minor style fix in DrvInfo::setMeta
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: I0655ecc675239b3d90e5adc305c3f37c1a904cf5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1181
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 00:37:29 +00:00
Vincent Ambo
d8ee0afda9 refactor(3p/nix): Compare attribute sets by pointer equality
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: I418e9127c5d9d31559c59e461f17726ddbc051c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1180
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 00:37:29 +00:00
Vincent Ambo
5cd7cf93fc refactor(3p/nix): Use a static empty Bindings for 0-element attrs
A significant fraction of all created attribute sets are empty; hence
this is an easy optimisation to make.

Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: I0884194d04c1ee95b2b239a253515f2152bc0856
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1179
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 00:37:29 +00:00
Vincent Ambo
1390827b9e refactor(3p/nix): Revert VectorBindings implementation
This reverts parts of the CLs splitting the backing implementation for
Bindings and moves back to only the BTreeMap-backed implementation.

Our evaluation has indicated that the Vector-backed implementation
does not match the performance of the plain array used upstream, and
in my view the complexity introduced by it is not worth the relatively
small (single-digit percentage) performance increase with a
pivot-point close to the number of attributes yielded by
stdenv.mkDerivation.

Going forward we will trial implementations of attribute sets backed
by HAMTs, and investigate other mechanisms of speeding up the language.

Some changes from the previous CLs are retained, for example the
removal of insert_or_assign and the passing of capacity.

Change-Id: I6eb4b075b453949583360755055c21a29d7ff642
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1172
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-15 00:16:06 +00:00
Vincent Ambo
c5de35ee62 refactor(3p/nix/libexpr): Backport upstream VectorBindings merge sort
Since one of the two implementations essentially uses the same shape
as the upstream Bindings, we backport their merge sort implementation
to ensure that we're doing the same thing semantically.

Paired-With: Luke Granger-Brown <git@lukegb.com>
Paired-With: Vincent Ambo <mail@tazj.in>
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: I0d865897991eec0c4dd84d9bd0415cd1ca437792
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1162
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-13 23:51:29 +00:00
Vincent Ambo
6166c9daf2 fix(3p/nix/libexpr): Remove default & copy constructors for VectorBindings
Paired-With: Luke Granger-Brown <git@lukegb.com>
Paired-With: Vincent Ambo <mail@tazj.in>
Paired-With: Perry Lorier <isomer@tvl.fyi>

Change-Id: Ief2e59d461452ce599abc63f6ebcfa07a7062491
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1161
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-07-13 23:51:29 +00:00
Vincent Ambo
d94690b537 fix(3p/nix/libexpr): Fix post-increment operator in BindingsIterator
We accidentally returned the incremented iterator in the
post-increment, this fixes it.

Paired-With: Luke Granger-Brown <git@lukegb.com>
Paired-With: Vincent Ambo <mail@tazj.in>
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: I36c79eb56359bb12a78ad3489e7d7d2eb2053510
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1140
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-13 23:51:29 +00:00
Griffin Smith
785ca36316 refactor(3p/nix): Remove unused insert_or_assign
This function in never called, so let's just remove it

Paired-With: Luke Granger-Brown <git@lukegb.com>
Paired-With: Vincent Ambo <mail@tazj.in>
Change-Id: I79125866254d90dd0842bc86830d2103ac313cb6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1125
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-13 23:50:24 +00:00
Griffin Smith
d5597b4784 feat(3p/nix): Statically pass bindings capacity where possible
To aid in making the decision of where to (currently just statically)
use a vector or btree as the backing implementation, add an extra
constructor argument to Bindings::NewGC for a capacity, and use
a (currently hardcoded at 32, for no good reason other than it felt like
a reasonable number) pivot to switch between our possible backing
implementations. Then, update all the call sites where it feels
reasonable that we know the capacity statically to *pass* that capacity
to the constructor.

Paired-With: Luke Granger-Brown <git@lukegb.com>
Paired-With: Vincent Ambo <mail@tazj.in>
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: I1858c161301a1cd0e83aeeb9a58839378869e71d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1124
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-13 23:50:24 +00:00
Griffin Smith
d5505fcff9 feat(3p/nix): Add vector-backed impl for Bindings
Add an alternative impl of the now-abstract Bindings base class that is
backed by a std::vector, somewhat similar but stylistically a little
superior to the array-backed implementation in upstream nix. The
underlying iterator type in BindingsIterator is now backed by a
std::variant that we std::visit an overload over in order to implement
the various bits of the iterator interface.

Paired-With: Luke Granger-Brown <git@lukegb.com>
Paired-With: Vincent Ambo <mail@tazj.in>
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: I7fbd1f4d5c449e2f9b82102a701b0bacd5e80672
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1123
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-13 23:50:24 +00:00
Griffin Smith
98148e6711 refactor(3p/nix): Abstract away concrete bindings impl
To pave the way for the thing we want to do eventually which is use a
linear-time array for bindings (aka attribute sets) that are statically
known to be small enough to get a performance benefit from doing so,
make the Bindings class abstract, and define a BTreeBindings class that
inherits from it and is (currently always) returned from the static
initializer. The idea is that we'll have an ArrayBindings class as well
later that we can dispatch to conditionally based on an optional
"capacity" parameter or something like that.

There was some difficulty here in getting the iterator to work - the
approach we settled on ended up making a concrete BindingsIterator class
which will wrap a std::variant of either a btree iterator or something
else later, but right now just wraps a btree iterator.

Paired-With: Luke Granger-Brown <git@lukegb.com>
Paired-With: Vincent Ambo <mail@tazj.in>
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: Ie02ca5a1c55e8ebf99ab1e957110bd9284278907
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1121
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-13 23:50:24 +00:00
Vincent Ambo
ae16a8e9cf docs(3p/nix): Minor semantic correction to a comment
Change-Id: I1cfbb7e933da54198115b28ac509b0d04870fd8f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1127
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-13 21:09:36 +00:00
Vincent Ambo
fa161e9a38 refactor(3p/nix/libexpr): Remove the nix::Symbol default constructor
Having a default constructor for this causes a variety of annoying
situations across the codebase in which this is initialised to an
unexpected value, leading to constant guarding against those
conditions.

It turns out there's actually no intrinsic reason that this default
constructor needs to exist. The biggest one was addressed in CL/1138
and this commit cleans up the remaining bits.

Change-Id: I4a847f50bc90e72f028598196592a7d8730a4e01
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1139
Reviewed-by: isomer <isomer@tvl.fyi>
Tested-by: BuildkiteCI
2020-07-13 21:06:43 +00:00
isomer
afd1367337 fix(nix-channel): Actually strip whitespace.
This has been providing a warning and it's been bothering me.

Change-Id: I0548059950ec4250d7cf0938f9deae09eafe593c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1141
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: isomer <isomer@tvl.fyi>
Tested-by: BuildkiteCI
2020-07-13 20:18:26 +00:00
Vincent Ambo
f3165f48aa refactor(3p/nix/libexpr): Make nix::AttrName a std::variant
nix:AttrName was one of the few classes that relied on the default
constructor of nix::Symbol (which I am trying to remove in a separate
change).

The class essentially represents the name of an attribute in a set,
which is either just a string expression or a dynamically evaluated
expression (e.g. string interpolation).

Previously it would be constructed by only setting one of the fields
and defaulting the other, now it is an explicit std::variant.

Note that there are several code paths where not all eventualities are
handled and this code is bug-for-bug compatible with those, except
that unknown conditions (which should never work) are now throwing
instead of silently doing ... something.

The language tests pass with this change, and the depot derivations
that I tested with evaluated successfully.

Change-Id: Icf1ee60a5f8308f4ab18a82749e00cf37a938a8f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1138
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-13 20:14:49 +00:00
Vincent Ambo
693661fe87 feat(3p/nix/libutil): Add util::overloaded class for std::visitor
Implements the fairly common lambda overload class used for std::visit
over variants and other things that require groups of callables.

Change-Id: Ia7448b7e1bd349b4909974758e6e6303a80d86d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1137
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: edef <edef@edef.eu>
2020-07-13 20:14:49 +00:00
Eelco Dolstra
5e2151b521 fix(3p/nix/repl): Scan NixRepl for GC roots
Backport of b244e65cdb

Fixes https://github.com/NixOS/nix/issues/3175

------

Original fault description by puck:

I suspect I found the issue: The Nix Command structs are stored on the
heap (using ref<T>, a wrapper around std::shared_ptr<T>), which means
that any pointers that the NixRepl struct contains are eligible to be
reaped by the GC. This includes, but is not limited to, the Env
pointer, which seems to cause most of the random segfaults, or random
other values in the environment, which seems to be what @arianvp
experienced too.

Change-Id: I376d7cfd432daaa6f1fbbf77788ff048082f34e5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1001
Tested-by: BuildkiteCI
Reviewed-by: Kane York <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-10 22:25:30 +00:00
Andreas Rammhold
1937bc8624 fix(3p/nix/libstore): use exception instead of assertion when storepath is malformed
Previously the nix-daemon would crash if a user fed it invalid store
paths for drv files. The crash was due to the changed assertion
triggering. Whenever that assertion would hit the nix-daemon process
along with all it's current childs (running builds from all users) would
be interrupted.

Before this patch:

 $ nix-store --realise /nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.drv
 don't know how to build these paths:
   /nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.drv
 error: unexpected end-of-file

 < nix-daemon terminates >

With this patch:

 $ nix-store --realise /nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.drv
 don't know how to build these paths:
   /nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.drv
 error: path '/nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.drv' is not a valid store path

 < nix-daemon does *NOT* terminate >

Change-Id: I01c5048c8a43a8b9154bdeb781d05b7744869ec0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/981
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-07-10 19:41:52 +00:00
Vincent Ambo
b485203b8c feat(3p/nix): Add additional worker protocol actions to proto
This takes us about halfway through worker-protocol.hh

I have left out the documentation strings for some of these items
because I don't feel that I can currently write an unambigous
description of them. For now I am just attempting to match the types.

Change-Id: Iae64b1676152fe4ea069e2021b75ad76465cf368
Reviewed-on: https://cl.tvl.fyi/c/depot/+/960
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-10 17:12:39 +00:00
Vincent Ambo
93eb10566a docs(3p/nix): Update links & other things in README
Change-Id: I1a111c11a83f0506e0973615848cc9c0ef4765e2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/959
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-07-09 22:53:28 +00:00
Vincent Ambo
8f4dea27a6 chore(3p/nix/libstore): Link to nixproto library
This isn't actually used yet, but forces the protos to be included in
the build which is useful for iteration.

Change-Id: I2abcaf297f34ae741f00ad0c929b226d5603c9d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/928
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-05 19:17:12 +00:00
Vincent Ambo
0064e55486 feat(3p/nix): Add protobuf & gRPC dependencies and generate sources
Adds dependencies on the gRPC & protobuf libraries, and implements Nix
code to generate the C++ sources from the included proto definitions.

This is theoretically supported via CMake, but practically doesn't
work and I don't care to debug why.

Doing it like this lets us instead add a CMake library target for our
proto definitions based on the sources generated by Nix.

Pros:
  * no need to deal with the gRPC CMake mess
  * it works!

Cons: * iteration requires nix-shell restart
Change-Id: Ie1fe9807fc96c49cb8f7161ba59d093456062b15
Reviewed-on: https://cl.tvl.fyi/c/depot/+/927
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-05 19:17:12 +00:00
Vincent Ambo
a6da980a0b feat(3p/nix): Add gRPC definitions for worker protocol
Adds initial gRPC definitions for the Nix worker protocol, which is
currently defined messily across the following files:

    src/libstore/worker-protocol.hh
    src/libstore/remote-store.cc
    src/nix-daemon/nix-daemon.cc

The protocol definition is basically a big enum with the signatures of
the calls being implicit in the various client/server implementation
functions.

The definitions in this file are slowly reversed from these implicit
signatures, and are likely to contain an error or two which will be
weeded out when this is taken into use.

Only a handful of the calls are included in this commit, it is
intended to get us up and running first.

Change-Id: Ibc9b2ab4b91a064c8935f09f7ac72bb8150fb476
Reviewed-on: https://cl.tvl.fyi/c/depot/+/926
Reviewed-by: isomer <isomer@tvl.fyi>
Tested-by: BuildkiteCI
2020-07-05 19:17:12 +00:00
Vincent Ambo
7e8a48c82c fix(3p/nix): Link to glog built by Nix
Same as cl/921, it seems that the trick to making this work is indeed
overriding the C++ standard used in the dependency.

Change-Id: I3c5984d71014d774c161ecc283844f504fd44719
Reviewed-on: https://cl.tvl.fyi/c/depot/+/922
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-07-05 16:54:12 +00:00
Vincent Ambo
5cb6e6ee21 fix(3p/nix): Link to Abseil built by Nix
This didn't work previously ... but now it does. I think setting the
standard explicitly is what did the trick, but it's slightly unclear
to me why.

Either way this means that Abseil is no longer constantly getting
recompiled when building Nix, which is nice.

Change-Id: I377f7b68bf1ef9045df6a2eee8fdd0c92f243547
Reviewed-on: https://cl.tvl.fyi/c/depot/+/921
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-07-05 16:54:12 +00:00
Vincent Ambo
4455f28e42 chore(3p/nix/libexpr): Remove dead code from attribute sets
These bits are no longer required with the hashmap-backed
implementation of attribute sets.

Change-Id: I8b936d8d438a00bad4ccf8e0b4dd719c559ce8c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/912
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2020-07-04 03:12:16 +00:00
glittershark
ec84fde6b1 fix(3p/nix): Revert "feat(3p/nix): Add installCheckPhase"
This reverts commit 368e8d1edd.

Reason for revert: Didn't mean to submit, and the phase is currently failing (which breaks the otherwise-functional derivation)

Change-Id: I515b2fb45188dc90f09ae2458453192487c74d71
Reviewed-on: https://cl.tvl.fyi/c/depot/+/581
Reviewed-by: tazjin <mail@tazj.in>
2020-06-23 15:23:04 +00:00
Griffin Smith
368e8d1edd feat(3p/nix): Add installCheckPhase
Add an installCheckPhase that runs the appropriate substituteAll on
common.sh and runs the lang.sh tests with the build artifacts in the
PATH.

Change-Id: I2df5a93b8f3ffdfdc194a0e7d6b6669ef520c345
Reviewed-on: https://cl.tvl.fyi/c/depot/+/561
Reviewed-by: glittershark <grfn@gws.fyi>
2020-06-23 14:25:37 +00:00
Griffin Smith
b94d9f89dc fix(3p/nix): Make traces log, not dlog
We don't want traces compiled out since they're an actual language
feature that're used in userspace - also their absence is breaking the
tests

Change-Id: Icaefca8f52e94001785f724fdc0c10a7586b24e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/562
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: Kane York <rikingcoding@gmail.com>
Reviewed-by: lukegbot <bot@lukegb.com>
2020-06-23 12:37:48 +00:00
Luke Granger-Brown
5252623ca7 fix(3p/nix): fix readTree by fixing readDir by deleting some code
Change-Id: I4f44e6050fc5121072f4cde7758defe2dcbd4e92
Reviewed-on: https://cl.tvl.fyi/c/depot/+/552
Reviewed-by: tazjin <mail@tazj.in>
2020-06-22 01:12:53 +00:00
Luke Granger-Brown
4de63f784c fix(3p/nix): propagate dependencies which are needed by people linking in nix
This includes absl, which we install into the output, and boost and the boehm GC,
which are moved to propagated deps.

Change-Id: I8f9f9795ff92e26b2320359064241d7fd59c2d33
Reviewed-on: https://cl.tvl.fyi/c/depot/+/549
Reviewed-by: tazjin <mail@tazj.in>
2020-06-21 23:49:19 +00:00
Luke Granger-Brown
b99829f142 fix(3p/nix): install nix_config.h and config.nix
This also installs the rest of corepkgs as a side-effect.

Change-Id: I67a42d45793d5e8fdad51c1f306eebf63e9c2868
Reviewed-on: https://cl.tvl.fyi/c/depot/+/548
Reviewed-by: tazjin <mail@tazj.in>
2020-06-21 23:49:19 +00:00
Luke Granger-Brown
4ea8cc3730 fix(3p/nix): allow blank lines in nix.conf
Change-Id: I5ab040a62e123c57fe712b252fbf84fe5a8bc026
Reviewed-on: https://cl.tvl.fyi/c/depot/+/547
Reviewed-by: tazjin <mail@tazj.in>
2020-06-21 23:49:19 +00:00
Luke Granger-Brown
8b83e0f31b feat(3p/nix): install pkgconfig files for libexpr, libmain and libstore
Also fixes the pkgconfig files to use the corresponding CMake variables.

Change-Id: I8095b8aff39ad91e592f3edc95555c9f1f1f153d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/545
Reviewed-by: tazjin <mail@tazj.in>
2020-06-21 20:42:28 +00:00
Luke Granger-Brown
739a45f9e4 chore(3p/nix): use GNUInstallDirs to determine output directory names
This ensures that we install both glog's .a and all the .so files we
generate into a single consistent output lib path (which is, err,
lib64, but whatever).

Change-Id: Ib6ac6eacf5f56e4b719cfb586db731efc122c31b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/544
Reviewed-by: tazjin <mail@tazj.in>
2020-06-21 20:42:28 +00:00
Luke Granger-Brown
a54f155c07 chore(3p/nix): export library headers to include/ in output.
Change-Id: I87eb6e59782d720015d351d8829dc7b8688e01f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/543
Reviewed-by: tazjin <mail@tazj.in>
2020-06-21 20:42:28 +00:00
edef
48f08e8cc0 fix(3p/nix): passAsFile: leave out the hash prefix
Having a colon in the path may cause issues, and having the hash
function indicated isn't actually necessary. We now verify the path
format in the tests to prevent regressions.

(cherry picked from commit c65a6fa86aef7bdf51fb4fba7bd31d265619ba3f)
2020-05-31 19:20:24 +01:00
Puck Meerburg
7cc7fdacbe fix(3p/nix): passAsFile: hash the attribute name instead of numbering sequentially
This makes the paths consistent without relying on ordering.

Co-authored-by: edef <edef@edef.eu>
(cherry picked from commit 515c0a263e137a00e82f7d981284dbe54db23247)
2020-05-31 19:18:07 +01:00
Vincent Ambo
3d939a2f63 chore(3p/nix): Clean up leftover Meson file 2020-05-30 23:30:46 +01:00
Vincent Ambo
699ab4e20a chore(3p/nix): Clean up some unused defines 2020-05-30 23:17:48 +01:00
Vincent Ambo
91ddc7603d feat(3p/nix): Configure CMake to generate compile commands 2020-05-29 22:31:07 +01:00
Vincent Ambo
e494a347b5 fix(3p/nix): Build glog as included CMake project 2020-05-29 22:04:06 +01:00
Vincent Ambo
25393d8080 refactor(3p/nix): Introduce CMake as the build system for Nix
Completes the switch from Meson to CMake for the core build system in
Nix.

Meson was added originally because someone else had already done the
work for integrating it in Nix and it was an upgrade from the previous
setup.

However over time it became clear that Meson is not quite mature
enough for projects like Nix that have occasionally peculiar
configuration constraints.

Some issues encountered with Meson (some of these are due to the Meson
setup in Nix):

* Difficulty with generating correct compile_commands.json for
  external tools like clangd
* Difficulty linking to libc++ when using clang
* Ugly shell invocations for certain parts of the build system (I want
  these to be gone!!!)

This CMake setup mimics the Meson configuration, but there are some
differences (some temporary):

* headers are now included separately for each library (see a previous
  commit that changes includes appropriately)
* autoheaders-style configuration is currently hardcoded. Before
  blindly copying this I want to evaluate how much of it actually exists
  for portability concerns that I don't have (such as support for OS
  X).
* Nix is built with libc++ by default.
* [libstore] SQL schema is now inlined via a generated header, not an
  included string literal

Abseil is still built as part of this build, rather than an external
dependency, because it chokes on differently configured compiler
invocations.

Note that because of the move to libc++ an unwanted behaviour is
introduced: glog log messages no longer have a body. I have yet to
debug what is going on there.
2020-05-28 00:11:25 +01:00
Vincent Ambo
3d7537da7f chore(3p/nix): Remove meson build files for core packages 2020-05-27 22:00:00 +01:00
Vincent Ambo
abe5df74a8 chore(3p/nix): Move Abseil symlink one layer up
This is required for upcoming build system changes.
2020-05-27 21:57:28 +01:00
Vincent Ambo
3652326ed2 refactor(3p/nix): Anchor local includes at src/
Previously all includes were anchored in one global mess of header
files. This moves the includes into filesystem "namespaces" (if you
will) for each sub-package of Nix.

Note: This commit does not introduce the relevant build system changes.
2020-05-27 21:56:34 +01:00
Vincent Ambo
2e32ceb523 fix(3p/nix): Default-import depot from a relative path 2020-05-27 00:06:59 +01:00
Vincent Ambo
14b52848f8 docs(3p/nix): Add a README explaining the goals of the fork 2020-05-25 16:39:18 +01:00
Vincent Ambo
bf452cbc2a refactor(3p/nix): Replace tokenizeStrings with absl::StrSplit
This function was a custom (and inefficient in the case of
single-character delimiters) string splitter which was used all over
the codebase. Abseil provides an appropriate replacement function.
2020-05-25 15:54:14 +01:00
Vincent Ambo
b99b368d17 refactor(3p/nix/libutil): Replace hasPrefix/Suffix with Abseil
Uses the equivalent absl::StartsWith and absl::EndsWith functions
instead.
2020-05-25 02:19:01 +01:00
Vincent Ambo
8cf1322a6f fix(3p/nix/libstore): Fix error condition when parsing generations 2020-05-25 01:24:37 +01:00
Vincent Ambo
98299da0fd refactor(3p/nix/libutil): Replace string2Int & trim functions
Replaces these functions with corresponding functions from Abseil,
namely absl::StripAsciiWhitespace and absl::SimpleAtoi.

In the course of doing this some minor things I encountered along the
way were also refactored.

This also changes the signatures of the various custom readFile
functions to use absl::string_view types.
2020-05-25 01:19:02 +01:00
Vincent Ambo
b371821db5 chore(3p/nix/libstore): Remove progress bar remnants from download 2020-05-24 23:36:49 +01:00
Vincent Ambo
f5aaa12f1d style(3p/nix): Remove 'using std::*' from xml-writer.hh
See previous commit for more details on why.
2020-05-24 23:19:09 +01:00
Vincent Ambo
838f86b0fd style(3p/nix): Remove 'using std::*' from types.hh
It is considered bad form to use things from includes in headers, as
these directives propagate to everywhere else and can make it
confusing.

types.hh (which is includes almost literally everywhere) had some of
these directives, which this commit removes.
2020-05-24 22:29:21 +01:00
Vincent Ambo
f30b2e610d chore(3p/nix): Enable cross-file-rename in clangd 2020-05-24 18:19:28 +01:00
Vincent Ambo
da9104767e chore(3p/nix): Make build type configurable 2020-05-24 17:53:42 +01:00
Vincent Ambo
06d7b4aebd refactor(3p/nix/libutil): Replace chomp() with absl::strings 2020-05-24 02:13:07 +01:00
Vincent Ambo
10481d2586 chore(3p/nix): Remove some OS X specific defines
This project will be dropping OS X support until the core is simplified.
2020-05-24 02:13:07 +01:00
Alyssa Ross
bac38f3c49 fix(3p/nix): Fix long paths permanently breaking GC
Suppose I have a path /nix/store/[hash]-[name]/a/a/a/a/a/[...]/a,
long enough that everything after "/nix/store/" is longer than 4096
(MAX_PATH) bytes.

Nix will happily allow such a path to be inserted into the store,
because it doesn't look at all the nested structure.  It just cares
about the /nix/store/[hash]-[name] part.  But, when the path is deleted,
we encounter a problem.  Nix will move the path to /nix/store/trash, but
then when it's trying to recursively delete the trash directory, it will
at some point try to unlink
/nix/store/trash/[hash]-[name]/a/a/a/a/a/[...]/a.  This will fail,
because the path is too long.  After this has failed, any store deletion
operation will never work again, because Nix needs to delete the trash
directory before recreating it to move new things to it.  (I assume this
is because otherwise a path being deleted could already exist in the
trash, and then moving it would fail.)

This means that if I can trick somebody into just fetching a tarball
containing a path of the right length, they won't be able to delete
store paths or garbage collect ever again, until the offending path is
manually removed from /nix/store/trash.  (And even fixing this manually
is quite difficult if you don't understand the issue, because the
absolute path that Nix says it failed to remove is also too long for
rm(1).)

This patch fixes the issue by making Nix's recursive delete operation
use unlinkat(2).  This function takes a relative path and a directory
file descriptor.  We ensure that the relative path is always just the
name of the directory entry, and therefore its length will never exceed
255 bytes.  This means that it will never even come close to AX_PATH,
and Nix will therefore be able to handle removing arbitrarily deep
directory hierachies.

Since the directory file descriptor is used for recursion after being
used in readDirectory, I made a variant of readDirectory that takes an
already open directory stream, to avoid the directory being opened
multiple times.  As we have seen from this issue, the less we have to
interact with paths, the better, and so it's good to reuse file
descriptors where possible.

I left _deletePath as succeeding even if the parent directory doesn't
exist, even though that feels wrong to me, because without that early
return, the linux-sandbox test failed.

Reported-by: Alyssa Ross <hi@alyssa.is>
Thanks-to: Puck Meerburg <puck@puckipedia.com>
Tested-by: Puck Meerburg <puck@puckipedia.com>
Reviewed-by: Puck Meerburg <puck@puckipedia.com>
(cherry picked from commit c05e20daa1abb3446e378331697938b78af2b3d7)
2020-05-24 00:12:38 +01:00
Vincent Ambo
f77945ca81 style(3p/nix): Rename derivation to 'tazjix'
This makes it easier to distinguish which thing I'm dealing with in
the store paths. It does not affect anything else.
2020-05-23 23:47:45 +01:00
Vincent Ambo
cac00656cb fix(3p/nix): Set Meson build type to 'release'
Gotta go fast ... (well, not while compiling)
2020-05-23 20:39:02 +01:00
Vincent Ambo
af762abadc fix(3p/nix/libexpr): Ensure ExprOpUpdate merges into destination
... this fixes nixpkgs eval!
2020-05-23 20:30:23 +01:00
Vincent Ambo
55b1a47647 refactor(3p/nix/libexpr): Make other 'const' in Bindings::merge 2020-05-23 20:29:05 +01:00
Vincent Ambo
ab1fbd4c6e fix(3p/nix/libexpr): Actually use AttributeMap type alias
Without this alias, the garbage-collecting allocator won't be used and
allocated attribute set values won't be visible during GC.
2020-05-23 20:19:13 +01:00
Vincent Ambo
39e72c27f1 docs(3p/nix/libexpr): Add comment on ExprSelect 2020-05-23 16:01:52 +01:00
Vincent Ambo
bca0e34859 docs(3p/nix/libexpr): Add some comments about function calls
These were things that took me a moment to realise.
2020-05-23 06:44:38 +01:00
Vincent Ambo
d27c722e9e chore(3p/nix/tests): Remove leftover '__overrides' tests 2020-05-23 06:19:49 +01:00
Vincent Ambo
3b903193be fix(3p/nix/libexpr): Fix attrNames/attrValues builtins for btree_map
Replaces the previous implementations which performed sorting with one
that instead walks through the map (which is already sorted) and
yields values from it.

This fixes a handful of language tests because the previous
implementation did not actually yield useful values on the new implementation.
2020-05-23 06:18:20 +01:00
Vincent Ambo
811c42d255 fix(3p/nix/libexpr): Ensure symbols are compared by value 2020-05-23 06:18:20 +01:00
Vincent Ambo
85a05a6f38 fix(3p/nix/libexpr): Remove the global empty attribute set
In the change to the backing structure of attribute sets, the
requirement to manually balance the capacity of the structure went
away.

This is a) because Abseil's data structures manage this on their own,
and b) because the new Bindings class is allocated using `new (GC)`
rather than writing into a predefined memory area.

As part of this change functions related to the capacity were
deprecated and set to 0 values, which in turn caused the creation of
new attribute sets to return the same (mutable!) default value in
various cases, leading to "side effects" that caused evaluation
failures.

FWIW, I'm not sure if this optimisation had noticeable performance
impact, but while untangling libexpr it definitely doesn't help trying
to follow what it's doing - so bye, bye!
2020-05-23 06:18:20 +01:00
Vincent Ambo
da4ca4e02f chore(3p/nix/libexpr): Minor readability improvements in eval/value 2020-05-23 06:18:20 +01:00
Vincent Ambo
92792264f7 chore(3p/nix/libexpr): Remove unused __overrides feature
This feature does not appear in nixpkgs, so I don't care about it. My
only goal is evaluating nixpkgs.
2020-05-23 01:09:35 +01:00
Vincent Ambo
6b447f4b25 chore(3p/nix/libexpr): Expose separate insert & "upsert" methods
Reading more through the old code, it seems like the intention
/sometimes/ is to replace values.
2020-05-23 00:52:20 +01:00
Vincent Ambo
8c28be1b69 fix(3p/nix/libexpr): Use gc_allocator<T> as the btree_map allocator
This will make all Attr values visible to the GC.
2020-05-22 23:43:44 +01:00
Vincent Ambo
5f728cb428 feat(3p/nix): Add a derivation to launch clangd
This wrapper derivation (which assumes that the depot is available at
~/depot) can be used to actually get clangd working with
//third_party/nix.

In my setup I can launch this with M-x eglot, followed by

  env
    CLANGD_FLAGS='--compile-commands-dir=/home/tazjin/projects/nix-build'
    nix-shell -A third_party.nix --run 'nix-clangd' /home/tazjin/depot
2020-05-22 23:00:23 +01:00
Vincent Ambo
c25281820f fix(3p/nix/libexpr): Do not allow duplicate attribute insertion
This is closer to bug-for-bug compatibility with the previous version,
which would put new elements at the end of the array and (due to the
linear scan) return previous ones.
2020-05-22 20:30:22 +01:00
Vincent Ambo
f459332f32 chore: Update from Clang 9 to Clang 10 for all projects 2020-05-22 18:29:47 +01:00
Vincent Ambo
618f9a6f07 refactor(3p/nix/libexpr): Use gc_cpp to allocate Value instances 2020-05-22 16:57:48 +01:00
Vincent Ambo
b3c9166b23 refactor(3p/nix/libexpr): state->allocBindings -> Bindings::NewGC
EvalState::allocBindings had little to do with Bindings, other than
returning them, and didn't belong in that class.
2020-05-22 16:57:36 +01:00
Vincent Ambo
e24466c795 fix(3p/nix/libexpr): Make new Bindings class visible to GC 2020-05-22 16:35:21 +01:00
Vincent Ambo
68e6e92a20 chore(3p/nix/libexpr): Delete Bindings::sort
This function does nothing anymore since the attributes are always
in-order.
2020-05-22 03:35:07 +01:00
Vincent Ambo
986a8f6b75 fix(3p/nix): Update for usage of new attribute set API
The new attribute set API uses the iterators of the btree_map
directly. This requires changes in various files because the internals
of libexpr are very entangled.

This code runs and compiles, but there is a bug causing empty
attribute sets to be assigned incorrectly.
2020-05-22 01:59:39 +01:00
Vincent Ambo
42205f27fc refactor(3p/nix/libexpr): Use absl::btree_map::iterator type
Instead of using a custom Args* iterator, use the one belonging to the
map type directly.
2020-05-22 01:59:36 +01:00
Vincent Ambo
ee4637e3a2 refactor(3p/nix/libexpr): Use absl::btree_map::merge for '//'
Instead of doing some sort of inline merge-sort of the two attribute
sets, use the attribute sets merge function.

This commit alone does not build and is not supposed to.
2020-05-22 01:52:03 +01:00
Vincent Ambo
28e347effe refactor(3p/nix/libexpr): Use absl::btree_map for AttrSets
This is the first step towards replacing the implementation of
attribute sets with an absl::btree_map.

Currently many access are done using array offsets and pointer
arithmetic, so this change is currently causing Nix to fail in various
ways.
2020-05-21 19:21:55 +01:00
Vincent Ambo
a162f4e825 refactor(3p/nix/libexpr): Use std::string as qualified type
Replaces most uses of `string` with `std::string`.

This came up because I removed the "types.hh" import from
"symbol-table.hh", which percolated through a bunch of files where
`string` was suddenly no longer defined ... *sigh*
2020-05-21 05:43:22 +01:00
Vincent Ambo
b97307056d docs(3p/nix/libexpr): Add clarifying comments to SymbolTable 2020-05-21 05:42:20 +01:00
Vincent Ambo
c395a48be2 fix(3p/nix): Compatibility with updated SymbolTable
The functions in SymbolTable have been renamed to match the Google
Style guide, and some debug-only functions have been removed.
2020-05-21 05:09:23 +01:00
Vincent Ambo
97e85f94e5 refactor(3p/nix/libexpr): Use absl::node_hash_set in SymbolTable
This replaces the previous use of std::unordered_set with
absl::node_hash_set.

This type was chosen because the current implementation requires
pointer stability.

This does not yet touch the 'Attr' struct.

As a bonus, the implementation of the SymbolTable struct is now
consolidated into a single header/implementation file pair.
2020-05-21 05:09:23 +01:00
Vincent Ambo
00017ace04 feat(3p/nix): Wrangle Meson/Nix/CMake into (temporary) submission
Meson is unable to use CMake in Nix to determine the internal
structure of the Abseil libraries.

This commit adds an explicit list of most of the Abseil targets that
are relevant (so far) and bundles them into a list that is linked
together.
2020-05-21 04:50:49 +01:00
Vincent Ambo
a3b48e6562 chore(3p/nix): Minor fixes to get rid of warnings 2020-05-21 00:24:12 +01:00
Vincent Ambo
9dfdf16a8d fix(3p/nix): Fix build of derivation if cmake is present
cmake automatically runs a configure hook which breaks the build,
since this isn't actually a cmake project. This hook is now disabled.

Additionally Abseil's sources are linked to an absolute derivation
path when the build launches, as opposed to the relative path used for
development builds.
2020-05-21 00:24:10 +01:00
Vincent Ambo
43677021e3 refactor(3p/nix): Apply clang-tidy's performance-* fixes
This applies the performance fixes listed here:

https://clang.llvm.org/extra/clang-tidy/checks/list.html
2020-05-20 22:58:43 +01:00
Vincent Ambo
689ef502f5 refactor(3p/nix): Apply clang-tidy's readability-* fixes
This applies the readability fixes listed here:

https://clang.llvm.org/extra/clang-tidy/checks/list.html
2020-05-20 22:27:37 +01:00
Vincent Ambo
d331d3a0b5 refactor(3p/nix): Apply clang-tidy's modernize-* fixes
This applies the modernization fixes listed here:

https://clang.llvm.org/extra/clang-tidy/checks/list.html

The 'modernize-use-trailing-return-type' fix was excluded due to my
personal preference (more specifically, I think the 'auto' keyword is
misleading in that position).
2020-05-20 04:33:07 +01:00
Vincent Ambo
fed31b2c9b feat(3p/nix): Add some initial Abseil build targets
These make it possible to link to Abseil strings.
2020-05-20 04:02:23 +01:00
Vincent Ambo
e2fad0f0d2 feat(3p/nix): Add Abseil as a CMAKE subproject to Meson
Yep.

This is accomplished by symlinking the sources into the location
expected by Meson for subprojects.
2020-05-20 03:03:08 +01:00