Moves a function that is not dependent on the generated code over to
parser.hh. This function also looks like it could be improved, but
that is left as an exercise for the reader.
Code that remains in lexer.l has been reformatted, while we're here.
Change-Id: I9c26bb4eed0772a720d0715029e8bc10ab16ac38
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1279
Tested-by: BuildkiteCI
Reviewed-by: Kane York <rikingcoding@gmail.com>
This moves the language test suite into Googletest by constructing
parameterised tests out of the same language snippets used for the
previous lang.sh evaluation.
So far this includes support for about 3/4 of all tests, specifically:
* all parser success/failure tests
* all evaluator failure tests
The evaluator success tests will be implemented in a subsequent commit,
because the output comparison contains a whole bunch of additional
logic that I did not want to cram in here.
Change-Id: Icec9f368366cdbaa53b4c7e4472b8b6e8dd72eba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1278
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: isomer <isomer@tvl.fyi>
Tested-by: BuildkiteCI
Unfortunately, to guarantee correct behaviour of some evaluation (!)
tests, addToStore needs to actually check whether passed in source
files exist and fail appropriately.
There is a chance that the dependency on this behaviour is actually a
bug in the upstream test suite, but my attempts at finding out more
about this from, say, the git history have so far been unsuccessful.
Change-Id: I311999ea28fcedf5da13a4e627b1c1c8e4e59cbd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1276
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
These files will be integrated into the evaluator unit tests instead
of running separately via a shell script.
Change-Id: I1d229e73b1d862777f5108c86891689900edefbe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1275
Tested-by: BuildkiteCI
Reviewed-by: Kane York <rikingcoding@gmail.com>
Reviewed-by: isomer <isomer@tvl.fyi>
These options only apply to nix-shell. The fact that nix-build
previously accepted them was an accident that resulted from the two
programs sharing an implementation.
Change-Id: I0047c98e2096010797316bff3ea4faf722fab86a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1273
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
I missed calling store_->assertStorePath on paths in several daemon
proto handlers, which the previous implementation did.
Change-Id: Ifad6eeb03b5a5babec7b4bcf7aca060813f15bb7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1272
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
I got the message proto wrong on this one as well - it needs both a path
and a signatures.
Change-Id: I9a489b1285bda61c15b2a3b47d9cfc3b50e387da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1270
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Implement the proto handler on the server side for
Worker::BuildDerivation. This includes several additions to the proto
which I had missed on the first pass, including the actual proto
definition for the Derivation itself and a few sequence number
reorderings which are fine because this is all provisional and not
deployed yet.
A couple things to note
- I implemented a couple constructors for nix classes that initialize
themselves based on their proto variants, which felt nice and didn't
end up causing any issues.
- I've made the conversions between the enum types in nix and in proto
explicit via switch statements rather than using a static_cast, out of
an abundance of caution that the error would get mismatched in the
future and we'd convert the wrong thing to the wrong thing - this is
verbose, but exceptionally future proof.
Change-Id: Iecf6b88e76bc37e49efa05fd65d6cd0cb0deffed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1249
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: Kane York <rikingcoding@gmail.com>
This silences a -Winconsistent-missing-override
Change-Id: I2492a8ab3c0b0a33a794f012bfee6bbd0c2f0a8e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1271
Reviewed-by: Kane York <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
Interpolating a path into a string will copy the referenced path into
the Nix store, so this got a dependency on all of src. By first
constructing a path to the src/proto directory using the + operator,
and then interpolating it, we limit what is copied to the store, and
therefore what code triggers a nix-proto-srcs rebuild.
Change-Id: I8dd750f6bc5902b74ffb56470bc8a5f2c01c8cf1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1263
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
This store implementation is required in all unit tests that use the
evaluator.
Change-Id: I1cfe8cecab8722cd66dc803747821a2be2b2619f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1269
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
This change does away with the previous special-casing of lists of
certain element sizes, and the use of raw C-style arrays.
Lists are now backed by a std::vector of nix::Value*, which uses the
traceable GC allocator.
This change is unfortunately quite noisy because the accessor methods
were updated/removed accordingly, so all callsites of Nix-related
lists have changed.
For some operations in primops.cc where keeping the previous code
structure would have been more difficult with a "proper" vector, the
implementation has been replaced with std::vector methods. For
example, list concatenation now uses appropriate range inserts.
Anecdotally the performance of this is about equal, to even slightly
better, than the previous implementation.
All language tests pass and the depot paths I've used for testing
still evaluate.
Change-Id: Ib5eca6c0207429cb323a330c838c3a2200b2c693
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1266
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: Kane York <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
This has several advantages:
* we can ensure that the vector is traced by the GC
* we don't need to unsafely allocate memory to make an Env
Note that there was previously a check about the size of the
environment, but it's unclear why this was the case (git history
yielded nothing interesting) and it seems to have no effect.
Change-Id: I4998b879a728a6fb68e1bd187c521e2304e5047e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1265
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: Kane York <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
all of the executables that get built during regular development depend
on this being set to a directory that contains the nix directory -
previously we had been doing it manually every time, this automates it.
Change-Id: I4c957c0abf0a92ca7122a47d3b141a8ede280e13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1258
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Configures the CMake build to load & run the GoogleTest tests.
I (grfn) also updated this to get the tests running as part of the nix
derivation, which required defining our own manual configurePhase and
installCheckPhase, rather than depending on the one provided by stdenv.
Not doing this would cause cmake to attempt to *run* the tests as part
of the buildPhase, which wouldn't work because the dynamic libraries
hadn't been put into a place where the test executables knew where to
find them. We're not sure *why* this fixes it, and for some reason
fixing this also breaks the automatic behavior of nixpkgs of passing
-j$NIX_BUILD_CORES -l$NIX_BUILD_CORES to make, but that's eaasy enough
to fix manually in a preBuild
Paired-With: Griffin Smith <grfn@gws.fyi>
Change-Id: I79d61854a3ff47301cdce8a40c76820a97bdf901
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1240
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Also adds the missing check_contents field to the VerifyStoreRequest
proto message, since it was missed in the original pass. This is done
using a renumbering, which is fine in this case since the proto hasn't
been deployed yet
Change-Id: I92bf4e48a71a25ae02ae02b3deaf6e7c71fe5da7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1237
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: Kane York <rikingcoding@gmail.com>
Change-Id: I8b1d84799a608a516d0b4980022a7edd545a1ca1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1235
Tested-by: BuildkiteCI
Reviewed-by: Kane York <rikingcoding@gmail.com>
Change-Id: I243d67b0bce29d54c7d6e08f5eee70bd395cf9a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1234
Tested-by: BuildkiteCI
Reviewed-by: Kane York <rikingcoding@gmail.com>
Backported from:
b3e5eea4a9fcd048a526
Intentionally skipped because we have not backported the JSON changes:
9f46f54de4
Did not apply changes ni primops.cc, because those look suspect and
are also based on something that we don't have in our tree.
Change-Id: I837787ce9f2c90267bc39fce15177980d209d4e9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1253
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
We want to *trace* the 'Value *' arrays, not garbage-collect them!
Otherwise the vectors/maps can end up pointing to nowhere.
Backported from:
10e17eaa58
Change-Id: I30dc94caa80c9d982e7a14bc67ba2d065e8203aa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1252
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Previously the memory would occasionally be collected during eval since
the GC doesn't consider the member variable as alive / doesn't scan the
region of memory where the pointer lives.
By using the traceable_allocator<T> allocator provided by Boehm GC we
can ensure the memory isn't collected. It should be properly freed when
SourceExprCommand goes out of scope.
Backported from:
d2c371927e
Change-Id: I1f7c745dbc66c7164bee50f4d9b0d437dbc7dd51
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1251
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
This is a blocker to enabling the linter.
Change-Id: I9f8d3cc19c7539086f53474a505362230fc56c04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1245
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
This is a bad idea, it shouldn't exist, nixpkgs doesn't use it.
Change-Id: Ic4d1b936d8f059d5c40f0567af165b02427d7e36
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1241
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
These are the queries that are handled in the confusing case statement
in the old daemon implementation, because they have very similar
structure.
Change-Id: Ie7143354f66cef4336dff8072ede9a56271a7e89
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1228
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
What it does I will never know. It takes a path and returns a path.
Something happens in the middle.
Change-Id: I499a9df700e5b954c9aaf6d694753ff34e773dfd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1210
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
These files were always implicitly depended upon by worker-protocol.hh, but just so happened also always be included in places where the other ones were already present.
We are likely getting rid of this file sooner rather than later, but in
the meantime this will cause clangd to shut up while I'm editing the
file.
Change-Id: I0d765f8b86828b6612e5483417e452221ea3c5b1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1207
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
Add QueryValidPaths and QuerySubstitutablePaths, both of which filter a
list of paths based on a set of criteria.
Change-Id: I6aa4647efe82b82dc9582a311643d5f9b6d521d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1203
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Replaces the previous uses of the (ordered!) std::map and std::set
with absl::flat_hash_{map|set}.
After some careful reading it seems that there is actually no
ordering dependency on these types, and the (drop-in) replacements
perform slightly better.
Overall this is not fixing a bottleneck, just a driveby thing.
Change-Id: Ided695dc75676bd58515aa9382df0be0a09c565e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1220
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: isomer <isomer@tvl.fyi>
Tested-by: BuildkiteCI
Several definitions of functions declared in eval.hh were previously
implemented in parser.y, this moves them over to parser.cc.
While this still isn't a reasonable place to keep them, the long-term
fix is more likely to be that eval.hh needs to be split up.
Before we get to that point however, this already gives us the ability
to use tooling with this code.
Change-Id: If06fb655325fe281564047ffab0a0a640428a0ee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1219
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Moves several of the static helper functions into a new parser.cc
file.
Once the rest of the code is usefully extracted, these will be moved
to a private namespace.
Change-Id: I0d7b53dcefe31bb5c6bad3ad7f5fcb48276bf799
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1218
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
First step (of many?) towards extracting all the inline code from the
Yacc file and keeping it somewhere more accessible instead.
Note that none of this code has previously been touched by a linter or
formatter, pretty much ever, so as it is extracted it also undergoes
similar changes to the whole codebase after the initial fork.
Change-Id: If3b7181f22e3b3fd8c58dfa9befa7ee2896ea06d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1217
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
