Commit graph

389 commits

Author SHA1 Message Date
Griffin Smith
785ca36316 refactor(3p/nix): Remove unused insert_or_assign
This function in never called, so let's just remove it

Paired-With: Luke Granger-Brown <git@lukegb.com>
Paired-With: Vincent Ambo <mail@tazj.in>
Change-Id: I79125866254d90dd0842bc86830d2103ac313cb6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1125
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-13 23:50:24 +00:00
Griffin Smith
d5597b4784 feat(3p/nix): Statically pass bindings capacity where possible
To aid in making the decision of where to (currently just statically)
use a vector or btree as the backing implementation, add an extra
constructor argument to Bindings::NewGC for a capacity, and use
a (currently hardcoded at 32, for no good reason other than it felt like
a reasonable number) pivot to switch between our possible backing
implementations. Then, update all the call sites where it feels
reasonable that we know the capacity statically to *pass* that capacity
to the constructor.

Paired-With: Luke Granger-Brown <git@lukegb.com>
Paired-With: Vincent Ambo <mail@tazj.in>
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: I1858c161301a1cd0e83aeeb9a58839378869e71d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1124
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-13 23:50:24 +00:00
Griffin Smith
d5505fcff9 feat(3p/nix): Add vector-backed impl for Bindings
Add an alternative impl of the now-abstract Bindings base class that is
backed by a std::vector, somewhat similar but stylistically a little
superior to the array-backed implementation in upstream nix. The
underlying iterator type in BindingsIterator is now backed by a
std::variant that we std::visit an overload over in order to implement
the various bits of the iterator interface.

Paired-With: Luke Granger-Brown <git@lukegb.com>
Paired-With: Vincent Ambo <mail@tazj.in>
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: I7fbd1f4d5c449e2f9b82102a701b0bacd5e80672
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1123
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-13 23:50:24 +00:00
Griffin Smith
98148e6711 refactor(3p/nix): Abstract away concrete bindings impl
To pave the way for the thing we want to do eventually which is use a
linear-time array for bindings (aka attribute sets) that are statically
known to be small enough to get a performance benefit from doing so,
make the Bindings class abstract, and define a BTreeBindings class that
inherits from it and is (currently always) returned from the static
initializer. The idea is that we'll have an ArrayBindings class as well
later that we can dispatch to conditionally based on an optional
"capacity" parameter or something like that.

There was some difficulty here in getting the iterator to work - the
approach we settled on ended up making a concrete BindingsIterator class
which will wrap a std::variant of either a btree iterator or something
else later, but right now just wraps a btree iterator.

Paired-With: Luke Granger-Brown <git@lukegb.com>
Paired-With: Vincent Ambo <mail@tazj.in>
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: Ie02ca5a1c55e8ebf99ab1e957110bd9284278907
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1121
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-13 23:50:24 +00:00
Vincent Ambo
ae16a8e9cf docs(3p/nix): Minor semantic correction to a comment
Change-Id: I1cfbb7e933da54198115b28ac509b0d04870fd8f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1127
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-13 21:09:36 +00:00
Vincent Ambo
fa161e9a38 refactor(3p/nix/libexpr): Remove the nix::Symbol default constructor
Having a default constructor for this causes a variety of annoying
situations across the codebase in which this is initialised to an
unexpected value, leading to constant guarding against those
conditions.

It turns out there's actually no intrinsic reason that this default
constructor needs to exist. The biggest one was addressed in CL/1138
and this commit cleans up the remaining bits.

Change-Id: I4a847f50bc90e72f028598196592a7d8730a4e01
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1139
Reviewed-by: isomer <isomer@tvl.fyi>
Tested-by: BuildkiteCI
2020-07-13 21:06:43 +00:00
isomer
afd1367337 fix(nix-channel): Actually strip whitespace.
This has been providing a warning and it's been bothering me.

Change-Id: I0548059950ec4250d7cf0938f9deae09eafe593c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1141
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: isomer <isomer@tvl.fyi>
Tested-by: BuildkiteCI
2020-07-13 20:18:26 +00:00
Vincent Ambo
f3165f48aa refactor(3p/nix/libexpr): Make nix::AttrName a std::variant
nix:AttrName was one of the few classes that relied on the default
constructor of nix::Symbol (which I am trying to remove in a separate
change).

The class essentially represents the name of an attribute in a set,
which is either just a string expression or a dynamically evaluated
expression (e.g. string interpolation).

Previously it would be constructed by only setting one of the fields
and defaulting the other, now it is an explicit std::variant.

Note that there are several code paths where not all eventualities are
handled and this code is bug-for-bug compatible with those, except
that unknown conditions (which should never work) are now throwing
instead of silently doing ... something.

The language tests pass with this change, and the depot derivations
that I tested with evaluated successfully.

Change-Id: Icf1ee60a5f8308f4ab18a82749e00cf37a938a8f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1138
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-13 20:14:49 +00:00
Vincent Ambo
693661fe87 feat(3p/nix/libutil): Add util::overloaded class for std::visitor
Implements the fairly common lambda overload class used for std::visit
over variants and other things that require groups of callables.

Change-Id: Ia7448b7e1bd349b4909974758e6e6303a80d86d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1137
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: edef <edef@edef.eu>
2020-07-13 20:14:49 +00:00
Eelco Dolstra
5e2151b521 fix(3p/nix/repl): Scan NixRepl for GC roots
Backport of b244e65cdb

Fixes https://github.com/NixOS/nix/issues/3175

------

Original fault description by puck:

I suspect I found the issue: The Nix Command structs are stored on the
heap (using ref<T>, a wrapper around std::shared_ptr<T>), which means
that any pointers that the NixRepl struct contains are eligible to be
reaped by the GC. This includes, but is not limited to, the Env
pointer, which seems to cause most of the random segfaults, or random
other values in the environment, which seems to be what @arianvp
experienced too.

Change-Id: I376d7cfd432daaa6f1fbbf77788ff048082f34e5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1001
Tested-by: BuildkiteCI
Reviewed-by: Kane York <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-10 22:25:30 +00:00
Andreas Rammhold
1937bc8624 fix(3p/nix/libstore): use exception instead of assertion when storepath is malformed
Previously the nix-daemon would crash if a user fed it invalid store
paths for drv files. The crash was due to the changed assertion
triggering. Whenever that assertion would hit the nix-daemon process
along with all it's current childs (running builds from all users) would
be interrupted.

Before this patch:

 $ nix-store --realise /nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.drv
 don't know how to build these paths:
   /nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.drv
 error: unexpected end-of-file

 < nix-daemon terminates >

With this patch:

 $ nix-store --realise /nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.drv
 don't know how to build these paths:
   /nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.drv
 error: path '/nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.drv' is not a valid store path

 < nix-daemon does *NOT* terminate >

Change-Id: I01c5048c8a43a8b9154bdeb781d05b7744869ec0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/981
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-07-10 19:41:52 +00:00
Vincent Ambo
b485203b8c feat(3p/nix): Add additional worker protocol actions to proto
This takes us about halfway through worker-protocol.hh

I have left out the documentation strings for some of these items
because I don't feel that I can currently write an unambigous
description of them. For now I am just attempting to match the types.

Change-Id: Iae64b1676152fe4ea069e2021b75ad76465cf368
Reviewed-on: https://cl.tvl.fyi/c/depot/+/960
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-10 17:12:39 +00:00
Vincent Ambo
93eb10566a docs(3p/nix): Update links & other things in README
Change-Id: I1a111c11a83f0506e0973615848cc9c0ef4765e2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/959
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-07-09 22:53:28 +00:00
Vincent Ambo
8f4dea27a6 chore(3p/nix/libstore): Link to nixproto library
This isn't actually used yet, but forces the protos to be included in
the build which is useful for iteration.

Change-Id: I2abcaf297f34ae741f00ad0c929b226d5603c9d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/928
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-05 19:17:12 +00:00
Vincent Ambo
0064e55486 feat(3p/nix): Add protobuf & gRPC dependencies and generate sources
Adds dependencies on the gRPC & protobuf libraries, and implements Nix
code to generate the C++ sources from the included proto definitions.

This is theoretically supported via CMake, but practically doesn't
work and I don't care to debug why.

Doing it like this lets us instead add a CMake library target for our
proto definitions based on the sources generated by Nix.

Pros:
  * no need to deal with the gRPC CMake mess
  * it works!

Cons: * iteration requires nix-shell restart
Change-Id: Ie1fe9807fc96c49cb8f7161ba59d093456062b15
Reviewed-on: https://cl.tvl.fyi/c/depot/+/927
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-05 19:17:12 +00:00
Vincent Ambo
a6da980a0b feat(3p/nix): Add gRPC definitions for worker protocol
Adds initial gRPC definitions for the Nix worker protocol, which is
currently defined messily across the following files:

    src/libstore/worker-protocol.hh
    src/libstore/remote-store.cc
    src/nix-daemon/nix-daemon.cc

The protocol definition is basically a big enum with the signatures of
the calls being implicit in the various client/server implementation
functions.

The definitions in this file are slowly reversed from these implicit
signatures, and are likely to contain an error or two which will be
weeded out when this is taken into use.

Only a handful of the calls are included in this commit, it is
intended to get us up and running first.

Change-Id: Ibc9b2ab4b91a064c8935f09f7ac72bb8150fb476
Reviewed-on: https://cl.tvl.fyi/c/depot/+/926
Reviewed-by: isomer <isomer@tvl.fyi>
Tested-by: BuildkiteCI
2020-07-05 19:17:12 +00:00
Vincent Ambo
7e8a48c82c fix(3p/nix): Link to glog built by Nix
Same as cl/921, it seems that the trick to making this work is indeed
overriding the C++ standard used in the dependency.

Change-Id: I3c5984d71014d774c161ecc283844f504fd44719
Reviewed-on: https://cl.tvl.fyi/c/depot/+/922
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-07-05 16:54:12 +00:00
Vincent Ambo
5cb6e6ee21 fix(3p/nix): Link to Abseil built by Nix
This didn't work previously ... but now it does. I think setting the
standard explicitly is what did the trick, but it's slightly unclear
to me why.

Either way this means that Abseil is no longer constantly getting
recompiled when building Nix, which is nice.

Change-Id: I377f7b68bf1ef9045df6a2eee8fdd0c92f243547
Reviewed-on: https://cl.tvl.fyi/c/depot/+/921
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-07-05 16:54:12 +00:00
Vincent Ambo
4455f28e42 chore(3p/nix/libexpr): Remove dead code from attribute sets
These bits are no longer required with the hashmap-backed
implementation of attribute sets.

Change-Id: I8b936d8d438a00bad4ccf8e0b4dd719c559ce8c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/912
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2020-07-04 03:12:16 +00:00
glittershark
ec84fde6b1 fix(3p/nix): Revert "feat(3p/nix): Add installCheckPhase"
This reverts commit 368e8d1edd.

Reason for revert: Didn't mean to submit, and the phase is currently failing (which breaks the otherwise-functional derivation)

Change-Id: I515b2fb45188dc90f09ae2458453192487c74d71
Reviewed-on: https://cl.tvl.fyi/c/depot/+/581
Reviewed-by: tazjin <mail@tazj.in>
2020-06-23 15:23:04 +00:00
Griffin Smith
368e8d1edd feat(3p/nix): Add installCheckPhase
Add an installCheckPhase that runs the appropriate substituteAll on
common.sh and runs the lang.sh tests with the build artifacts in the
PATH.

Change-Id: I2df5a93b8f3ffdfdc194a0e7d6b6669ef520c345
Reviewed-on: https://cl.tvl.fyi/c/depot/+/561
Reviewed-by: glittershark <grfn@gws.fyi>
2020-06-23 14:25:37 +00:00
Griffin Smith
b94d9f89dc fix(3p/nix): Make traces log, not dlog
We don't want traces compiled out since they're an actual language
feature that're used in userspace - also their absence is breaking the
tests

Change-Id: Icaefca8f52e94001785f724fdc0c10a7586b24e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/562
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: Kane York <rikingcoding@gmail.com>
Reviewed-by: lukegbot <bot@lukegb.com>
2020-06-23 12:37:48 +00:00
Luke Granger-Brown
5252623ca7 fix(3p/nix): fix readTree by fixing readDir by deleting some code
Change-Id: I4f44e6050fc5121072f4cde7758defe2dcbd4e92
Reviewed-on: https://cl.tvl.fyi/c/depot/+/552
Reviewed-by: tazjin <mail@tazj.in>
2020-06-22 01:12:53 +00:00
Luke Granger-Brown
4de63f784c fix(3p/nix): propagate dependencies which are needed by people linking in nix
This includes absl, which we install into the output, and boost and the boehm GC,
which are moved to propagated deps.

Change-Id: I8f9f9795ff92e26b2320359064241d7fd59c2d33
Reviewed-on: https://cl.tvl.fyi/c/depot/+/549
Reviewed-by: tazjin <mail@tazj.in>
2020-06-21 23:49:19 +00:00
Luke Granger-Brown
b99829f142 fix(3p/nix): install nix_config.h and config.nix
This also installs the rest of corepkgs as a side-effect.

Change-Id: I67a42d45793d5e8fdad51c1f306eebf63e9c2868
Reviewed-on: https://cl.tvl.fyi/c/depot/+/548
Reviewed-by: tazjin <mail@tazj.in>
2020-06-21 23:49:19 +00:00
Luke Granger-Brown
4ea8cc3730 fix(3p/nix): allow blank lines in nix.conf
Change-Id: I5ab040a62e123c57fe712b252fbf84fe5a8bc026
Reviewed-on: https://cl.tvl.fyi/c/depot/+/547
Reviewed-by: tazjin <mail@tazj.in>
2020-06-21 23:49:19 +00:00
Luke Granger-Brown
8b83e0f31b feat(3p/nix): install pkgconfig files for libexpr, libmain and libstore
Also fixes the pkgconfig files to use the corresponding CMake variables.

Change-Id: I8095b8aff39ad91e592f3edc95555c9f1f1f153d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/545
Reviewed-by: tazjin <mail@tazj.in>
2020-06-21 20:42:28 +00:00
Luke Granger-Brown
739a45f9e4 chore(3p/nix): use GNUInstallDirs to determine output directory names
This ensures that we install both glog's .a and all the .so files we
generate into a single consistent output lib path (which is, err,
lib64, but whatever).

Change-Id: Ib6ac6eacf5f56e4b719cfb586db731efc122c31b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/544
Reviewed-by: tazjin <mail@tazj.in>
2020-06-21 20:42:28 +00:00
Luke Granger-Brown
a54f155c07 chore(3p/nix): export library headers to include/ in output.
Change-Id: I87eb6e59782d720015d351d8829dc7b8688e01f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/543
Reviewed-by: tazjin <mail@tazj.in>
2020-06-21 20:42:28 +00:00
edef
48f08e8cc0 fix(3p/nix): passAsFile: leave out the hash prefix
Having a colon in the path may cause issues, and having the hash
function indicated isn't actually necessary. We now verify the path
format in the tests to prevent regressions.

(cherry picked from commit c65a6fa86aef7bdf51fb4fba7bd31d265619ba3f)
2020-05-31 19:20:24 +01:00
Puck Meerburg
7cc7fdacbe fix(3p/nix): passAsFile: hash the attribute name instead of numbering sequentially
This makes the paths consistent without relying on ordering.

Co-authored-by: edef <edef@edef.eu>
(cherry picked from commit 515c0a263e137a00e82f7d981284dbe54db23247)
2020-05-31 19:18:07 +01:00
Vincent Ambo
3d939a2f63 chore(3p/nix): Clean up leftover Meson file 2020-05-30 23:30:46 +01:00
Vincent Ambo
699ab4e20a chore(3p/nix): Clean up some unused defines 2020-05-30 23:17:48 +01:00
Vincent Ambo
91ddc7603d feat(3p/nix): Configure CMake to generate compile commands 2020-05-29 22:31:07 +01:00
Vincent Ambo
e494a347b5 fix(3p/nix): Build glog as included CMake project 2020-05-29 22:04:06 +01:00
Vincent Ambo
25393d8080 refactor(3p/nix): Introduce CMake as the build system for Nix
Completes the switch from Meson to CMake for the core build system in
Nix.

Meson was added originally because someone else had already done the
work for integrating it in Nix and it was an upgrade from the previous
setup.

However over time it became clear that Meson is not quite mature
enough for projects like Nix that have occasionally peculiar
configuration constraints.

Some issues encountered with Meson (some of these are due to the Meson
setup in Nix):

* Difficulty with generating correct compile_commands.json for
  external tools like clangd
* Difficulty linking to libc++ when using clang
* Ugly shell invocations for certain parts of the build system (I want
  these to be gone!!!)

This CMake setup mimics the Meson configuration, but there are some
differences (some temporary):

* headers are now included separately for each library (see a previous
  commit that changes includes appropriately)
* autoheaders-style configuration is currently hardcoded. Before
  blindly copying this I want to evaluate how much of it actually exists
  for portability concerns that I don't have (such as support for OS
  X).
* Nix is built with libc++ by default.
* [libstore] SQL schema is now inlined via a generated header, not an
  included string literal

Abseil is still built as part of this build, rather than an external
dependency, because it chokes on differently configured compiler
invocations.

Note that because of the move to libc++ an unwanted behaviour is
introduced: glog log messages no longer have a body. I have yet to
debug what is going on there.
2020-05-28 00:11:25 +01:00
Vincent Ambo
3d7537da7f chore(3p/nix): Remove meson build files for core packages 2020-05-27 22:00:00 +01:00
Vincent Ambo
abe5df74a8 chore(3p/nix): Move Abseil symlink one layer up
This is required for upcoming build system changes.
2020-05-27 21:57:28 +01:00
Vincent Ambo
3652326ed2 refactor(3p/nix): Anchor local includes at src/
Previously all includes were anchored in one global mess of header
files. This moves the includes into filesystem "namespaces" (if you
will) for each sub-package of Nix.

Note: This commit does not introduce the relevant build system changes.
2020-05-27 21:56:34 +01:00
Vincent Ambo
2e32ceb523 fix(3p/nix): Default-import depot from a relative path 2020-05-27 00:06:59 +01:00
Vincent Ambo
14b52848f8 docs(3p/nix): Add a README explaining the goals of the fork 2020-05-25 16:39:18 +01:00
Vincent Ambo
bf452cbc2a refactor(3p/nix): Replace tokenizeStrings with absl::StrSplit
This function was a custom (and inefficient in the case of
single-character delimiters) string splitter which was used all over
the codebase. Abseil provides an appropriate replacement function.
2020-05-25 15:54:14 +01:00
Vincent Ambo
b99b368d17 refactor(3p/nix/libutil): Replace hasPrefix/Suffix with Abseil
Uses the equivalent absl::StartsWith and absl::EndsWith functions
instead.
2020-05-25 02:19:01 +01:00
Vincent Ambo
8cf1322a6f fix(3p/nix/libstore): Fix error condition when parsing generations 2020-05-25 01:24:37 +01:00
Vincent Ambo
98299da0fd refactor(3p/nix/libutil): Replace string2Int & trim functions
Replaces these functions with corresponding functions from Abseil,
namely absl::StripAsciiWhitespace and absl::SimpleAtoi.

In the course of doing this some minor things I encountered along the
way were also refactored.

This also changes the signatures of the various custom readFile
functions to use absl::string_view types.
2020-05-25 01:19:02 +01:00
Vincent Ambo
b371821db5 chore(3p/nix/libstore): Remove progress bar remnants from download 2020-05-24 23:36:49 +01:00
Vincent Ambo
f5aaa12f1d style(3p/nix): Remove 'using std::*' from xml-writer.hh
See previous commit for more details on why.
2020-05-24 23:19:09 +01:00
Vincent Ambo
838f86b0fd style(3p/nix): Remove 'using std::*' from types.hh
It is considered bad form to use things from includes in headers, as
these directives propagate to everywhere else and can make it
confusing.

types.hh (which is includes almost literally everywhere) had some of
these directives, which this commit removes.
2020-05-24 22:29:21 +01:00
Vincent Ambo
f30b2e610d chore(3p/nix): Enable cross-file-rename in clangd 2020-05-24 18:19:28 +01:00
Vincent Ambo
da9104767e chore(3p/nix): Make build type configurable 2020-05-24 17:53:42 +01:00
Vincent Ambo
06d7b4aebd refactor(3p/nix/libutil): Replace chomp() with absl::strings 2020-05-24 02:13:07 +01:00
Vincent Ambo
10481d2586 chore(3p/nix): Remove some OS X specific defines
This project will be dropping OS X support until the core is simplified.
2020-05-24 02:13:07 +01:00
Alyssa Ross
bac38f3c49 fix(3p/nix): Fix long paths permanently breaking GC
Suppose I have a path /nix/store/[hash]-[name]/a/a/a/a/a/[...]/a,
long enough that everything after "/nix/store/" is longer than 4096
(MAX_PATH) bytes.

Nix will happily allow such a path to be inserted into the store,
because it doesn't look at all the nested structure.  It just cares
about the /nix/store/[hash]-[name] part.  But, when the path is deleted,
we encounter a problem.  Nix will move the path to /nix/store/trash, but
then when it's trying to recursively delete the trash directory, it will
at some point try to unlink
/nix/store/trash/[hash]-[name]/a/a/a/a/a/[...]/a.  This will fail,
because the path is too long.  After this has failed, any store deletion
operation will never work again, because Nix needs to delete the trash
directory before recreating it to move new things to it.  (I assume this
is because otherwise a path being deleted could already exist in the
trash, and then moving it would fail.)

This means that if I can trick somebody into just fetching a tarball
containing a path of the right length, they won't be able to delete
store paths or garbage collect ever again, until the offending path is
manually removed from /nix/store/trash.  (And even fixing this manually
is quite difficult if you don't understand the issue, because the
absolute path that Nix says it failed to remove is also too long for
rm(1).)

This patch fixes the issue by making Nix's recursive delete operation
use unlinkat(2).  This function takes a relative path and a directory
file descriptor.  We ensure that the relative path is always just the
name of the directory entry, and therefore its length will never exceed
255 bytes.  This means that it will never even come close to AX_PATH,
and Nix will therefore be able to handle removing arbitrarily deep
directory hierachies.

Since the directory file descriptor is used for recursion after being
used in readDirectory, I made a variant of readDirectory that takes an
already open directory stream, to avoid the directory being opened
multiple times.  As we have seen from this issue, the less we have to
interact with paths, the better, and so it's good to reuse file
descriptors where possible.

I left _deletePath as succeeding even if the parent directory doesn't
exist, even though that feels wrong to me, because without that early
return, the linux-sandbox test failed.

Reported-by: Alyssa Ross <hi@alyssa.is>
Thanks-to: Puck Meerburg <puck@puckipedia.com>
Tested-by: Puck Meerburg <puck@puckipedia.com>
Reviewed-by: Puck Meerburg <puck@puckipedia.com>
(cherry picked from commit c05e20daa1abb3446e378331697938b78af2b3d7)
2020-05-24 00:12:38 +01:00
Vincent Ambo
f77945ca81 style(3p/nix): Rename derivation to 'tazjix'
This makes it easier to distinguish which thing I'm dealing with in
the store paths. It does not affect anything else.
2020-05-23 23:47:45 +01:00
Vincent Ambo
cac00656cb fix(3p/nix): Set Meson build type to 'release'
Gotta go fast ... (well, not while compiling)
2020-05-23 20:39:02 +01:00
Vincent Ambo
af762abadc fix(3p/nix/libexpr): Ensure ExprOpUpdate merges into destination
... this fixes nixpkgs eval!
2020-05-23 20:30:23 +01:00
Vincent Ambo
55b1a47647 refactor(3p/nix/libexpr): Make other 'const' in Bindings::merge 2020-05-23 20:29:05 +01:00
Vincent Ambo
ab1fbd4c6e fix(3p/nix/libexpr): Actually use AttributeMap type alias
Without this alias, the garbage-collecting allocator won't be used and
allocated attribute set values won't be visible during GC.
2020-05-23 20:19:13 +01:00
Vincent Ambo
39e72c27f1 docs(3p/nix/libexpr): Add comment on ExprSelect 2020-05-23 16:01:52 +01:00
Vincent Ambo
bca0e34859 docs(3p/nix/libexpr): Add some comments about function calls
These were things that took me a moment to realise.
2020-05-23 06:44:38 +01:00
Vincent Ambo
d27c722e9e chore(3p/nix/tests): Remove leftover '__overrides' tests 2020-05-23 06:19:49 +01:00
Vincent Ambo
3b903193be fix(3p/nix/libexpr): Fix attrNames/attrValues builtins for btree_map
Replaces the previous implementations which performed sorting with one
that instead walks through the map (which is already sorted) and
yields values from it.

This fixes a handful of language tests because the previous
implementation did not actually yield useful values on the new implementation.
2020-05-23 06:18:20 +01:00
Vincent Ambo
811c42d255 fix(3p/nix/libexpr): Ensure symbols are compared by value 2020-05-23 06:18:20 +01:00
Vincent Ambo
85a05a6f38 fix(3p/nix/libexpr): Remove the global empty attribute set
In the change to the backing structure of attribute sets, the
requirement to manually balance the capacity of the structure went
away.

This is a) because Abseil's data structures manage this on their own,
and b) because the new Bindings class is allocated using `new (GC)`
rather than writing into a predefined memory area.

As part of this change functions related to the capacity were
deprecated and set to 0 values, which in turn caused the creation of
new attribute sets to return the same (mutable!) default value in
various cases, leading to "side effects" that caused evaluation
failures.

FWIW, I'm not sure if this optimisation had noticeable performance
impact, but while untangling libexpr it definitely doesn't help trying
to follow what it's doing - so bye, bye!
2020-05-23 06:18:20 +01:00
Vincent Ambo
da4ca4e02f chore(3p/nix/libexpr): Minor readability improvements in eval/value 2020-05-23 06:18:20 +01:00
Vincent Ambo
92792264f7 chore(3p/nix/libexpr): Remove unused __overrides feature
This feature does not appear in nixpkgs, so I don't care about it. My
only goal is evaluating nixpkgs.
2020-05-23 01:09:35 +01:00
Vincent Ambo
6b447f4b25 chore(3p/nix/libexpr): Expose separate insert & "upsert" methods
Reading more through the old code, it seems like the intention
/sometimes/ is to replace values.
2020-05-23 00:52:20 +01:00
Vincent Ambo
8c28be1b69 fix(3p/nix/libexpr): Use gc_allocator<T> as the btree_map allocator
This will make all Attr values visible to the GC.
2020-05-22 23:43:44 +01:00
Vincent Ambo
5f728cb428 feat(3p/nix): Add a derivation to launch clangd
This wrapper derivation (which assumes that the depot is available at
~/depot) can be used to actually get clangd working with
//third_party/nix.

In my setup I can launch this with M-x eglot, followed by

  env
    CLANGD_FLAGS='--compile-commands-dir=/home/tazjin/projects/nix-build'
    nix-shell -A third_party.nix --run 'nix-clangd' /home/tazjin/depot
2020-05-22 23:00:23 +01:00
Vincent Ambo
c25281820f fix(3p/nix/libexpr): Do not allow duplicate attribute insertion
This is closer to bug-for-bug compatibility with the previous version,
which would put new elements at the end of the array and (due to the
linear scan) return previous ones.
2020-05-22 20:30:22 +01:00
Vincent Ambo
f459332f32 chore: Update from Clang 9 to Clang 10 for all projects 2020-05-22 18:29:47 +01:00
Vincent Ambo
618f9a6f07 refactor(3p/nix/libexpr): Use gc_cpp to allocate Value instances 2020-05-22 16:57:48 +01:00
Vincent Ambo
b3c9166b23 refactor(3p/nix/libexpr): state->allocBindings -> Bindings::NewGC
EvalState::allocBindings had little to do with Bindings, other than
returning them, and didn't belong in that class.
2020-05-22 16:57:36 +01:00
Vincent Ambo
e24466c795 fix(3p/nix/libexpr): Make new Bindings class visible to GC 2020-05-22 16:35:21 +01:00
Vincent Ambo
68e6e92a20 chore(3p/nix/libexpr): Delete Bindings::sort
This function does nothing anymore since the attributes are always
in-order.
2020-05-22 03:35:07 +01:00
Vincent Ambo
986a8f6b75 fix(3p/nix): Update for usage of new attribute set API
The new attribute set API uses the iterators of the btree_map
directly. This requires changes in various files because the internals
of libexpr are very entangled.

This code runs and compiles, but there is a bug causing empty
attribute sets to be assigned incorrectly.
2020-05-22 01:59:39 +01:00
Vincent Ambo
42205f27fc refactor(3p/nix/libexpr): Use absl::btree_map::iterator type
Instead of using a custom Args* iterator, use the one belonging to the
map type directly.
2020-05-22 01:59:36 +01:00
Vincent Ambo
ee4637e3a2 refactor(3p/nix/libexpr): Use absl::btree_map::merge for '//'
Instead of doing some sort of inline merge-sort of the two attribute
sets, use the attribute sets merge function.

This commit alone does not build and is not supposed to.
2020-05-22 01:52:03 +01:00
Vincent Ambo
28e347effe refactor(3p/nix/libexpr): Use absl::btree_map for AttrSets
This is the first step towards replacing the implementation of
attribute sets with an absl::btree_map.

Currently many access are done using array offsets and pointer
arithmetic, so this change is currently causing Nix to fail in various
ways.
2020-05-21 19:21:55 +01:00
Vincent Ambo
a162f4e825 refactor(3p/nix/libexpr): Use std::string as qualified type
Replaces most uses of `string` with `std::string`.

This came up because I removed the "types.hh" import from
"symbol-table.hh", which percolated through a bunch of files where
`string` was suddenly no longer defined ... *sigh*
2020-05-21 05:43:22 +01:00
Vincent Ambo
b97307056d docs(3p/nix/libexpr): Add clarifying comments to SymbolTable 2020-05-21 05:42:20 +01:00
Vincent Ambo
c395a48be2 fix(3p/nix): Compatibility with updated SymbolTable
The functions in SymbolTable have been renamed to match the Google
Style guide, and some debug-only functions have been removed.
2020-05-21 05:09:23 +01:00
Vincent Ambo
97e85f94e5 refactor(3p/nix/libexpr): Use absl::node_hash_set in SymbolTable
This replaces the previous use of std::unordered_set with
absl::node_hash_set.

This type was chosen because the current implementation requires
pointer stability.

This does not yet touch the 'Attr' struct.

As a bonus, the implementation of the SymbolTable struct is now
consolidated into a single header/implementation file pair.
2020-05-21 05:09:23 +01:00
Vincent Ambo
00017ace04 feat(3p/nix): Wrangle Meson/Nix/CMake into (temporary) submission
Meson is unable to use CMake in Nix to determine the internal
structure of the Abseil libraries.

This commit adds an explicit list of most of the Abseil targets that
are relevant (so far) and bundles them into a list that is linked
together.
2020-05-21 04:50:49 +01:00
Vincent Ambo
a3b48e6562 chore(3p/nix): Minor fixes to get rid of warnings 2020-05-21 00:24:12 +01:00
Vincent Ambo
9dfdf16a8d fix(3p/nix): Fix build of derivation if cmake is present
cmake automatically runs a configure hook which breaks the build,
since this isn't actually a cmake project. This hook is now disabled.

Additionally Abseil's sources are linked to an absolute derivation
path when the build launches, as opposed to the relative path used for
development builds.
2020-05-21 00:24:10 +01:00
Vincent Ambo
43677021e3 refactor(3p/nix): Apply clang-tidy's performance-* fixes
This applies the performance fixes listed here:

https://clang.llvm.org/extra/clang-tidy/checks/list.html
2020-05-20 22:58:43 +01:00
Vincent Ambo
689ef502f5 refactor(3p/nix): Apply clang-tidy's readability-* fixes
This applies the readability fixes listed here:

https://clang.llvm.org/extra/clang-tidy/checks/list.html
2020-05-20 22:27:37 +01:00
Vincent Ambo
d331d3a0b5 refactor(3p/nix): Apply clang-tidy's modernize-* fixes
This applies the modernization fixes listed here:

https://clang.llvm.org/extra/clang-tidy/checks/list.html

The 'modernize-use-trailing-return-type' fix was excluded due to my
personal preference (more specifically, I think the 'auto' keyword is
misleading in that position).
2020-05-20 04:33:07 +01:00
Vincent Ambo
fed31b2c9b feat(3p/nix): Add some initial Abseil build targets
These make it possible to link to Abseil strings.
2020-05-20 04:02:23 +01:00
Vincent Ambo
e2fad0f0d2 feat(3p/nix): Add Abseil as a CMAKE subproject to Meson
Yep.

This is accomplished by symlinking the sources into the location
expected by Meson for subprojects.
2020-05-20 03:03:08 +01:00
Vincent Ambo
ffb2ae54be chore(third_party/nix): Dump of minor accumulated changes
None of these are worthy of a specific commit, or even have a real
reason behind them, but I didn't want to lose them.
2020-05-20 01:10:44 +01:00
Vincent Ambo
88f337588c refactor(3p/nix): Make all single-argument constructors explicit
Implicit constructors can be confusing, especially in a codebase that
is already as unintentionally obfuscated as this one.

https://google.github.io/styleguide/cppguide.html#Explicit_Constructors
2020-05-19 22:02:23 +01:00
Vincent Ambo
3908732181 style(3p/nix): Final act in the brace-wrapping saga
This last change set was generated by a full clang-tidy run (including
compilation):

    clang-tidy -p ~/projects/nix-build/ \
      -checks=-*,readability-braces-around-statements -fix src/*/*.cc

Actually running clang-tidy requires some massaging to make it play
nice with Nix + meson, I'll be adding a wrapper or something for that soon.
2020-05-19 20:51:32 +01:00
Vincent Ambo
cf40d08908 chore(3p/nix): Remove OSX only 'resolve-system-dependencies' tool 2020-05-19 20:34:12 +01:00
Vincent Ambo
83f4c37adf style(3p/nix/libstore): Additional if/for bracing fixes 2020-05-19 19:40:31 +01:00
Vincent Ambo
0b8415124d fix(3p/nix/libstore): Fix mistake introduced by bracing changes
This statement got included in a loop when it shouldn't have been. At
least it led to some funny derivation files!
2020-05-19 19:37:06 +01:00
Vincent Ambo
1841d93ccb style(3p/nix): Add braces around single-line for-loops
These were not caught by the previous clang-tidy invocation, but were
instead sorted out using amber[0] as such:

    ambr --regex 'for (\(.+\))\s([a-z].*;)' 'for $1 { $2 }'

[0]: https://github.com/dalance/amber
2020-05-19 19:04:08 +01:00
Vincent Ambo
867055133d style(3p/nix): Add braces around single-line conditionals
These were not caught by the previous clang-tidy invocation, but were
instead sorted out using amber[0] as such:

    ambr --regex 'if (\(.+\))\s([a-z].*;)' 'if $1 { $2 }'

[0]: https://github.com/dalance/amber
2020-05-19 18:55:58 +01:00
Vincent Ambo
c6a31838cd fix(3p/nix/libexpr): Declare value union types explicitly
Previously these structs were declared anonymously inside of the -
anonymous - union. This is not actually supported by the C++ standard,
but is merely a compiler-specific extension.

Unfortunately untangling this required a forward-declaration of the
Value type.
2020-05-19 18:19:35 +01:00
Vincent Ambo
b37ff365ad fix(3p/nix/libexpr): Use noexcept instead of throw() 2020-05-19 18:19:23 +01:00
Vincent Ambo
09cbc431cc fix(3p/nix): Fix incorrectly braced conditionals and loops
Fixes mistakes introduced by clang-tidy in the previous commit.
2020-05-19 18:01:12 +01:00
Vincent Ambo
b490742a51 style(3p/nix): Enforce braces around loops and conditionals
This change was generated with:

  fd -e cc -e hh | xargs -I{} clang-tidy {} -p ~/projects/nix-build/ \
    --checks='-*,readability-braces-around-statements' --fix \
    -fix-errors

Some manual fixes were applied because some convoluted unbraced
statements couldn't be untangled by clang-tidy.

This commit still includes invalid files, but I decided to clean them
up in a subsequent commit so that it becomes more obvious where
clang-tidy failed. Maybe this will allow for a bug-report to
clang-tidy.
2020-05-19 17:38:04 +01:00
Vincent Ambo
c758de9d22 style(3p/nix): Reformat all includes to match new style 2020-05-19 15:55:11 +01:00
Vincent Ambo
b508f5b91a style(3p/nix): Change include formatting config for clang-format
Changes the configuration to regroup all includes. The include groups
will be (in this order):

1. (in .cc): Include of the corresponding header
2. Includes of C++ standard library headers
3. Includes of other external headers
4. Includes of local headers
2020-05-19 15:55:09 +01:00
Vincent Ambo
62c18c9898 fix(3p/nix): Set C++ style to "Google" in Emacs locals 2020-05-19 05:13:47 +01:00
Vincent Ambo
95a57f15ca refactor(3p/nix/nix-*): Replace logging with glog in binaries 2020-05-19 04:52:47 +01:00
Vincent Ambo
9aa2ecd78c refactor(3p/nix/nix-daemon): Remove activities from protocol
Removes the activity transfer that was previously nulled out from the
daemon protocol completely.

This might actually break Nix completely, I haven't tried yet, but
that's fine because this will be replaced with gRPC.
2020-05-19 04:51:45 +01:00
Vincent Ambo
c7be956f3f style(3p/nix/libexpr): Reformat with clang-format 2020-05-19 04:51:01 +01:00
Vincent Ambo
f8deeb1442 chore(3p/nix/nix): Remove progress bar implementation
The progress bar has lots of complexity for little benefit. The
previous activity tracking stuff has been deleted as part of the
logging refactoring and I am not going to implement support for this
again for now.
2020-05-19 04:49:39 +01:00
Vincent Ambo
939dd9f817 refactor(3p/nix/libexpr): Replace logging.h with glog 2020-05-19 01:24:50 +01:00
Vincent Ambo
d0c44425e1 refactor(3p/nix/libmain): Replace logging.h with glog 2020-05-19 01:04:14 +01:00
Vincent Ambo
505b6b044b refactor(3p/nix/libstore): Replace logging.h with glog 2020-05-19 01:04:14 +01:00
Vincent Ambo
6dc6c29fa4 refactor(3p/nix/libutil): Replace internal logging library with glog 2020-05-18 02:34:41 +01:00
Vincent Ambo
c584480cd4 chore(3p/nix/libutil): Remove logging implementation 2020-05-18 00:39:50 +01:00
Vincent Ambo
0335b91b93 feat(3p/nix): Add glog dependency 2020-05-17 23:58:51 +01:00
Vincent Ambo
e797b4d09b chore(3p/nix): Always enable the garbage collector
It's unclear why this toggle existed, now it doesn't.
2020-05-17 22:27:53 +01:00
Vincent Ambo
168ef124ef fix(3p/nix): Minor fixes to derivation
* version must be set to use this as the system Nix
* missing busybox path is now set
* fixed build output names
2020-05-17 21:25:50 +01:00
Vincent Ambo
8944370b7f feat(3p/nix): Add new Meson-based derivation for building Nix
This builds the mesonified Nix and is compatible with the depot
structure and nix-shell.
2020-05-17 20:46:06 +01:00
Vincent Ambo
9d865dbb1f chore(3p/nix): Remove symlink setup from Meson build
This belongs in the derivation itself, IMO.
2020-05-17 20:45:47 +01:00
Vincent Ambo
96700c9522 chore(3p/nix): Remove old build derivations
Removes the autotools-based build derivations, in favour of using a
simpler build that just wraps Meson.
2020-05-17 20:24:18 +01:00
Vincent Ambo
40395860c7 chore(3p/nix): Remove old build system and other miscellanies
There are still remnants of the old build system (for example, the
build derivations are not yet updated at all), but we'll get there.
2020-05-17 16:43:21 +01:00
Vincent Ambo
0f2cf531f7 style(3p/nix): Reformat project in Google C++ style
Reformatted with:

    fd . -e hh -e cc | xargs clang-format -i
2020-05-17 16:31:57 +01:00
Vincent Ambo
65a1aae98c fix(3p/nix): Fix Meson configuration for libexpr 2020-05-17 16:28:17 +01:00
Vincent Ambo
c94ef8d00b chore(3p/nix): Use packages from depot tree 2020-05-17 16:28:15 +01:00
John Ericson
939cec746e chore(3p/nix): Clean up libseccomp_dep
(cherry picked from commit b380f75867d48e6af461513d26803b61250d9c80)
2020-05-17 15:57:09 +01:00
John Ericson
adff9913c0 docs(3p/nix): Clean up descriptions in Meson
1. Fix that for libsodium

  2. Consistently capitalize

(cherry picked from commit 12ce28bdf21d1f261132bdefe3dac25b1aab7dcc)
2020-05-17 15:57:04 +01:00
John Ericson
8481ff4351 fix(3p/nix): non-existent wrapper dependency .found() should be false too
(cherry picked from commit 90f9f4aef781fc5c3ead8652fc408a431f480905)
2020-05-17 15:56:50 +01:00
John Ericson
9c21176f57 chore(3p/nix): Make libsodium an auto dependency
This means do autodetection if the user doesn't specify.

(cherry picked from commit 1be6ab4d3eb4d6b04c322b9ed0a108c2b5354ec5)
2020-05-17 15:56:36 +01:00
Vincent Ambo
3f6f3356e8 feat(3p/nix/build): Add meson & ninja as nativeBuildDeps 2020-05-17 15:56:32 +01:00
John Ericson
6cbc248b3f refactor(3p/nix): Remove abstractions in HAVE_STRUCT_DIRENT_D_TYPE code
(cherry picked from commit 727dc569a7a13b038d550b8c98a79d5928424d9b)
2020-05-17 15:56:28 +01:00
John Ericson
4b50bd28a0 feat(3p/nix): meson: Don't look for libraries in lib dir
1. First of all, this doesn't work in nixpkgs. Per [1], gcc ignores `-L`
   for purposes of `--print-file-dirs`, which breaks horribly on linux. But
   if we don't pass extra dirs, meosn first just tries `-l...`, which does
   work.

2. Even if it did work, `libdir` means where we are installing libs, not
   where libs are expected to be found. Those are not necessarily the
   same (again, nixpkgs), and even when they are and non-standard, it is
   better to use DESTDIR or have a modified toolchain.

[1]: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87758

(cherry picked from commit a142164e746644e20f66908c156ca913bef4664f)
2020-05-17 15:56:23 +01:00
John Ericson
d64f60c733 feat(3p/nix): meson: Define HAVE_STRUCT_DIRENT_D_TYPE macro
(cherry picked from commit 7ae66bbef5e92f4a0bba49780d8c316f134884f2)
2020-05-17 15:56:19 +01:00
John Ericson
55e9f61041 chore(3p/nix): Add Meson and Ninja dependencies
(cherry picked from commit 3a968f5152d6f65ea3d9fdd36df3c188a4e20e24)
2020-05-17 15:56:13 +01:00
p01arst0rm
e52ce543a4 chore(3p/nix): more improvements
(cherry picked from commit 507d28fd226e018d5927b49b1bd07833cbd49d25)
2020-05-17 15:55:31 +01:00
p01arst0rm
9974eca8a0 feat(3p/nix): minor improvements
(cherry picked from commit f6f7046acfaf90c19c597687d384b491c852b4c6)
2020-05-17 15:55:27 +01:00
p01arst0rm
055139ac6a feat(3p/nix): added meson support
(cherry picked from commit 086a81b7a5bbe1fc022efb5935ff68f6ad71ddaf)
2020-05-17 15:55:13 +01:00
Vincent Ambo
c455f7a987 feat(third_party/nix): Add clang-format configuration 2020-05-17 15:53:25 +01:00
Vincent Ambo
7994fd1d54 Add 'third_party/nix/' from commit 'be66c7a6b24e3c3c6157fd37b86c7203d14acf10'
git-subtree-dir: third_party/nix
git-subtree-mainline: cf8cd640c1
git-subtree-split: be66c7a6b24e3c3c6157fd37b86c7203d14acf10
2020-05-17 15:52:38 +01:00