Commit graph

123 commits

Author SHA1 Message Date
Vincent Ambo
7e8a48c82c fix(3p/nix): Link to glog built by Nix
Same as cl/921, it seems that the trick to making this work is indeed
overriding the C++ standard used in the dependency.

Change-Id: I3c5984d71014d774c161ecc283844f504fd44719
Reviewed-on: https://cl.tvl.fyi/c/depot/+/922
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-07-05 16:54:12 +00:00
Vincent Ambo
5cb6e6ee21 fix(3p/nix): Link to Abseil built by Nix
This didn't work previously ... but now it does. I think setting the
standard explicitly is what did the trick, but it's slightly unclear
to me why.

Either way this means that Abseil is no longer constantly getting
recompiled when building Nix, which is nice.

Change-Id: I377f7b68bf1ef9045df6a2eee8fdd0c92f243547
Reviewed-on: https://cl.tvl.fyi/c/depot/+/921
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-07-05 16:54:12 +00:00
Vincent Ambo
4455f28e42 chore(3p/nix/libexpr): Remove dead code from attribute sets
These bits are no longer required with the hashmap-backed
implementation of attribute sets.

Change-Id: I8b936d8d438a00bad4ccf8e0b4dd719c559ce8c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/912
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2020-07-04 03:12:16 +00:00
glittershark
ec84fde6b1 fix(3p/nix): Revert "feat(3p/nix): Add installCheckPhase"
This reverts commit 368e8d1edd.

Reason for revert: Didn't mean to submit, and the phase is currently failing (which breaks the otherwise-functional derivation)

Change-Id: I515b2fb45188dc90f09ae2458453192487c74d71
Reviewed-on: https://cl.tvl.fyi/c/depot/+/581
Reviewed-by: tazjin <mail@tazj.in>
2020-06-23 15:23:04 +00:00
Griffin Smith
368e8d1edd feat(3p/nix): Add installCheckPhase
Add an installCheckPhase that runs the appropriate substituteAll on
common.sh and runs the lang.sh tests with the build artifacts in the
PATH.

Change-Id: I2df5a93b8f3ffdfdc194a0e7d6b6669ef520c345
Reviewed-on: https://cl.tvl.fyi/c/depot/+/561
Reviewed-by: glittershark <grfn@gws.fyi>
2020-06-23 14:25:37 +00:00
Griffin Smith
b94d9f89dc fix(3p/nix): Make traces log, not dlog
We don't want traces compiled out since they're an actual language
feature that're used in userspace - also their absence is breaking the
tests

Change-Id: Icaefca8f52e94001785f724fdc0c10a7586b24e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/562
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: Kane York <rikingcoding@gmail.com>
Reviewed-by: lukegbot <bot@lukegb.com>
2020-06-23 12:37:48 +00:00
Luke Granger-Brown
5252623ca7 fix(3p/nix): fix readTree by fixing readDir by deleting some code
Change-Id: I4f44e6050fc5121072f4cde7758defe2dcbd4e92
Reviewed-on: https://cl.tvl.fyi/c/depot/+/552
Reviewed-by: tazjin <mail@tazj.in>
2020-06-22 01:12:53 +00:00
Luke Granger-Brown
4de63f784c fix(3p/nix): propagate dependencies which are needed by people linking in nix
This includes absl, which we install into the output, and boost and the boehm GC,
which are moved to propagated deps.

Change-Id: I8f9f9795ff92e26b2320359064241d7fd59c2d33
Reviewed-on: https://cl.tvl.fyi/c/depot/+/549
Reviewed-by: tazjin <mail@tazj.in>
2020-06-21 23:49:19 +00:00
Luke Granger-Brown
b99829f142 fix(3p/nix): install nix_config.h and config.nix
This also installs the rest of corepkgs as a side-effect.

Change-Id: I67a42d45793d5e8fdad51c1f306eebf63e9c2868
Reviewed-on: https://cl.tvl.fyi/c/depot/+/548
Reviewed-by: tazjin <mail@tazj.in>
2020-06-21 23:49:19 +00:00
Luke Granger-Brown
4ea8cc3730 fix(3p/nix): allow blank lines in nix.conf
Change-Id: I5ab040a62e123c57fe712b252fbf84fe5a8bc026
Reviewed-on: https://cl.tvl.fyi/c/depot/+/547
Reviewed-by: tazjin <mail@tazj.in>
2020-06-21 23:49:19 +00:00
Luke Granger-Brown
8b83e0f31b feat(3p/nix): install pkgconfig files for libexpr, libmain and libstore
Also fixes the pkgconfig files to use the corresponding CMake variables.

Change-Id: I8095b8aff39ad91e592f3edc95555c9f1f1f153d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/545
Reviewed-by: tazjin <mail@tazj.in>
2020-06-21 20:42:28 +00:00
Luke Granger-Brown
739a45f9e4 chore(3p/nix): use GNUInstallDirs to determine output directory names
This ensures that we install both glog's .a and all the .so files we
generate into a single consistent output lib path (which is, err,
lib64, but whatever).

Change-Id: Ib6ac6eacf5f56e4b719cfb586db731efc122c31b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/544
Reviewed-by: tazjin <mail@tazj.in>
2020-06-21 20:42:28 +00:00
Luke Granger-Brown
a54f155c07 chore(3p/nix): export library headers to include/ in output.
Change-Id: I87eb6e59782d720015d351d8829dc7b8688e01f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/543
Reviewed-by: tazjin <mail@tazj.in>
2020-06-21 20:42:28 +00:00
edef
48f08e8cc0 fix(3p/nix): passAsFile: leave out the hash prefix
Having a colon in the path may cause issues, and having the hash
function indicated isn't actually necessary. We now verify the path
format in the tests to prevent regressions.

(cherry picked from commit c65a6fa86aef7bdf51fb4fba7bd31d265619ba3f)
2020-05-31 19:20:24 +01:00
Puck Meerburg
7cc7fdacbe fix(3p/nix): passAsFile: hash the attribute name instead of numbering sequentially
This makes the paths consistent without relying on ordering.

Co-authored-by: edef <edef@edef.eu>
(cherry picked from commit 515c0a263e137a00e82f7d981284dbe54db23247)
2020-05-31 19:18:07 +01:00
Vincent Ambo
3d939a2f63 chore(3p/nix): Clean up leftover Meson file 2020-05-30 23:30:46 +01:00
Vincent Ambo
699ab4e20a chore(3p/nix): Clean up some unused defines 2020-05-30 23:17:48 +01:00
Vincent Ambo
91ddc7603d feat(3p/nix): Configure CMake to generate compile commands 2020-05-29 22:31:07 +01:00
Vincent Ambo
e494a347b5 fix(3p/nix): Build glog as included CMake project 2020-05-29 22:04:06 +01:00
Vincent Ambo
25393d8080 refactor(3p/nix): Introduce CMake as the build system for Nix
Completes the switch from Meson to CMake for the core build system in
Nix.

Meson was added originally because someone else had already done the
work for integrating it in Nix and it was an upgrade from the previous
setup.

However over time it became clear that Meson is not quite mature
enough for projects like Nix that have occasionally peculiar
configuration constraints.

Some issues encountered with Meson (some of these are due to the Meson
setup in Nix):

* Difficulty with generating correct compile_commands.json for
  external tools like clangd
* Difficulty linking to libc++ when using clang
* Ugly shell invocations for certain parts of the build system (I want
  these to be gone!!!)

This CMake setup mimics the Meson configuration, but there are some
differences (some temporary):

* headers are now included separately for each library (see a previous
  commit that changes includes appropriately)
* autoheaders-style configuration is currently hardcoded. Before
  blindly copying this I want to evaluate how much of it actually exists
  for portability concerns that I don't have (such as support for OS
  X).
* Nix is built with libc++ by default.
* [libstore] SQL schema is now inlined via a generated header, not an
  included string literal

Abseil is still built as part of this build, rather than an external
dependency, because it chokes on differently configured compiler
invocations.

Note that because of the move to libc++ an unwanted behaviour is
introduced: glog log messages no longer have a body. I have yet to
debug what is going on there.
2020-05-28 00:11:25 +01:00
Vincent Ambo
3d7537da7f chore(3p/nix): Remove meson build files for core packages 2020-05-27 22:00:00 +01:00
Vincent Ambo
abe5df74a8 chore(3p/nix): Move Abseil symlink one layer up
This is required for upcoming build system changes.
2020-05-27 21:57:28 +01:00
Vincent Ambo
3652326ed2 refactor(3p/nix): Anchor local includes at src/
Previously all includes were anchored in one global mess of header
files. This moves the includes into filesystem "namespaces" (if you
will) for each sub-package of Nix.

Note: This commit does not introduce the relevant build system changes.
2020-05-27 21:56:34 +01:00
Vincent Ambo
2e32ceb523 fix(3p/nix): Default-import depot from a relative path 2020-05-27 00:06:59 +01:00
Vincent Ambo
14b52848f8 docs(3p/nix): Add a README explaining the goals of the fork 2020-05-25 16:39:18 +01:00
Vincent Ambo
bf452cbc2a refactor(3p/nix): Replace tokenizeStrings with absl::StrSplit
This function was a custom (and inefficient in the case of
single-character delimiters) string splitter which was used all over
the codebase. Abseil provides an appropriate replacement function.
2020-05-25 15:54:14 +01:00
Vincent Ambo
b99b368d17 refactor(3p/nix/libutil): Replace hasPrefix/Suffix with Abseil
Uses the equivalent absl::StartsWith and absl::EndsWith functions
instead.
2020-05-25 02:19:01 +01:00
Vincent Ambo
8cf1322a6f fix(3p/nix/libstore): Fix error condition when parsing generations 2020-05-25 01:24:37 +01:00
Vincent Ambo
98299da0fd refactor(3p/nix/libutil): Replace string2Int & trim functions
Replaces these functions with corresponding functions from Abseil,
namely absl::StripAsciiWhitespace and absl::SimpleAtoi.

In the course of doing this some minor things I encountered along the
way were also refactored.

This also changes the signatures of the various custom readFile
functions to use absl::string_view types.
2020-05-25 01:19:02 +01:00
Vincent Ambo
b371821db5 chore(3p/nix/libstore): Remove progress bar remnants from download 2020-05-24 23:36:49 +01:00
Vincent Ambo
f5aaa12f1d style(3p/nix): Remove 'using std::*' from xml-writer.hh
See previous commit for more details on why.
2020-05-24 23:19:09 +01:00
Vincent Ambo
838f86b0fd style(3p/nix): Remove 'using std::*' from types.hh
It is considered bad form to use things from includes in headers, as
these directives propagate to everywhere else and can make it
confusing.

types.hh (which is includes almost literally everywhere) had some of
these directives, which this commit removes.
2020-05-24 22:29:21 +01:00
Vincent Ambo
f30b2e610d chore(3p/nix): Enable cross-file-rename in clangd 2020-05-24 18:19:28 +01:00
Vincent Ambo
da9104767e chore(3p/nix): Make build type configurable 2020-05-24 17:53:42 +01:00
Vincent Ambo
06d7b4aebd refactor(3p/nix/libutil): Replace chomp() with absl::strings 2020-05-24 02:13:07 +01:00
Vincent Ambo
10481d2586 chore(3p/nix): Remove some OS X specific defines
This project will be dropping OS X support until the core is simplified.
2020-05-24 02:13:07 +01:00
Alyssa Ross
bac38f3c49 fix(3p/nix): Fix long paths permanently breaking GC
Suppose I have a path /nix/store/[hash]-[name]/a/a/a/a/a/[...]/a,
long enough that everything after "/nix/store/" is longer than 4096
(MAX_PATH) bytes.

Nix will happily allow such a path to be inserted into the store,
because it doesn't look at all the nested structure.  It just cares
about the /nix/store/[hash]-[name] part.  But, when the path is deleted,
we encounter a problem.  Nix will move the path to /nix/store/trash, but
then when it's trying to recursively delete the trash directory, it will
at some point try to unlink
/nix/store/trash/[hash]-[name]/a/a/a/a/a/[...]/a.  This will fail,
because the path is too long.  After this has failed, any store deletion
operation will never work again, because Nix needs to delete the trash
directory before recreating it to move new things to it.  (I assume this
is because otherwise a path being deleted could already exist in the
trash, and then moving it would fail.)

This means that if I can trick somebody into just fetching a tarball
containing a path of the right length, they won't be able to delete
store paths or garbage collect ever again, until the offending path is
manually removed from /nix/store/trash.  (And even fixing this manually
is quite difficult if you don't understand the issue, because the
absolute path that Nix says it failed to remove is also too long for
rm(1).)

This patch fixes the issue by making Nix's recursive delete operation
use unlinkat(2).  This function takes a relative path and a directory
file descriptor.  We ensure that the relative path is always just the
name of the directory entry, and therefore its length will never exceed
255 bytes.  This means that it will never even come close to AX_PATH,
and Nix will therefore be able to handle removing arbitrarily deep
directory hierachies.

Since the directory file descriptor is used for recursion after being
used in readDirectory, I made a variant of readDirectory that takes an
already open directory stream, to avoid the directory being opened
multiple times.  As we have seen from this issue, the less we have to
interact with paths, the better, and so it's good to reuse file
descriptors where possible.

I left _deletePath as succeeding even if the parent directory doesn't
exist, even though that feels wrong to me, because without that early
return, the linux-sandbox test failed.

Reported-by: Alyssa Ross <hi@alyssa.is>
Thanks-to: Puck Meerburg <puck@puckipedia.com>
Tested-by: Puck Meerburg <puck@puckipedia.com>
Reviewed-by: Puck Meerburg <puck@puckipedia.com>
(cherry picked from commit c05e20daa1abb3446e378331697938b78af2b3d7)
2020-05-24 00:12:38 +01:00
Vincent Ambo
f77945ca81 style(3p/nix): Rename derivation to 'tazjix'
This makes it easier to distinguish which thing I'm dealing with in
the store paths. It does not affect anything else.
2020-05-23 23:47:45 +01:00
Vincent Ambo
cac00656cb fix(3p/nix): Set Meson build type to 'release'
Gotta go fast ... (well, not while compiling)
2020-05-23 20:39:02 +01:00
Vincent Ambo
af762abadc fix(3p/nix/libexpr): Ensure ExprOpUpdate merges into destination
... this fixes nixpkgs eval!
2020-05-23 20:30:23 +01:00
Vincent Ambo
55b1a47647 refactor(3p/nix/libexpr): Make other 'const' in Bindings::merge 2020-05-23 20:29:05 +01:00
Vincent Ambo
ab1fbd4c6e fix(3p/nix/libexpr): Actually use AttributeMap type alias
Without this alias, the garbage-collecting allocator won't be used and
allocated attribute set values won't be visible during GC.
2020-05-23 20:19:13 +01:00
Vincent Ambo
39e72c27f1 docs(3p/nix/libexpr): Add comment on ExprSelect 2020-05-23 16:01:52 +01:00
Vincent Ambo
bca0e34859 docs(3p/nix/libexpr): Add some comments about function calls
These were things that took me a moment to realise.
2020-05-23 06:44:38 +01:00
Vincent Ambo
d27c722e9e chore(3p/nix/tests): Remove leftover '__overrides' tests 2020-05-23 06:19:49 +01:00
Vincent Ambo
3b903193be fix(3p/nix/libexpr): Fix attrNames/attrValues builtins for btree_map
Replaces the previous implementations which performed sorting with one
that instead walks through the map (which is already sorted) and
yields values from it.

This fixes a handful of language tests because the previous
implementation did not actually yield useful values on the new implementation.
2020-05-23 06:18:20 +01:00
Vincent Ambo
811c42d255 fix(3p/nix/libexpr): Ensure symbols are compared by value 2020-05-23 06:18:20 +01:00
Vincent Ambo
85a05a6f38 fix(3p/nix/libexpr): Remove the global empty attribute set
In the change to the backing structure of attribute sets, the
requirement to manually balance the capacity of the structure went
away.

This is a) because Abseil's data structures manage this on their own,
and b) because the new Bindings class is allocated using `new (GC)`
rather than writing into a predefined memory area.

As part of this change functions related to the capacity were
deprecated and set to 0 values, which in turn caused the creation of
new attribute sets to return the same (mutable!) default value in
various cases, leading to "side effects" that caused evaluation
failures.

FWIW, I'm not sure if this optimisation had noticeable performance
impact, but while untangling libexpr it definitely doesn't help trying
to follow what it's doing - so bye, bye!
2020-05-23 06:18:20 +01:00
Vincent Ambo
da4ca4e02f chore(3p/nix/libexpr): Minor readability improvements in eval/value 2020-05-23 06:18:20 +01:00
Vincent Ambo
92792264f7 chore(3p/nix/libexpr): Remove unused __overrides feature
This feature does not appear in nixpkgs, so I don't care about it. My
only goal is evaluating nixpkgs.
2020-05-23 01:09:35 +01:00