Commit graph

99 commits

Author SHA1 Message Date
Vincent Ambo
14b52848f8 docs(3p/nix): Add a README explaining the goals of the fork 2020-05-25 16:39:18 +01:00
Vincent Ambo
bf452cbc2a refactor(3p/nix): Replace tokenizeStrings with absl::StrSplit
This function was a custom (and inefficient in the case of
single-character delimiters) string splitter which was used all over
the codebase. Abseil provides an appropriate replacement function.
2020-05-25 15:54:14 +01:00
Vincent Ambo
b99b368d17 refactor(3p/nix/libutil): Replace hasPrefix/Suffix with Abseil
Uses the equivalent absl::StartsWith and absl::EndsWith functions
instead.
2020-05-25 02:19:01 +01:00
Vincent Ambo
8cf1322a6f fix(3p/nix/libstore): Fix error condition when parsing generations 2020-05-25 01:24:37 +01:00
Vincent Ambo
98299da0fd refactor(3p/nix/libutil): Replace string2Int & trim functions
Replaces these functions with corresponding functions from Abseil,
namely absl::StripAsciiWhitespace and absl::SimpleAtoi.

In the course of doing this some minor things I encountered along the
way were also refactored.

This also changes the signatures of the various custom readFile
functions to use absl::string_view types.
2020-05-25 01:19:02 +01:00
Vincent Ambo
b371821db5 chore(3p/nix/libstore): Remove progress bar remnants from download 2020-05-24 23:36:49 +01:00
Vincent Ambo
f5aaa12f1d style(3p/nix): Remove 'using std::*' from xml-writer.hh
See previous commit for more details on why.
2020-05-24 23:19:09 +01:00
Vincent Ambo
838f86b0fd style(3p/nix): Remove 'using std::*' from types.hh
It is considered bad form to use things from includes in headers, as
these directives propagate to everywhere else and can make it
confusing.

types.hh (which is includes almost literally everywhere) had some of
these directives, which this commit removes.
2020-05-24 22:29:21 +01:00
Vincent Ambo
f30b2e610d chore(3p/nix): Enable cross-file-rename in clangd 2020-05-24 18:19:28 +01:00
Vincent Ambo
da9104767e chore(3p/nix): Make build type configurable 2020-05-24 17:53:42 +01:00
Vincent Ambo
06d7b4aebd refactor(3p/nix/libutil): Replace chomp() with absl::strings 2020-05-24 02:13:07 +01:00
Vincent Ambo
10481d2586 chore(3p/nix): Remove some OS X specific defines
This project will be dropping OS X support until the core is simplified.
2020-05-24 02:13:07 +01:00
Alyssa Ross
bac38f3c49 fix(3p/nix): Fix long paths permanently breaking GC
Suppose I have a path /nix/store/[hash]-[name]/a/a/a/a/a/[...]/a,
long enough that everything after "/nix/store/" is longer than 4096
(MAX_PATH) bytes.

Nix will happily allow such a path to be inserted into the store,
because it doesn't look at all the nested structure.  It just cares
about the /nix/store/[hash]-[name] part.  But, when the path is deleted,
we encounter a problem.  Nix will move the path to /nix/store/trash, but
then when it's trying to recursively delete the trash directory, it will
at some point try to unlink
/nix/store/trash/[hash]-[name]/a/a/a/a/a/[...]/a.  This will fail,
because the path is too long.  After this has failed, any store deletion
operation will never work again, because Nix needs to delete the trash
directory before recreating it to move new things to it.  (I assume this
is because otherwise a path being deleted could already exist in the
trash, and then moving it would fail.)

This means that if I can trick somebody into just fetching a tarball
containing a path of the right length, they won't be able to delete
store paths or garbage collect ever again, until the offending path is
manually removed from /nix/store/trash.  (And even fixing this manually
is quite difficult if you don't understand the issue, because the
absolute path that Nix says it failed to remove is also too long for
rm(1).)

This patch fixes the issue by making Nix's recursive delete operation
use unlinkat(2).  This function takes a relative path and a directory
file descriptor.  We ensure that the relative path is always just the
name of the directory entry, and therefore its length will never exceed
255 bytes.  This means that it will never even come close to AX_PATH,
and Nix will therefore be able to handle removing arbitrarily deep
directory hierachies.

Since the directory file descriptor is used for recursion after being
used in readDirectory, I made a variant of readDirectory that takes an
already open directory stream, to avoid the directory being opened
multiple times.  As we have seen from this issue, the less we have to
interact with paths, the better, and so it's good to reuse file
descriptors where possible.

I left _deletePath as succeeding even if the parent directory doesn't
exist, even though that feels wrong to me, because without that early
return, the linux-sandbox test failed.

Reported-by: Alyssa Ross <hi@alyssa.is>
Thanks-to: Puck Meerburg <puck@puckipedia.com>
Tested-by: Puck Meerburg <puck@puckipedia.com>
Reviewed-by: Puck Meerburg <puck@puckipedia.com>
(cherry picked from commit c05e20daa1abb3446e378331697938b78af2b3d7)
2020-05-24 00:12:38 +01:00
Vincent Ambo
f77945ca81 style(3p/nix): Rename derivation to 'tazjix'
This makes it easier to distinguish which thing I'm dealing with in
the store paths. It does not affect anything else.
2020-05-23 23:47:45 +01:00
Vincent Ambo
cac00656cb fix(3p/nix): Set Meson build type to 'release'
Gotta go fast ... (well, not while compiling)
2020-05-23 20:39:02 +01:00
Vincent Ambo
af762abadc fix(3p/nix/libexpr): Ensure ExprOpUpdate merges into destination
... this fixes nixpkgs eval!
2020-05-23 20:30:23 +01:00
Vincent Ambo
55b1a47647 refactor(3p/nix/libexpr): Make other 'const' in Bindings::merge 2020-05-23 20:29:05 +01:00
Vincent Ambo
ab1fbd4c6e fix(3p/nix/libexpr): Actually use AttributeMap type alias
Without this alias, the garbage-collecting allocator won't be used and
allocated attribute set values won't be visible during GC.
2020-05-23 20:19:13 +01:00
Vincent Ambo
39e72c27f1 docs(3p/nix/libexpr): Add comment on ExprSelect 2020-05-23 16:01:52 +01:00
Vincent Ambo
bca0e34859 docs(3p/nix/libexpr): Add some comments about function calls
These were things that took me a moment to realise.
2020-05-23 06:44:38 +01:00
Vincent Ambo
d27c722e9e chore(3p/nix/tests): Remove leftover '__overrides' tests 2020-05-23 06:19:49 +01:00
Vincent Ambo
3b903193be fix(3p/nix/libexpr): Fix attrNames/attrValues builtins for btree_map
Replaces the previous implementations which performed sorting with one
that instead walks through the map (which is already sorted) and
yields values from it.

This fixes a handful of language tests because the previous
implementation did not actually yield useful values on the new implementation.
2020-05-23 06:18:20 +01:00
Vincent Ambo
811c42d255 fix(3p/nix/libexpr): Ensure symbols are compared by value 2020-05-23 06:18:20 +01:00
Vincent Ambo
85a05a6f38 fix(3p/nix/libexpr): Remove the global empty attribute set
In the change to the backing structure of attribute sets, the
requirement to manually balance the capacity of the structure went
away.

This is a) because Abseil's data structures manage this on their own,
and b) because the new Bindings class is allocated using `new (GC)`
rather than writing into a predefined memory area.

As part of this change functions related to the capacity were
deprecated and set to 0 values, which in turn caused the creation of
new attribute sets to return the same (mutable!) default value in
various cases, leading to "side effects" that caused evaluation
failures.

FWIW, I'm not sure if this optimisation had noticeable performance
impact, but while untangling libexpr it definitely doesn't help trying
to follow what it's doing - so bye, bye!
2020-05-23 06:18:20 +01:00
Vincent Ambo
da4ca4e02f chore(3p/nix/libexpr): Minor readability improvements in eval/value 2020-05-23 06:18:20 +01:00
Vincent Ambo
92792264f7 chore(3p/nix/libexpr): Remove unused __overrides feature
This feature does not appear in nixpkgs, so I don't care about it. My
only goal is evaluating nixpkgs.
2020-05-23 01:09:35 +01:00
Vincent Ambo
6b447f4b25 chore(3p/nix/libexpr): Expose separate insert & "upsert" methods
Reading more through the old code, it seems like the intention
/sometimes/ is to replace values.
2020-05-23 00:52:20 +01:00
Vincent Ambo
8c28be1b69 fix(3p/nix/libexpr): Use gc_allocator<T> as the btree_map allocator
This will make all Attr values visible to the GC.
2020-05-22 23:43:44 +01:00
Vincent Ambo
5f728cb428 feat(3p/nix): Add a derivation to launch clangd
This wrapper derivation (which assumes that the depot is available at
~/depot) can be used to actually get clangd working with
//third_party/nix.

In my setup I can launch this with M-x eglot, followed by

  env
    CLANGD_FLAGS='--compile-commands-dir=/home/tazjin/projects/nix-build'
    nix-shell -A third_party.nix --run 'nix-clangd' /home/tazjin/depot
2020-05-22 23:00:23 +01:00
Vincent Ambo
c25281820f fix(3p/nix/libexpr): Do not allow duplicate attribute insertion
This is closer to bug-for-bug compatibility with the previous version,
which would put new elements at the end of the array and (due to the
linear scan) return previous ones.
2020-05-22 20:30:22 +01:00
Vincent Ambo
f459332f32 chore: Update from Clang 9 to Clang 10 for all projects 2020-05-22 18:29:47 +01:00
Vincent Ambo
618f9a6f07 refactor(3p/nix/libexpr): Use gc_cpp to allocate Value instances 2020-05-22 16:57:48 +01:00
Vincent Ambo
b3c9166b23 refactor(3p/nix/libexpr): state->allocBindings -> Bindings::NewGC
EvalState::allocBindings had little to do with Bindings, other than
returning them, and didn't belong in that class.
2020-05-22 16:57:36 +01:00
Vincent Ambo
e24466c795 fix(3p/nix/libexpr): Make new Bindings class visible to GC 2020-05-22 16:35:21 +01:00
Vincent Ambo
68e6e92a20 chore(3p/nix/libexpr): Delete Bindings::sort
This function does nothing anymore since the attributes are always
in-order.
2020-05-22 03:35:07 +01:00
Vincent Ambo
986a8f6b75 fix(3p/nix): Update for usage of new attribute set API
The new attribute set API uses the iterators of the btree_map
directly. This requires changes in various files because the internals
of libexpr are very entangled.

This code runs and compiles, but there is a bug causing empty
attribute sets to be assigned incorrectly.
2020-05-22 01:59:39 +01:00
Vincent Ambo
42205f27fc refactor(3p/nix/libexpr): Use absl::btree_map::iterator type
Instead of using a custom Args* iterator, use the one belonging to the
map type directly.
2020-05-22 01:59:36 +01:00
Vincent Ambo
ee4637e3a2 refactor(3p/nix/libexpr): Use absl::btree_map::merge for '//'
Instead of doing some sort of inline merge-sort of the two attribute
sets, use the attribute sets merge function.

This commit alone does not build and is not supposed to.
2020-05-22 01:52:03 +01:00
Vincent Ambo
28e347effe refactor(3p/nix/libexpr): Use absl::btree_map for AttrSets
This is the first step towards replacing the implementation of
attribute sets with an absl::btree_map.

Currently many access are done using array offsets and pointer
arithmetic, so this change is currently causing Nix to fail in various
ways.
2020-05-21 19:21:55 +01:00
Vincent Ambo
a162f4e825 refactor(3p/nix/libexpr): Use std::string as qualified type
Replaces most uses of `string` with `std::string`.

This came up because I removed the "types.hh" import from
"symbol-table.hh", which percolated through a bunch of files where
`string` was suddenly no longer defined ... *sigh*
2020-05-21 05:43:22 +01:00
Vincent Ambo
b97307056d docs(3p/nix/libexpr): Add clarifying comments to SymbolTable 2020-05-21 05:42:20 +01:00
Vincent Ambo
c395a48be2 fix(3p/nix): Compatibility with updated SymbolTable
The functions in SymbolTable have been renamed to match the Google
Style guide, and some debug-only functions have been removed.
2020-05-21 05:09:23 +01:00
Vincent Ambo
97e85f94e5 refactor(3p/nix/libexpr): Use absl::node_hash_set in SymbolTable
This replaces the previous use of std::unordered_set with
absl::node_hash_set.

This type was chosen because the current implementation requires
pointer stability.

This does not yet touch the 'Attr' struct.

As a bonus, the implementation of the SymbolTable struct is now
consolidated into a single header/implementation file pair.
2020-05-21 05:09:23 +01:00
Vincent Ambo
00017ace04 feat(3p/nix): Wrangle Meson/Nix/CMake into (temporary) submission
Meson is unable to use CMake in Nix to determine the internal
structure of the Abseil libraries.

This commit adds an explicit list of most of the Abseil targets that
are relevant (so far) and bundles them into a list that is linked
together.
2020-05-21 04:50:49 +01:00
Vincent Ambo
a3b48e6562 chore(3p/nix): Minor fixes to get rid of warnings 2020-05-21 00:24:12 +01:00
Vincent Ambo
9dfdf16a8d fix(3p/nix): Fix build of derivation if cmake is present
cmake automatically runs a configure hook which breaks the build,
since this isn't actually a cmake project. This hook is now disabled.

Additionally Abseil's sources are linked to an absolute derivation
path when the build launches, as opposed to the relative path used for
development builds.
2020-05-21 00:24:10 +01:00
Vincent Ambo
43677021e3 refactor(3p/nix): Apply clang-tidy's performance-* fixes
This applies the performance fixes listed here:

https://clang.llvm.org/extra/clang-tidy/checks/list.html
2020-05-20 22:58:43 +01:00
Vincent Ambo
689ef502f5 refactor(3p/nix): Apply clang-tidy's readability-* fixes
This applies the readability fixes listed here:

https://clang.llvm.org/extra/clang-tidy/checks/list.html
2020-05-20 22:27:37 +01:00
Vincent Ambo
d331d3a0b5 refactor(3p/nix): Apply clang-tidy's modernize-* fixes
This applies the modernization fixes listed here:

https://clang.llvm.org/extra/clang-tidy/checks/list.html

The 'modernize-use-trailing-return-type' fix was excluded due to my
personal preference (more specifically, I think the 'auto' keyword is
misleading in that position).
2020-05-20 04:33:07 +01:00
Vincent Ambo
fed31b2c9b feat(3p/nix): Add some initial Abseil build targets
These make it possible to link to Abseil strings.
2020-05-20 04:02:23 +01:00