I missed calling store_->assertStorePath on paths in several daemon
proto handlers, which the previous implementation did.
Change-Id: Ifad6eeb03b5a5babec7b4bcf7aca060813f15bb7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1272
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
I got the message proto wrong on this one as well - it needs both a path
and a signatures.
Change-Id: I9a489b1285bda61c15b2a3b47d9cfc3b50e387da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1270
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Implement the proto handler on the server side for
Worker::BuildDerivation. This includes several additions to the proto
which I had missed on the first pass, including the actual proto
definition for the Derivation itself and a few sequence number
reorderings which are fine because this is all provisional and not
deployed yet.
A couple things to note
- I implemented a couple constructors for nix classes that initialize
themselves based on their proto variants, which felt nice and didn't
end up causing any issues.
- I've made the conversions between the enum types in nix and in proto
explicit via switch statements rather than using a static_cast, out of
an abundance of caution that the error would get mismatched in the
future and we'd convert the wrong thing to the wrong thing - this is
verbose, but exceptionally future proof.
Change-Id: Iecf6b88e76bc37e49efa05fd65d6cd0cb0deffed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1249
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: Kane York <rikingcoding@gmail.com>
This silences a -Winconsistent-missing-override
Change-Id: I2492a8ab3c0b0a33a794f012bfee6bbd0c2f0a8e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1271
Reviewed-by: Kane York <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
Interpolating a path into a string will copy the referenced path into
the Nix store, so this got a dependency on all of src. By first
constructing a path to the src/proto directory using the + operator,
and then interpolating it, we limit what is copied to the store, and
therefore what code triggers a nix-proto-srcs rebuild.
Change-Id: I8dd750f6bc5902b74ffb56470bc8a5f2c01c8cf1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1263
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
This store implementation is required in all unit tests that use the
evaluator.
Change-Id: I1cfe8cecab8722cd66dc803747821a2be2b2619f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1269
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
This change does away with the previous special-casing of lists of
certain element sizes, and the use of raw C-style arrays.
Lists are now backed by a std::vector of nix::Value*, which uses the
traceable GC allocator.
This change is unfortunately quite noisy because the accessor methods
were updated/removed accordingly, so all callsites of Nix-related
lists have changed.
For some operations in primops.cc where keeping the previous code
structure would have been more difficult with a "proper" vector, the
implementation has been replaced with std::vector methods. For
example, list concatenation now uses appropriate range inserts.
Anecdotally the performance of this is about equal, to even slightly
better, than the previous implementation.
All language tests pass and the depot paths I've used for testing
still evaluate.
Change-Id: Ib5eca6c0207429cb323a330c838c3a2200b2c693
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1266
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: Kane York <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
This has several advantages:
* we can ensure that the vector is traced by the GC
* we don't need to unsafely allocate memory to make an Env
Note that there was previously a check about the size of the
environment, but it's unclear why this was the case (git history
yielded nothing interesting) and it seems to have no effect.
Change-Id: I4998b879a728a6fb68e1bd187c521e2304e5047e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1265
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: Kane York <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
This breaks tooling for some people, and we aren't actually using the
git submodules.
Change-Id: I5b4dfd4ad76bf72e9dbc7de29f17f28ebf6b8ba0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1255
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
all of the executables that get built during regular development depend
on this being set to a directory that contains the nix directory -
previously we had been doing it manually every time, this automates it.
Change-Id: I4c957c0abf0a92ca7122a47d3b141a8ede280e13
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1258
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Configures the CMake build to load & run the GoogleTest tests.
I (grfn) also updated this to get the tests running as part of the nix
derivation, which required defining our own manual configurePhase and
installCheckPhase, rather than depending on the one provided by stdenv.
Not doing this would cause cmake to attempt to *run* the tests as part
of the buildPhase, which wouldn't work because the dynamic libraries
hadn't been put into a place where the test executables knew where to
find them. We're not sure *why* this fixes it, and for some reason
fixing this also breaks the automatic behavior of nixpkgs of passing
-j$NIX_BUILD_CORES -l$NIX_BUILD_CORES to make, but that's eaasy enough
to fix manually in a preBuild
Paired-With: Griffin Smith <grfn@gws.fyi>
Change-Id: I79d61854a3ff47301cdce8a40c76820a97bdf901
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1240
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Also adds the missing check_contents field to the VerifyStoreRequest
proto message, since it was missed in the original pass. This is done
using a renumbering, which is fine in this case since the proto hasn't
been deployed yet
Change-Id: I92bf4e48a71a25ae02ae02b3deaf6e7c71fe5da7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1237
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: Kane York <rikingcoding@gmail.com>
Change-Id: I8b1d84799a608a516d0b4980022a7edd545a1ca1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1235
Tested-by: BuildkiteCI
Reviewed-by: Kane York <rikingcoding@gmail.com>
Change-Id: I243d67b0bce29d54c7d6e08f5eee70bd395cf9a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1234
Tested-by: BuildkiteCI
Reviewed-by: Kane York <rikingcoding@gmail.com>
Backported from:
b3e5eea4a9fcd048a526
Intentionally skipped because we have not backported the JSON changes:
9f46f54de4
Did not apply changes ni primops.cc, because those look suspect and
are also based on something that we don't have in our tree.
Change-Id: I837787ce9f2c90267bc39fce15177980d209d4e9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1253
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
We want to *trace* the 'Value *' arrays, not garbage-collect them!
Otherwise the vectors/maps can end up pointing to nowhere.
Backported from:
10e17eaa58
Change-Id: I30dc94caa80c9d982e7a14bc67ba2d065e8203aa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1252
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Previously the memory would occasionally be collected during eval since
the GC doesn't consider the member variable as alive / doesn't scan the
region of memory where the pointer lives.
By using the traceable_allocator<T> allocator provided by Boehm GC we
can ensure the memory isn't collected. It should be properly freed when
SourceExprCommand goes out of scope.
Backported from:
d2c371927e
Change-Id: I1f7c745dbc66c7164bee50f4d9b0d437dbc7dd51
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1251
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
This is a blocker to enabling the linter.
Change-Id: I9f8d3cc19c7539086f53474a505362230fc56c04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1245
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
This is a bad idea, it shouldn't exist, nixpkgs doesn't use it.
Change-Id: Ic4d1b936d8f059d5c40f0567af165b02427d7e36
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1241
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
These are the queries that are handled in the confusing case statement
in the old daemon implementation, because they have very similar
structure.
Change-Id: Ie7143354f66cef4336dff8072ede9a56271a7e89
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1228
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
What it does I will never know. It takes a path and returns a path.
Something happens in the middle.
Change-Id: I499a9df700e5b954c9aaf6d694753ff34e773dfd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1210
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
These files were always implicitly depended upon by worker-protocol.hh, but just so happened also always be included in places where the other ones were already present.
We are likely getting rid of this file sooner rather than later, but in
the meantime this will cause clangd to shut up while I'm editing the
file.
Change-Id: I0d765f8b86828b6612e5483417e452221ea3c5b1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1207
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
Add QueryValidPaths and QuerySubstitutablePaths, both of which filter a
list of paths based on a set of criteria.
Change-Id: I6aa4647efe82b82dc9582a311643d5f9b6d521d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1203
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Replaces the previous uses of the (ordered!) std::map and std::set
with absl::flat_hash_{map|set}.
After some careful reading it seems that there is actually no
ordering dependency on these types, and the (drop-in) replacements
perform slightly better.
Overall this is not fixing a bottleneck, just a driveby thing.
Change-Id: Ided695dc75676bd58515aa9382df0be0a09c565e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1220
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: isomer <isomer@tvl.fyi>
Tested-by: BuildkiteCI
Several definitions of functions declared in eval.hh were previously
implemented in parser.y, this moves them over to parser.cc.
While this still isn't a reasonable place to keep them, the long-term
fix is more likely to be that eval.hh needs to be split up.
Before we get to that point however, this already gives us the ability
to use tooling with this code.
Change-Id: If06fb655325fe281564047ffab0a0a640428a0ee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1219
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Moves several of the static helper functions into a new parser.cc
file.
Once the rest of the code is usefully extracted, these will be moved
to a private namespace.
Change-Id: I0d7b53dcefe31bb5c6bad3ad7f5fcb48276bf799
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1218
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
First step (of many?) towards extracting all the inline code from the
Yacc file and keeping it somewhere more accessible instead.
Note that none of this code has previously been touched by a linter or
formatter, pretty much ever, so as it is extracted it also undergoes
similar changes to the whole codebase after the initial fork.
Change-Id: If3b7181f22e3b3fd8c58dfa9befa7ee2896ea06d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1217
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Instead of manually iterating over the two bindings to be combined,
this adds a new static method on the Bindings class which merges two
attribute sets by calling the range insertion operator over them.
In some anecdotal tests, this can lead to a ~10% speed bump -
depending on the specific operation.
Change-Id: I5dea03b0589a83a789d3a8a0fc81d0d9e6598371
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1216
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Changes the derivation name & README overview to say "Tvix" instead of
"tazjix".
The previous name was mostly intended as a joke, and a way for me to
distinguish output paths. It's certainly not the intention to have a
portmanteau with my name here, especially now that several people are
contributing to the fork.
Change-Id: Icface5484d52355111eca23b2f6bd3b9e5567275
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1212
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
They're not mentioned anywhere on the client or the server.
Change-Id: Ia78c8bde49326f6bf69b4aeded083cfd3235131f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1188
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
On the chopping block this time:
* requests/responses that return one or more store paths, and contain
nothing else, have been changed to use the same types (StorePath and
StorePaths, respectively)
* QuerySubstitutablePathInfos has been added. It should be noted that
legacy Nix has two versions of this call, one that only queries a
single info (deprecated) and one that queries multiple. We have only
implemented the latter.
* QueryDerivationOutputs has been added.
Change-Id: Iccc9041e7064e141cf593467ecdcc327581c4056
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1186
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Slight performance optimisation of nix::printString by copying chunks
of the input string which do not need escaping as contiguous blocks.
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: I48bad90c8f2831ae4524c814a12b1982989922f9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1184
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: glittershark <grfn@gws.fyi>
A significant fraction of all created attribute sets are empty; hence
this is an easy optimisation to make.
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: I0884194d04c1ee95b2b239a253515f2152bc0856
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1179
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Add tools.perf-flamegraph, which collects the base case execution of
perf piped through stackcollapse-perf and flamegraph to flamegraph the
execution of an external command via perf.
Change-Id: I671fe254dc374b6cd7deca2d3bdea266164de025
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1176
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
Change-Id: If580b76b7b7170ce7840d3a97c3a547c29405a6f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1174
Tested-by: BuildkiteCI
Reviewed-by: Kane York <rikingcoding@gmail.com>
This reverts parts of the CLs splitting the backing implementation for
Bindings and moves back to only the BTreeMap-backed implementation.
Our evaluation has indicated that the Vector-backed implementation
does not match the performance of the plain array used upstream, and
in my view the complexity introduced by it is not worth the relatively
small (single-digit percentage) performance increase with a
pivot-point close to the number of attributes yielded by
stdenv.mkDerivation.
Going forward we will trial implementations of attribute sets backed
by HAMTs, and investigate other mechanisms of speeding up the language.
Some changes from the previous CLs are retained, for example the
removal of insert_or_assign and the passing of capacity.
Change-Id: I6eb4b075b453949583360755055c21a29d7ff642
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1172
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
Since one of the two implementations essentially uses the same shape
as the upstream Bindings, we backport their merge sort implementation
to ensure that we're doing the same thing semantically.
Paired-With: Luke Granger-Brown <git@lukegb.com>
Paired-With: Vincent Ambo <mail@tazj.in>
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: I0d865897991eec0c4dd84d9bd0415cd1ca437792
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1162
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
We accidentally returned the incremented iterator in the
post-increment, this fixes it.
Paired-With: Luke Granger-Brown <git@lukegb.com>
Paired-With: Vincent Ambo <mail@tazj.in>
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: I36c79eb56359bb12a78ad3489e7d7d2eb2053510
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1140
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
This function in never called, so let's just remove it
Paired-With: Luke Granger-Brown <git@lukegb.com>
Paired-With: Vincent Ambo <mail@tazj.in>
Change-Id: I79125866254d90dd0842bc86830d2103ac313cb6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1125
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: isomer <isomer@tvl.fyi>
To aid in making the decision of where to (currently just statically)
use a vector or btree as the backing implementation, add an extra
constructor argument to Bindings::NewGC for a capacity, and use
a (currently hardcoded at 32, for no good reason other than it felt like
a reasonable number) pivot to switch between our possible backing
implementations. Then, update all the call sites where it feels
reasonable that we know the capacity statically to *pass* that capacity
to the constructor.
Paired-With: Luke Granger-Brown <git@lukegb.com>
Paired-With: Vincent Ambo <mail@tazj.in>
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: I1858c161301a1cd0e83aeeb9a58839378869e71d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1124
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: isomer <isomer@tvl.fyi>
Add an alternative impl of the now-abstract Bindings base class that is
backed by a std::vector, somewhat similar but stylistically a little
superior to the array-backed implementation in upstream nix. The
underlying iterator type in BindingsIterator is now backed by a
std::variant that we std::visit an overload over in order to implement
the various bits of the iterator interface.
Paired-With: Luke Granger-Brown <git@lukegb.com>
Paired-With: Vincent Ambo <mail@tazj.in>
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: I7fbd1f4d5c449e2f9b82102a701b0bacd5e80672
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1123
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
To pave the way for the thing we want to do eventually which is use a
linear-time array for bindings (aka attribute sets) that are statically
known to be small enough to get a performance benefit from doing so,
make the Bindings class abstract, and define a BTreeBindings class that
inherits from it and is (currently always) returned from the static
initializer. The idea is that we'll have an ArrayBindings class as well
later that we can dispatch to conditionally based on an optional
"capacity" parameter or something like that.
There was some difficulty here in getting the iterator to work - the
approach we settled on ended up making a concrete BindingsIterator class
which will wrap a std::variant of either a btree iterator or something
else later, but right now just wraps a btree iterator.
Paired-With: Luke Granger-Brown <git@lukegb.com>
Paired-With: Vincent Ambo <mail@tazj.in>
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: Ie02ca5a1c55e8ebf99ab1e957110bd9284278907
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1121
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Having a default constructor for this causes a variety of annoying
situations across the codebase in which this is initialised to an
unexpected value, leading to constant guarding against those
conditions.
It turns out there's actually no intrinsic reason that this default
constructor needs to exist. The biggest one was addressed in CL/1138
and this commit cleans up the remaining bits.
Change-Id: I4a847f50bc90e72f028598196592a7d8730a4e01
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1139
Reviewed-by: isomer <isomer@tvl.fyi>
Tested-by: BuildkiteCI
This has been providing a warning and it's been bothering me.
Change-Id: I0548059950ec4250d7cf0938f9deae09eafe593c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1141
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: isomer <isomer@tvl.fyi>
Tested-by: BuildkiteCI
nix:AttrName was one of the few classes that relied on the default
constructor of nix::Symbol (which I am trying to remove in a separate
change).
The class essentially represents the name of an attribute in a set,
which is either just a string expression or a dynamically evaluated
expression (e.g. string interpolation).
Previously it would be constructed by only setting one of the fields
and defaulting the other, now it is an explicit std::variant.
Note that there are several code paths where not all eventualities are
handled and this code is bug-for-bug compatible with those, except
that unknown conditions (which should never work) are now throwing
instead of silently doing ... something.
The language tests pass with this change, and the depot derivations
that I tested with evaluated successfully.
Change-Id: Icf1ee60a5f8308f4ab18a82749e00cf37a938a8f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1138
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
Implements the fairly common lambda overload class used for std::visit
over variants and other things that require groups of callables.
Change-Id: Ia7448b7e1bd349b4909974758e6e6303a80d86d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1137
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: edef <edef@edef.eu>
This also incidentally includes a fix for _not_ loading fonts from
Google Fonts, but I don't really care about that.
Change-Id: I6e00791d0ba06cb1e3c029e1b8617c33000d2ab1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1041
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Backport of b244e65cdb
Fixes https://github.com/NixOS/nix/issues/3175
------
Original fault description by puck:
I suspect I found the issue: The Nix Command structs are stored on the
heap (using ref<T>, a wrapper around std::shared_ptr<T>), which means
that any pointers that the NixRepl struct contains are eligible to be
reaped by the GC. This includes, but is not limited to, the Env
pointer, which seems to cause most of the random segfaults, or random
other values in the environment, which seems to be what @arianvp
experienced too.
Change-Id: I376d7cfd432daaa6f1fbbf77788ff048082f34e5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1001
Tested-by: BuildkiteCI
Reviewed-by: Kane York <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
Previously the nix-daemon would crash if a user fed it invalid store
paths for drv files. The crash was due to the changed assertion
triggering. Whenever that assertion would hit the nix-daemon process
along with all it's current childs (running builds from all users) would
be interrupted.
Before this patch:
$ nix-store --realise /nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.drv
don't know how to build these paths:
/nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.drv
error: unexpected end-of-file
< nix-daemon terminates >
With this patch:
$ nix-store --realise /nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.drv
don't know how to build these paths:
/nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.drv
error: path '/nix/store/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.drv' is not a valid store path
< nix-daemon does *NOT* terminate >
Change-Id: I01c5048c8a43a8b9154bdeb781d05b7744869ec0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/981
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
This takes us about halfway through worker-protocol.hh
I have left out the documentation strings for some of these items
because I don't feel that I can currently write an unambigous
description of them. For now I am just attempting to match the types.
Change-Id: Iae64b1676152fe4ea069e2021b75ad76465cf368
Reviewed-on: https://cl.tvl.fyi/c/depot/+/960
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
- X-Forwarded-Proto support so it knows it's behind TLS
- Remove extraneous logs and just log to stdout so it's caught be systemd
Change-Id: I650777bbfd24a1922f26967ffff7da06d14b6639
Reviewed-on: https://cl.tvl.fyi/c/depot/+/952
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
This reverts commit 8b58593193.
Reason for revert: Google has fixed the upstream issue and this is not necessary anymore.
Change-Id: Ib1f7cfbc0e7e0997239514e9be75d2deab8c3c67
Reviewed-on: https://cl.tvl.fyi/c/depot/+/961
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Integrate the keyboard layout for my Ergodox EZ, which is a layout for
QMK, with the depot, including exposing several cross-compilation
packages necessary to compile qmk for avr in third_party.
Change-Id: Idd43169a0a3cf0be2bd1a578fdaff70388a58bfc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/947
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
There is a temporary issue affecting the retrieval of discovery
documents in the Google Mail API:
https://issuetracker.google.com/issues/160441983
It's possible to work around this by hardcoding the document instead
of retrieving it over the network, as all other API calls still work
as they should.
This does exactly that by generating a patch to apply to lieer, from
the file checked in to the depot with this commit.
This workaround should be reverted once Google has fixed the issue
upstream.
Change-Id: I0063d0bc67753ffa5261c2aa059c7bfd09136ba0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/945
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
This isn't actually used yet, but forces the protos to be included in
the build which is useful for iteration.
Change-Id: I2abcaf297f34ae741f00ad0c929b226d5603c9d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/928
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Adds dependencies on the gRPC & protobuf libraries, and implements Nix
code to generate the C++ sources from the included proto definitions.
This is theoretically supported via CMake, but practically doesn't
work and I don't care to debug why.
Doing it like this lets us instead add a CMake library target for our
proto definitions based on the sources generated by Nix.
Pros:
* no need to deal with the gRPC CMake mess
* it works!
Cons: * iteration requires nix-shell restart
Change-Id: Ie1fe9807fc96c49cb8f7161ba59d093456062b15
Reviewed-on: https://cl.tvl.fyi/c/depot/+/927
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Adds initial gRPC definitions for the Nix worker protocol, which is
currently defined messily across the following files:
src/libstore/worker-protocol.hh
src/libstore/remote-store.cc
src/nix-daemon/nix-daemon.cc
The protocol definition is basically a big enum with the signatures of
the calls being implicit in the various client/server implementation
functions.
The definitions in this file are slowly reversed from these implicit
signatures, and are likely to contain an error or two which will be
weeded out when this is taken into use.
Only a handful of the calls are included in this commit, it is
intended to get us up and running first.
Change-Id: Ibc9b2ab4b91a064c8935f09f7ac72bb8150fb476
Reviewed-on: https://cl.tvl.fyi/c/depot/+/926
Reviewed-by: isomer <isomer@tvl.fyi>
Tested-by: BuildkiteCI
gRPC 1.29.0 can not be built with our precompiled libs because it
accidentally overrides the C++ standard specified by the users.
This was fixed in 1.30.0.
Change-Id: I9a0390b3f5ec8dae6c295562f3a94b3dfa681dd1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/925
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Same as cl/921, it seems that the trick to making this work is indeed
overriding the C++ standard used in the dependency.
Change-Id: I3c5984d71014d774c161ecc283844f504fd44719
Reviewed-on: https://cl.tvl.fyi/c/depot/+/922
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
This didn't work previously ... but now it does. I think setting the
standard explicitly is what did the trick, but it's slightly unclear
to me why.
Either way this means that Abseil is no longer constantly getting
recompiled when building Nix, which is nice.
Change-Id: I377f7b68bf1ef9045df6a2eee8fdd0c92f243547
Reviewed-on: https://cl.tvl.fyi/c/depot/+/921
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
These bits are no longer required with the hashmap-backed
implementation of attribute sets.
Change-Id: I8b936d8d438a00bad4ccf8e0b4dd719c559ce8c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/912
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Add expression for building haskell-language-server, based on a vendored
version of https://github.com/korayal/hls-nix with hashes updated to
work with our nixpkgs version and ghc 8.8.3. Also add that to CI
builds, so whitby will build it for me (thanks whitby).
Change-Id: I373f8a7cb67974b8aa043b116436c074591b8d57
Reviewed-on: https://cl.tvl.fyi/c/depot/+/897
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: BuildkiteCI
At present, we don't return HTML titles if there's a trailing slash,
or a patchset. Instead, just consume the / and anything after it.
This also fixes /123, because this is HTTP redirected to the full path
*with a trailing slash* which otherwise wouldn't get the title
injected.
Change-Id: Idfd0e67752880a37dce0b400a3c1cfc53fac2912
Reviewed-on: https://cl.tvl.fyi/c/depot/+/859
Reviewed-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
This enables support for the Argon2 password hashing mechanism in
OpenLDAP. Note that we also need to configure the LDAP module to load
this, so this change is not yet sufficient for actually using Argon2
hashes.
Change-Id: I151b854b777daa924b22224a43851432a88a2760
Reviewed-on: https://cl.tvl.fyi/c/depot/+/830
Reviewed-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Tested-by: BuildkiteCI
... and remove a package that doesn't exist anymore (at this location)
from the nixpkgs allowlist.
Change-Id: I663c84c387fb04bb3b47448132ad768ed5352474
Reviewed-on: https://cl.tvl.fyi/c/depot/+/829
Reviewed-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Tested-by: BuildkiteCI
We have this nice `runExecline` now, so we don’t need to use
`runCommand` (which spawns bash) just to write a simple script.
Change-Id: I2941ed8c1448fa1d7cc02dc18b24a8a945b2c38b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/704
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: BuildkiteCI
I've done a small amount of investigation and settled on this as my
favorite gitignore source filter function out of the several that are
available.
Change-Id: Idf1f2f643acc7f8e44de6c0c8702b16e0d37face
Reviewed-on: https://cl.tvl.fyi/c/depot/+/762
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Add a few relatively uncontroversial patches to fix some broken packages
that I had developed for xanthous to the top-level third_party tree, so
they can be reused by other people in the monorepo
Change-Id: I68740477bda278c5dcc123080029ee4bd2cae37a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/740
Reviewed-by: Profpatsch <mail@profpatsch.de>
Reviewed-by: tazjin <mail@tazj.in>
The most trivial of all derivations. It is more useful than it looks.
Can be used to bind nix expressions (e.g. test suites) to a
derivation, so that `nix-build` does not crap itself.
Change-Id: I61c24d8c129c9505733161207f3c30e820f5b15e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/665
Reviewed-by: tazjin <mail@tazj.in>
This is a simple-stupid “unix import system” for nix, for referencing
binaries in `/bin/` by their name and lifting them to a Nix attrset.
Allows for simple aliasing of executable names.
Change-Id: Ifa23cb377201c3b08050c5026e9751e736afaf56
Reviewed-on: https://cl.tvl.fyi/c/depot/+/664
Reviewed-by: tazjin <mail@tazj.in>
This is a writer, similar to `pkgs.writeBashScript` or
`pkgs.writers.writePython3`.
The difference is that we can correctly write all execline scripts by
using nix lists of lists, so the user doesn’t have to care about
escaping arguments (like they have to in bash scripts with
`lib.escapeShellArg` for example).
Change-Id: I2f2874cf61170ddca07b89b692f762725f4a75dc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/625
Reviewed-by: Kane York <rikingcoding@gmail.com>
Reviewed-by: tazjin <mail@tazj.in>
This adds configuration which generates the structure expected for
Buildkite pipelines, which can then be dynamically ingested by
Buildkite when a pipeline is triggered.
Change-Id: I61e3dc3affb19c1f2550ef827fa73b17f8d8ae47
Reviewed-on: https://cl.tvl.fyi/c/depot/+/627
Reviewed-by: ericvolp12 <ericvolp12@gmail.com>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Create a store path where the executable `exe` is linked to
$out/bin/${name}. This is useful for e.g. including it as a “package”
in `buildInputs` of a shell.nix.
For example, if I have the exeutable /nix/store/…-hello, I can make it
into /nix/store/…-binify-hello/bin/hello with
`binify { exe = …; name = "hello" }`.
Change-Id: I600bdcd8f143bca2dd8dfbb165a9a5a8d6397622
Reviewed-on: https://cl.tvl.fyi/c/depot/+/624
Reviewed-by: tazjin <mail@tazj.in>
This script creates a pulseaudio sink that will cancel noise in audio
streams sent to it, and then move it on to the default sink.
This means that other people's crackling, static background, gulping,
keyboard sounds, fan whirring, construction noise etc. are removed.
Some preliminary tests on TVL suggest that this actually works. The
parameter might need some tweaking ("50" is just the default value),
as there is some occasional crackling at the beginning/end of a speech
segment, but this is already *much* better than before.
Change-Id: I9d4e2b39cfc2b878b4b7c5458788b8d46fb801af
Reviewed-on: https://cl.tvl.fyi/c/depot/+/577
Reviewed-by: nyanotech <nyanotechnology@gmail.com>
This is a real-time noise suppression plugin for pulseaudio. I want to
cancel other people's noise with this.
Change-Id: Ia3031435e0db35eb80b626ab60d7a62b892d295d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/576
Reviewed-by: Kane York <rikingcoding@gmail.com>
This reverts commit 368e8d1edd.
Reason for revert: Didn't mean to submit, and the phase is currently failing (which breaks the otherwise-functional derivation)
Change-Id: I515b2fb45188dc90f09ae2458453192487c74d71
Reviewed-on: https://cl.tvl.fyi/c/depot/+/581
Reviewed-by: tazjin <mail@tazj.in>
Add an installCheckPhase that runs the appropriate substituteAll on
common.sh and runs the lang.sh tests with the build artifacts in the
PATH.
Change-Id: I2df5a93b8f3ffdfdc194a0e7d6b6669ef520c345
Reviewed-on: https://cl.tvl.fyi/c/depot/+/561
Reviewed-by: glittershark <grfn@gws.fyi>
We don't want traces compiled out since they're an actual language
feature that're used in userspace - also their absence is breaking the
tests
Change-Id: Icaefca8f52e94001785f724fdc0c10a7586b24e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/562
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: Kane York <rikingcoding@gmail.com>
Reviewed-by: lukegbot <bot@lukegb.com>
This includes absl, which we install into the output, and boost and the boehm GC,
which are moved to propagated deps.
Change-Id: I8f9f9795ff92e26b2320359064241d7fd59c2d33
Reviewed-on: https://cl.tvl.fyi/c/depot/+/549
Reviewed-by: tazjin <mail@tazj.in>
This also installs the rest of corepkgs as a side-effect.
Change-Id: I67a42d45793d5e8fdad51c1f306eebf63e9c2868
Reviewed-on: https://cl.tvl.fyi/c/depot/+/548
Reviewed-by: tazjin <mail@tazj.in>
Also fixes the pkgconfig files to use the corresponding CMake variables.
Change-Id: I8095b8aff39ad91e592f3edc95555c9f1f1f153d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/545
Reviewed-by: tazjin <mail@tazj.in>
This ensures that we install both glog's .a and all the .so files we
generate into a single consistent output lib path (which is, err,
lib64, but whatever).
Change-Id: Ib6ac6eacf5f56e4b719cfb586db731efc122c31b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/544
Reviewed-by: tazjin <mail@tazj.in>
Without this patch, this board is not included in the pin map
overrides and audio basically doesn't work.
With this patch, the audio still doesn't work, but it can at least
correctly detect what is and isn't plugged in - so that's progress.
Change-Id: I66ca7d7a1e0e25a0212b9659381875ad4c590ffc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/542
Reviewed-by: lukegb <lukegb@tvl.fyi>
Because this is using %h, we're subject to the whims of the remote
in trying to make sure that we get a consistent commit hash length
in the VERSION stamp.
Change-Id: I716193c1440ec367880b6a5f7dfa4f85a11c19a9
The Go language authors have released an experimental version of Go
that has a type system: https://blog.golang.org/generics-next-step
This overrides the existing Go derivation to build the typed Go. The
next step is a buildTypedGo set of functions that wrap buildGo.
Change-Id: Idb8a4868bca003d821ed5cb324af633398faf002
Reviewed-on: https://cl.tvl.fyi/c/depot/+/443
Reviewed-by: lukegb <lukegb@tvl.fyi>
We can always revert this if we want it back.
Change-Id: I1332b6dd541199584b7b5b94a8651172d79e53a9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/442
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
- This imports the tvldb (actually a thing called 'paroxysm') code
from https://git.theta.eu.org/eta/paroxysm into the monorepo.
- Additionally, I did a nix thing, yay! \o/
(well, with tazjin's help)
- 3p/default.nix needed modifying to whitelist pgsql.
Change-Id: Icdf13ca221650dde376f632bd2dd8a087af451bf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/389
Reviewed-by: tazjin <mail@tazj.in>
I need it to deploy my website
Change-Id: I5df8d76d6e0a3d8892ae8bc69d2b46b310f147a5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/399
Reviewed-by: tazjin <mail@tazj.in>
This derivation can add arbitrary new Sublime syntaxes to bat's syntax
file, which is used by cheddar.
Included is a Prolog syntax. It is kind of mediocre, but better than
nothing.
Change-Id: I6ceecbc86a5dfba5886cd27bd5114721845446a7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/348
Reviewed-by: lukegb <lukegb@tvl.fyi>
This plugin just blindly assigns everyone and, as q3k has already
pointed out, just isn't particularly useful.
We might want to roll our own, for example:
19: 40:41 <+Remosi> I want the virtual owner thing, we could call it
Gerrit Workgroup Synthesizer Queuing, or gwsq for short.
Change-Id: Ib12a921ae4047ac6a734035dd0900c8964fb12d8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/350
Reviewed-by: riking <rikingcoding@gmail.com>
This builds git-bug, a distributed issue tracker that uses git as its
data store.
It also installs its man pages and shell completions. It is
recommended that users add git-bug to their system closure for these
extras to work, as they will not be picked up by the dispatch script.
Change-Id: I1595368e61b0bae8a9497abd023085cb90a521a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/345
Reviewed-by: lukegb <lukegb@tvl.fyi>
Without these changes, the NixOS module isn't able to use the new
Gerrit derivation.
These changes are already deployed as I needed to make them to get
Gerrit back up.
Change-Id: Iad3aa6158789a014134fddccd40b508b81486100
Reviewed-on: https://cl.tvl.fyi/c/depot/+/301
Reviewed-by: lukegb <lukegb@tvl.fyi>
This adds support for overriding the detected languages based on the
filename - we assume here that rules.pl will always map to a Prolog
file.
I could've overridden the entire default language to Prolog, since it's
unlikely that we'll have any Perl here, but given the relative
popularity of the two languages I opted to just override the file we
know we'll have (because it's used by Gerrit itself).
https://usercontent.irccloud-cdn.com/file/yhZZx1nd/highlighted_prolog.png
Change-Id: I26a7e6dab191e0b80a027b026f884020a1f07178
Reviewed-on: https://cl.tvl.fyi/c/depot/+/254
Reviewed-by: tazjin <mail@tazj.in>
highlight.js supports syntax highlighting .nix files, but the Gerrit
diff components doesn't map the .nix mimetype onto the nix language.
.nix appears to already be taken by another mimetype that isn't
Nix-related, but we just map that onto nix anyway.
https://usercontent.irccloud-cdn.com/file/AVhoPvrb/highlighted.png
Change-Id: I842b29c78355e5bec580e711e25d693284ab6f59
Reviewed-on: https://cl.tvl.fyi/c/depot/+/253
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: q3k <q3k@q3k.org>
gerrit.Watcher is a class which watches the Gerrit stream-events SSH
connection and produces events.
There's a basic CLBot binary as well, to demonstrate driving it to
produce messages on the logging output. It doesn't really do anything
else.
Change-Id: I274fe0a77c8329f79456425405e2fbdc3ca2edf0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/245
Reviewed-by: tazjin <mail@tazj.in>
This uses the actual Bazel build, using a variety of tricks and hacks to
make it actually work.
Bazel really wants to download linux binaries from the internet and run
them. In lieu of trying to fix the build system to not do this, we
instead put bazel inside an FHS environment, which allows the binaries
to find their dependencies.
We also have to patch a few things:
* We use build --nobuild instead of fetch, so we only fetch the
dependencies we actually need for the build and not, say, Windows
binaries.
* We don't remove rules_cc, because we need it as an external
dependency, not bundled.
* We do some manual fixes on the cache before packing, because we need
to remove some in-tree sources (so they don't cause the hash to break,
since the hashes differ each time they're generated), and also remove
some extraneous files.
* We explicitly turn off the repository and disk caches, because the
.bazelrc at the root of the Gerrit tree turns them on, with paths
pointing into the user's home directory.
* detzip is used instead of the zip binary for packing bower_components
into an archive. detzip doesn't create entries for directories, and
also doesn't store most metadata (timestamps, etc.), and uses store
(i.e. uncompressed) compression only. It also sorts the file tree
before writing them into the file.
Change-Id: I572c43f7175067ecb1b85cdf40dda13a52de1439
Reviewed-on: https://cl.tvl.fyi/c/depot/+/252
Reviewed-by: tazjin <mail@tazj.in>
detzip will be used in a patch for the Gerrit bower repository helper, which allows
us to get consistent hashes for the output of fetching the dependencies for the
Bazel build.
Change-Id: I6c87b19815b9d747064108aecbb57ed875d2623b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/251
Reviewed-by: lukegb <lukegb@tvl.fyi>
This adds a little tool that can be used to relay mail to Gmail (and
other SMTP servers). It is intended to be used by Gerrit, which is
incompatible with Gmail's SMTP servers.
Configuration has been tested by performing a few sends through the
tvlbot@tazj.in account.
Note that this is using the standard Gmail SMTP server. Using the
smtp-relay server relies on IP whitelisting, but camden.tazj.in has a
larger number of IPv6 addresses than can be whitelisted (the maximum
is 65k). This means that we are limited to 2000 mails per recipient
per day, which should be fine.
Change-Id: Ie43564d753030f5c800a9cdb4ae98292877d80dc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/101
Reviewed-by: edef <edef@edef.eu>
Clang treats function-like macros "correctly", in that, per the C11 spec:
"Each subsequent instance of the function-like macro name followed by a (
[...] is replaced by the replacement list [...]".
Additionally, fprintf is also permitted to be defined as a function-like
macro rather than as a true function: "Any function declared in a header
may be additionally implemented as a function-like macro defined in the
header [...]". The specification then suggests surrounding the name of the
function in parens to avoid this, which is the technique we use here to avoid
the function-like macro being invoked.
The other fix here is to use uintptr_t for some arithmetic, since Git
is expecting an int as the value here and not a pointer.
Having a colon in the path may cause issues, and having the hash
function indicated isn't actually necessary. We now verify the path
format in the tests to prevent regressions.
(cherry picked from commit c65a6fa86aef7bdf51fb4fba7bd31d265619ba3f)
This makes the paths consistent without relying on ordering.
Co-authored-by: edef <edef@edef.eu>
(cherry picked from commit 515c0a263e137a00e82f7d981284dbe54db23247)
Completes the switch from Meson to CMake for the core build system in
Nix.
Meson was added originally because someone else had already done the
work for integrating it in Nix and it was an upgrade from the previous
setup.
However over time it became clear that Meson is not quite mature
enough for projects like Nix that have occasionally peculiar
configuration constraints.
Some issues encountered with Meson (some of these are due to the Meson
setup in Nix):
* Difficulty with generating correct compile_commands.json for
external tools like clangd
* Difficulty linking to libc++ when using clang
* Ugly shell invocations for certain parts of the build system (I want
these to be gone!!!)
This CMake setup mimics the Meson configuration, but there are some
differences (some temporary):
* headers are now included separately for each library (see a previous
commit that changes includes appropriately)
* autoheaders-style configuration is currently hardcoded. Before
blindly copying this I want to evaluate how much of it actually exists
for portability concerns that I don't have (such as support for OS
X).
* Nix is built with libc++ by default.
* [libstore] SQL schema is now inlined via a generated header, not an
included string literal
Abseil is still built as part of this build, rather than an external
dependency, because it chokes on differently configured compiler
invocations.
Note that because of the move to libc++ an unwanted behaviour is
introduced: glog log messages no longer have a body. I have yet to
debug what is going on there.
Previously all includes were anchored in one global mess of header
files. This moves the includes into filesystem "namespaces" (if you
will) for each sub-package of Nix.
Note: This commit does not introduce the relevant build system changes.
This function was a custom (and inefficient in the case of
single-character delimiters) string splitter which was used all over
the codebase. Abseil provides an appropriate replacement function.
Replaces these functions with corresponding functions from Abseil,
namely absl::StripAsciiWhitespace and absl::SimpleAtoi.
In the course of doing this some minor things I encountered along the
way were also refactored.
This also changes the signatures of the various custom readFile
functions to use absl::string_view types.
It is considered bad form to use things from includes in headers, as
these directives propagate to everywhere else and can make it
confusing.
types.hh (which is includes almost literally everywhere) had some of
these directives, which this commit removes.
Suppose I have a path /nix/store/[hash]-[name]/a/a/a/a/a/[...]/a,
long enough that everything after "/nix/store/" is longer than 4096
(MAX_PATH) bytes.
Nix will happily allow such a path to be inserted into the store,
because it doesn't look at all the nested structure. It just cares
about the /nix/store/[hash]-[name] part. But, when the path is deleted,
we encounter a problem. Nix will move the path to /nix/store/trash, but
then when it's trying to recursively delete the trash directory, it will
at some point try to unlink
/nix/store/trash/[hash]-[name]/a/a/a/a/a/[...]/a. This will fail,
because the path is too long. After this has failed, any store deletion
operation will never work again, because Nix needs to delete the trash
directory before recreating it to move new things to it. (I assume this
is because otherwise a path being deleted could already exist in the
trash, and then moving it would fail.)
This means that if I can trick somebody into just fetching a tarball
containing a path of the right length, they won't be able to delete
store paths or garbage collect ever again, until the offending path is
manually removed from /nix/store/trash. (And even fixing this manually
is quite difficult if you don't understand the issue, because the
absolute path that Nix says it failed to remove is also too long for
rm(1).)
This patch fixes the issue by making Nix's recursive delete operation
use unlinkat(2). This function takes a relative path and a directory
file descriptor. We ensure that the relative path is always just the
name of the directory entry, and therefore its length will never exceed
255 bytes. This means that it will never even come close to AX_PATH,
and Nix will therefore be able to handle removing arbitrarily deep
directory hierachies.
Since the directory file descriptor is used for recursion after being
used in readDirectory, I made a variant of readDirectory that takes an
already open directory stream, to avoid the directory being opened
multiple times. As we have seen from this issue, the less we have to
interact with paths, the better, and so it's good to reuse file
descriptors where possible.
I left _deletePath as succeeding even if the parent directory doesn't
exist, even though that feels wrong to me, because without that early
return, the linux-sandbox test failed.
Reported-by: Alyssa Ross <hi@alyssa.is>
Thanks-to: Puck Meerburg <puck@puckipedia.com>
Tested-by: Puck Meerburg <puck@puckipedia.com>
Reviewed-by: Puck Meerburg <puck@puckipedia.com>
(cherry picked from commit c05e20daa1abb3446e378331697938b78af2b3d7)
Replaces the previous implementations which performed sorting with one
that instead walks through the map (which is already sorted) and
yields values from it.
This fixes a handful of language tests because the previous
implementation did not actually yield useful values on the new implementation.
In the change to the backing structure of attribute sets, the
requirement to manually balance the capacity of the structure went
away.
This is a) because Abseil's data structures manage this on their own,
and b) because the new Bindings class is allocated using `new (GC)`
rather than writing into a predefined memory area.
As part of this change functions related to the capacity were
deprecated and set to 0 values, which in turn caused the creation of
new attribute sets to return the same (mutable!) default value in
various cases, leading to "side effects" that caused evaluation
failures.
FWIW, I'm not sure if this optimisation had noticeable performance
impact, but while untangling libexpr it definitely doesn't help trying
to follow what it's doing - so bye, bye!
This wrapper derivation (which assumes that the depot is available at
~/depot) can be used to actually get clangd working with
//third_party/nix.
In my setup I can launch this with M-x eglot, followed by
env
CLANGD_FLAGS='--compile-commands-dir=/home/tazjin/projects/nix-build'
nix-shell -A third_party.nix --run 'nix-clangd' /home/tazjin/depot
This is closer to bug-for-bug compatibility with the previous version,
which would put new elements at the end of the array and (due to the
linear scan) return previous ones.
Emacs is currently subtly broken on nixos-unstable, but I don't care
about debugging that.
To work around it, this reintroduces the NixOS stable channel (20.03)
but as a separate attribute set from which attributes like Emacs can be
picked into //third_party.
The new attribute set API uses the iterators of the btree_map
directly. This requires changes in various files because the internals
of libexpr are very entangled.
This code runs and compiles, but there is a bug causing empty
attribute sets to be assigned incorrectly.
Instead of doing some sort of inline merge-sort of the two attribute
sets, use the attribute sets merge function.
This commit alone does not build and is not supposed to.
This is the first step towards replacing the implementation of
attribute sets with an absl::btree_map.
Currently many access are done using array offsets and pointer
arithmetic, so this change is currently causing Nix to fail in various
ways.
Replaces most uses of `string` with `std::string`.
This came up because I removed the "types.hh" import from
"symbol-table.hh", which percolated through a bunch of files where
`string` was suddenly no longer defined ... *sigh*
This replaces the previous use of std::unordered_set with
absl::node_hash_set.
This type was chosen because the current implementation requires
pointer stability.
This does not yet touch the 'Attr' struct.
As a bonus, the implementation of the SymbolTable struct is now
consolidated into a single header/implementation file pair.
Meson is unable to use CMake in Nix to determine the internal
structure of the Abseil libraries.
This commit adds an explicit list of most of the Abseil targets that
are relevant (so far) and bundles them into a list that is linked
together.
cmake automatically runs a configure hook which breaks the build,
since this isn't actually a cmake project. This hook is now disabled.
Additionally Abseil's sources are linked to an absolute derivation
path when the build launches, as opposed to the relative path used for
development builds.
This applies the modernization fixes listed here:
https://clang.llvm.org/extra/clang-tidy/checks/list.html
The 'modernize-use-trailing-return-type' fix was excluded due to my
personal preference (more specifically, I think the 'auto' keyword is
misleading in that position).
This last change set was generated by a full clang-tidy run (including
compilation):
clang-tidy -p ~/projects/nix-build/ \
-checks=-*,readability-braces-around-statements -fix src/*/*.cc
Actually running clang-tidy requires some massaging to make it play
nice with Nix + meson, I'll be adding a wrapper or something for that soon.
These were not caught by the previous clang-tidy invocation, but were
instead sorted out using amber[0] as such:
ambr --regex 'for (\(.+\))\s([a-z].*;)' 'for $1 { $2 }'
[0]: https://github.com/dalance/amber
These were not caught by the previous clang-tidy invocation, but were
instead sorted out using amber[0] as such:
ambr --regex 'if (\(.+\))\s([a-z].*;)' 'if $1 { $2 }'
[0]: https://github.com/dalance/amber
Previously these structs were declared anonymously inside of the -
anonymous - union. This is not actually supported by the C++ standard,
but is merely a compiler-specific extension.
Unfortunately untangling this required a forward-declaration of the
Value type.
This change was generated with:
fd -e cc -e hh | xargs -I{} clang-tidy {} -p ~/projects/nix-build/ \
--checks='-*,readability-braces-around-statements' --fix \
-fix-errors
Some manual fixes were applied because some convoluted unbraced
statements couldn't be untangled by clang-tidy.
This commit still includes invalid files, but I decided to clean them
up in a subsequent commit so that it becomes more obvious where
clang-tidy failed. Maybe this will allow for a bug-report to
clang-tidy.
Changes the configuration to regroup all includes. The include groups
will be (in this order):
1. (in .cc): Include of the corresponding header
2. Includes of C++ standard library headers
3. Includes of other external headers
4. Includes of local headers
Removes the activity transfer that was previously nulled out from the
daemon protocol completely.
This might actually break Nix completely, I haven't tried yet, but
that's fine because this will be replaced with gRPC.
The progress bar has lots of complexity for little benefit. The
previous activity tracking stuff has been deleted as part of the
logging refactoring and I am not going to implement support for this
again for now.
1. First of all, this doesn't work in nixpkgs. Per [1], gcc ignores `-L`
for purposes of `--print-file-dirs`, which breaks horribly on linux. But
if we don't pass extra dirs, meosn first just tries `-l...`, which does
work.
2. Even if it did work, `libdir` means where we are installing libs, not
where libs are expected to be found. Those are not necessarily the
same (again, nixpkgs), and even when they are and non-standard, it is
better to use DESTDIR or have a modified toolchain.
[1]: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87758
(cherry picked from commit a142164e746644e20f66908c156ca913bef4664f)
Reverts "Bump channel to a NixOS 20.03 release commit".
This reverts commit a629d7fdd4.
This commit moves to a 20.03 release commit, in which building
Chromium with VAAPI is broken.
These patches enable hardware-accelerated video decoding, which is
useful for Stadia.
The main issue with this is that Hydra doesn't currently cache
Chromium with these patches, which means that it is built from scratch
which takes in the order of 5 hours on an otherwise unused nugget.
Builds ffmpeg with CUDA Toolkit as a dependency, which includes a
library called "libnpp" that provides something related to hardware
accelerated video stream resizing.
v0v
Adds the proto definitions required for the Stackdriver Logging API.
This compiles, but I'm unsure whether it's actually correct because
there seems to be a lot of copy & paste in the build setup.
Updates the build process for googleapis in C++ to read the proto
sources from the GOOGLEAPIS_DIR environment variable (injected by Nix)
instead of attempting to download them at build time.
This adds very basic capability[0] and message tag[1] support to rcirc
which is used to implement support for the IRCv3 server-time[2] spec.
During connection setup, the server is asked to list its capabilities
and the `server-time` capability is then blindly requested from
it (the CAP handler code does not check whether server-time is
actually part of the listed capabilities). rcirc does not need to know
whether this negotiation succeeded, because server time tags will
either be sent or not.
By default rcirc prints all timestamps at current-time. A new variable
`rcirc-last-message-time` has been added which, if set, overrides this
timestamp. It is set by the message handler after parsing IRCv3 tags.
Thanks to William Cummings for nudging me in the direction of his post
about adding ZNC playback support to rcirc[4], from which some parts
of this code were taken.
This has been tested with IRCCloud's bouncers.
[0]: https://ircv3.net/specs/core/capability-negotiation
[1]: https://ircv3.net/specs/extensions/message-tags
[2]: https://ircv3.net/specs/extensions/server-time-3.2.html
This change makes cgit pass the current repo and vpath inside of the
repo on to the about cmd, which makes it possible for it to correctly
render `tree`, `log` and other links to the same vpath.
This removes the ASDF system definition for Gemma and switches the
code over to buildLisp.
The program builds (including some terrifying hacks to get the
frontend to work), but there are some bizarre runtime issues that I
need to debug.
