Commit graph

10949 commits

Author SHA1 Message Date
Eelco Dolstra
014436eb4a fix(3p/nix): ValueMap, VectorVector: Use traceable_allocator
We want to *trace* the 'Value *' arrays, not garbage-collect them!
Otherwise the vectors/maps can end up pointing to nowhere.

Backported from:
10e17eaa58

Change-Id: I30dc94caa80c9d982e7a14bc67ba2d065e8203aa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1252
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-17 20:04:14 +00:00
Andreas Rammhold
5218c83b9f fix(3p/nix): allocate the vSourceExpr via uncollectable memory
Previously the memory would occasionally be collected during eval since
the GC doesn't consider the member variable as alive / doesn't scan the
region of memory where the pointer lives.

By using the traceable_allocator<T> allocator provided by Boehm GC we
can ensure the memory isn't collected. It should be properly freed when
SourceExprCommand goes out of scope.

Backported from:
d2c371927e

Change-Id: I1f7c745dbc66c7164bee50f4d9b0d437dbc7dd51
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1251
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-17 20:04:14 +00:00
Kane York
6c98d3d81f fix(nix/buildGo/example): use correct package name
Change-Id: Ie191e4154afe5aa47bb9ea5322010db014a1fa42
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1246
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-17 20:01:29 +00:00
Kane York
af47a89d6f chore(3p/nix/proto): Fix style violations in new proto defs
This is a blocker to enabling the linter.

Change-Id: I9f8d3cc19c7539086f53474a505362230fc56c04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1245
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-17 20:01:29 +00:00
Kane York
39ab9d8fc6 feat(buf.yaml): add protobuf linter config
Change-Id: I0506d0625769caa9f9bcd623722440bebe8b28b5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1244
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-17 20:01:29 +00:00
Kane York
f6a9706b3b feat(3p/bufbuild): Import buf
Change-Id: Ibb35980ce3a398dd7af1c14df9eba84f0fa414b7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1243
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-17 20:01:29 +00:00
isomer
cbff4fab9e chore(whitby): += Isomer
Change-Id: I446ab16d009dc24340606ab2f411197af24d79c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1142
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-17 19:38:51 +00:00
Vincent Ambo
8d5857abbc chore(3p/nix): Remove builtins.exec
This is a bad idea, it shouldn't exist, nixpkgs doesn't use it.

Change-Id: Ic4d1b936d8f059d5c40f0567af165b02427d7e36
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1241
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-17 19:03:59 +00:00
Vincent Ambo
a2870d05a3 feat(3p/gtest): Bump version & override stdenv to libcxx
Change-Id: I19a34e80f51add5a7c6cde2650aae60c1919ad43
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1238
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-17 02:59:42 +00:00
Kane York
a556913079 chore(3p/nix): add glittershark to OWNERS
Change-Id: I71228383c8c7810ff8032e67bde0f45339809934
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1236
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-17 01:24:38 +00:00
Griffin Smith
678022ad59 feat(gs/emacs): Add capture template for email tasks
The link here doesn't work, which is frustrating.

Change-Id: I70f5770226b1c39a5401d09037cce31a52f5cada
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1225
Reviewed-by: lukegb <lukegb@tvl.fyi>
Tested-by: BuildkiteCI
2020-07-16 23:58:12 +00:00
Griffin Smith
d1b3514910 feat(3p/nix/nix-daemon): Implement Worker::QueryAllValidPaths handler
Change-Id: If9d2144ae872183abf47d84c94cc5c5fa23ced87
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1229
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 23:57:49 +00:00
Griffin Smith
ec859afe95 feat(3p/nix/nix-daemon): Implement Worker::QueryMissing handler
Change-Id: I004a0c7969e1fe27b844adfed2d3ba3e6c5279bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1227
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 23:25:25 +00:00
Vincent Ambo
5c9af8faee feat(3p/nix): Implement similar group of Worker::Query* handlers
These are the queries that are handled in the confusing case statement
in the old daemon implementation, because they have very similar
structure.

Change-Id: Ie7143354f66cef4336dff8072ede9a56271a7e89
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1228
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 23:20:15 +00:00
Vincent Ambo
32735eebdd feat(3p/nix/nix-daemon): Implement Worker::HasSubstitutes handler
Change-Id: I0b01b554eee4b4e31416aabb498ea3d8b8a8092b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1226
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 22:48:09 +00:00
Vincent Ambo
bb66262068 feat(3p/nix/nix-daemon): Implement Worker::IsValidPath handler
Change-Id: I741c2b9b58f234a21850640e2b0c071ff4441234
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1223
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 22:48:09 +00:00
Griffin Smith
f1aa62c9ee feat(gs/emacs): Install protobuf-mode
Change-Id: I711976b81b1063a79f15129973da8be12239751a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1224
Reviewed-by: lukegb <lukegb@tvl.fyi>
Tested-by: BuildkiteCI
2020-07-16 22:41:57 +00:00
Vincent Ambo
ba11727ce0 feat(3p/nix/proto): Add QueryMissing RPC call
Change-Id: I6a29c56a0f945afdef7434da539b4b91b2e4d2e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1222
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-16 22:10:22 +00:00
Vincent Ambo
afe99225b9 feat(3p/nix/proto): Add AddToStoreNar RPC call
Change-Id: I335630644205adb280d83e006baf9d497ac1ac95
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1221
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 22:10:22 +00:00
Griffin Smith
cf6c48ef47 feat(3p/nix): Add NarFromPath
What it does I will never know. It takes a path and returns a path.
Something happens in the middle.

Change-Id: I499a9df700e5b954c9aaf6d694753ff34e773dfd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1210
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 21:41:40 +00:00
Griffin Smith
c3e96bcfa2 feat(3p/nix): Add AddSignatures
Change-Id: Ic37409c7987c261dc3eb91382d40f458fae6ea0c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1209
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-07-16 21:41:40 +00:00
Griffin Smith
f4baaf79e2 feat(3p/nix): Add BuildDerivation proto
Change-Id: Ib66b4ff2da72677e74b6a713186c72f66d9f6f88
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1208
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 21:41:40 +00:00
Griffin Smith
db4ac8ea7c chore(3p/nix/worker-protocol): Add missing includes
These files were always implicitly depended upon by worker-protocol.hh, but just so happened also always be included in places where the other ones were already present.

We are likely getting rid of this file sooner rather than later, but in
the meantime this will cause clangd to shut up while I'm editing the
file.

Change-Id: I0d765f8b86828b6612e5483417e452221ea3c5b1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1207
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-07-16 21:40:58 +00:00
Griffin Smith
4468e411fd feat(3p/nix): Add VerifyStore
Change-Id: I6a0596e986867e94857da461c746a80750830f7b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1206
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 21:40:58 +00:00
Griffin Smith
3999439fbb feat(3p/nix): Add OptimiseStore
Change-Id: I0d254504801283e018eef41bd5ea9850b17c0d80
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1205
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 21:40:58 +00:00
Griffin Smith
b6cb681c0b feat(3p/nix): Add QueryValidDerivers
Change-Id: I52965bdbf143ae11201e1fafed1c25b3d8f1bee9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1204
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 21:40:58 +00:00
Griffin Smith
143ef52592 feat(3p/nix): Add path query ops
Add QueryValidPaths and QuerySubstitutablePaths, both of which filter a
list of paths based on a set of criteria.

Change-Id: I6aa4647efe82b82dc9582a311643d5f9b6d521d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1203
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 21:40:58 +00:00
Griffin Smith
863a77dd2a feat(3p/nix): Add QueryPathFromHashPart
Change-Id: I0a31557f4cf585d3e539e29ef1a07b443aeec9fc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1202
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 21:40:58 +00:00
Vincent Ambo
d470ec0d29 refactor(3p/nix/libexpr): Use Abseil collection types for parser state
Replaces the previous uses of the (ordered!) std::map and std::set
with absl::flat_hash_{map|set}.

After some careful reading it seems that there is actually no
ordering dependency on these types, and the (drop-in) replacements
perform slightly better.

Overall this is not fixing a bottleneck, just a driveby thing.

Change-Id: Ided695dc75676bd58515aa9382df0be0a09c565e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1220
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: isomer <isomer@tvl.fyi>
Tested-by: BuildkiteCI
2020-07-16 21:03:52 +00:00
Griffin Smith
1b42504a12 feat(gs/shell): Add "nix build local" alias
For when I'm building stuff I don't want to build on whitby.

Change-Id: Ic377fe0d68436a81ee479ff4aa029a51e0a5babf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1215
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-16 21:03:13 +00:00
Vincent Ambo
e05eb20102 refactor(3p/nix/libexpr): Move EvalState:* implementations out of parser.y
Several definitions of functions declared in eval.hh were previously
implemented in parser.y, this moves them over to parser.cc.

While this still isn't a reasonable place to keep them, the long-term
fix is more likely to be that eval.hh needs to be split up.

Before we get to that point however, this already gives us the ability
to use tooling with this code.

Change-Id: If06fb655325fe281564047ffab0a0a640428a0ee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1219
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-16 19:53:51 +00:00
Vincent Ambo
5ae2a2b847 refactor(3p/nix/libexpr): Extract static helpers out of parser.y
Moves several of the static helper functions into a new parser.cc
file.

Once the rest of the code is usefully extracted, these will be moved
to a private namespace.

Change-Id: I0d7b53dcefe31bb5c6bad3ad7f5fcb48276bf799
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1218
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-16 19:53:51 +00:00
Vincent Ambo
d579ceb783 refactor(3p/nix/libexpr): Extract parser header out of Yacc file
First step (of many?) towards extracting all the inline code from the
Yacc file and keeping it somewhere more accessible instead.

Note that none of this code has previously been touched by a linter or
formatter, pretty much ever, so as it is extracted it also undergoes
similar changes to the whole codebase after the initial fork.

Change-Id: If3b7181f22e3b3fd8c58dfa9befa7ee2896ea06d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1217
Tested-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
2020-07-16 19:53:51 +00:00
Vincent Ambo
cb3d967508 refactor(3p/nix/libexpr): Use range insertion to merge nix::Bindings
Instead of manually iterating over the two bindings to be combined,
this adds a new static method on the Bindings class which merges two
attribute sets by calling the range insertion operator over them.

In some anecdotal tests, this can lead to a ~10% speed bump -
depending on the specific operation.

Change-Id: I5dea03b0589a83a789d3a8a0fc81d0d9e6598371
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1216
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 18:51:47 +00:00
Vincent Ambo
1ba5aa293b feat(tazjin/emacs): Rotate left screen counterclockwise
Change-Id: I0318e394b32c341a04471f687ee4c6d5beb29cd1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1214
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 17:57:51 +00:00
Griffin Smith
4c340cbeb8 feat(gws.fyi): Reimport cert after renewing
Change-Id: Iebd5c9bc3a62838a862cc4b37b2f43f62ad8018e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1213
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-16 15:46:16 +00:00
Vincent Ambo
9a4a4058a8 style(3p/nix): Rename the project to tvix
Changes the derivation name & README overview to say "Tvix" instead of
"tazjix".

The previous name was mostly intended as a joke, and a way for me to
distinguish output paths. It's certainly not the intention to have a
portmanteau with my name here, especially now that several people are
contributing to the fork.

Change-Id: Icface5484d52355111eca23b2f6bd3b9e5567275
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1212
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-16 15:28:52 +00:00
Griffin Smith
a87de918af fix(gs/emacs): Prevent accidentally sending :q
If I try to send a message containing :q to ERC, just drop it.

Change-Id: I563044f0b837e2dbf4d5db94f9e906196ca91b49
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1211
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 15:20:19 +00:00
Griffin Smith
14e9e161c9 feat(3p/nix/proto): Add QueryDerivationOutputNames
Change-Id: I5452d0f76441a61d70031f567c9cb75249569c2a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1201
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-07-16 01:54:05 +00:00
Griffin Smith
17c0962802 feat(3p/nix): Add proto for QueryPathInfo
Change-Id: I10ec338ef2d5360954abdc5bb3d4789f34b031fc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1200
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-07-16 01:54:05 +00:00
Griffin Smith
0386fad78f chore(3p/nix): {query,clear}FailedPaths are obsolete
They're not mentioned anywhere on the client or the server.

Change-Id: Ia78c8bde49326f6bf69b4aeded083cfd3235131f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1188
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 01:51:19 +00:00
Griffin Smith
150e2bc908 feat(3p/nix/worker.proto): Add QueryAllValidPaths
takes nothing, returns paths

Change-Id: I4087b9b5a25d9edc76b97d0a0f553259693cca3a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1187
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-16 01:51:19 +00:00
Vincent Ambo
622250bff9 feat(3p/nix/proto): Add additional worker-protocol operations
On the chopping block this time:

* requests/responses that return one or more store paths, and contain
  nothing else, have been changed to use the same types (StorePath and
  StorePaths, respectively)
* QuerySubstitutablePathInfos has been added. It should be noted that
  legacy Nix has two versions of this call, one that only queries a
  single info (deprecated) and one that queries multiple. We have only
  implemented the latter.
* QueryDerivationOutputs has been added.

Change-Id: Iccc9041e7064e141cf593467ecdcc327581c4056
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1186
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 01:19:38 +00:00
Vincent Ambo
1bd08e73cd fix(tools/perf-flamegraph): Get rid of most 'unknown's
This makes two main changes:

1) It disables profiling of things that aren't part of the invoked
   process (-a). This isn't particularly useful in most cases because
   it will show things like Chrome's VP9 decoding while you're
   profiling something else.

2) It tells perf about the debug symbol format, which for some reason
   isn't DWARF2 by default.

Change-Id: Ida6a54b74f1ec635fec930db7e2a2aa039dd1443
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1185
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-07-16 00:37:29 +00:00
Vincent Ambo
eff165425a refactor(3p/libstore): Optimise nix::printString
Slight performance optimisation of nix::printString by copying chunks
of the input string which do not need escaping as contiguous blocks.

Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: I48bad90c8f2831ae4524c814a12b1982989922f9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1184
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 00:37:29 +00:00
Vincent Ambo
e92c824330 chore(tazjin/frog): Lower 'kernel.perf_event_paranoid' to -1
This allows access to profiling events by (almost?) all users.

Change-Id: I10e31ea9978f0b552a3973c29e98b43a6db45d1d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1183
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-07-16 00:37:29 +00:00
Vincent Ambo
4303cedde4 style(3p/nix): Minor style fix in DrvInfo::setMeta
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: I0655ecc675239b3d90e5adc305c3f37c1a904cf5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1181
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 00:37:29 +00:00
Vincent Ambo
d8ee0afda9 refactor(3p/nix): Compare attribute sets by pointer equality
Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: I418e9127c5d9d31559c59e461f17726ddbc051c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1180
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 00:37:29 +00:00
Vincent Ambo
5cd7cf93fc refactor(3p/nix): Use a static empty Bindings for 0-element attrs
A significant fraction of all created attribute sets are empty; hence
this is an easy optimisation to make.

Paired-With: Perry Lorier <isomer@tvl.fyi>
Change-Id: I0884194d04c1ee95b2b239a253515f2152bc0856
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1179
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-07-16 00:37:29 +00:00
Griffin Smith
04ae293360 feat(tools): Add script to flamegraph a command
Add tools.perf-flamegraph, which collects the base case execution of
perf piped through stackcollapse-perf and flamegraph to flamegraph the
execution of an external command via perf.

Change-Id: I671fe254dc374b6cd7deca2d3bdea266164de025
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1176
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-07-15 20:21:34 +00:00