Commit graph

697 commits

Author SHA1 Message Date
Vincent Ambo
082c006c04 merge(3p/absl): subtree merge of Abseil up to e19260f
... notably, this includes Abseil's own StatusOr type, which
conflicted with our implementation (that was taken from TensorFlow).

Change-Id: Ie7d6764b64055caaeb8dc7b6b9d066291e6b538f
2020-11-21 15:48:57 +01:00
Vincent Ambo
cc27324d02 chore(tvix): Use StatusOr API available in Abseil's version
The Abseil version of `StatusOr` does not come with the status macros
or the `Consume*` family of functions.

This change modifies the existing code to use the common denominator
of the API that is available between Abseil's own implementation of
`StatusOr` and the one from Tensorflow that we are currently using.

Change-Id: I5c37f68636a1fd54d153f95d7303ab8644abb774
2020-11-21 15:26:02 +01:00
Griffin Smith
8d24a975f1 fix(tvix): Use copy constructor to add strings to protos
Passing a string directly to add_paths like this causes the proto class
to take ownership over the string, meaning when it is destructed it
will *explicitly* free the string. When the string's actual owner (the
derivation struct) then goes out of scope it'll get freed again, causing
a double-free. This fixes that to instead use the copy constructor to
assign to a pointer to a new path, and covers the whole to_proto method
with a rapidcheck test.

Fixes: b/64
Change-Id: I84235bed9104ff430a0acf686d4a96f1e2e9a897
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2106
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-11-19 00:29:42 +00:00
Griffin Smith
20e206a3f6 fix(tvix): Use mutable field for assigning proto fields
This was accidentally using the proto arena API to assign the derivation
field of a BuildDerivationRequest. We *thought* this was causing a
double free, but even with this change that's still happening. That
said, this change is probably still a good idea since it's using the
proto API as intended.

References: b/64
Change-Id: I950a4eafb214e9113639ea54d2dfd4659b7be931
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2104
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-11-19 00:29:42 +00:00
Vincent Ambo
8a6d00aceb feat(3p/irccat): init at '06a985y4'
This is to be used for forwarding messages to our IRC channels.

Change-Id: I6362c6f50a22f504588a7161d41a170f4e7a6edc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2073
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-11-08 18:38:08 +00:00
Vincent Ambo
1d71c29919 chore(3p/exwm): Update derivation to build local sources
... this should also update my system EXWM.

Change-Id: Idfbbda67613ac678dc2d5f82533e1c6176ab4a28
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2072
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-11-08 01:26:53 +00:00
Vincent Ambo
6104f6514f chore(3p/emacs/exwm): Import from commit '0368127976bda29d35eed788edfe74644ecd3845'
Imported from 0368127976

git-subtree-dir: third_party/emacs/exwm
git-subtree-mainline: e84f9ef0ad
git-subtree-split: 0368127976
Change-Id: Id3af5610254180f42947d71265aad89def7c6a3d
2020-11-08 01:01:28 +01:00
Vincent Ambo
4f7bb40a31 chore(3p/rr): Upgrade to latest master with Zen workarounds
* bumps to HEAD of master
* include the zen_workaround script in the drv output

Change-Id: I8f5f33a61b9395295c02f3ea5e930a81b3ffa7ac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1990
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-09-23 15:23:25 +00:00
Griffin Smith
96dbf4d686 fix(tvix): Convert INVALID_ARGUMENT to InvalidPath
The code that calls queryPathInfoUncached explicitly catches the
InvalidPath exception and translates it into a null result - but the RPC
code was throwing a regular old Error for invalid paths. At some point
we should get rid of all the exception-driven control flow in this whole
thing, but in the meantime this gets us back to functional.

Change-Id: I2a38790ee0c691ab0c8394c7738d7693fa42aa10
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1980
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-09-16 20:17:01 +00:00
Kane York
fedbe693b9 refactor(3p/tvix): make Value.primOp a std::shared_ptr
Change-Id: I04cc42ae62d2ff71b07fde8e3710dcc9b4363a05
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1973
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-09-15 21:52:56 +00:00
Kane York
d9c7f32a9c fix(3p/tvix): properly zero the largest member of Value
The static_assert is present to force us to change the constructor when a member grows.

Change-Id: Ifa4f4a03eb7ce13cff109cf26ecf0032045905e9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1972
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-09-15 21:42:54 +00:00
Kane York
2c19bd6662 refactor(3p/tvix): convert NixList usage to shared_ptr
Starting to reclaim memory. Fairly simple mechanical replacement.

Change-Id: I6b4c5c5596729470d1a049eba61e69e8097decf4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1971
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-09-15 21:42:54 +00:00
Griffin Smith
381ce8a666 refactor(tvix): Make static strings constexpr string_views
Make all static std::strings constexpr std::string_views, and replace
concatenation with absl::StrCat where necessary.

Technically all of these are constant, so they really don't need to be
top-level statics - and since I'm trying to get rid of as much global
state as possible in preparation for making the nix daemon properly
multithreaded I figured I'd knock these out while I was at it.

Change-Id: Ibd3ad9ef68f0a0eacb135541b39fdb13dae042e1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1939
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-09-14 21:38:30 +00:00
Griffin Smith
65967bbbb1 refactor(tvix): Convert refLength to constexpr
This is constant, so let's make it a constexpr.

Change-Id: I6a9eb9f740979740f730ebe142d2cbf23dbcfd70
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1961
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-09-10 17:03:45 +00:00
Griffin Smith
5dcf64c128 fix(tvix): Make working flag thread_local
Since the daemon is running in threads now rather than forking a process
per connection (thanks to grpc) this static flag to prevent accidentally
initializing a new worker during a build is getting stepped on by
multiple threads. This converts it to thread-local, and also adds an
actual message to the check so that if it hits in the future we know
what's going on.

Fixes: b/58
Change-Id: I07a2f1582e56709c104f79935e5405fa24888f59
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1940
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: V <v@anomalous.eu>
2020-09-10 17:03:45 +00:00
Griffin Smith
2ca20da43f chore(tvix): dontStrip tvix build
This is mostly to help in tracing through the core dumps for b/58, but
is also generally a good idea while we're still in hardcore
debugging-mode.

Fixes: b/63
Change-Id: Ib816a2a98bf3b8afa6e8b255a74ccb93af6508c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1937
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-09-06 22:21:12 +00:00
Griffin Smith
8f4d89e429 feat(tvix): Implement EnsurePath RPC handler
Missed this one

Fixes: b/62
Change-Id: I6f3bbcd3e2a7db5786283761b57466da33a497f7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1936
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-09-06 21:41:39 +00:00
Griffin Smith
053a98a7d7 revert(tvix): "Disable sandboxing by default until its time comes"
This reverts commit a41c3dedb1.

Reason for revert: I believe its time has come.

Change-Id: I4133dc8a3b57b40b280646d1622ed7a0409c5246
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1941
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-09-06 21:12:11 +00:00
Griffin Smith
8b7dcfaeba feat(tvix): Write even more build logs to client
Write *all* build logs (essentially everything that wasn't a DLOG), not
just the stdout of the builder, to the client-bound log_sink_ that's
threaded through build.cc. These logs are significantly more useful on
the client, both when debugging tvix itself and (more importantly) when
building things with it.

To make this all work, this commit also refactors the passing-around of
the log sink in build.cc slightly - the main reference is now owned by
the Worker class, which already stores pointers to itself on the Goals
it creates, so those Goals can then load the log sink from the Worker.
This change was originally made to allow for logging to the log_sink
from instance methods of the worker, but actually feels significantly
more natural than storing the sink on the goal anyway.

Change-Id: I461685bc7f5c0a326054940ac5971b568dd43f8e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1925
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-09-06 21:12:11 +00:00
Griffin Smith
c4136447b3 refactor(tvix): Remove default buildMode arguments from libstore
These were not used anyway, and *may* have been causing a weird series
of problems leading to things sometimes aborting. Or maybe it has
nothing to do with it - the only correlation we've found is from
disassembling the output of a core dump. Regardless, this feels
stylistically like a good idea.

Change-Id: I288b50945e74ea90c118a21767fa4387c9f47f7d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1921
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-09-01 22:05:21 +00:00
Griffin Smith
adebfbbf24 feat(tvix): Print "Tvix" in the version string
Rather than printing "Nix" in the version string, print "Tvix". This is
cosmetically nice, but also is very useful when making sure I actually
have installed tvix on a system successfully.

Change-Id: Idd1a9954bd66509327f52b7457cfba2f090ab30c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1924
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-09-01 03:30:12 +00:00
Eelco Dolstra
785cb3a754 refactor(tvix): getEnv(): Return std::optional
This allows distinguishing between an empty value and no value.

Patch ported from upstream at
ba87b08f85

Change-Id: I061cc8e16b1a7a0341adfc3b0edca1c0c51d5c97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1884
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-09-01 02:21:53 +00:00
Griffin Smith
c5f3b12f04 feat(tvix): Support systemd socket activation
When the nix daemon starts up, first check (using sd_listen_fds) whether
we have been systemd socket-activated. If so, instead of passing the nix
daemon socket path to grpc, start a manual accept(2) loop, passing the
client file descriptors to grpc via AddInsecureChannelFromFd. There's an
open grpc issue at https://github.com/grpc/grpc/issues/19133 for
building support into grpc to do this automatically, but as of right now
this appears to be the only way to make this happen.

Making this happen, by the way, was a bit of a journey - at one point I
attempted to ServerBuilder's experimental AddExternalConnectionAcceptor
API, and that didn't work either - it appears that the final missing
piece to getting this working was explicitly fcntl(2)ing the client file
descriptors to set O_NONBLOCK before passing them into gRPC. With that
set, this all works inside of the test vm.

Fixes: b/56
Change-Id: I5d2ab2b5b02eb570249b30a9674e115c61b0ab0e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1882
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-09-01 01:07:11 +00:00
Griffin Smith
e472aa016e feat(tvix): Expose depot in the test vm
Expose depot to the test vm via NIX_PATH, so we can test the build of
depot packages from inside the test vm.

Change-Id: I26314634895ab1339f5715c0b0b261028eefdaa3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1923
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-09-01 01:07:11 +00:00
Griffin Smith
2bc564bd0d chore(tvix): Add riking to owners
Change-Id: Icefafe6967cb83a43c9e417a29ce8cc97fb3432d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1892
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <mail@tazj.in>
2020-08-31 23:01:44 +00:00
Griffin Smith
2279419ede refactor(tvix): Drop meson.build files
These are unused, and I'm not sure they ever actually worked.

Change-Id: I6bb6b4257b7815932581ded7929437dba80961c9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1885
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: tazjin <mail@tazj.in>
2020-08-31 15:38:16 +00:00
Griffin Smith
de84e491cf fix(tvix): localstatedir should be /nix/var
The various bits (including the unix socket that we listen on) append
/nix to this - so it isn't expected to be in there.

Change-Id: I0d95f2cc5641f610c1bdb03b036fac61dac9842d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1890
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-08-30 16:39:25 +00:00
Griffin Smith
1c64930e9f feat(tvix): Install profile scripts into the output
The nix package is expected to install /etc/profile.d/nix.sh and
/etc/profile.d/nix-daemon.sh into its output - these set the environment
variables expected by the client-side nix user commands, eg NIX_DAEMON.

Fixes: b/52
Change-Id: I5fb964c076f092d0ef7e1d49aca37875c866fb90
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1889
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-08-30 16:39:25 +00:00
Griffin Smith
74a8c3d359 fix(tvix): Chunk the AddTextToStore request
Rather than sending the entire AddTextToStore request along in a single
message, send it in a stream of chunks using the same metadata-first
approach we've been using for the other store gRPC requests. This fixes
a bug where certain builds could send more data than the maximum gRPC
request size (4194304 bytes, it would appear), resulting in a
RESOURCE_EXHAUSTED error.

The initial chunk size, which is currently constant but should be made
dynamic at some point in the future, has been chosen based on the IPC
bandwidth delay product for tazjin's desktop, rounded up.

Change-Id: I6f0232cdbc98653484816b39855126873fc59a03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1835
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-29 14:29:45 +00:00
Griffin Smith
059d90dd6d feat(tvix): Add scripts for local nix daemon testing
These are the scripts I use to test the nix daemon interaction with a
non-/nix store directory during development, copied almost verbatim from
my cmake build directory. As such, there's likely a *lot* of cleanup and
deduplication to be done here, but I'm committing these as is in the
hope that others can benefit from them somehow.

Change-Id: I42a10a85e6731fa2014c7ea9738224d678a8376b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1881
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-08-29 01:52:29 +00:00
Griffin Smith
a7cd92770d revert(glittershark): "Temporarily skip grfn's subtree"
This reverts commit 475d41f698. I'd like
my derivations back, please.

Changes necessary to get this working:

- Don't depend on `nixpkgs` being in the NIX_PATH for my website - it's
  not necessary anyway since emacs 27 is mainline now
- .skip-subtrees on things that shouldn't be evaluated anyway
- Get rid of system/pkgs, and move the one thing in there that *wasn't*
  already in third_party (alsi) to third_party
- Drop notifymuch for now - it's not working, and I'll probably get it
  landed in nixpkgs before I manage to get it working
- Add __readTree = true to my systems so they get built.
- explicitly disable ci for xanthous, which is failing to build and had
  been omitted previously

Change-Id: I20f5e81d6eb7ffe040091a08d75d0cb15304f707
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1864
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-28 21:24:36 +00:00
Griffin Smith
fb0528c5da fix(tvix): Pass all args when auto-calling a function with an ellipsis
The command line options --arg and --argstr that are used by a bunch of
CLI commands to pass arguments to top-level functions in files go
through the same code-path as auto-calling top-level functions with
their default arguments - this, however, was only passing the arguments
that were *explicitly* mentioned in the formals of the function - in the
case of an as-pattern with an ellipsis (eg args @ { ... }) extra passed
arguments would get omitted. This fixes that to instead pass *all*
specified auto args in the case that our function has an ellipsis.

Submitted upstream at https://github.com/NixOS/nix/pull/3965

Fixes: #46
Change-Id: I32b7ee0e5bacf75b2bc43a3f0796f533f4bd5959
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1863
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-08-28 01:18:19 +00:00
Griffin Smith
3bada1d41e refactor(tvix): Clean up MixEvalArgs internals
Previously, MixEvalArgs (a generic data type used to handle --arg,
--argstr, and -I arguments to `nix-build`, `nix eval`, etc.) was storing
the difference between --arg and --argstr by prepending a single
character (either 'E' or 'S') to the value of the arg. This is messy and
un-type-safe, so this commit refactors that to use a proper enum and a
std::pair, which allows us to add a switch and get totality checking.
yay, types!

Change-Id: I883f8db3136d05dda190fac0a1b494386c2ff87b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1862
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-28 01:18:19 +00:00
Vincent Ambo
eb3c4a3c11 fix(3p/haskell_overlay): Fix imports and remove unused file
Change-Id: I1973bc1b11442dc8150c3b07bd06a6f94c4c4380
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1851
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-26 23:49:32 +00:00
Vincent Ambo
7e6b3e1b42 chore: Add meta.ci=false to various broken projects
These projects, which are not currently included in CI runs, don't
build at the moment.

Upcoming logic changes would mean that we would start including them
in CI, which is undesirable until they're fixed - but I'm not going to
be doing that now.

Change-Id: I7c337e098be8bff00db6d99fc7236a695f5a85f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1850
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-26 23:49:32 +00:00
Vincent Ambo
8acbde4e15 chore(third_party): Skip subtrees of large external projects
Change-Id: Icef3900f9d1e55fd15c5286a794293368c33492c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1846
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-26 23:49:32 +00:00
Vincent Ambo
1cf11317ca refactor(tvix/libutil): Mark single-argument constructors explicit
This is the clang-tidy lint 'google-explicit-constructor'.

There's a whole bunch of breakage that was introduced by this, and we
had to opt out a few types of this (esp. the string formatting crap).

In some cases minor other changes have been done to keep the code
working, instead of converting between types (e.g. an explicit
comparison operator implementation for nix::Pid).

Change-Id: I12e1ca51a6bc2c882dba81a2526b9729d26988e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1832
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-23 11:58:44 +00:00
Vincent Ambo
1443298657 style(tvix): Add missing braces in expressions
The previous clang-tidy invocation missed some header files, which has
now been rectified.

Change-Id: I31547754fbf52f439dc7aeefb08ab90bd50c4156
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1831
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-21 03:55:56 +00:00
Vincent Ambo
7edbe59c6c style(tvix): Replace remaining typedefs with using statements
Change (mostly) created via:

  fd . -e hh -e cc | parallel clang-tidy -p ~/builds/nix --fix "'--checks=-*,modernize-use-using'"

Change-Id: I90ad6e64a5399f2005dfb9956074ec51c2c78cea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1830
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-21 03:55:56 +00:00
Vincent Ambo
ec72ca1961 fix(tvix): Add newlines to build logs
.. turns out producing an entire build log on a single line is not
useful.

This does not use `<< std::endl` because we have run into buffering
issues with the implementation of the logs->gRPC sink, but intend to
replace this in the future using a structured sink for BuildEvent
protos rather than a raw stream.

Change-Id: Ia9b05fa804391d389e2ef53ab4436c0ec5cc452e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1828
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-21 01:35:08 +00:00
Vincent Ambo
06681c35a9 refactor(tvix): Replace several DiscardLogsSink() with std::cerr
Introduces the actual log sink in several places where we actually
want the build logs to thread through correctly.

This should cover *most* build paths.

Change-Id: I735dff8a79f7e35a5874eb89b4abb980f9703dc2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1827
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-21 01:35:08 +00:00
Vincent Ambo
262af89f81 feat(tvix): Write build logs into the build log sink
This *should* wire up the builder's logs all the way back through the
gRPC client, where they are then conveniently discarded.

Change-Id: I65f22526d0b5a8b8d90f28665bc1b4bc7f7c802a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1825
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-21 00:57:18 +00:00
Vincent Ambo
e08f36c32f chore(tvix): Thread std::ostream through builder goals
This passes an output stream for build logs to almost all relevant
functions inside of build.cc by threading it through the
`Goal`-abstraction.

Store calls that create goals but don't have a sink available use the
DiscardLogsSink().

Change-Id: I2c0cb1aec1f9150f33113f4752055cea518ede8b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1824
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-21 00:37:16 +00:00
Vincent Ambo
8893c30114 chore(tvix): Add TODO comment for std::ostream in buildDerivation
Change-Id: I1e3b6abd4606d86f00ddaa8e2a1809f51e27126d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1823
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-21 00:37:16 +00:00
Vincent Ambo
375c1c2a4e style(tvix): Move a comment to fix a formatting issue
Change-Id: Idd490b978c07d1b1113ee97b5d19719a27436fba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1822
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-21 00:37:16 +00:00
Vincent Ambo
33e8b0f975 chore(tvix): Thread a std::ostream through Store::buildPaths
This part of the store API needs to carry a handle to the log sink
from now on, so that it can be passed in as appropriate from the gRPC
handlers.

In all places where there is no such handler available at the moment,
the discarding log sink has been inserted. This can be used as a
convenient grep target in the future.

Change-Id: I26628e30b4c6437dccdf8f722ca2e8ed827dfc19
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1797
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-20 11:48:40 +00:00
Vincent Ambo
f7fa77f14d style(tvix): typedef -> using in build.cc/store-api.hh
this shuts up a few lints ...

Change-Id: I51ee9418767e97247f98b170cdad137e5b12b44d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1796
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-20 11:48:40 +00:00
Vincent Ambo
a75c0278db refactor(tvix): Remove some default values from virtual methods
This is not actually legal code, but it kind of ... works. There are
more of these around, these were just the ones Griffin stumbled upon
while working on the build logs.

Change-Id: Iff9821d8fe145dd426648a8ff4510a73f67c9b7d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1795
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-20 11:48:40 +00:00
Griffin Smith
e09a6262d5 feat(tvix): Implement std::streambuf for a build log -> gRPC sink
Introduces a class which implements std::streambuf by sending build
log lines to the provided gRPC stream writer as individual messages.

This can be used in the implementations of calls which trigger builds
to forward logs back to the clients.

Change-Id: I3cecba2219cc24d56692056079c7d7e4e0fc1e2c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1794
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-20 11:48:40 +00:00
Vincent Ambo
19e874a985 feat(tvix): Introduce build event streams in worker protocol
Introduces a new `BuildEvent` proto type which is streamed in response
to calls that trigger builds of derivations.

This type can currently supply build statuses, log lines and
information about builds starting.

This is in preparation for threading build logs through the processes.

Since we have nowhere to send the logs (yet), a null sink is used
instead.

Co-authored-by: Griffin Smith <grfn@gws.fyi>
Change-Id: If7332337b89506c7e404cd20174acdaa1a3be4e8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1793
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-20 11:48:40 +00:00
Vincent Ambo
883de9b8d7 feat(tvix): Add a no-op stream buffer for discarding build logs
In some cases we don't have anywhere for the build logs to go. Until
we understand those cases fully and can get rid of them, this null
sink implementation can be used.

Change-Id: Ib93c43caf268e2c01c43d59737a829e8c43d223e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1792
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-20 11:48:40 +00:00
edef
50b200f21a chore(3p/loxy): add OWNERS
Change-Id: I054711b07307ce9c0b3ce0f85319e58616db3639
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1790
Tested-by: BuildkiteCI
Reviewed-by: V <v@anomalous.eu>
Reviewed-by: tazjin <mail@tazj.in>
2020-08-19 13:20:04 +00:00
Kane York
92d4554b62 feat(tools/tvlc): init project
tvlc is a tool for managing sparse git worktrees of the TVL depot.
It is still in development; near-term tasks include a setup script,
Nix dependency resolution, worktree removal, and the dispatch
script.

See cs.tvl.fyi/depot/docs/designs/SPARSE_CHECKOUTS.md for more info.

Change-Id: Iad96656f0206178980fe7dcadd3dffe70d690f8f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1760
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-19 01:23:54 +00:00
kanepyork
32d9e6c8d0 revert(3p/loxy): Add link to source repository
This reverts commit 4d113ae9f9.

Reason for revert: Source repository link has been added to the homepage

Change-Id: I21667fab928621ba8973c8598d01920ecbf512ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1801
Reviewed-by: lukegb <lukegb@tvl.fyi>
Tested-by: BuildkiteCI
2020-08-18 22:36:18 +00:00
Vincent Ambo
4ed096b7d7 chore(3p/emacs): Enable cairo for Emacs 27.1
... this in turn enables support for Harfbuzz, which enables support
for ligatures - maybe this will make grfn happy!

Change-Id: I331c5171a829800fe532ca0b0d4910478e63ef1b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1783
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-18 22:06:16 +00:00
Kane York
290df663af fix(tvix/config): properly handle nonexistent config files
If the global nix config, or any available user nix config location, does not exist, then the loadConfFile() function will throw and not finish initalizing the Nix configuration.

Change-Id: I6db83bca54d9b66699356d107720603476e32c23
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1657
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-18 12:05:41 +00:00
Kane York
b76cd7253a feat(tvix): Re-enable language tests that needed a store
Now that we have access to a store in tests, we can enable the tests that needed a store.

Additionally, move the expected output files for disabled tests into the disabled folder.

Change-Id: I2492d49d43b93c7c9b0463e4d3d2855a5a51365d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1758
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-18 03:58:13 +00:00
Kane York
fea4df560b chore(tvix/tests): port upstream add.sh test
Change-Id: I5151d142d6b2b7f1df37b170b0160b8f77a89120
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1755
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-18 03:58:13 +00:00
Kane York
010dba3495 chore(tvix/tests): Add OpenTemporaryStore() helper
Change-Id: Ia655175c255f9cf5a47e4e7a20373a4ba4315ed9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1753
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-18 03:58:13 +00:00
Vincent Ambo
c344e03705 chore(3p): Provide Emacs 27.1
This is now available in the upstream channel, and I need it on one
machine for $reasons.

Normally the Emacs attribute is versioned to make it clear which
version we're dealing with, but not in this case. I've added an assert
to check that it is indeed 27.

Change-Id: I827df0d36350066bf4e177150d32af0c8d123b9b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1764
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-17 18:14:40 +00:00
Vincent Ambo
6f5659c88b chore(3p): Bump nixos-unstable channel
This bumps the channel to a commit that includes fixes for an nginx
issue we have been seeing:

https://github.com/NixOS/nixpkgs/pull/95264

Includes the following compatibility fixes:

- tests disabled in third_party.bufbuild: These were enabled
  unexpectedly by the update, but don't run in the sandbox because
  they want to download things from github

Change-Id: I98a3b5de57f62f1fd3a37701fa1896eddeedff85
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1759
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-08-17 09:54:18 +00:00
Vincent Ambo
4d113ae9f9 chore(3p/loxy): Add link to source repository
This is not obvious otherwise.

Change-Id: Ic07993a58e545345dae4529d281d6fdae35c19aa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1761
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-17 09:54:18 +00:00
Kane York
1fc9ba4885 refactor(tvix): always pass Bindings by ptr, use shared/unique_ptr
Value now carries a shared_ptr<Bindings>, and all Bindings constructors return a unique_ptr<Bindings>.

The test that wanted to compare two Bindings by putting them into Values has been modified to use the new Equal() method on Bindings (extracted from EvalState).

Change-Id: I8dfb60e65fdabb717e3b3e5d56d5b3fc82f70883
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1744
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-08-17 02:23:49 +00:00
Kane York
38f2ea34f4 refactor(tvix): add explicit copy/move constructors for Value
This is in preparation for making some of Value's members into refcounted ('smart') pointers.

Change-Id: Ibc54e23ac35766a2fd4e14871c9a7c936a603778
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1743
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-17 02:23:49 +00:00
Vincent Ambo
cbebc75d94 chore(3p): Remove telega derivation
The upstream telega.el derivation (under emacsPackages) now includes
the server component.

Change-Id: I49d85ea16d2fce10a505e2459fcd396706e39923
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1748
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-15 12:11:36 +00:00
Vincent Ambo
1a48f39078 fix(tvix): Ensure prim_sort actually uses the right outlist
Previously the outlist would not be used, and it would sort god knows
what in the out value.

This was probably introduced by the std::vector refactoring, and the
language test for builtins.sort was disabled. Whatever reason there
was for disabling it seems to be gone, so we're re-enabling it.

Change-Id: I98941c2cad78df58ff7bea1ece3aaa4133e94bf8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1757
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-08-15 01:58:44 +00:00
Kane York
e458e5255a feat(tvix/tests): add gtest matchers for absl::Status
This allows you to write EXPECT_OK(statusor), as well as
EXPECT_THAT(status, IsStatusCode(StatusCode::kInvalidArgument).

Change-Id: I53bed694d812c501eb305ed4ddb358e1f9a68277
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1704
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-15 00:52:54 +00:00
Vincent Ambo
d44565d17e chore(3p): Pick Emacs & friends from the default channel
I had previously pinned Emacs to the (older) stable channel, because
of a heretofore undiagnosed issue in newer versions.

It turns out that the older Emacs is linking against X11-related
libraries, namely <unintelligible>, which cause even *weirder*
undiagnosed issues if paired with the rest of the recent packages.

This commit removes the exception for Emacs and just picks it from the
same channel as everything else.

Change-Id: Ifce21c99f0204d4cf131bacb366dd9617fc301b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1751
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-15 00:26:42 +00:00
Griffin Smith
d1c38d9597 refactor(tvix): Make Store::buildPaths return a Status
Make Store::buildPaths return a Status with [[nodiscard]] rather than
throwing exceptions to signal failure. This is the beginning of a long
road to refactor the entire store API to be status/statusor based
instead of using exceptions.

Change-Id: I2e32371c95a25b87ad129987c217d49c6d6e0c85
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1745
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-14 03:06:46 +00:00
Kane York
0124691fdd fix(tvix): use correct value of SANDBOX_SHELL
Change-Id: I8e471c0f0a30337681042beeff633821eee67346
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1746
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-14 02:56:04 +00:00
Kane York
72e61aa584 refactor(tvix): completely remove boehm gc
We have decided that leaking memory is a better fate than random,
non-debuggable memory corruption. Future CLs will begin changing
various fields to std::unique_ptr and std::shared_ptr.

It turns out that disabling the GC does not have disasterous impact.
The Nix evaluator only runs on the client CLI, never in any long-
running process. Even the REPL does not leak too badly under this
change, because it uses one EvalState for the duration of the REPL.

Building an explicitly tracing garbage collector is likely in the
future of this project, but that giant amount of work cannot be
done under a nix evaluator that is constantly crashing. We need to
restore development velocity here, and this is the best way we've
figured out to do it.

Change-Id: I2fcda8fcee853c15a9a5e22eca7c5a784bc2bf76
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1720
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-08-14 00:35:18 +00:00
Vincent Ambo
e38015f7b5 refactor(tvix): Very minor lint fixes
Change-Id: I4aca504d98c79f931b7faadff86f40d27ed5a2c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1719
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-13 01:39:03 +00:00
Kane York
906f5c1d2d fix(3p/nix): inherit Expr from gc, make parser state traceable
The parser contained vectors, and the primary parser state, that
were not participating in GC tracing.

Change-Id: Ie198592cd7acffd390e3e2ae9595138b56416838
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1706
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-10 22:16:42 +00:00
Vincent Ambo
42bdaacca6 style(tvix): Sort out minor formatting errors
Invocations of the MakeError macro that were not followed by a
semicolon messed up indentation in the next lines.

Change-Id: I03d7d1443f062a38af2c7da3da8928e0ed05e274
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1708
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-10 17:40:18 +00:00
Vincent Ambo
a1965e9182 fix(tvix): Print invalid store path when encountered
... instead of silently swallowing it.

Change-Id: I084c51d633d632bd4e3f1ca42987dd6068fbb2a9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1707
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2020-08-10 16:49:46 +00:00
Griffin Smith
d1653533a6 test(tvix): Cover scanForReferences in a test
Aded a few test cases covering the scanForReferences function, which had
been accidentally broken in 976a36c (which is now partially-reverted).
As part of this, since the test needed to generate hashes for store
paths, the logic in MakeStorePath to compress a sha256 hash down to 20
bytes and convert it to base32 has been extracted to a member function
on the Hash class.

Fixes: #34
Change-Id: Ie2d914688a80f42d0234d351a7cc0714fd15709e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1698
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-09 17:26:19 +00:00
Griffin Smith
ec7d368546 fix(tvix): Make nix value structs inherit from GC
All of the miscellanious structs that make up a nix Value should inherit
from gc, as they contain pointers to GC'ed things as members

Fixes: #42
Change-Id: I057d8c9f7dafbee7de7644ff152c9cba1aa6bc03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1696
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-09 02:22:44 +00:00
Griffin Smith
059468ba15 refactor(tvix): Remove UDSRemoteStore
Now that we've fully implemented the RPC-based store client, we can get
rid of the UDSRemoteStore, whose only use was connecting to the locally
running nix daemon. The RemoteStore still needs to be around to connect
to remote upstream nix stores over SSH.

Change-Id: I0699819803cbfe966b9a46786f2c927d8e4bf1a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1693
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-09 02:22:44 +00:00
Griffin Smith
e440f60b6c feat(tvix): Implement all remaining RPC calls
Implement all remaining RPC calls on the RpcSstore client, remove a few
stub methods we had added that weren't actually present in the old
RemoteStore implementation, and add one more RPC call for getBuildLog
that is present in the store API, but that we hadn't added as a
stub *or* to the proto.

Fixes: #29
Change-Id: Id827f51a393ece4bc7bbecaf38aee9eb4b329770
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1692
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-08-09 02:22:44 +00:00
Griffin Smith
747dc65154 refactor(tvix): Factor-out proto utilities
Factor out the shared utilities for interacting with protobufs/grpc from
libstore to a proto.hh header in libproto.

Change-Id: I1cb8d94867d5d4b63a9994be0b53f8f612eb8e3a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1691
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-09 01:52:59 +00:00
Griffin Smith
ef7894273b feat(tvix): Pass function name to SuccessOrThrow
To aid in debugging RPC call failures, pass the name of the function
being called to the SuccessOrThrow util funcion in the RpcStore

Change-Id: I523dacfab896b85a3dbe6050c07ee6bd5906fa44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1690
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-09 01:17:09 +00:00
Kane York
1de00e6c42 chore(3p/nix): apply google-readability-casting
Command run: jq <compile_commands.json -r 'map(.file)|.[]' | grep -v '/generated/' | parallel clang-tidy -p compile_commands.json -checks=-*,google-readability-casting --fix

Manual fixes applied in src/nix-env/nix-env.cc, src/libstore/store-api.cc

Change-Id: I406b4be9368c557ca59329bf6f7002704e955f8d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1557
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2020-08-08 22:16:02 +00:00
Griffin Smith
053a138002 fix(tvix): Wrap remaining RPCs in HandleExceptions
Wrap the BuildPaths and AddTextToStore RPC handlers in HandleExceptions.
These were missed in the original pass due to a merge.

Change-Id: Ie5be45e6098fba7a2b6b1c1be81578cb742c2880
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1689
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-08 20:03:59 +00:00
edef
3049f31d28 chore(3p/nix): don't optional-wrap EvalState::file_access_trace_fn further
std::function has a natural null we can't eliminate anyway, so this was
effectively std::optional<std::optional<non_nullable_function>>.

Change-Id: If99f139146021edb25d133dad7f0f6e125ef53df
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1688
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-08 18:01:11 +00:00
Griffin Smith
7db734afad feat(tvix): Implement AddToStoreNar
Implement both the client and server sides of AddToStoreNar, using a
templated generalization of the sources and sinks we were using for
AddToStore on both ends.

Change-Id: I73d0ed34118c711b125851dff99a7518ced4af35
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1686
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-08 17:51:13 +00:00
Kane York
a5dae62e85 fix(3p/nix): Use SkipEmpty in all calls to absl::StrSplit
The behavior to return a list containing a single empty string when
provided an empty string is a behavior that absl inherited from legacy
code. However, the behavior expected by legacy code in Nix is the
behavior provided by the SkipEmpty option. Switch all calls to use
SkipEmpty, except for the call already using SkipWhitespace.

See also commit 26a59482d2, with the
partly-prophetic message: "there may be other places we need to
fix this as well."

Change-Id: I6e94856a12cfb1b7e4a3b4e221769ed446648861
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1687
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-06 20:50:47 +00:00
Vincent Ambo
6a97206ceb refactor(tvix): Use absl::btree_map for BasicDerivation's env
Change-Id: I111a9a268debea322f23fdced3bed9ff3e8ed3b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1685
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-06 02:49:58 +00:00
Vincent Ambo
3ec3661678 refactor(tvix): Use absl::btree_map for DerivationInputs
Change-Id: If160ab1b09161969d9080d5d0d6f82e44a953e3c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1684
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-06 02:30:28 +00:00
Vincent Ambo
91bd7ce73a refactor(tvix): Use absl::btree_map for DerivationOutputs
This container implementation is much faster than std::map. We have
stuck to an ordered container because it's unclear whether the
accesses of this field (of which there are *many*) are actually
ordering dependent.

Also includes an Arbitrary implementation for absl::btree_map (for any
K, V that are also Arbitrary).

Change-Id: I04f58ca0ce32b9ae1759313b01508b0e44bae793
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1683
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-06 02:20:36 +00:00
Vincent Ambo
a41c3dedb1 chore(tvix): Disable sandboxing by default until its time comes
Change-Id: Ifcc776841924abcb43fb115f3de569fe500756c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1682
Reviewed-by: glittershark <grfn@gws.fyi>
Tested-by: BuildkiteCI
2020-08-06 02:20:36 +00:00
Vincent Ambo
d42c3dd72f chore(3p): Bump nixpkgs channels
Bumps both nixos-unstable and nixos-20.03 to today's versions, as per
status.nixos.org

Contains minor fixes to things that broke because of the update:

* tazjin/frog: hardware.u2f is a deprecated setting
* glittershark/system: modSha256 in Go modules is now vendorSha256
* glittershark/owothia: removed version constraint on relude

Change-Id: Ib3e9612b1b06ed547b90e4f8b0ffe5ed7fe0a5c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1642
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-06 00:15:33 +00:00
Kane York
68b5306c56 feat(3p/nix): add --trace-file-access to nix-instantiate
This builds on edef's work with depot-scan by adding a dedicated
flag to the command. We piggyback on upstream's restricted-mode
implementation, the checkSourcePath function.

Change-Id: I52bb613549f40dbca1e8caa036635910c1a3d6d0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1654
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 22:49:22 +00:00
Griffin Smith
ea488b5705 feat(3p/nix): Implement FindRoots, CollectGarbage
Implement the RPC client and server handlers for the FindRoots and
CollectGarbage RPC calls

Change-Id: Ifa5d582c6a33bd1e7661ac2fc860505ef404dad0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1656
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-05 22:28:34 +00:00
Kane York
f9df9b4733 fix(3p/nix): fix usage error of absl::Base64Unescape
The source and destination strings cannot be the same string - absl
will write to the destination in a streaming manner, causing the
source to become invalid.

Change-Id: I3578cf1f8789a51d85e0950f7987c398f0a00953
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1659
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 18:29:22 +00:00
Kane York
f10d60a454 fix(3p/nix/store): fix race condition in downloader
In certain circumstances, the decompression thread could race ahead of the downloader thread and process the same chunk twice. Clear the data buffer while the lock is held to prevent this kind of incident.

Change-Id: I19a84a0c5768d1228c6c18a7664a7b8893ef96de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1658
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 18:29:22 +00:00
Vincent Ambo
3f923b2aa0 refactor(tvix): Use SANDBOX_SHELL from environment in CMake
This makes the configuration work out-of-the-box using CMake in a Nix
shell, rather than having to pass the additional variable to CMake on
the command line.

Change-Id: I04d3cc4f2e5ecf47bf2ee459d5e48588b84ae4dd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1643
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 11:37:20 +00:00
Kane York
ffa8e7a998 feat(3p/nix): add tests for BinaryCacheStore
These tests are in preparation for factoring the Store away from libutil's Callback to absl::StatusOr.
They use the newly added MockBinaryCacheStore.

Updates: #25

Change-Id: I30c207589ec38254806ebc9a983f35668e353ae9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1595
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 04:11:57 +00:00
Kane York
3c3527e16b feat(3p/nix): add MockBinaryCacheStore
This adds an implementation of BinaryCacheStore to be used by tests exercising both the logic inherent to BinaryCacheStore and more general Store tests.

A new library target, nixstoremock, is created to indicate that this file is intended only for use in tests and not in user-facing code.

Change-Id: Ib68f777238843a4f3a2303db8a69735fbc22d161
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1645
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 04:11:57 +00:00
Griffin Smith
28c7e926ea feat(3p/nix): Implement AddToStore rpc client
Implement the AddToStore RPC client method, which uses an
AddToStorePathWriterSink (the dual of the AddToStorePathReaderSource on
the server side) to hook into the dumpPath machinery (which we should
refactor not to use sinks or sources, but not yet) and write dumped
paths as binary data over gRPC. With this commit and sandboxing
disabled, the following derivation builds cleanly:

    derivation {
      name = "test";
      builder = ./build.sh;
      system = "x86_64-linux";
    }

where build.sh has chmod +x and contains:

    #!/bin/sh
    echo 1 > $out

Change-Id: I94cab86f0825a3a9993262a9807130645c13bf44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1638
Reviewed-by: kanepyork <rikingcoding@gmail.com>
Tested-by: BuildkiteCI
2020-08-05 03:24:36 +00:00
Kane York
b15b447fcb fix(3p/nix): fix clang-tidy for int-string appends
Change-Id: I276de7a5fd1c705c87d35dd616e5980c747190aa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1597
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-05 02:54:47 +00:00