mdebray/tvl-depot
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
tvl-depot/third_party/nix/src/libstore/serve-protocol.hh

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

25 lines
487 B
C++
Raw Normal View History

nix-store --serve: Use a versioned protocol Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-10 13:43:13 +01:00
#pragma once
namespace nix {
#define SERVE_MAGIC_1 0x390c9deb
#define SERVE_MAGIC_2 0x5452eecb
Make adding paths via nix-store --serve run in constant memory It adds a new operation, cmdAddToStoreNar, that does the same thing as the corresponding nix-daemon operation, i.e. call addToStore(). This replaces cmdImportPaths, which has the major issue that it sends the NAR first and the store path second, thus requiring us to store the incoming NAR either in memory or on disk until we decide what to do with it. For example, this reduces the memory usage of $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79 from 267 MiB to 12 MiB. Probably fixes #1988.
2018-08-03 21:10:03 +02:00
#define SERVE_PROTOCOL_VERSION 0x205
nix-store --serve: Use a versioned protocol Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-10 13:43:13 +01:00
#define GET_PROTOCOL_MAJOR(x) ((x)&0xff00)
#define GET_PROTOCOL_MINOR(x) ((x)&0x00ff)
style(tvix): Replace remaining typedefs with using statements Change (mostly) created via: fd . -e hh -e cc | parallel clang-tidy -p ~/builds/nix --fix "'--checks=-*,modernize-use-using'" Change-Id: I90ad6e64a5399f2005dfb9956074ec51c2c78cea Reviewed-on: https://cl.tvl.fyi/c/depot/+/1830 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-21 04:23:04 +02:00
using ServeCommand = enum {
nix-copy-closure: Fix race condition There is a long-standing race condition when copying a closure to a remote machine, particularly affecting build-remote.pl: the client first asks the remote machine which paths it already has, then copies over the missing paths. If the garbage collector kicks in on the remote machine between the first and second step, the already-present paths may be deleted. The missing paths may then refer to deleted paths, causing nix-copy-closure to fail. The client now performs both steps using a single remote Nix call (using ‘nix-store --serve’), locking all paths in the closure while querying. I changed the --serve protocol a bit (getting rid of QueryCommand), so this breaks the SSH substituter from older versions. But it was marked experimental anyway. Fixes #141.
2014-07-10 11:51:22 +02:00
cmdQueryValidPaths = 1,
cmdQueryPathInfos = 2,
cmdDumpStorePath = 3,
cmdImportPaths = 4,
build-remote.pl: Use ‘nix-store --serve’ on the remote side This makes things more efficient (we don't need to use an SSH master connection, and we only start a single remote process) and gets rid of locking issues (the remote nix-store process will keep inputs and outputs locked as long as they're needed). It also makes it more or less secure to connect directly to the root account on the build machine, using a forced command (e.g. ‘command="nix-store --serve --write"’). This bypasses the Nix daemon and is therefore more efficient. Also, don't call nix-store to import the output paths.
2014-07-11 16:02:19 +02:00
cmdExportPaths = 5,
cmdBuildPaths = 6,
Implement nix-copy-closure --from via nix-store --serve
2014-07-24 16:00:29 +02:00
cmdQueryClosure = 7,
Allow remote builds without sending the derivation closure Previously, to build a derivation remotely, we had to copy the entire closure of the .drv file to the remote machine, even though we only need the top-level derivation. This is very wasteful: the closure can contain thousands of store paths, and in some Hydra use cases, include source paths that are very large (e.g. Git/Mercurial checkouts). So now there is a new operation, StoreAPI::buildDerivation(), that performs a build from an in-memory representation of a derivation (BasicDerivation) rather than from a on-disk .drv file. The only files that need to be in the Nix store are the sources of the derivation (drv.inputSrcs), and the needed output paths of the dependencies (as described by drv.inputDrvs). "nix-store --serve" exposes this interface. Note that this is a privileged operation, because you can construct a derivation that builds any store path whatsoever. Fixing this will require changing the hashing scheme (i.e., the output paths should be computed from the other fields in BasicDerivation, allowing them to be verified without access to other derivations). However, this would be quite nice because it would allow .drv-free building (e.g. "nix-env -i" wouldn't have to write any .drv files to disk). Fixes #173.
2015-07-17 17:57:40 +02:00
cmdBuildDerivation = 8,
Make adding paths via nix-store --serve run in constant memory It adds a new operation, cmdAddToStoreNar, that does the same thing as the corresponding nix-daemon operation, i.e. call addToStore(). This replaces cmdImportPaths, which has the major issue that it sends the NAR first and the store path second, thus requiring us to store the incoming NAR either in memory or on disk until we decide what to do with it. For example, this reduces the memory usage of $ nix copy --to 'ssh://localhost?remote-store=/tmp/nix' /nix/store/95cwv4q54dc6giaqv6q6p4r02ia2km35-blender-2.79 from 267 MiB to 12 MiB. Probably fixes #1988.
2018-08-03 21:10:03 +02:00
cmdAddToStoreNar = 9,
style(tvix): Replace remaining typedefs with using statements Change (mostly) created via: fd . -e hh -e cc | parallel clang-tidy -p ~/builds/nix --fix "'--checks=-*,modernize-use-using'" Change-Id: I90ad6e64a5399f2005dfb9956074ec51c2c78cea Reviewed-on: https://cl.tvl.fyi/c/depot/+/1830 Tested-by: BuildkiteCI Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-21 04:23:04 +02:00
};
nix-store --serve: Use a versioned protocol Signed-off-by: Shea Levy <shea@shealevy.com>
2014-02-10 13:43:13 +01:00
} // namespace nix
Reference in a new issue Copy permalink