tvl-depot/tvix/default.nix

247 lines
7.7 KiB
Nix
Raw Normal View History

refactor(tvix): share a Cargo.lock file between Rust projects This relates to the (abandoned) cl/7256. Introduces a Cargo workspace at //tvix that is primarily intended to be used as a workaround for the annoying Nix+Rust tooling while having a consistent set of dependencies. This is driven in part by a desire to adopt crate2nix and get more granular Nix builds for Tvix's Rust projects, and in part by a need to split //tvix/eval into something providing the CLI (REPL etc.), and a library providing eval, without significantly altering the structure of build targets. To accomplish this the workspace has been designed to allow projects to remain independent build targets. I want to avoid lumping all the projects together - something like //tvix/eval should always be independent of other parts of tvix. A helper function in //tvix/default.nix lets downstream naersk projects construct a sparse root for the project which combines the workspace's `Cargo.lock` with the project's own `Cargo.toml`. Note that cargo commands in the workspace itself require the build dependencies of _all_ projects to be present, which is currently a bit annoying to accomplish. This introduces some breakage: 1. It breaks usage of rust-analyser without being in a shell with the dependencies of *all* Tvix projects, as it is not capable of respecting only the subset of dependencies for a part of the workspace. 2. It is no longer possible to run tests using `cargo test`, as the test generation crate we use does not work with workspaces: https://github.com/frehberg/test-generator/issues/6 This still works in the Nix build as we construct a Cargo project that looks like it's not in a workspace there. Until somebody fixes that crate / writes a new macro / does something else with the test suite, the way to run the tests is through the Nix build. Long-term we'll probably want to get rid of cargo completely, it's just a big wart and most tooling works without it if correctly configured, but we don't have time for that now. Change-Id: I846bff7a8429a25c077fd1e9ef4e3c34a299a4a1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7533 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-06 14:47:02 +01:00
# Nix helpers for projects under //tvix
{ pkgs, lib, depot, ... }:
refactor(tvix): share a Cargo.lock file between Rust projects This relates to the (abandoned) cl/7256. Introduces a Cargo workspace at //tvix that is primarily intended to be used as a workaround for the annoying Nix+Rust tooling while having a consistent set of dependencies. This is driven in part by a desire to adopt crate2nix and get more granular Nix builds for Tvix's Rust projects, and in part by a need to split //tvix/eval into something providing the CLI (REPL etc.), and a library providing eval, without significantly altering the structure of build targets. To accomplish this the workspace has been designed to allow projects to remain independent build targets. I want to avoid lumping all the projects together - something like //tvix/eval should always be independent of other parts of tvix. A helper function in //tvix/default.nix lets downstream naersk projects construct a sparse root for the project which combines the workspace's `Cargo.lock` with the project's own `Cargo.toml`. Note that cargo commands in the workspace itself require the build dependencies of _all_ projects to be present, which is currently a bit annoying to accomplish. This introduces some breakage: 1. It breaks usage of rust-analyser without being in a shell with the dependencies of *all* Tvix projects, as it is not capable of respecting only the subset of dependencies for a part of the workspace. 2. It is no longer possible to run tests using `cargo test`, as the test generation crate we use does not work with workspaces: https://github.com/frehberg/test-generator/issues/6 This still works in the Nix build as we construct a Cargo project that looks like it's not in a workspace there. Until somebody fixes that crate / writes a new macro / does something else with the test suite, the way to run the tests is through the Nix build. Long-term we'll probably want to get rid of cargo completely, it's just a big wart and most tooling works without it if correctly configured, but we don't have time for that now. Change-Id: I846bff7a8429a25c077fd1e9ef4e3c34a299a4a1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7533 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-06 14:47:02 +01:00
let
# crate override for crates that need protobuf
protobufDep = prev: (prev.nativeBuildInputs or [ ]) ++ [ pkgs.buildPackages.protobuf ];
# On Darwin, some crates producing binaries need to be able to link against security.
darwinDeps = lib.optionals pkgs.stdenv.isDarwin (with pkgs.buildPackages.darwin.apple_sdk.frameworks; [
Security
SystemConfiguration
]);
# Load the crate2nix crate tree.
crates = pkgs.callPackage ./Cargo.nix {
defaultCrateOverrides = pkgs.defaultCrateOverrides // {
nix-compat = prev: {
src = depot.tvix.utils.filterRustCrateSrc rec {
root = prev.src.origSrc;
extraFileset = (root + "/testdata");
};
};
tvix-build = prev: {
src = depot.tvix.utils.filterRustCrateSrc rec {
root = prev.src.origSrc;
extraFileset = (lib.fileset.fileFilter (f: f.hasExt "proto") root);
};
PROTO_ROOT = depot.tvix.build.protos.protos;
nativeBuildInputs = protobufDep prev;
buildInputs = darwinDeps;
};
tvix-castore = prev: {
src = depot.tvix.utils.filterRustCrateSrc rec {
root = prev.src.origSrc;
extraFileset = (lib.fileset.fileFilter (f: f.hasExt "proto") root);
};
PROTO_ROOT = depot.tvix.castore.protos.protos;
nativeBuildInputs = protobufDep prev;
};
tvix-cli = prev: {
src = depot.tvix.utils.filterRustCrateSrc { root = prev.src.origSrc; };
buildInputs = prev.buildInputs or [ ] ++ darwinDeps;
};
tvix-store = prev: {
src = depot.tvix.utils.filterRustCrateSrc rec {
root = prev.src.origSrc;
extraFileset = (lib.fileset.fileFilter (f: f.hasExt "proto") root);
};
PROTO_ROOT = depot.tvix.store.protos.protos;
nativeBuildInputs = protobufDep prev;
# fuse-backend-rs uses DiskArbitration framework to handle mount/unmount on Darwin
buildInputs = prev.buildInputs or [ ]
++ darwinDeps
++ lib.optional pkgs.stdenv.isDarwin pkgs.buildPackages.darwin.apple_sdk.frameworks.DiskArbitration;
};
tvix-eval-builtin-macros = prev: {
src = depot.tvix.utils.filterRustCrateSrc { root = prev.src.origSrc; };
};
tvix-eval = prev: {
src = depot.tvix.utils.filterRustCrateSrc rec {
root = prev.src.origSrc;
extraFileset = (root + "/proptest-regressions");
};
};
tvix-glue = prev: {
src = depot.tvix.utils.filterRustCrateSrc {
root = prev.src.origSrc;
};
};
tvix-serde = prev: {
src = depot.tvix.utils.filterRustCrateSrc { root = prev.src.origSrc; };
};
tvix-tracing = prev: {
src = depot.tvix.utils.filterRustCrateSrc { root = prev.src.origSrc; };
};
};
};
# Cargo dependencies to be used with nixpkgs rustPlatform functions.
cargoDeps = pkgs.rustPlatform.importCargoLock {
lockFile = ./Cargo.lock;
# Extract the hashes from `crates` / Cargo.nix, we already get them from cargo2nix.
# This returns an attribute set containing "${crateName}-${version}" as key,
# and the outputHash as value.
outputHashes = builtins.listToAttrs
(map
(crateName:
(lib.nameValuePair "${crateName}-${crates.internal.crates.${crateName}.version}" crates.internal.crates.${crateName}.src.outputHash)
) [
"wu-manber"
]);
};
# The cleaned sources.
src = depot.third_party.gitignoreSource ./.;
# Target containing *all* tvix proto files.
# Useful for workspace-wide cargo invocations (doc, clippy)
protos = pkgs.symlinkJoin {
name = "tvix-all-protos";
paths = [
depot.tvix.build.protos.protos
depot.tvix.castore.protos.protos
depot.tvix.store.protos.protos
];
};
in
{
inherit crates protos;
# Run crate2nix generate, ensure the output doesn't differ afterwards
# (and doesn't fail).
#
# Currently this re-downloads every crate every time
# crate2nix-check (but not crate2nix) is built.
# TODO(amjoseph): be less wasteful with bandwidth.
#
crate2nix-check =
let
outputHashAlgo = "sha256";
in
pkgs.stdenv.mkDerivation {
inherit src;
# Important: we include the hash of all Cargo related files in the derivation name.
# This forces the FOD to be rebuilt/re-verified whenever one of them changes.
name = "tvix-crate2nix-check-" + builtins.substring 0 8 (builtins.hashString "sha256"
(lib.concatMapStrings (f: builtins.hashFile "sha256" f)
([ ./Cargo.toml ./Cargo.lock ] ++ (map (m: ./. + "/${m}/Cargo.toml") (lib.importTOML ./Cargo.toml).workspace.members))
)
);
nativeBuildInputs = with pkgs; [ git cacert cargo ];
buildPhase = ''
export CARGO_HOME=$(mktemp -d)
# The following command can be omitted, in which case
# crate2nix-generate will run it automatically, but won't show the
# output, which makes it look like the build is somehow "stuck" for a
# minute or two.
cargo metadata > /dev/null
${pkgs.crate2nix}/bin/crate2nix generate --all-features
${pkgs.treefmt}/bin/treefmt Cargo.nix \
--no-cache \
--on-unmatched=debug \
--config-file=${depot.tools.depotfmt.config} \
--tree-root=.
# technically unnecessary, but provides more-helpful output in case of error
diff -ur Cargo.nix ${src}/Cargo.nix
# the FOD hash will check that the (re-)generated Cargo.nix matches the committed Cargo.nix
cp Cargo.nix $out
'';
# This is an FOD in order to allow `cargo` to perform network access.
outputHashMode = "flat";
inherit outputHashAlgo;
outputHash = builtins.hashFile outputHashAlgo ./Cargo.nix;
env.SSL_CERT_FILE = "${pkgs.cacert.out}/etc/ssl/certs/ca-bundle.crt";
};
# Provide the Tvix logo in both .webp and .png format.
logo = pkgs.runCommand "logo"
{
nativeBuildInputs = [ pkgs.imagemagick ];
} ''
mkdir -p $out
cp ${./logo.webp} $out/logo.webp
convert $out/logo.webp $out/logo.png
'';
# Provide a shell for the combined dependencies of all Tvix Rust
# projects. Note that as this is manually maintained it may be
# lacking something, but it is required for some people's workflows.
#
# This shell can be entered with e.g. `mg shell //tvix:shell`.
2023-10-07 05:40:24 +02:00
# This is a separate file, so it can be used individually in the tvix josh
# workspace too.
shell = (import ./shell.nix { inherit pkgs; });
# Build the Rust documentation for publishing on docs.tvix.dev.
rust-docs = pkgs.stdenv.mkDerivation {
inherit cargoDeps src;
name = "tvix-rust-docs";
PROTO_ROOT = protos;
nativeBuildInputs = with pkgs; [
cargo
pkg-config
protobuf
rustc
rustPlatform.cargoSetupHook
];
buildInputs = [
pkgs.fuse
] ++ lib.optional pkgs.stdenv.isDarwin pkgs.libiconv;
buildPhase = ''
cargo doc --document-private-items
mv target/doc $out
'';
};
# Run cargo clippy. We run it with -Dwarnings, so warnings cause a nonzero
# exit code.
clippy = pkgs.stdenv.mkDerivation {
inherit cargoDeps src;
name = "tvix-clippy";
PROTO_ROOT = protos;
buildInputs = [
pkgs.fuse
];
nativeBuildInputs = with pkgs; [
cargo
clippy
pkg-config
protobuf
rustc
rustPlatform.cargoSetupHook
];
buildPhase = "cargo clippy --tests --all-features --benches --examples -- -Dwarnings | tee $out";
};
meta.ci.targets = [
"clippy"
"crate2nix-check"
"shell"
"rust-docs"
];
utils = import ./utils.nix { inherit lib depot; };
refactor(tvix): share a Cargo.lock file between Rust projects This relates to the (abandoned) cl/7256. Introduces a Cargo workspace at //tvix that is primarily intended to be used as a workaround for the annoying Nix+Rust tooling while having a consistent set of dependencies. This is driven in part by a desire to adopt crate2nix and get more granular Nix builds for Tvix's Rust projects, and in part by a need to split //tvix/eval into something providing the CLI (REPL etc.), and a library providing eval, without significantly altering the structure of build targets. To accomplish this the workspace has been designed to allow projects to remain independent build targets. I want to avoid lumping all the projects together - something like //tvix/eval should always be independent of other parts of tvix. A helper function in //tvix/default.nix lets downstream naersk projects construct a sparse root for the project which combines the workspace's `Cargo.lock` with the project's own `Cargo.toml`. Note that cargo commands in the workspace itself require the build dependencies of _all_ projects to be present, which is currently a bit annoying to accomplish. This introduces some breakage: 1. It breaks usage of rust-analyser without being in a shell with the dependencies of *all* Tvix projects, as it is not capable of respecting only the subset of dependencies for a part of the workspace. 2. It is no longer possible to run tests using `cargo test`, as the test generation crate we use does not work with workspaces: https://github.com/frehberg/test-generator/issues/6 This still works in the Nix build as we construct a Cargo project that looks like it's not in a workspace there. Until somebody fixes that crate / writes a new macro / does something else with the test suite, the way to run the tests is through the Nix build. Long-term we'll probably want to get rid of cargo completely, it's just a big wart and most tooling works without it if correctly configured, but we don't have time for that now. Change-Id: I846bff7a8429a25c077fd1e9ef4e3c34a299a4a1 Reviewed-on: https://cl.tvl.fyi/c/depot/+/7533 Reviewed-by: flokli <flokli@flokli.de> Autosubmit: tazjin <tazjin@tvl.su> Tested-by: BuildkiteCI
2022-12-06 14:47:02 +01:00
}