ecoppens/colmena
1
0
Fork 0
forked from DGNum/colmena
Code Pull requests Activity
410 commits 2 branches 0 tags 994 KiB
Commit graph

170 commits

Author SHA1 Message Date
Zhaofeng Li
85b0107b59 eval.nix: Skip recursive type checking for nodes in colmena eval as well 
This matches the behavior of the `nodes` argument in NixOS modules.
 2022-02-16 01:11:06 -08:00
Linus Heckemann
efa7322587
streaming evaluator: respect eval-nodes-limit (#58) 2022-02-16 01:10:52 -08:00
Zhaofeng Li
c088925e0d Misc doc tweaks 2022-02-05 18:43:43 -08:00
Zhaofeng Li
45b135c1b1 nix_eval_jobs: Add timeouts to tests 
Sometimes nix-eval-jobs would lock up indefinitely, so let's make it
fail fast in CI.

Also ignore test_json_global_error while we investigate.
 2022-02-04 20:58:17 -08:00
Zhaofeng Li
f674ddf173 key: Compute full path to key file in Nix 2022-01-25 14:22:26 -08:00
Zhaofeng Li
e58dde1be0 eval.nix: Don't override modulesPath 
`eval-config.nix` will set the canonical path for us that will then be used
to strip the module path prefixes in `nixos/modules/misc/documentation.nix` [1].
This should fix split docs build problems described in #50.

[1] 1511e72b75/nixos/modules/misc/documentation.nix (L79)
 2022-01-25 14:22:26 -08:00
Zhaofeng Li
7cd9b6f70c eval.nix: Don't pass baseModules to modules 
This is unnecessary and breaks the separation of user `modules`
and `baseModules` in `eval-config.nix`.
 2022-01-25 14:22:26 -08:00
Zhaofeng Li
9251019723 nix_eval_jobs: Errors may not have an attribute name attached 2022-01-25 14:22:26 -08:00
Zhaofeng Li
c0107b21e0 General code cleanup 2022-01-25 14:22:26 -08:00
Paul Haerle
5b8971a0f4
eval.nix: expose nixosModules.deploymentOptions (#49) 
Allow flake users to import .#nixosModules.deploymentOptions
into their flake, so that the same expression can be used for both,
.#colmena.$host as well as .#nixosConfiguration.$host, without the
latter complaining about undefined options in "deployment".
 2022-01-23 10:06:41 -08:00
Zhaofeng Li
cf9a72a1d4 nix: Add initial nix-eval-job integration 
A DrvSetEvaluator is able to evaluate attribute sets of derivations,
streaming results as they come in.
 2022-01-22 17:50:53 -08:00
Zhaofeng Li
3e40e84e19 nix: Add NixExpression abstraction 
This decouples expression generation from evaluation. `NixExpression`s
that evaluate to a set of derivations can be fed to `DrvSetEvaluator`s
which may be able to parallelize evaluation.
 2022-01-22 17:50:53 -08:00
Zhaofeng Li
55ce6d078e Remove unused stuff 2022-01-22 17:50:53 -08:00
Zhaofeng Li
d3e556027f deployment: Move chunked mode into self-contained function 2022-01-21 00:45:12 -08:00
Zhaofeng Li
82361e5ea5 hive: Clean up, make way for nix-eval-jobs 2022-01-21 00:45:12 -08:00
Zhaofeng Li
3fa0dee6b0 hive: Convert derivations to drvPath in separate attribute 
This is to make way for the nix-eval-jobs streaming evaluator.
 2022-01-21 00:45:12 -08:00
Zhaofeng Li
abb74b9f49 Add EditorConfig rules, fix trailing whitespaces 2022-01-08 01:20:36 -08:00
Zhaofeng Li
deca292b53 Refactor NixOptions 2022-01-08 01:20:36 -08:00
Zhaofeng Li
31fd1e49ac Move nix::{NixResult, NixError} to error::{ColmenaResult, ColmenaError} 2022-01-08 01:20:36 -08:00
Zhaofeng Li
16ed9d8c66 Move nix::NixCommand to util::CommandExt 2022-01-08 01:20:36 -08:00
Zhaofeng Li
f92236da46 Refactor current profile detection 2022-01-08 01:20:36 -08:00
Zhaofeng Li
da7907c874 General code cleanup 2022-01-05 14:01:05 -08:00
Zhaofeng Li
f9fab83030 deployment: Remove useless Arc clones 2022-01-02 13:16:24 -08:00
Zhaofeng Li
7bf57fd1ad nix/node_filter: Forgot to add 2022-01-01 16:47:38 -08:00
Zhaofeng Li
98897bf4de Support building on target nodes 
This partially addresses #33, and allows Colmena to be used more
easily on bandwidth-constrained hosts and macOS.

With `deployment.buildOnTarget = true;` deployment works fine from
macOS without designated builders, except when IFD is involved.
 2022-01-01 16:41:35 -08:00
Zhaofeng Li
872f944743 eval.nix: Skip recursive type checking when cross-referencing configs through the nodes argument 
This makes evaluation faster especially when `nodes` is heavily used in
the configuration (like in my auto-meshing setup). This matches the
behavior of Morph.
 2022-01-01 16:41:35 -08:00
Zhaofeng Li
cdbb69617f eval.nix: Support specifying a list of configs 2022-01-01 16:41:35 -08:00
Zhaofeng Li
f9f4dd5f63 Remove another ugly toJSON hack 2022-01-01 16:41:35 -08:00
Zhaofeng Li
f809d3b21c nix/host: Always copy outputs to remote along with derivations when realizing 
This prevents useless rebuilds when trying to realize a derivation on
a remote host. This code path isn't actually used by Colmena at the
moment.
 2022-01-01 16:41:35 -08:00
Zhaofeng Li
2b652f7236 hive: Add another test for nixpkgs.system 2022-01-01 16:41:35 -08:00
Zhaofeng Li
8f77184d58 eval.nix: Inherit pkgs.system in evalConfig 2021-12-18 14:35:06 -08:00
Zhaofeng Li
bd4493da73 nix: Move eval.nix tests into /hive 2021-12-18 14:35:06 -08:00
Zhaofeng Li
31fc98cfa7 nix: Remove ugly toJSON hack 
We now evaluate with --strict.
 2021-12-07 23:13:31 -08:00
Zhaofeng Li
eebded1786 Build each node individually 
Now nodes that take a long time to build won't bottleneck the
deployment of other nodes in the same chunk.

Fixes #47.
 2021-12-07 23:13:31 -08:00
Zhaofeng Li
39f597f778 hive: Pass --read-write-mode to nix-instantiate --eval 
This enables the use of IFD inside `meta` and node `deployment.*`
options, from which we obtain the values with `nix-instantiate --eval`.

Fixes #45.
 2021-12-05 12:34:44 -08:00
Zhaofeng Li
7433661aed Add deployment.keys.<name>.name 2021-12-05 01:14:12 -08:00
Zhaofeng Li
0f8873027f utils: Don't panic in capture_stream 2021-12-05 01:14:12 -08:00
Zhaofeng Li
ddccad4fb9 eval.nix: Auto-call functors as well 
This makes some very cursed setup work :P
 2021-12-04 01:03:26 -08:00
Zhaofeng Li
064432f38e nix/deployment: Misc cleanup 2021-12-04 01:03:26 -08:00
Zhaofeng Li
a3e292d7fb nix: Another deployment parallelization oops 2021-12-03 00:06:17 -08:00
Zhaofeng Li
2bba64a002 nix: Fix deployment parallelization 2021-12-02 23:49:46 -08:00
Zhaofeng Li
a42e8c5bf0 Misc cleanup 2021-11-23 14:12:01 -08:00
Zhaofeng Li
ec51f5703f deployment: A few oops 2021-11-22 01:53:08 -08:00
Zhaofeng Li
572f77dbf5 A couple of fixes to progress output 
Need to have some integration tests.
 2021-11-21 16:02:45 -08:00
Zhaofeng Li
0cb3f8e968 Redesign deployment process (again) 
We now ship Events from different parts of the deployment process
via a channel to a job monitor.
 2021-11-21 00:11:37 -08:00
Zhaofeng Li
5c84134af3 Refactor node names 2021-11-20 23:34:52 -08:00
Zhaofeng Li
47ccc6c9c4 nix/info.rs: Update message about nixUnstable for 21.11 2021-11-18 15:41:02 -08:00
Zhaofeng Li
e5665775b2 Preserve elapsed time in store path listing after build 
Fixes #36.
 2021-11-18 01:38:58 -08:00
Zhaofeng Li
4497ef296e Add manual 
We finally have some real documentation :)
 2021-11-17 22:21:00 -08:00
Zhaofeng Li
1535857acc Also disallow pinning to a Nixpkgs lambda in Flakes 
Somehow missed this one *shrugs*
 2021-11-17 22:21:00 -08:00
Zhaofeng Li
006cb2c5ee eval.nix: Make the uninitialized nixpkgs error more informative 2021-11-16 21:01:40 -08:00
Zhaofeng Li
f7eb121260 Disallow uninitialized meta.nixpkgs in Flakes 2021-10-28 17:10:58 -07:00
Zhaofeng Li
765f42fa24 introspect: Support actually instantiating the expression 2021-10-28 14:09:35 -07:00
Zhaofeng Li
0e0a1e84f0 Make flake resolution (slightly) less terrible 
Instead of using `path:` which always copies the entire directory,
we now try to resolve the Flake URI using `nix flake metadata` which
may give us a `git+file:`.
 2021-10-25 23:38:10 -07:00
Zhaofeng Li
b48753239a hive.rs: Canonicalize flake path 
Relative paths are no longer allowed in newer Nix versions.
 2021-10-25 21:53:38 -07:00
Zhaofeng Li
6d6e33fcd4 nix: Remove unneeded ok() 2021-10-23 20:49:14 -07:00
Bjørn Forsman
4106a73e75 Allow selecting ssh user dynamically 
...by setting `deployment.targetUser = null`.

This allows sharing a deployment file (hive.nix/flake.nix) between
multiple admins, without having to use a shared root account.
 2021-10-23 15:06:56 +02:00
Zhaofeng Li
37b43cd6d7 eval.nix: Support autocall for hive configuration 2021-08-26 19:59:22 -07:00
Zhaofeng Li
7cc6552ee3 hive.rs: Remove unwrap in builder_args() 2021-08-26 19:59:22 -07:00
Zhaofeng Li
7b69946d98 Ensure key ownerships are set correctly 
Depending on when keys are uploaded (`deployment.keys.<name>.uploadAt`):

`pre-activation`:
We set the ownerships in the uploader script opportunistically and
continue if the user/group does not exist. Then, in the activation
script, we set the ownerships of all pre-activation keys.

`post-activation`:
We set the ownerships in the uploader script and fail if the
user/group does not exist.

The ownerships will be correct regardless of which mode is in use.

Fixes #23. Also a more complete solution to #10.
 2021-08-26 12:54:41 -07:00
Zhaofeng Li
24339bcca7 Add deployment.keys.<name>.uploadAt 
This mirrors the functionality recently added in morph and allows
for the uploading of keys after system profile activation.

Fixes #10.
 2021-08-24 23:25:46 -07:00
Zhaofeng Li
135a42b20f eval.nix: Add meta.specialArgs 2021-07-16 22:52:23 -07:00
Zhaofeng Li
671cf38796 hive.rs: Pass --builders to nix-instantiate as well 2021-07-13 01:38:52 -07:00
Zhaofeng Li
67db0e73d1 Add check for Flakes support 2021-06-29 01:02:43 -07:00
Zhaofeng Li
e50ba82bf2 Add basic Flakes support 
Co-authored-by: Alex Zero <joseph@marsden.space>
 2021-06-29 01:02:43 -07:00
Zhaofeng Li
22ae18f5e7 Exit with non-zero code if any node fails to deploy 
The exit codes are in flux and should not be relied upon.

Fixes #28.
 2021-05-24 00:15:38 -07:00
Zhaofeng Li
960af8f793 Add deployment.privilegeEscalationCommand 
This adds a NixOps-equivalent option for non-root deployment
on remote hosts.

Fixes #27.
 2021-05-24 00:15:38 -07:00
Zhaofeng Li
39d612a5e7 ssh: Remove dead code 2021-05-24 00:15:38 -07:00
Zhaofeng Li
99ba8db335
Merge pull request #21 from jasonrm/machines-file 
eval.nix: Adds meta.machinesFile option that is passed to Nix as builder option
 2021-05-07 16:25:36 -07:00
Zhaofeng Li
16ccdbc700 Better handling of killed processes 2021-04-28 15:09:40 -07:00
Zhaofeng Li
44b421c2c7 key.rs: Fix typo (user -> group) 
Fixes #22.
 2021-04-19 15:40:19 -07:00
Jason R. McNeil
3ee97c2a76 apply: Add deployment.replaceUnknownProfiles option and --force-replace-unknown-profiles switch 
If `deployment.replaceUnknownProfiles` is set to false, a diverged hive
config (in a shared git repo for example) won't result in accidentally
undoing another applied configuration profile.

The deployment option is set to true so that fiction is minimized from
aggressive garbage collection, first time profile application and low
contention hives.
 2021-04-10 13:42:38 -07:00
Jason R. McNeil
e0465567b2 eval.nix: Adds meta.machinesFile option that is passed to Nix as builders argument 2021-04-09 23:54:13 -07:00
Zhaofeng Li
8abcd5d53b "Successfully built" -> "Build successful" for consistency 2021-03-18 15:13:34 -07:00
Zhaofeng Li
b44dd1f877 apply_local: Don't bother evaluating other hosts 2021-03-18 15:05:05 -07:00
Zhaofeng Li
e9487ced9e host: Use the key uploader script for both SSH and local 2021-03-17 22:39:05 -07:00
Zhaofeng Li
610a725ba2 Add --keep-result to create GC roots for profiles 
This resembles the behavior of morph.

Reference: #18
 2021-03-17 14:59:57 -07:00
Zhaofeng Li
81375e71b2 deployment: Display the resulting paths if the goal is to build only 
Reference: #18
 2021-03-17 14:59:43 -07:00
Jason R. McNeil
4098bf73bc Makes SSH options available to nix-copy-closure 2021-03-14 22:20:47 -07:00
Zhaofeng Li
082a033443 eval.nix: Exclude internal Nixpkgs config options from node override warning 2021-02-17 23:06:22 -08:00
Zhaofeng Li
e32e130621 Always print the entire log for failures in eval and build 
This makes it easier to debug configuration issues without -v.

Fixes #14.
 2021-02-17 22:48:26 -08:00
Zhaofeng Li
d16a13654c Merge nixpkgs.config and nixpkgs.overlays 
This replaces #12, and allows for Nixpkgs overlays and config to be overridden
in machine configs. With #12, overlays set in machine configurations
(`nixpkgs.overlays`) get silently ignored.
 2021-02-17 22:46:01 -08:00
Zhaofeng Li
9eae937b42 apply: Disable configuration of build process limit 2021-02-17 08:09:15 -08:00
Justinas Stankevicius
10f98d715f Propagate same 'pkgs' to all modules 2021-02-16 20:53:43 +02:00
Zhaofeng Li
30dc352eb9 eval.nix: Add type checking to meta/network 
Primarily to make the evaluation error out if the configuration
tries to use non-existent options (e.g., pinning Nixpkgs with
morph-specific options).
 2021-02-12 14:52:09 -08:00
Zhaofeng Li
d0bba90d04 ssh: Fix shell escaping 
The previous `sh -c` invocation was incorrect and just happened
to work on hosts with a Bourne-compatible shell set as the login
shell. Commands in the deploy script were being executed in the
login shell.
 2021-02-12 13:55:44 -08:00
Zhaofeng Li
95ddbcbfd6 ssh/deploy-key: Skip chown if the user/group doesn't exist 
This matches the behavior of NixOps.

Potential solution to #10.
 2021-02-12 13:54:17 -08:00
Zhaofeng Li
dbd66d7c7c Add initial set of tests 2021-02-11 13:27:21 -08:00
Justinas Stankevicius
4c7f8eb838 keyCommand: on error, do not upload key, report 2021-02-11 21:16:56 +02:00
Zhaofeng Li
e49e9367c0 key: Serialize KeySource through an intermediate struct 
Well, still better than `if/else`-ing all the way. Also we
definitely need unit tests.

See #8.
 2021-02-11 00:51:11 -08:00
Zhaofeng Li
2886662e18 nix: Key names can contain one path component only 
Well, I changed my mind and this should be cleaner.
 2021-02-10 18:17:55 -08:00
Zhaofeng Li
52622ecd27 Add 'deployment.keys.<key>.keyCommand' support 
Fixes #3.
 2021-02-10 18:08:47 -08:00
Zhaofeng Li
ce9f639a53 key: Make the key source better typed 2021-02-10 17:34:52 -08:00
Zhaofeng Li
62753ea138 progress: Let's just call them "tasks" instead of "processes"... 2021-02-10 17:20:49 -08:00
Zhaofeng Li
9d59a6a288
Merge pull request #5 from justinas/keys-keyfile 
Add 'deployment.keys.<key>.keyFile' option
 2021-02-10 17:20:28 -08:00
Justinas Stankevicius
d90fc56bc3 Implement key upload from local file 2021-02-10 20:37:54 +02:00
Justinas Stankevicius
f521f19629 Add deployment.keys.<key>.keyFile option 2021-02-10 20:37:54 +02:00
Zhaofeng Li
afabd8c6f9 Minor tokio cleanup 2021-02-10 10:29:17 -08:00
Zhaofeng Li
9f4d5a2221 Target stable toolchain 2021-02-10 00:41:02 -08:00
Zhaofeng Li
78a6825be6 Add exec command 2021-02-09 22:07:10 -08:00