ecoppens/colmena
1
0
Fork 0
forked from DGNum/colmena
Code Pull requests Activity
487 commits 2 branches 0 tags 994 KiB
Commit graph

261 commits

Author SHA1 Message Date
Zhaofeng Li
f674ddf173 key: Compute full path to key file in Nix 2022-01-25 14:22:26 -08:00
Zhaofeng Li
e58dde1be0 eval.nix: Don't override modulesPath 
`eval-config.nix` will set the canonical path for us that will then be used
to strip the module path prefixes in `nixos/modules/misc/documentation.nix` [1].
This should fix split docs build problems described in #50.

[1] 1511e72b75/nixos/modules/misc/documentation.nix (L79)
 2022-01-25 14:22:26 -08:00
Zhaofeng Li
7cd9b6f70c eval.nix: Don't pass baseModules to modules 
This is unnecessary and breaks the separation of user `modules`
and `baseModules` in `eval-config.nix`.
 2022-01-25 14:22:26 -08:00
Zhaofeng Li
9251019723 nix_eval_jobs: Errors may not have an attribute name attached 2022-01-25 14:22:26 -08:00
Zhaofeng Li
c0107b21e0 General code cleanup 2022-01-25 14:22:26 -08:00
Paul Haerle
5b8971a0f4
eval.nix: expose nixosModules.deploymentOptions (#49) 
Allow flake users to import .#nixosModules.deploymentOptions
into their flake, so that the same expression can be used for both,
.#colmena.$host as well as .#nixosConfiguration.$host, without the
latter complaining about undefined options in "deployment".
 2022-01-23 10:06:41 -08:00
Zhaofeng Li
0d9198c351 spinner: Work around the lack of newline after completion 2022-01-22 17:50:53 -08:00
Zhaofeng Li
87189ec3df nix_info: Display pinned nix-eval-jobs path 2022-01-22 17:50:53 -08:00
Zhaofeng Li
cf9a72a1d4 nix: Add initial nix-eval-job integration 
A DrvSetEvaluator is able to evaluate attribute sets of derivations,
streaming results as they come in.
 2022-01-22 17:50:53 -08:00
Zhaofeng Li
3e40e84e19 nix: Add NixExpression abstraction 
This decouples expression generation from evaluation. `NixExpression`s
that evaluate to a set of derivations can be fed to `DrvSetEvaluator`s
which may be able to parallelize evaluation.
 2022-01-22 17:50:53 -08:00
Zhaofeng Li
9fbb6b133b job: Make it possible to have a JobHandle not connected to a JobMonitor 2022-01-22 17:50:53 -08:00
Zhaofeng Li
55ce6d078e Remove unused stuff 2022-01-22 17:50:53 -08:00
Zhaofeng Li
d3e556027f deployment: Move chunked mode into self-contained function 2022-01-21 00:45:12 -08:00
Zhaofeng Li
82361e5ea5 hive: Clean up, make way for nix-eval-jobs 2022-01-21 00:45:12 -08:00
Zhaofeng Li
3fa0dee6b0 hive: Convert derivations to drvPath in separate attribute 
This is to make way for the nix-eval-jobs streaming evaluator.
 2022-01-21 00:45:12 -08:00
Zhaofeng Li
abb74b9f49 Add EditorConfig rules, fix trailing whitespaces 2022-01-08 01:20:36 -08:00
Zhaofeng Li
deca292b53 Refactor NixOptions 2022-01-08 01:20:36 -08:00
Zhaofeng Li
31fd1e49ac Move nix::{NixResult, NixError} to error::{ColmenaResult, ColmenaError} 2022-01-08 01:20:36 -08:00
Zhaofeng Li
16ed9d8c66 Move nix::NixCommand to util::CommandExt 2022-01-08 01:20:36 -08:00
Zhaofeng Li
f92236da46 Refactor current profile detection 2022-01-08 01:20:36 -08:00
Zhaofeng Li
6401ce4c3c cli: Remove target from logs 
format_target is enabled by default in env_logger 0.9.0. It makes
the logs look cluttered and is not very useful for our use case.
 2022-01-05 14:01:05 -08:00
Zhaofeng Li
da7907c874 General code cleanup 2022-01-05 14:01:05 -08:00
Zhaofeng Li
f234e16e80 manual: Fix colorized CLI help 
In clap 3.0, it's no longer possible to get `App::write_long_help` to
output colorized text (it doesn't invoke the Colorizer at all). So let's
move the generation outside of Rust.
 2022-01-03 10:37:03 -08:00
Zhaofeng Li
d2762757f0 command/eval: Fix introspect alias 
On clap 3.0, resetting the App::name still causes clap to complain
that the subcommand doesn't exist in debug mode.
 2022-01-03 10:37:03 -08:00
Zhaofeng Li
79d1010356 command: Change wording for consistency 2022-01-03 10:37:03 -08:00
Zhaofeng Li
81e2982bbb Migrate to clap 0.3 2022-01-03 10:37:03 -08:00
Zhaofeng Li
f9fab83030 deployment: Remove useless Arc clones 2022-01-02 13:16:24 -08:00
Zhaofeng Li
7bf57fd1ad nix/node_filter: Forgot to add 2022-01-01 16:47:38 -08:00
Zhaofeng Li
98897bf4de Support building on target nodes 
This partially addresses #33, and allows Colmena to be used more
easily on bandwidth-constrained hosts and macOS.

With `deployment.buildOnTarget = true;` deployment works fine from
macOS without designated builders, except when IFD is involved.
 2022-01-01 16:41:35 -08:00
Zhaofeng Li
872f944743 eval.nix: Skip recursive type checking when cross-referencing configs through the nodes argument 
This makes evaluation faster especially when `nodes` is heavily used in
the configuration (like in my auto-meshing setup). This matches the
behavior of Morph.
 2022-01-01 16:41:35 -08:00
Zhaofeng Li
cdbb69617f eval.nix: Support specifying a list of configs 2022-01-01 16:41:35 -08:00
Zhaofeng Li
bc3450b3ce cli: Disable apply-local on non-Linux platforms 2022-01-01 16:41:35 -08:00
Zhaofeng Li
f9f4dd5f63 Remove another ugly toJSON hack 2022-01-01 16:41:35 -08:00
Zhaofeng Li
111bc9dea7 Remove test-progress subcommand from release builds 2022-01-01 16:41:35 -08:00
Zhaofeng Li
f809d3b21c nix/host: Always copy outputs to remote along with derivations when realizing 
This prevents useless rebuilds when trying to realize a derivation on
a remote host. This code path isn't actually used by Colmena at the
moment.
 2022-01-01 16:41:35 -08:00
Zhaofeng Li
2b652f7236 hive: Add another test for nixpkgs.system 2022-01-01 16:41:35 -08:00
Zhaofeng Li
8f77184d58 eval.nix: Inherit pkgs.system in evalConfig 2021-12-18 14:35:06 -08:00
Zhaofeng Li
bd4493da73 nix: Move eval.nix tests into /hive 2021-12-18 14:35:06 -08:00
Zhaofeng Li
31fc98cfa7 nix: Remove ugly toJSON hack 
We now evaluate with --strict.
 2021-12-07 23:13:31 -08:00
Zhaofeng Li
eebded1786 Build each node individually 
Now nodes that take a long time to build won't bottleneck the
deployment of other nodes in the same chunk.

Fixes #47.
 2021-12-07 23:13:31 -08:00
Zhaofeng Li
39f597f778 hive: Pass --read-write-mode to nix-instantiate --eval 
This enables the use of IFD inside `meta` and node `deployment.*`
options, from which we obtain the values with `nix-instantiate --eval`.

Fixes #45.
 2021-12-05 12:34:44 -08:00
Zhaofeng Li
7433661aed Add deployment.keys.<name>.name 2021-12-05 01:14:12 -08:00
Zhaofeng Li
0f8873027f utils: Don't panic in capture_stream 2021-12-05 01:14:12 -08:00
Zhaofeng Li
ddccad4fb9 eval.nix: Auto-call functors as well 
This makes some very cursed setup work :P
 2021-12-04 01:03:26 -08:00
Zhaofeng Li
7757a0e45d command/apply: Use verbose output when goal is "dry-activate" 
It doesn't make sense to use the spinners since you can't see
the full output from the activation script.

Fixes #44.
 2021-12-04 01:03:26 -08:00
Zhaofeng Li
7b1feffb9e util: Log the configuration in use 2021-12-04 01:03:26 -08:00
Zhaofeng Li
9cd2af9438 cli: Re-order arguments in help message 2021-12-04 01:03:26 -08:00
Zhaofeng Li
f253e6eb18 cli: Allow configuring output colorization 
We now follow the <https://bixense.com/clicolors> standard.
 2021-12-04 01:03:26 -08:00
Zhaofeng Li
dc57b489ea progress: Restore tty-detection after refactoring 2021-12-04 01:03:26 -08:00
Zhaofeng Li
064432f38e nix/deployment: Misc cleanup 2021-12-04 01:03:26 -08:00
Zhaofeng Li
04976e5fda job: Make event formatting less ugly 2021-12-03 00:06:21 -08:00
Zhaofeng Li
a3e292d7fb nix: Another deployment parallelization oops 2021-12-03 00:06:17 -08:00
Zhaofeng Li
2bba64a002 nix: Fix deployment parallelization 2021-12-02 23:49:46 -08:00
Zhaofeng Li
a42e8c5bf0 Misc cleanup 2021-11-23 14:12:01 -08:00
Zhaofeng Li
ec51f5703f deployment: A few oops 2021-11-22 01:53:08 -08:00
Zhaofeng Li
572f77dbf5 A couple of fixes to progress output 
Need to have some integration tests.
 2021-11-21 16:02:45 -08:00
Zhaofeng Li
b3016ad3a7 progress/plain: Fix label alignment 2021-11-21 16:02:45 -08:00
Zhaofeng Li
46090598eb apply-local: Fix output after rewrite 2021-11-21 15:30:52 -08:00
Zhaofeng Li
0cb3f8e968 Redesign deployment process (again) 
We now ship Events from different parts of the deployment process
via a channel to a job monitor.
 2021-11-21 00:11:37 -08:00
Zhaofeng Li
5c84134af3 Refactor node names 2021-11-20 23:34:52 -08:00
Zhaofeng Li
47ccc6c9c4 nix/info.rs: Update message about nixUnstable for 21.11 2021-11-18 15:41:02 -08:00
Zhaofeng Li
c271780b63 Improve error reporting 2021-11-18 13:15:20 -08:00
Zhaofeng Li
fae58994e4 Prepare for 0.2.0 release 2021-11-18 13:15:20 -08:00
Zhaofeng Li
e5665775b2 Preserve elapsed time in store path listing after build 
Fixes #36.
 2021-11-18 01:38:58 -08:00
Zhaofeng Li
80a71d9023 cli.rs: Point user at manual 2021-11-17 22:21:00 -08:00
Zhaofeng Li
4497ef296e Add manual 
We finally have some real documentation :)
 2021-11-17 22:21:00 -08:00
Zhaofeng Li
c42c97d2f6 Add internal command to render CLI usage text for manual 2021-11-17 22:21:00 -08:00
Zhaofeng Li
1535857acc Also disallow pinning to a Nixpkgs lambda in Flakes 
Somehow missed this one *shrugs*
 2021-11-17 22:21:00 -08:00
Zhaofeng Li
006cb2c5ee eval.nix: Make the uninitialized nixpkgs error more informative 2021-11-16 21:01:40 -08:00
Zhaofeng Li
f716daa3a1 Migrate to indicatif 0.7 beta 
Now there is no need for the ugly hack where we drove the spinner
in a separate thread :)
 2021-11-16 21:01:33 -08:00
Zhaofeng Li
7d15d08d6d Add test command for progress spinners 2021-11-16 21:01:33 -08:00
Zhaofeng Li
86eeeece3c command: Rename introspect to eval 2021-10-28 17:27:30 -07:00
Zhaofeng Li
f7eb121260 Disallow uninitialized meta.nixpkgs in Flakes 2021-10-28 17:10:58 -07:00
Zhaofeng Li
765f42fa24 introspect: Support actually instantiating the expression 2021-10-28 14:09:35 -07:00
Zhaofeng Li
58f2bf391f src/util.rs: No need to list 2021-10-28 14:09:35 -07:00
Zhaofeng Li
0e0a1e84f0 Make flake resolution (slightly) less terrible 
Instead of using `path:` which always copies the entire directory,
we now try to resolve the Flake URI using `nix flake metadata` which
may give us a `git+file:`.
 2021-10-25 23:38:10 -07:00
Zhaofeng Li
b48753239a hive.rs: Canonicalize flake path 
Relative paths are no longer allowed in newer Nix versions.
 2021-10-25 21:53:38 -07:00
Zhaofeng Li
6d6e33fcd4 nix: Remove unneeded ok() 2021-10-23 20:49:14 -07:00
Bjørn Forsman
4106a73e75 Allow selecting ssh user dynamically 
...by setting `deployment.targetUser = null`.

This allows sharing a deployment file (hive.nix/flake.nix) between
multiple admins, without having to use a shared root account.
 2021-10-23 15:06:56 +02:00
Zhaofeng Li
37b43cd6d7 eval.nix: Support autocall for hive configuration 2021-08-26 19:59:22 -07:00
Zhaofeng Li
7cc6552ee3 hive.rs: Remove unwrap in builder_args() 2021-08-26 19:59:22 -07:00
Zhaofeng Li
7b69946d98 Ensure key ownerships are set correctly 
Depending on when keys are uploaded (`deployment.keys.<name>.uploadAt`):

`pre-activation`:
We set the ownerships in the uploader script opportunistically and
continue if the user/group does not exist. Then, in the activation
script, we set the ownerships of all pre-activation keys.

`post-activation`:
We set the ownerships in the uploader script and fail if the
user/group does not exist.

The ownerships will be correct regardless of which mode is in use.

Fixes #23. Also a more complete solution to #10.
 2021-08-26 12:54:41 -07:00
Zhaofeng Li
24339bcca7 Add deployment.keys.<name>.uploadAt 
This mirrors the functionality recently added in morph and allows
for the uploading of keys after system profile activation.

Fixes #10.
 2021-08-24 23:25:46 -07:00
Zhaofeng Li
135a42b20f eval.nix: Add meta.specialArgs 2021-07-16 22:52:23 -07:00
Zhaofeng Li
671cf38796 hive.rs: Pass --builders to nix-instantiate as well 2021-07-13 01:38:52 -07:00
Zhaofeng Li
c644f79ad1 cli.rs: Indicate support for flakes 2021-06-29 01:19:13 -07:00
Zhaofeng Li
67db0e73d1 Add check for Flakes support 2021-06-29 01:02:43 -07:00
Zhaofeng Li
e50ba82bf2 Add basic Flakes support 
Co-authored-by: Alex Zero <joseph@marsden.space>
 2021-06-29 01:02:43 -07:00
Zhaofeng Li
22ae18f5e7 Exit with non-zero code if any node fails to deploy 
The exit codes are in flux and should not be relied upon.

Fixes #28.
 2021-05-24 00:15:38 -07:00
Zhaofeng Li
960af8f793 Add deployment.privilegeEscalationCommand 
This adds a NixOps-equivalent option for non-root deployment
on remote hosts.

Fixes #27.
 2021-05-24 00:15:38 -07:00
Zhaofeng Li
39d612a5e7 ssh: Remove dead code 2021-05-24 00:15:38 -07:00
Zhaofeng Li
99ba8db335
Merge pull request #21 from jasonrm/machines-file 
eval.nix: Adds meta.machinesFile option that is passed to Nix as builder option
 2021-05-07 16:25:36 -07:00
Zhaofeng Li
16ccdbc700 Better handling of killed processes 2021-04-28 15:09:40 -07:00
Zhaofeng Li
44b421c2c7 key.rs: Fix typo (user -> group) 
Fixes #22.
 2021-04-19 15:40:19 -07:00
Jason R. McNeil
3ee97c2a76 apply: Add deployment.replaceUnknownProfiles option and --force-replace-unknown-profiles switch 
If `deployment.replaceUnknownProfiles` is set to false, a diverged hive
config (in a shared git repo for example) won't result in accidentally
undoing another applied configuration profile.

The deployment option is set to true so that fiction is minimized from
aggressive garbage collection, first time profile application and low
contention hives.
 2021-04-10 13:42:38 -07:00
Jason R. McNeil
e0465567b2 eval.nix: Adds meta.machinesFile option that is passed to Nix as builders argument 2021-04-09 23:54:13 -07:00
Zhaofeng Li
0927fe9dc1 cli: Add hidden command to generate shell autocompletion 2021-03-23 14:14:04 -07:00
Zhaofeng Li
53b55a102e cli: Set bin_name to be lower case 2021-03-23 14:14:04 -07:00
Zhaofeng Li
ba2574755a Separate global CLI setup into module 2021-03-23 14:14:04 -07:00
Zhaofeng Li
8abcd5d53b "Successfully built" -> "Build successful" for consistency 2021-03-18 15:13:34 -07:00
Zhaofeng Li
b44dd1f877 apply_local: Don't bother evaluating other hosts 2021-03-18 15:05:05 -07:00
Zhaofeng Li
e9487ced9e host: Use the key uploader script for both SSH and local 2021-03-17 22:39:05 -07:00
Zhaofeng Li
29cfd45141 Miscellaneous doc fixes 2021-03-17 19:07:26 -07:00
Zhaofeng Li
610a725ba2 Add --keep-result to create GC roots for profiles 
This resembles the behavior of morph.

Reference: #18
 2021-03-17 14:59:57 -07:00
Zhaofeng Li
81375e71b2 deployment: Display the resulting paths if the goal is to build only 
Reference: #18
 2021-03-17 14:59:43 -07:00
Zhaofeng Li
180d2f2435
Merge pull request #17 from jasonrm/nix-copy-ssh-options 
Makes SSH options available to nix-copy-closure
 2021-03-14 23:06:04 -07:00
Jason R. McNeil
4098bf73bc Makes SSH options available to nix-copy-closure 2021-03-14 22:20:47 -07:00
Zhaofeng Li
fb5ff6f9c9 Allow specifying alternative commands for privilege escalation 
Fixes #16.
 2021-03-10 08:42:51 -08:00
Zhaofeng Li
082a033443 eval.nix: Exclude internal Nixpkgs config options from node override warning 2021-02-17 23:06:22 -08:00
Zhaofeng Li
3b005b0949 src/progress.rs: Oops, forgot to add 2021-02-17 22:56:34 -08:00
Zhaofeng Li
e32e130621 Always print the entire log for failures in eval and build 
This makes it easier to debug configuration issues without -v.

Fixes #14.
 2021-02-17 22:48:26 -08:00
Zhaofeng Li
d16a13654c Merge nixpkgs.config and nixpkgs.overlays 
This replaces #12, and allows for Nixpkgs overlays and config to be overridden
in machine configs. With #12, overlays set in machine configurations
(`nixpkgs.overlays`) get silently ignored.
 2021-02-17 22:46:01 -08:00
Zhaofeng Li
9eae937b42 apply: Disable configuration of build process limit 2021-02-17 08:09:15 -08:00
Justinas Stankevicius
10f98d715f Propagate same 'pkgs' to all modules 2021-02-16 20:53:43 +02:00
Zhaofeng Li
6a9282e121 exec: Remove outdated doc 2021-02-12 16:21:38 -08:00
Zhaofeng Li
30dc352eb9 eval.nix: Add type checking to meta/network 
Primarily to make the evaluation error out if the configuration
tries to use non-existent options (e.g., pinning Nixpkgs with
morph-specific options).
 2021-02-12 14:52:09 -08:00
Zhaofeng Li
d0bba90d04 ssh: Fix shell escaping 
The previous `sh -c` invocation was incorrect and just happened
to work on hosts with a Bourne-compatible shell set as the login
shell. Commands in the deploy script were being executed in the
login shell.
 2021-02-12 13:55:44 -08:00
Zhaofeng Li
95ddbcbfd6 ssh/deploy-key: Skip chown if the user/group doesn't exist 
This matches the behavior of NixOps.

Potential solution to #10.
 2021-02-12 13:54:17 -08:00
Zhaofeng Li
dbd66d7c7c Add initial set of tests 2021-02-11 13:27:21 -08:00
Justinas Stankevicius
4c7f8eb838 keyCommand: on error, do not upload key, report 2021-02-11 21:16:56 +02:00
Zhaofeng Li
e49e9367c0 key: Serialize KeySource through an intermediate struct 
Well, still better than `if/else`-ing all the way. Also we
definitely need unit tests.

See #8.
 2021-02-11 00:51:11 -08:00
Zhaofeng Li
2886662e18 nix: Key names can contain one path component only 
Well, I changed my mind and this should be cleaner.
 2021-02-10 18:17:55 -08:00
Zhaofeng Li
52622ecd27 Add 'deployment.keys.<key>.keyCommand' support 
Fixes #3.
 2021-02-10 18:08:47 -08:00
Zhaofeng Li
ce9f639a53 key: Make the key source better typed 2021-02-10 17:34:52 -08:00
Zhaofeng Li
62753ea138 progress: Let's just call them "tasks" instead of "processes"... 2021-02-10 17:20:49 -08:00
Zhaofeng Li
9d59a6a288
Merge pull request #5 from justinas/keys-keyfile 
Add 'deployment.keys.<key>.keyFile' option
 2021-02-10 17:20:28 -08:00
Justinas Stankevicius
d90fc56bc3 Implement key upload from local file 2021-02-10 20:37:54 +02:00
Justinas Stankevicius
f521f19629 Add deployment.keys.<key>.keyFile option 2021-02-10 20:37:54 +02:00
Zhaofeng Li
afabd8c6f9 Minor tokio cleanup 2021-02-10 10:29:17 -08:00
Zhaofeng Li
9f4d5a2221 Target stable toolchain 2021-02-10 00:41:02 -08:00
Zhaofeng Li
9a588815c8 exec.rs: Wording 2021-02-10 00:22:17 -08:00
Zhaofeng Li
d9d9bf48f6 apply.rs: Remove unused build limit flag 2021-02-09 22:33:45 -08:00
Zhaofeng Li
78a6825be6 Add exec command 2021-02-09 22:07:10 -08:00
Zhaofeng Li
1c9e7cdb83 Allow customization of SSH configurations 2021-02-09 21:02:00 -08:00
Zhaofeng Li
a2fa8f1da7 Clean up logging / progress display 2021-02-09 19:28:45 -08:00
Zhaofeng Li
8934726664 More fixes to key deployment and logging 2021-02-09 14:57:11 -08:00
Zhaofeng Li
84aa165aef Refactoring and deployment.keys implementation 
More refactoring of the deployment process, as well as an initial
implementation of `deployment.keys`.

Fixes #2.
 2021-02-08 19:00:29 -08:00
Zhaofeng Li
21c2bef3ad Small fixes to eval logging 2021-02-05 02:20:57 -08:00
Zhaofeng Li
406e5a6443 apply-local: Allow overriding the node name 2021-02-05 02:20:08 -08:00
Zhaofeng Li
6e909477ae nix/deployment: Well, that's an embarrassing typo... 2021-01-29 21:24:05 -08:00
Zhaofeng Li
68ecb095b8 nix: Small fixes to logging 2021-01-28 23:58:54 -08:00
Zhaofeng Li
ade2095919 Remove unused --no-build-substitutes 
Ended up not implementing this.
 2021-01-24 17:26:50 -08:00
Zhaofeng Li
506b894be6 Redesign deployment process 
Now evaluation can be automatically split into chunks based on available
RAM. All three stages of the deployment process (evaluate, build,
apply) can happen concurrently.

Fixes #1.
 2021-01-24 14:08:48 -08:00
Zhaofeng Li
f53ebef41c nix/eval.nix: Disallow setting network and meta at the same time 2021-01-17 00:06:45 -08:00
Zhaofeng Li
2050e84bb8 nix/eval.nix: Support setting meta.nixpkgs to a .nix that returns an initialized Nixpkgs attrset 2021-01-17 00:06:20 -08:00
Zhaofeng Li
f3bf3dc492 Allow disabling --use-substitutes and --gzip during copying 2021-01-13 12:20:34 -08:00
Zhaofeng Li
2cb429ed8d Minor fixes to CLI help messages 2021-01-01 20:45:41 -08:00
Zhaofeng Li
be8e30cebf deployment.rs: Strip line endings for error logs 2020-12-30 16:54:07 -08:00
Zhaofeng Li
1125eb6d1b nix: Suppress GC warnings 
The derivations and built closures are intentionally not added as GC
roots. Maybe we can provide an option for those who want GC roots
to be created.
 2020-12-29 12:10:00 -08:00
Zhaofeng Li
4c1ce95c15 apply_local.rs: s/Coult/Could 2020-12-29 12:05:06 -08:00