ecoppens/colmena
1
0
Fork 0
forked from DGNum/colmena
Code Pull requests Activity
392 commits 2 branches 0 tags 994 KiB
Commit graph

220 commits

Author SHA1 Message Date
Zhaofeng Li
26b2203da9 Update deps 2022-03-07 22:02:04 -08:00
Zhaofeng Li
108c135700 Rename: NixFailure -> ChildFailure, NixKilled -> ChildKilled 2022-03-07 22:02:04 -08:00
Glenn McDonald
9c179b0db8
apply_local: Fix nixos detection from os-release (#63) 2022-03-07 20:52:22 -08:00
Zhaofeng Li
2b281286d0
eval.nix: Add "${name}-key.service" units for keys (#51) 
Fixes #48.
 2022-02-16 10:23:27 -08:00
Zhaofeng Li
85b0107b59 eval.nix: Skip recursive type checking for nodes in colmena eval as well 
This matches the behavior of the `nodes` argument in NixOS modules.
 2022-02-16 01:11:06 -08:00
Linus Heckemann
efa7322587
streaming evaluator: respect eval-nodes-limit (#58) 2022-02-16 01:10:52 -08:00
Zhaofeng Li
c088925e0d Misc doc tweaks 2022-02-05 18:43:43 -08:00
Zhaofeng Li
45b135c1b1 nix_eval_jobs: Add timeouts to tests 
Sometimes nix-eval-jobs would lock up indefinitely, so let's make it
fail fast in CI.

Also ignore test_json_global_error while we investigate.
 2022-02-04 20:58:17 -08:00
Zhaofeng Li
856f82644e apply_local: Fix argument name typo 
Caught by debug assertions in clap 3.0. We should switch to the Derive
API where possible to prevent this from happening in the future.
 2022-01-27 18:48:25 -08:00
Zhaofeng Li
f674ddf173 key: Compute full path to key file in Nix 2022-01-25 14:22:26 -08:00
Zhaofeng Li
e58dde1be0 eval.nix: Don't override modulesPath 
`eval-config.nix` will set the canonical path for us that will then be used
to strip the module path prefixes in `nixos/modules/misc/documentation.nix` [1].
This should fix split docs build problems described in #50.

[1] 1511e72b75/nixos/modules/misc/documentation.nix (L79)
 2022-01-25 14:22:26 -08:00
Zhaofeng Li
7cd9b6f70c eval.nix: Don't pass baseModules to modules 
This is unnecessary and breaks the separation of user `modules`
and `baseModules` in `eval-config.nix`.
 2022-01-25 14:22:26 -08:00
Zhaofeng Li
9251019723 nix_eval_jobs: Errors may not have an attribute name attached 2022-01-25 14:22:26 -08:00
Zhaofeng Li
c0107b21e0 General code cleanup 2022-01-25 14:22:26 -08:00
Paul Haerle
5b8971a0f4
eval.nix: expose nixosModules.deploymentOptions (#49) 
Allow flake users to import .#nixosModules.deploymentOptions
into their flake, so that the same expression can be used for both,
.#colmena.$host as well as .#nixosConfiguration.$host, without the
latter complaining about undefined options in "deployment".
 2022-01-23 10:06:41 -08:00
Zhaofeng Li
0d9198c351 spinner: Work around the lack of newline after completion 2022-01-22 17:50:53 -08:00
Zhaofeng Li
87189ec3df nix_info: Display pinned nix-eval-jobs path 2022-01-22 17:50:53 -08:00
Zhaofeng Li
cf9a72a1d4 nix: Add initial nix-eval-job integration 
A DrvSetEvaluator is able to evaluate attribute sets of derivations,
streaming results as they come in.
 2022-01-22 17:50:53 -08:00
Zhaofeng Li
3e40e84e19 nix: Add NixExpression abstraction 
This decouples expression generation from evaluation. `NixExpression`s
that evaluate to a set of derivations can be fed to `DrvSetEvaluator`s
which may be able to parallelize evaluation.
 2022-01-22 17:50:53 -08:00
Zhaofeng Li
9fbb6b133b job: Make it possible to have a JobHandle not connected to a JobMonitor 2022-01-22 17:50:53 -08:00
Zhaofeng Li
55ce6d078e Remove unused stuff 2022-01-22 17:50:53 -08:00
Zhaofeng Li
d3e556027f deployment: Move chunked mode into self-contained function 2022-01-21 00:45:12 -08:00
Zhaofeng Li
82361e5ea5 hive: Clean up, make way for nix-eval-jobs 2022-01-21 00:45:12 -08:00
Zhaofeng Li
3fa0dee6b0 hive: Convert derivations to drvPath in separate attribute 
This is to make way for the nix-eval-jobs streaming evaluator.
 2022-01-21 00:45:12 -08:00
Zhaofeng Li
abb74b9f49 Add EditorConfig rules, fix trailing whitespaces 2022-01-08 01:20:36 -08:00
Zhaofeng Li
deca292b53 Refactor NixOptions 2022-01-08 01:20:36 -08:00
Zhaofeng Li
31fd1e49ac Move nix::{NixResult, NixError} to error::{ColmenaResult, ColmenaError} 2022-01-08 01:20:36 -08:00
Zhaofeng Li
16ed9d8c66 Move nix::NixCommand to util::CommandExt 2022-01-08 01:20:36 -08:00
Zhaofeng Li
f92236da46 Refactor current profile detection 2022-01-08 01:20:36 -08:00
Zhaofeng Li
6401ce4c3c cli: Remove target from logs 
format_target is enabled by default in env_logger 0.9.0. It makes
the logs look cluttered and is not very useful for our use case.
 2022-01-05 14:01:05 -08:00
Zhaofeng Li
da7907c874 General code cleanup 2022-01-05 14:01:05 -08:00
Zhaofeng Li
f234e16e80 manual: Fix colorized CLI help 
In clap 3.0, it's no longer possible to get `App::write_long_help` to
output colorized text (it doesn't invoke the Colorizer at all). So let's
move the generation outside of Rust.
 2022-01-03 10:37:03 -08:00
Zhaofeng Li
d2762757f0 command/eval: Fix introspect alias 
On clap 3.0, resetting the App::name still causes clap to complain
that the subcommand doesn't exist in debug mode.
 2022-01-03 10:37:03 -08:00
Zhaofeng Li
79d1010356 command: Change wording for consistency 2022-01-03 10:37:03 -08:00
Zhaofeng Li
81e2982bbb Migrate to clap 0.3 2022-01-03 10:37:03 -08:00
Zhaofeng Li
f9fab83030 deployment: Remove useless Arc clones 2022-01-02 13:16:24 -08:00
Zhaofeng Li
7bf57fd1ad nix/node_filter: Forgot to add 2022-01-01 16:47:38 -08:00
Zhaofeng Li
98897bf4de Support building on target nodes 
This partially addresses #33, and allows Colmena to be used more
easily on bandwidth-constrained hosts and macOS.

With `deployment.buildOnTarget = true;` deployment works fine from
macOS without designated builders, except when IFD is involved.
 2022-01-01 16:41:35 -08:00
Zhaofeng Li
872f944743 eval.nix: Skip recursive type checking when cross-referencing configs through the nodes argument 
This makes evaluation faster especially when `nodes` is heavily used in
the configuration (like in my auto-meshing setup). This matches the
behavior of Morph.
 2022-01-01 16:41:35 -08:00
Zhaofeng Li
cdbb69617f eval.nix: Support specifying a list of configs 2022-01-01 16:41:35 -08:00
Zhaofeng Li
bc3450b3ce cli: Disable apply-local on non-Linux platforms 2022-01-01 16:41:35 -08:00
Zhaofeng Li
f9f4dd5f63 Remove another ugly toJSON hack 2022-01-01 16:41:35 -08:00
Zhaofeng Li
111bc9dea7 Remove test-progress subcommand from release builds 2022-01-01 16:41:35 -08:00
Zhaofeng Li
f809d3b21c nix/host: Always copy outputs to remote along with derivations when realizing 
This prevents useless rebuilds when trying to realize a derivation on
a remote host. This code path isn't actually used by Colmena at the
moment.
 2022-01-01 16:41:35 -08:00
Zhaofeng Li
2b652f7236 hive: Add another test for nixpkgs.system 2022-01-01 16:41:35 -08:00
Zhaofeng Li
8f77184d58 eval.nix: Inherit pkgs.system in evalConfig 2021-12-18 14:35:06 -08:00
Zhaofeng Li
bd4493da73 nix: Move eval.nix tests into /hive 2021-12-18 14:35:06 -08:00
Zhaofeng Li
31fc98cfa7 nix: Remove ugly toJSON hack 
We now evaluate with --strict.
 2021-12-07 23:13:31 -08:00
Zhaofeng Li
eebded1786 Build each node individually 
Now nodes that take a long time to build won't bottleneck the
deployment of other nodes in the same chunk.

Fixes #47.
 2021-12-07 23:13:31 -08:00
Zhaofeng Li
39f597f778 hive: Pass --read-write-mode to nix-instantiate --eval 
This enables the use of IFD inside `meta` and node `deployment.*`
options, from which we obtain the values with `nix-instantiate --eval`.

Fixes #45.
 2021-12-05 12:34:44 -08:00
Zhaofeng Li
7433661aed Add deployment.keys.<name>.name 2021-12-05 01:14:12 -08:00
Zhaofeng Li
0f8873027f utils: Don't panic in capture_stream 2021-12-05 01:14:12 -08:00
Zhaofeng Li
ddccad4fb9 eval.nix: Auto-call functors as well 
This makes some very cursed setup work :P
 2021-12-04 01:03:26 -08:00
Zhaofeng Li
7757a0e45d command/apply: Use verbose output when goal is "dry-activate" 
It doesn't make sense to use the spinners since you can't see
the full output from the activation script.

Fixes #44.
 2021-12-04 01:03:26 -08:00
Zhaofeng Li
7b1feffb9e util: Log the configuration in use 2021-12-04 01:03:26 -08:00
Zhaofeng Li
9cd2af9438 cli: Re-order arguments in help message 2021-12-04 01:03:26 -08:00
Zhaofeng Li
f253e6eb18 cli: Allow configuring output colorization 
We now follow the <https://bixense.com/clicolors> standard.
 2021-12-04 01:03:26 -08:00
Zhaofeng Li
dc57b489ea progress: Restore tty-detection after refactoring 2021-12-04 01:03:26 -08:00
Zhaofeng Li
064432f38e nix/deployment: Misc cleanup 2021-12-04 01:03:26 -08:00
Zhaofeng Li
04976e5fda job: Make event formatting less ugly 2021-12-03 00:06:21 -08:00
Zhaofeng Li
a3e292d7fb nix: Another deployment parallelization oops 2021-12-03 00:06:17 -08:00
Zhaofeng Li
2bba64a002 nix: Fix deployment parallelization 2021-12-02 23:49:46 -08:00
Zhaofeng Li
a42e8c5bf0 Misc cleanup 2021-11-23 14:12:01 -08:00
Zhaofeng Li
ec51f5703f deployment: A few oops 2021-11-22 01:53:08 -08:00
Zhaofeng Li
572f77dbf5 A couple of fixes to progress output 
Need to have some integration tests.
 2021-11-21 16:02:45 -08:00
Zhaofeng Li
b3016ad3a7 progress/plain: Fix label alignment 2021-11-21 16:02:45 -08:00
Zhaofeng Li
46090598eb apply-local: Fix output after rewrite 2021-11-21 15:30:52 -08:00
Zhaofeng Li
0cb3f8e968 Redesign deployment process (again) 
We now ship Events from different parts of the deployment process
via a channel to a job monitor.
 2021-11-21 00:11:37 -08:00
Zhaofeng Li
5c84134af3 Refactor node names 2021-11-20 23:34:52 -08:00
Zhaofeng Li
47ccc6c9c4 nix/info.rs: Update message about nixUnstable for 21.11 2021-11-18 15:41:02 -08:00
Zhaofeng Li
c271780b63 Improve error reporting 2021-11-18 13:15:20 -08:00
Zhaofeng Li
fae58994e4 Prepare for 0.2.0 release 2021-11-18 13:15:20 -08:00
Zhaofeng Li
e5665775b2 Preserve elapsed time in store path listing after build 
Fixes #36.
 2021-11-18 01:38:58 -08:00
Zhaofeng Li
80a71d9023 cli.rs: Point user at manual 2021-11-17 22:21:00 -08:00
Zhaofeng Li
4497ef296e Add manual 
We finally have some real documentation :)
 2021-11-17 22:21:00 -08:00
Zhaofeng Li
c42c97d2f6 Add internal command to render CLI usage text for manual 2021-11-17 22:21:00 -08:00
Zhaofeng Li
1535857acc Also disallow pinning to a Nixpkgs lambda in Flakes 
Somehow missed this one *shrugs*
 2021-11-17 22:21:00 -08:00
Zhaofeng Li
006cb2c5ee eval.nix: Make the uninitialized nixpkgs error more informative 2021-11-16 21:01:40 -08:00
Zhaofeng Li
f716daa3a1 Migrate to indicatif 0.7 beta 
Now there is no need for the ugly hack where we drove the spinner
in a separate thread :)
 2021-11-16 21:01:33 -08:00
Zhaofeng Li
7d15d08d6d Add test command for progress spinners 2021-11-16 21:01:33 -08:00
Zhaofeng Li
86eeeece3c command: Rename introspect to eval 2021-10-28 17:27:30 -07:00
Zhaofeng Li
f7eb121260 Disallow uninitialized meta.nixpkgs in Flakes 2021-10-28 17:10:58 -07:00
Zhaofeng Li
765f42fa24 introspect: Support actually instantiating the expression 2021-10-28 14:09:35 -07:00
Zhaofeng Li
58f2bf391f src/util.rs: No need to list 2021-10-28 14:09:35 -07:00
Zhaofeng Li
0e0a1e84f0 Make flake resolution (slightly) less terrible 
Instead of using `path:` which always copies the entire directory,
we now try to resolve the Flake URI using `nix flake metadata` which
may give us a `git+file:`.
 2021-10-25 23:38:10 -07:00
Zhaofeng Li
b48753239a hive.rs: Canonicalize flake path 
Relative paths are no longer allowed in newer Nix versions.
 2021-10-25 21:53:38 -07:00
Zhaofeng Li
6d6e33fcd4 nix: Remove unneeded ok() 2021-10-23 20:49:14 -07:00
Bjørn Forsman
4106a73e75 Allow selecting ssh user dynamically 
...by setting `deployment.targetUser = null`.

This allows sharing a deployment file (hive.nix/flake.nix) between
multiple admins, without having to use a shared root account.
 2021-10-23 15:06:56 +02:00
Zhaofeng Li
37b43cd6d7 eval.nix: Support autocall for hive configuration 2021-08-26 19:59:22 -07:00
Zhaofeng Li
7cc6552ee3 hive.rs: Remove unwrap in builder_args() 2021-08-26 19:59:22 -07:00
Zhaofeng Li
7b69946d98 Ensure key ownerships are set correctly 
Depending on when keys are uploaded (`deployment.keys.<name>.uploadAt`):

`pre-activation`:
We set the ownerships in the uploader script opportunistically and
continue if the user/group does not exist. Then, in the activation
script, we set the ownerships of all pre-activation keys.

`post-activation`:
We set the ownerships in the uploader script and fail if the
user/group does not exist.

The ownerships will be correct regardless of which mode is in use.

Fixes #23. Also a more complete solution to #10.
 2021-08-26 12:54:41 -07:00
Zhaofeng Li
24339bcca7 Add deployment.keys.<name>.uploadAt 
This mirrors the functionality recently added in morph and allows
for the uploading of keys after system profile activation.

Fixes #10.
 2021-08-24 23:25:46 -07:00
Zhaofeng Li
135a42b20f eval.nix: Add meta.specialArgs 2021-07-16 22:52:23 -07:00
Zhaofeng Li
671cf38796 hive.rs: Pass --builders to nix-instantiate as well 2021-07-13 01:38:52 -07:00
Zhaofeng Li
c644f79ad1 cli.rs: Indicate support for flakes 2021-06-29 01:19:13 -07:00
Zhaofeng Li
67db0e73d1 Add check for Flakes support 2021-06-29 01:02:43 -07:00
Zhaofeng Li
e50ba82bf2 Add basic Flakes support 
Co-authored-by: Alex Zero <joseph@marsden.space>
 2021-06-29 01:02:43 -07:00
Zhaofeng Li
22ae18f5e7 Exit with non-zero code if any node fails to deploy 
The exit codes are in flux and should not be relied upon.

Fixes #28.
 2021-05-24 00:15:38 -07:00
Zhaofeng Li
960af8f793 Add deployment.privilegeEscalationCommand 
This adds a NixOps-equivalent option for non-root deployment
on remote hosts.

Fixes #27.
 2021-05-24 00:15:38 -07:00
Zhaofeng Li
39d612a5e7 ssh: Remove dead code 2021-05-24 00:15:38 -07:00