ecoppens/colmena
1
0
Fork 0
forked from DGNum/colmena
Code Pull requests Activity
443 commits 2 branches 0 tags 994 KiB
Commit graph

245 commits

Author SHA1 Message Date
Zhaofeng Li
dd3f4b252c host/local: Upload keys with sudo as well 2022-06-10 11:29:25 -07:00
Zhaofeng Li
d7aa4ab864 hive/tests: Fix editorconfig 2022-06-09 11:35:02 -07:00
Zhaofeng Li
9db25bd914 deployment: Move GC root creation after build 
Previously GC roots were only created after successful activation.
 2022-06-09 11:31:49 -07:00
Zhaofeng Li
2b0bafa155 eval.nix: Restore signature in introspect 
It was broken in 9bd5e7bb25.
 2022-06-09 11:31:49 -07:00
Zhaofeng Li
b5629dca83 eval.nix: Enable config merging for meta.nixpkgs on 21.11+ 
Ref: #74 #15
 2022-06-03 23:51:32 -07:00
Zhaofeng Li
1cd3aa6401 eval.nix: Accept path-likes in meta.nixpkgs as well 
This fixes usage with Niv.
 2022-06-03 23:51:32 -07:00
Zhaofeng Li
b87da69558 Don't import impure overlays if meta.nixpkgs is a path 
Also update documentations to recommend specifying `overlays = []`
explicitly.

Fixes #39.
 2022-06-03 23:51:32 -07:00
Zhaofeng Li
ca12be27ed apply-local: Escalate privileges only during activation 
Fixes #85.
 2022-06-03 23:51:32 -07:00
Zhaofeng Li
fa07814abf hive/assets.rs: Use OpenOptions::new() instead 2022-06-03 23:51:32 -07:00
Zhaofeng Li
4037b61a2b apply: Fix typo in help 2022-06-03 23:51:32 -07:00
Zhaofeng Li
af3247faeb Update docs after #91 2022-06-03 23:51:32 -07:00
Zhaofeng Li
ed91898e59 Clippy 2022-06-03 23:51:32 -07:00
Zhaofeng Li
ac8852dcdf hive: Refactor static file creation 2022-06-02 16:39:45 -07:00
Wanja Hentze
6fe48ccd04 nix-eval-jobs: Strip quotes surrounding attribute names 
Fixes #92.
 2022-06-02 16:39:45 -07:00
Victor Nawothnig
e937eb8faa Do not use an explicit user for ssh when deploymentUser is null 2022-06-01 12:30:49 +02:00
Zhaofeng Li
429a0f5aa1 eval.nix: Suppress _module.args docs 2022-05-22 02:15:33 -07:00
Zhaofeng Li
1be16d552d apply: Add --reboot flag 
Fixes #46.
 2022-05-22 02:15:33 -07:00
Zhaofeng Li
e239cbd260 host: Add get_current_system_profile 2022-05-22 02:15:33 -07:00
Zhaofeng Li
c8b79e7e2d ssh: Clean up 2022-05-22 02:15:33 -07:00
David Arnold
9bd5e7bb25 imp: soc of eval, modules & options 
- the reason for this change is to have more transparent separation
  of concern between effectuations of the module system and pre-module
  system effectuations

- with improved flakes support down the line, pre-module system
  effectuations will get more complex

- this also allows to patch the aspects of the evaluation individually
  while tracking other components from upstream. eg. path options & eval
  but not modules
 2022-05-22 01:54:52 -07:00
Zhaofeng Li
ea4f2ba6dc Migrate other doc links to new domain 2022-05-13 19:14:58 -07:00
Zhaofeng Li
3e7341a98c utils.rs: Tolerate invalid UTF-8 in streams 
This will substitute any invalid UTF-8 sequence with `\u{fffd}`. Such
substitutions are okay for human-readable log outputs.

Correctness: A side-effect of this is that it also affects the code
that captures filesystem paths from stdout, in which case any substitution
is unacceptable. Currently we only capture store paths from Nix, and Nix
restricts characters allowed in derivation names. Nevertheless we will
refactor this for correctness.

Fixes #75.
 2022-04-25 00:03:57 -07:00
Zhaofeng Li
6a64625fda eval.nix: Ignore typed allowAliases config when checking unoverridden config keys 
This was added in NixOS/nixpkgs#166792.

Fixes #74.
 2022-04-25 00:03:57 -07:00
Zhaofeng Li
ab754ac2e3 Squelch dead code warnings on non-Linux platforms (aka macOS) 2022-04-19 00:29:35 -07:00
Zhaofeng Li
5a81996afb ssh: Enable BatchMode 2022-04-19 00:29:35 -07:00
Zhaofeng Li
26b2203da9 Update deps 2022-03-07 22:02:04 -08:00
Zhaofeng Li
108c135700 Rename: NixFailure -> ChildFailure, NixKilled -> ChildKilled 2022-03-07 22:02:04 -08:00
Glenn McDonald
9c179b0db8
apply_local: Fix nixos detection from os-release (#63) 2022-03-07 20:52:22 -08:00
Zhaofeng Li
2b281286d0
eval.nix: Add "${name}-key.service" units for keys (#51) 
Fixes #48.
 2022-02-16 10:23:27 -08:00
Zhaofeng Li
85b0107b59 eval.nix: Skip recursive type checking for nodes in colmena eval as well 
This matches the behavior of the `nodes` argument in NixOS modules.
 2022-02-16 01:11:06 -08:00
Linus Heckemann
efa7322587
streaming evaluator: respect eval-nodes-limit (#58) 2022-02-16 01:10:52 -08:00
Zhaofeng Li
c088925e0d Misc doc tweaks 2022-02-05 18:43:43 -08:00
Zhaofeng Li
45b135c1b1 nix_eval_jobs: Add timeouts to tests 
Sometimes nix-eval-jobs would lock up indefinitely, so let's make it
fail fast in CI.

Also ignore test_json_global_error while we investigate.
 2022-02-04 20:58:17 -08:00
Zhaofeng Li
856f82644e apply_local: Fix argument name typo 
Caught by debug assertions in clap 3.0. We should switch to the Derive
API where possible to prevent this from happening in the future.
 2022-01-27 18:48:25 -08:00
Zhaofeng Li
f674ddf173 key: Compute full path to key file in Nix 2022-01-25 14:22:26 -08:00
Zhaofeng Li
e58dde1be0 eval.nix: Don't override modulesPath 
`eval-config.nix` will set the canonical path for us that will then be used
to strip the module path prefixes in `nixos/modules/misc/documentation.nix` [1].
This should fix split docs build problems described in #50.

[1] 1511e72b75/nixos/modules/misc/documentation.nix (L79)
 2022-01-25 14:22:26 -08:00
Zhaofeng Li
7cd9b6f70c eval.nix: Don't pass baseModules to modules 
This is unnecessary and breaks the separation of user `modules`
and `baseModules` in `eval-config.nix`.
 2022-01-25 14:22:26 -08:00
Zhaofeng Li
9251019723 nix_eval_jobs: Errors may not have an attribute name attached 2022-01-25 14:22:26 -08:00
Zhaofeng Li
c0107b21e0 General code cleanup 2022-01-25 14:22:26 -08:00
Paul Haerle
5b8971a0f4
eval.nix: expose nixosModules.deploymentOptions (#49) 
Allow flake users to import .#nixosModules.deploymentOptions
into their flake, so that the same expression can be used for both,
.#colmena.$host as well as .#nixosConfiguration.$host, without the
latter complaining about undefined options in "deployment".
 2022-01-23 10:06:41 -08:00
Zhaofeng Li
0d9198c351 spinner: Work around the lack of newline after completion 2022-01-22 17:50:53 -08:00
Zhaofeng Li
87189ec3df nix_info: Display pinned nix-eval-jobs path 2022-01-22 17:50:53 -08:00
Zhaofeng Li
cf9a72a1d4 nix: Add initial nix-eval-job integration 
A DrvSetEvaluator is able to evaluate attribute sets of derivations,
streaming results as they come in.
 2022-01-22 17:50:53 -08:00
Zhaofeng Li
3e40e84e19 nix: Add NixExpression abstraction 
This decouples expression generation from evaluation. `NixExpression`s
that evaluate to a set of derivations can be fed to `DrvSetEvaluator`s
which may be able to parallelize evaluation.
 2022-01-22 17:50:53 -08:00
Zhaofeng Li
9fbb6b133b job: Make it possible to have a JobHandle not connected to a JobMonitor 2022-01-22 17:50:53 -08:00
Zhaofeng Li
55ce6d078e Remove unused stuff 2022-01-22 17:50:53 -08:00
Zhaofeng Li
d3e556027f deployment: Move chunked mode into self-contained function 2022-01-21 00:45:12 -08:00
Zhaofeng Li
82361e5ea5 hive: Clean up, make way for nix-eval-jobs 2022-01-21 00:45:12 -08:00
Zhaofeng Li
3fa0dee6b0 hive: Convert derivations to drvPath in separate attribute 
This is to make way for the nix-eval-jobs streaming evaluator.
 2022-01-21 00:45:12 -08:00
Zhaofeng Li
abb74b9f49 Add EditorConfig rules, fix trailing whitespaces 2022-01-08 01:20:36 -08:00
Zhaofeng Li
deca292b53 Refactor NixOptions 2022-01-08 01:20:36 -08:00
Zhaofeng Li
31fd1e49ac Move nix::{NixResult, NixError} to error::{ColmenaResult, ColmenaError} 2022-01-08 01:20:36 -08:00
Zhaofeng Li
16ed9d8c66 Move nix::NixCommand to util::CommandExt 2022-01-08 01:20:36 -08:00
Zhaofeng Li
f92236da46 Refactor current profile detection 2022-01-08 01:20:36 -08:00
Zhaofeng Li
6401ce4c3c cli: Remove target from logs 
format_target is enabled by default in env_logger 0.9.0. It makes
the logs look cluttered and is not very useful for our use case.
 2022-01-05 14:01:05 -08:00
Zhaofeng Li
da7907c874 General code cleanup 2022-01-05 14:01:05 -08:00
Zhaofeng Li
f234e16e80 manual: Fix colorized CLI help 
In clap 3.0, it's no longer possible to get `App::write_long_help` to
output colorized text (it doesn't invoke the Colorizer at all). So let's
move the generation outside of Rust.
 2022-01-03 10:37:03 -08:00
Zhaofeng Li
d2762757f0 command/eval: Fix introspect alias 
On clap 3.0, resetting the App::name still causes clap to complain
that the subcommand doesn't exist in debug mode.
 2022-01-03 10:37:03 -08:00
Zhaofeng Li
79d1010356 command: Change wording for consistency 2022-01-03 10:37:03 -08:00
Zhaofeng Li
81e2982bbb Migrate to clap 0.3 2022-01-03 10:37:03 -08:00
Zhaofeng Li
f9fab83030 deployment: Remove useless Arc clones 2022-01-02 13:16:24 -08:00
Zhaofeng Li
7bf57fd1ad nix/node_filter: Forgot to add 2022-01-01 16:47:38 -08:00
Zhaofeng Li
98897bf4de Support building on target nodes 
This partially addresses #33, and allows Colmena to be used more
easily on bandwidth-constrained hosts and macOS.

With `deployment.buildOnTarget = true;` deployment works fine from
macOS without designated builders, except when IFD is involved.
 2022-01-01 16:41:35 -08:00
Zhaofeng Li
872f944743 eval.nix: Skip recursive type checking when cross-referencing configs through the nodes argument 
This makes evaluation faster especially when `nodes` is heavily used in
the configuration (like in my auto-meshing setup). This matches the
behavior of Morph.
 2022-01-01 16:41:35 -08:00
Zhaofeng Li
cdbb69617f eval.nix: Support specifying a list of configs 2022-01-01 16:41:35 -08:00
Zhaofeng Li
bc3450b3ce cli: Disable apply-local on non-Linux platforms 2022-01-01 16:41:35 -08:00
Zhaofeng Li
f9f4dd5f63 Remove another ugly toJSON hack 2022-01-01 16:41:35 -08:00
Zhaofeng Li
111bc9dea7 Remove test-progress subcommand from release builds 2022-01-01 16:41:35 -08:00
Zhaofeng Li
f809d3b21c nix/host: Always copy outputs to remote along with derivations when realizing 
This prevents useless rebuilds when trying to realize a derivation on
a remote host. This code path isn't actually used by Colmena at the
moment.
 2022-01-01 16:41:35 -08:00
Zhaofeng Li
2b652f7236 hive: Add another test for nixpkgs.system 2022-01-01 16:41:35 -08:00
Zhaofeng Li
8f77184d58 eval.nix: Inherit pkgs.system in evalConfig 2021-12-18 14:35:06 -08:00
Zhaofeng Li
bd4493da73 nix: Move eval.nix tests into /hive 2021-12-18 14:35:06 -08:00
Zhaofeng Li
31fc98cfa7 nix: Remove ugly toJSON hack 
We now evaluate with --strict.
 2021-12-07 23:13:31 -08:00
Zhaofeng Li
eebded1786 Build each node individually 
Now nodes that take a long time to build won't bottleneck the
deployment of other nodes in the same chunk.

Fixes #47.
 2021-12-07 23:13:31 -08:00
Zhaofeng Li
39f597f778 hive: Pass --read-write-mode to nix-instantiate --eval 
This enables the use of IFD inside `meta` and node `deployment.*`
options, from which we obtain the values with `nix-instantiate --eval`.

Fixes #45.
 2021-12-05 12:34:44 -08:00
Zhaofeng Li
7433661aed Add deployment.keys.<name>.name 2021-12-05 01:14:12 -08:00
Zhaofeng Li
0f8873027f utils: Don't panic in capture_stream 2021-12-05 01:14:12 -08:00
Zhaofeng Li
ddccad4fb9 eval.nix: Auto-call functors as well 
This makes some very cursed setup work :P
 2021-12-04 01:03:26 -08:00
Zhaofeng Li
7757a0e45d command/apply: Use verbose output when goal is "dry-activate" 
It doesn't make sense to use the spinners since you can't see
the full output from the activation script.

Fixes #44.
 2021-12-04 01:03:26 -08:00
Zhaofeng Li
7b1feffb9e util: Log the configuration in use 2021-12-04 01:03:26 -08:00
Zhaofeng Li
9cd2af9438 cli: Re-order arguments in help message 2021-12-04 01:03:26 -08:00
Zhaofeng Li
f253e6eb18 cli: Allow configuring output colorization 
We now follow the <https://bixense.com/clicolors> standard.
 2021-12-04 01:03:26 -08:00
Zhaofeng Li
dc57b489ea progress: Restore tty-detection after refactoring 2021-12-04 01:03:26 -08:00
Zhaofeng Li
064432f38e nix/deployment: Misc cleanup 2021-12-04 01:03:26 -08:00
Zhaofeng Li
04976e5fda job: Make event formatting less ugly 2021-12-03 00:06:21 -08:00
Zhaofeng Li
a3e292d7fb nix: Another deployment parallelization oops 2021-12-03 00:06:17 -08:00
Zhaofeng Li
2bba64a002 nix: Fix deployment parallelization 2021-12-02 23:49:46 -08:00
Zhaofeng Li
a42e8c5bf0 Misc cleanup 2021-11-23 14:12:01 -08:00
Zhaofeng Li
ec51f5703f deployment: A few oops 2021-11-22 01:53:08 -08:00
Zhaofeng Li
572f77dbf5 A couple of fixes to progress output 
Need to have some integration tests.
 2021-11-21 16:02:45 -08:00
Zhaofeng Li
b3016ad3a7 progress/plain: Fix label alignment 2021-11-21 16:02:45 -08:00
Zhaofeng Li
46090598eb apply-local: Fix output after rewrite 2021-11-21 15:30:52 -08:00
Zhaofeng Li
0cb3f8e968 Redesign deployment process (again) 
We now ship Events from different parts of the deployment process
via a channel to a job monitor.
 2021-11-21 00:11:37 -08:00
Zhaofeng Li
5c84134af3 Refactor node names 2021-11-20 23:34:52 -08:00
Zhaofeng Li
47ccc6c9c4 nix/info.rs: Update message about nixUnstable for 21.11 2021-11-18 15:41:02 -08:00
Zhaofeng Li
c271780b63 Improve error reporting 2021-11-18 13:15:20 -08:00
Zhaofeng Li
fae58994e4 Prepare for 0.2.0 release 2021-11-18 13:15:20 -08:00
Zhaofeng Li
e5665775b2 Preserve elapsed time in store path listing after build 
Fixes #36.
 2021-11-18 01:38:58 -08:00
Zhaofeng Li
80a71d9023 cli.rs: Point user at manual 2021-11-17 22:21:00 -08:00
Zhaofeng Li
4497ef296e Add manual 
We finally have some real documentation :)
 2021-11-17 22:21:00 -08:00