mdebray/tvl-depot
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
tvl-depot/tvix/boot
History
Florian Klink fca3926a62 feat(tvix/boot/tests): make urls and paths configurable 
Fork off a long-living tvix-store daemon in the background, and have
both the import, as well as the virtiofs invocation connect to that one,
rather than use the underlying storage directly.

This also opens up the potential to have other things seed the store
(like nar-bridge).

Change-Id: Ie13aedaf7ab31e5ebe78a7be42f144af3920fa9c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11186
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-18 16:10:05 +00:00
..
tests feat(tvix/boot/tests): make urls and paths configurable 2024-03-18 16:10:05 +00:00
default.nix chore(tvix/tests): rename to //tvix/boot 2023-09-25 12:46:10 +00:00
README.md refactor(tvix/nar-bridge): mv nar-bridge{,-http} 2023-10-05 06:17:10 +00:00
tvix-init.go chore(tvix/tests): rename to //tvix/boot 2023-09-25 12:46:10 +00:00

README.md

tvix/boot

This directory provides tooling to boot VMs with /nix/store provided by virtiofs.

In the tests/ subdirectory, there's some integration tests.

//tvix/boot:runVM

A script spinning up a tvix-store virtiofs daemon, then starting a cloud- hypervisor VM.

The cloud-hypervisor VM is using a (semi-)minimal kernel image with virtiofs support, and a custom initrd (using u-root). It supports various command line options, to be able to do VM tests, act as an interactive shell or exec a binary from a closure.

It supports the following env vars:

  • CH_NUM_CPUS=1 controls the number of CPUs available to the VM
  • CH_MEM_SIZE=512M controls the memory availabe to the VM
  • CH_CMDLINE= controls the kernel cmdline (which can be used to control the boot)

Usage

First, ensure you have tvix-store in $PATH, as that's what run-tvix-vm expects:

Assuming you ran cargo build --profile=release-with-debug before, and are in the tvix directory:

export PATH=$PATH:$PWD/target/release-with-debug

Secondly, configure tvix to use the local backend:

export BLOB_SERVICE_ADDR=sled://$PWD/blobs.sled
export DIRECTORY_SERVICE_ADDR=sled://$PWD/directories.sled
export PATH_INFO_SERVICE_ADDR=sled://$PWD/pathinfo.sled

Potentially copy some data into tvix-store (via nar-bridge):

mg run //tvix:store -- daemon &
$(mg build //tvix:nar-bridge)/bin/nar-bridge-http &
rm -Rf ~/.cache/nix; nix copy --to http://localhost:9000\?compression\=none $(mg build //third_party/nixpkgs:hello)
pkill nar-bridge-http; pkill tvix-store

Interactive shell

Run the VM like this:

CH_CMDLINE=tvix.shell mg run //tvix/boot:runVM --

You'll get dropped into an interactive shell, from which you can do things with the store:

  ______      _         ____      _ __
 /_  __/   __(_)  __   /  _/___  (_) /_
  / / | | / / / |/_/   / // __ \/ / __/
 / /  | |/ / />  <   _/ // / / / / /_
/_/   |___/_/_/|_|  /___/_/ /_/_/\__/

/# ls -la /nix/store/
dr-xr-xr-x root 0 0   Jan  1 00:00 .
dr-xr-xr-x root 0 989 Jan  1 00:00 aw2fw9ag10wr9pf0qk4nk5sxi0q0bn56-glibc-2.37-8
dr-xr-xr-x root 0 3   Jan  1 00:00 jbwb8d8l28lg9z0xzl784wyb9vlbwss6-xgcc-12.3.0-libgcc
dr-xr-xr-x root 0 82  Jan  1 00:00 k8ivghpggjrq1n49xp8sj116i4sh8lia-libidn2-2.3.4
dr-xr-xr-x root 0 141 Jan  1 00:00 mdi7lvrn2mx7rfzv3fdq3v5yw8swiks6-hello-2.12.1
dr-xr-xr-x root 0 5   Jan  1 00:00 s2gi8pfjszy6rq3ydx0z1vwbbskw994i-libunistring-1.1

Once you exit the shell, the VM will power off itself.

Execute a specific binary

Run the VM like this:

hello_cmd=$(mg build //third_party/nixpkgs:hello)/bin/hello
CH_CMDLINE=tvix.run=$hello_cmd mg run //tvix/boot:runVM --

Observe it executing the file (and closure) from the tvix-store:

[    0.277486] Run /init as init process
  ______      _         ____      _ __
 /_  __/   __(_)  __   /  _/___  (_) /_
  / / | | / / / |/_/   / // __ \/ / __/
 / /  | |/ / />  <   _/ // / / / / /_
/_/   |___/_/_/|_|  /___/_/ /_/_/\__/

Hello, world!
2023/09/24 21:10:19 Nothing left to be done, powering off.
[    0.299122] ACPI: PM: Preparing to enter system sleep state S5
[    0.299422] reboot: Power down

Execute a NixOS system closure

It's also possible to invoke a system closure. To do this, tvix-init honors the init= cmdline option, and will switch_root to it.

CH_CMDLINE=init=/nix/store/…-nixos-system-…/init mg run //tvix/boot:runVM --

  ______      _         ____      _ __
 /_  __/   __(_)  __   /  _/___  (_) /_
  / / | | / / / |/_/   / // __ \/ / __/
 / /  | |/ / />  <   _/ // / / / / /_
/_/   |___/_/_/|_|  /___/_/ /_/_/\__/

2023/09/24 21:16:43 switch_root: moving mounts
2023/09/24 21:16:43 switch_root: Skipping "/run" as the dir does not exist
2023/09/24 21:16:43 switch_root: Changing directory
2023/09/24 21:16:43 switch_root: Moving /
2023/09/24 21:16:43 switch_root: Changing root!
2023/09/24 21:16:43 switch_root: Deleting old /
2023/09/24 21:16:43 switch_root: executing init

<<< NixOS Stage 2 >>>

[    0.322096] booting system configuration /nix/store/g657sdxinpqfcdv0162zmb8vv9b5c4c5-nixos-system-client-23.11.git.82102fc37da
running activation script...
setting up /etc...
starting systemd...
[    0.980740] systemd[1]: systemd 253.6 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified)

This effectively replaces the NixOS Stage 1 entirely.