403 lines
15 KiB
ReStructuredText
403 lines
15 KiB
ReStructuredText
Development
|
|
###########
|
|
|
|
As a developer working on Liminix, or implementing a service or
|
|
module, you probably want to test your changes more conveniently
|
|
than by building and flashing a new image every time. This section
|
|
documents various affordances for iteration and experiments.
|
|
|
|
In general, packages and tools that run on the "build" machine are
|
|
available in the ``buildEnv`` derivation and can most easily
|
|
be added to your environment by running :command:`nix-shell`.
|
|
|
|
|
|
|
|
Emulated devices
|
|
****************
|
|
|
|
Liminix has a ``qemu`` device, which generates images suitable for
|
|
running on your build machine using the free `QEMU machine emulator <http://www.qemu.org>`_.
|
|
This is useful for developing userland without needing to keep
|
|
flashing or messing with U-Boot: it also enables testing against
|
|
emulated network peers using `QEMU socket networking <https://wiki.qemu.org/Documentation/Networking#Socket>`_,
|
|
which may be preferable to letting Liminix loose on your actual LAN.
|
|
To build it,
|
|
|
|
.. code-block:: console
|
|
|
|
nix-build -I liminix-config=path/to/your/configuration.nix --arg device "import ./devices/qemu" -A outputs.default
|
|
|
|
This creates a :file:`result/` directory containing a :file:`vmlinux`
|
|
and a :file:`rootfs`, and also a shell script :file:`run.sh` which
|
|
invokes QEMU to run that kernel with that filesystem. It connects the Liminix
|
|
serial console and the `QEMU monitor <https://www.qemu.org/docs/master/system/monitor.html>`_ to stdin/stdout. Use ^P (not ^A) to switch to the monitor.
|
|
|
|
If you run with ``--background /path/to/some/directory`` as the first
|
|
parameter, it will fork into the background and open Unix sockets in
|
|
that directory for console and monitor. Use :command:`nix-shell --run
|
|
connect-vm` to connect to either of these sockets, and ^O to
|
|
disconnect.
|
|
|
|
.. _qemu-networking:
|
|
|
|
Networking
|
|
==========
|
|
|
|
VMs can network with each other using QEMU
|
|
socket networking. We observe these conventions, so that we can run
|
|
multiple emulated instances and have them wired up to each other in
|
|
the right way:
|
|
|
|
* multicast 230.0.0.1:1234 : access (interconnect between router and "isp")
|
|
* multicast 230.0.0.1:1235 : lan
|
|
* multicast 230.0.0.1:1236 : world (the internet)
|
|
|
|
Any VM started by a :command:`run.sh` script is connected to "lan" and
|
|
"access", and the emulated border network gateway (see below) runs
|
|
PPPoE and is connected to "access" and "world".
|
|
|
|
.. _border-network-gateway:
|
|
|
|
Border Network Gateway
|
|
----------------------
|
|
|
|
In pkgs/routeros there is a derivation to install and configure
|
|
`Mikrotik RouterOS <https://mikrotik.com/software>`_ as a PPPoE access
|
|
concentrator connected to the ``access`` and ``world`` networks, so that
|
|
Liminix PPPoE client support can be tested without actual hardware.
|
|
|
|
This is made available as the :command:`routeros` command in
|
|
``buildEnv``, so you can do something like::
|
|
|
|
mkdir ros-sockets
|
|
nix-shell
|
|
nix-shell$ routeros ros-sockets
|
|
nix-shell$ connect-vm ./ros-sockets/console
|
|
|
|
to start it and connect to it. Note that by default it runs in the
|
|
background. It is connected to "access" and "world" virtual networks
|
|
and runs a PPPoE service on "access" - so a Liminix VM with a
|
|
PPPOE client can connect to it and thus reach the virtual internet.
|
|
[ check, but pretty sure this is not the actual internet ]
|
|
|
|
`Liminix does not provide RouterOS licences and it is your own
|
|
responsibility if you use this to ensure you're compliant with the
|
|
terms of Mikrotik's licencing. It may be supplemented or replaced in
|
|
time with configurations for RP-PPPoE and/or Accel PPP.`
|
|
|
|
Hardware devices
|
|
****************
|
|
|
|
.. _serial:
|
|
|
|
U-Boot and serial shenanigans
|
|
=============================
|
|
|
|
Every device that we have so far encountered in Liminix uses `U-Boot,
|
|
the "Universal Boot Loader" <https://docs.u-boot.org/en/latest/>`_ so
|
|
it's worth knowing a bit about it. "Universal" is in this context a
|
|
bit of a misnomer, though: encountering *mainline* U-Boot is very rare
|
|
and often you'll find it is a fork from some version last updated
|
|
in 2008. Upgrading U-Boot is more or less complicated depending on the
|
|
device and is outside scope for Liminix.
|
|
|
|
To speak to U-Boot on your device you'll usually need a serial
|
|
connection to it. This is device-specific. Usually it involves
|
|
opening the box, locating the serial header pins (TX, RX and GND) and
|
|
connecting a USB TTL converter to them.
|
|
|
|
The Rolls Royce of USB/UART cables is the `FTDI cable
|
|
<https://cpc.farnell.com/ftdi/ttl-232r-rpi/cable-debug-ttl-232-usb-rpi/dp/SC12825?st=usb%20to%20uart%20cable>`_,
|
|
but there are cheaper alternatives based on the PL2303 and CP2102 chipsets. Or
|
|
get creative and use the `UART GPIO pins <https://pinout.xyz/>`_ on a Raspberry Pi. Whatever you do, make sure
|
|
that the voltages are compatible: if your device is 3.3V (this is
|
|
typical but not universal), you don't want to be sending it 5v or
|
|
(even worse) 12v.
|
|
|
|
Run a terminal emulator such as Minicom on the computer at other end
|
|
of the link. 115200 8N1 is the typical speed.
|
|
|
|
.. NOTE::
|
|
|
|
TTL serial connections typically have no form of flow control and
|
|
so don't always like having massive chunks of text pasted into
|
|
them - and U-Boot may drop characters while it's busy. So don't
|
|
necessarily expect to copy-paste large chunks of text into the
|
|
terminal emulator and have it work just like that.
|
|
|
|
If using Minicom, you may find it helps to bring up the "Termimal
|
|
settings" dialog (C^A T), then configure "Newline tx delay" to
|
|
some small but non-zero value.
|
|
|
|
When you turn the router on you should be greeted with some messages
|
|
from U-Boot, followed by the instruction to hit some key to stop
|
|
autoboot. Do this and you will get to the prompt. If you didn't see
|
|
anything, the strong likelihood is that TX and RX are the wrong way
|
|
around. If you see garbage, try a different speed.
|
|
|
|
Interesting commands to try first in U-Boot are :command:`help` and
|
|
:command:`printenv`.
|
|
|
|
To do anything useful with U-Boot you will probably need a way to get
|
|
large binary files onto the device, and the usual way to do this is by
|
|
adding a network connection and using TFTP to download them. It's
|
|
quite common that the device's U-Boot doesn't speak DHCP so it will
|
|
need a static LAN address. You might also want to keep it away from
|
|
your "real" LAN: see :ref:`bng` for some potentially useful tooling
|
|
to use it on an isolated network.
|
|
|
|
|
|
TFTP
|
|
====
|
|
|
|
.. _tftp server:
|
|
|
|
How you get your image onto hardware will vary according to the
|
|
device, but is likely to involve taking it apart to add wires to
|
|
serial console pads/headers, then using U-Boot to fetch images over
|
|
TFTP. The OpenWrt documentation has a `good explanation <https://openwrt.org/docs/techref/hardware/port.serial>`_ of what you may expect to find on
|
|
the device.
|
|
|
|
There is a rudimentary TFTP server bundled with the system which runs
|
|
from the command line, has an allowlist for client connections, and
|
|
follows symlinks, so you can have your device download images direct
|
|
from the :file:`./result` directory without exposing :file:`/nix/store/` to the
|
|
internet or mucking about copying files to :file:`/tftproot`. If the
|
|
permitted device is to be given the IP address 192.168.8.251 you might
|
|
do something like this:
|
|
|
|
.. code-block:: console
|
|
|
|
nix-shell --run "tufted -a 192.168.8.251 result"
|
|
|
|
Now add the device and server IP addresses to your configuration:
|
|
|
|
.. code-block:: nix
|
|
|
|
boot.tftp = {
|
|
serverip = "192.168.8.111";
|
|
ipaddr = "192.168.8.251";
|
|
};
|
|
|
|
and then build the derivation for ``outputs.default`` or
|
|
``outputs.mtdimage`` (for which it will be an alias on any device
|
|
where this is applicable). You should find it has created
|
|
|
|
* :file:`result/firmware.bin` which is the file you are going to flash
|
|
* :file:`result/flash.scr` which is a set of instructions to U-Boot to
|
|
download the image and write it to flash after erasing the appropriate
|
|
flash partition.
|
|
|
|
.. NOTE::
|
|
|
|
TTL serial connections typically have no form of flow control and
|
|
so don't always like having massive chunks of text pasted into
|
|
them - and U-Boot may drop characters while it's busy. So don't
|
|
necessarily expect to copy-paste the whole of :file:`boot.scr` into
|
|
a terminal emulator and have it work just like that. You may need
|
|
to paste each line one at a time, or even retype it.
|
|
|
|
|
|
For a faster edit-compile-test cycle, you can build a TFTP-bootable
|
|
image instead of flashing. In your device configuration add
|
|
|
|
.. code-block:: nix
|
|
|
|
imports = [
|
|
./modules/tftpboot.nix
|
|
];
|
|
|
|
and then build ``outputs.tftpboot``. This creates a file in
|
|
``result/`` called ``boot.scr``, which you can copy and paste into
|
|
U-Boot to transfer the kernel and filesystem over TFTP and boot the
|
|
kernel from RAM.
|
|
|
|
|
|
.. _bng:
|
|
|
|
Networking
|
|
==========
|
|
|
|
You probably don't want to be testing a device that might serve DHCP,
|
|
DNS and routing protocols on the same LAN as you (or your colleagues,
|
|
employees, or family) are using for anything else, because it will
|
|
interfere. You also might want to test the device against an
|
|
"upstream" connection without having to unplug your regular home
|
|
router from the internet so you can borrow the cable/fibre/DSL.
|
|
|
|
``bordervm`` is included for this purpose. You will need
|
|
|
|
* a Linux machine with a spare (PCI or USB) ethernet device which you can dedicate to Liminix
|
|
|
|
* an L2TP service such as https://www.aa.net.uk/broadband/l2tp-service/
|
|
|
|
You need to "hide" the Ethernet device from the host - for PCI this
|
|
means configuring it for VFIO passthru; for USB you need to unload the
|
|
module(s) it uses. I have this segment in configuration.nix which you
|
|
may be able to adapt:
|
|
|
|
.. code-block:: nix
|
|
|
|
boot = {
|
|
kernelParams = [ "intel_iommu=on" ];
|
|
kernelModules = [
|
|
"kvm-intel" "vfio_virqfd" "vfio_pci" "vfio_iommu_type1" "vfio"
|
|
];
|
|
|
|
postBootCommands = ''
|
|
# modprobe -i vfio-pci
|
|
# echo vfio-pci > /sys/bus/pci/devices/0000:01:00.0/driver_override
|
|
'';
|
|
blacklistedKernelModules = [
|
|
"r8153_ecm" "cdc_ether"
|
|
];
|
|
};
|
|
services.udev.extraRules = ''
|
|
SUBSYSTEM=="usb", ATTRS{idVendor}=="0bda", ATTRS{idProduct}=="8153", OWNER="dan"
|
|
'';
|
|
|
|
Then
|
|
you can execute :command:`run-border-vm` in a ``buildEnv`` shell,
|
|
which starts up QEMU using the NixOS configuration in
|
|
:file:`bordervm-configuration.nix`.
|
|
|
|
In this VM
|
|
|
|
* your Liminix checkout is mounted under :file:`/home/liminix/liminix`
|
|
|
|
* TFTP is listening on the ethernet device and serving
|
|
:file:`/home/liminix/liminix`. The server IP address is 10.0.0.1
|
|
|
|
* a PPPOE-L2TP relay is running on the same ethernet card. When the
|
|
connected Liminix device makes PPPoE requests, the relay spawns
|
|
L2TPv2 Access Concentrator sessions to your specified L2TP LNS.
|
|
Note that authentication is expected at the PPP layer not the L2TP
|
|
layer, so the PAP/CHAP credentials provided by your L2TP service can
|
|
be configured into your test device - bordervm doesn't need to know
|
|
about them.
|
|
|
|
To configure bordervm, you need a file called :file:`bordervm.conf.nix`
|
|
which you can create by copying and appropriately editing :file:`bordervm.conf-example.nix`
|
|
|
|
.. note::
|
|
|
|
If you make changes to the bordervm configuration after executing
|
|
:command:`run-border-vm`, you need to remove the :file:`border.qcow2` disk
|
|
image file otherwise the changes won't get picked up.
|
|
|
|
|
|
Running tests
|
|
*************
|
|
|
|
You can run all of the tests by evaluating :file:`ci.nix`, which is the
|
|
input I use in Hydra. Note that it expects Nixpkgs stable `and` unstable
|
|
as inputs, because it builds the qemu device against both.
|
|
|
|
.. code-block:: console
|
|
|
|
nix-build --argstr liminix `pwd` --arg nixpkgs "<nixpkgs>" \
|
|
--argstr unstable `pwd`/../unstable-nixpkgs/ ci.nix
|
|
|
|
To run a single named test, use the ``-A`` flag. For example, ``-A pppoe``
|
|
|
|
|
|
|
|
|
|
Troubleshooting
|
|
***************
|
|
|
|
Diagnosing unexpectedly large images
|
|
====================================
|
|
|
|
Sometimes you can add a package and it causes the image size to balloon
|
|
because it has dependencies on other things you didn't know about. Build the
|
|
``outputs.manifest`` attribute, which is a JSON representation of the
|
|
filesystem, and you can run :command:`nix-store --query` on it.
|
|
|
|
.. code-block:: console
|
|
|
|
nix-build -I liminix-config=path/to/your/configuration.nix \
|
|
--arg device "import ./devices/qemu" -A outputs.manifest \
|
|
-o manifest
|
|
nix-store -q --tree manifest
|
|
|
|
|
|
Contributing
|
|
************
|
|
|
|
Contributions are welcome, though in these early days there may be a
|
|
bit of back and forth involved before patches are merged:
|
|
Please get in touch somehow `before` you invest a lot of time into a
|
|
code contribution I haven't asked for. Just so I know it's expected
|
|
and you're not wasting time doing something I won't accept or have
|
|
already started on.
|
|
|
|
|
|
Nix language style
|
|
==================
|
|
|
|
This section describes some Nix language style points that we
|
|
attempt to adhere to in this repo.
|
|
|
|
* favour ``callPackage`` over raw ``import`` for calling derivations
|
|
or any function that may generate one - any code that might need
|
|
``pkgs`` or parts of it.
|
|
|
|
* prefer ``let inherit (quark) up down strange charm`` over
|
|
``with quark``, in any context where the scope is more than a single
|
|
expression or there is more than one reference to ``up``, ``down``
|
|
etc. ``with pkgs; [ foo bar baz]`` is OK,
|
|
``with lib; stdenv.mkDerivation { ... }`` is usually not.
|
|
|
|
* ``<liminix>`` is defined only when running tests, so don't refer to it
|
|
in "application" code
|
|
|
|
* the parameters to a derivation are sorted alphabetically, except for
|
|
``lib``, ``stdenv`` and maybe other non-package "special cases"
|
|
|
|
* indentation is whatever emacs nix-mode says it is.
|
|
|
|
* where a ``let`` form defines multiple names, put a newline after the
|
|
token ``let``, and indent each name two characters
|
|
|
|
* to decide whether some code should be a package or a module?
|
|
Packages are self-contained - they live in ``/nix/store/eeeeeee-name``
|
|
and don't directly change system behaviour by their presence or
|
|
absense. modules can add to
|
|
``/etc`` or ``/bin`` or other global state, create services, all that
|
|
side-effecty stuff. Generally it should be a package unless it
|
|
can't be.
|
|
|
|
|
|
|
|
Copyright
|
|
=========
|
|
|
|
The Nix code in Liminix is MIT-licenced (same as Nixpkgs), but the
|
|
code it combines from other places (e.g. Linux, OpenWrt) may have a
|
|
variety of licences. I have no intention of asking for copyright
|
|
assignment: just like when submitting to the Linux kernel you retain
|
|
the copyright on the code you contribute.
|
|
|
|
|
|
Code of Conduct
|
|
===============
|
|
|
|
Please govern yourself in Liminix project venues according to the
|
|
`Code of Conduct <https://gti.telent.net/dan/liminix/src/commit/7bcf6b15c3fdddafeda13f65b3cd4a422dc52cd3/CODE-OF-CONDUCT.md>`_
|
|
|
|
|
|
Where to send patches
|
|
=====================
|
|
|
|
|
|
Liminix' primary repo is https://gti.telent.net/dan/liminix but you
|
|
can't send code there directly because it doesn't have open registrations.
|
|
|
|
* There's a `mirror on Github <https://github.com/telent/liminix>`_ for
|
|
convenience and visibility: you can open PRs against that
|
|
|
|
* or, you can send me your patch by email using `git send-email <https://git-send-email.io/>`_
|
|
|
|
* or in the future, some day, we will have federated Gitea using
|
|
ActivityPub.
|