diff --git a/.gitignore b/.gitignore index 4b19f73..bec8fc1 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,6 @@ *.img *.bin result -result-* \ No newline at end of file +result-* +*.qcow2 +_build \ No newline at end of file diff --git a/README.md b/README.md index ef27b6e..372cd96 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,8 @@ # Liminix -A Nix-based system for configuring consumer wifi routers. - -## What is this? - -This is a Nix-based collection of software tailored for domestic wifi -router or IoT device devices, of the kind that OpenWrt or DD-WRT or -Gargoyle or Tomato run on. It's a reboot/restart/rewrite of NixWRT. +A Nix-based system for configuring consumer wifi routers or IoT device +devices, of the kind that OpenWrt or DD-WRT or Gargoyle or Tomato run +on. It's a reboot/restart/rewrite of NixWRT. This is not NixOS-on-your-router: it's aimed at devices that are underpowered for the full NixOS experience. It uses busybox tools, @@ -17,184 +13,14 @@ of "limen", or "of the threshold". Your router stands at the threshold of your (online) home and everything you send to/receive from the outside word goes across it. -### What about NixWRT? +## What about NixWRT? This is an in-progress rewrite of NixWRT, incorporating Lessons -Learned. That said, as of today (September 2022) it is not yet -anywhere _near_ feature parity. +Learned. -Liminix will eventually provide these differentiators over NixWRT: +## Documentation -* a writable filesystem so that software updates or reconfiguration - (e.g. changing passwords) don't require taking the device offline to - reflash it. +Documentation is in the [doc](doc/) directory. You can build it +by running -* more flexible service management with dependencies, to allow - configurations such as "route through PPPoE if it is healthy, with - fallback to LTE" - -* a spec for valid configuration options (a la NixOS module options) - to that we can detect errors at evaluation time instead of producing - a bad image. - -* a network-based mechanism for secrets management so that changes can - be pushed from a central location to several Liminix devices at once - -* send device metrics and logs to a monitoring/alerting/o11y - infrastructure - -Today though, it does approximately none of these things and certainly -not on real hardware. - - -## Building - -### For the device - -These instructions assume you have nixpkgs checked out in a peer -directory of this one. - -You need a `configuration.nix` file pointed to by ``, a -hardware device definition as argument `device`, and to choose an -appropriate output attribute depending on what your device is and how -you plan to install onto it. For example: - - NIX_PATH=nixpkgs=../nixpkgs:$NIX_PATH nix-build -I liminix-config=./tests/smoke/configuration.nix --arg device "import ./devices/qemu" -A outputs.default - -`outputs.default` is intended to do something appropriate for the -device, whatever that is. For the qemu device, it creates a directory -containing a squashfs root image and a kernel. - -### For the build machine - -Liminix also includes some tools intended for the build machine. You can -run - - nix-shell -A buildEnv --arg device '(import ./devices/qemu)' - -to get a shell environment with (currently) a tftp server and -a script to start a PPPoE server in QEMU for testing against. - - -#### QEMU - -QEMU is useful for developing userland without needing to keep -flashing or messing with U-Boot: it also enables testing against -emulated network peers using [QEMU socket networking](https://wiki.qemu.org/Documentation/Networking#Socket), -which may be preferable to letting Liminix loose on your actual LAN. - -We observe these conventions for QEMU network sockets, so that we can -run multiple emulated instances and have them wired up to each other -in the right way - -* multicast 230.0.0.1:1234 : access (interconnect between router and "isp") -* multicast 230.0.0.1:1235 : lan -* multicast 230.0.0.1:1236 : world (the internet) - -### Running Liminix in Qemu - -In a `buildEnv` nix-shell, you can use the `mips-vm` command -to run Qemu with appropriate config for two ethernet interfaces -hooked up to "lan" and "access" respectively. It connects the Liminix -serial console and the [QEMU monitor](https://www.qemu.org/docs/master/system/monitor.html) to stdin/stdout. Use ^P (not ^A) to switch to the monitor. - - nix-shell -A buildEnv --arg device '(import ./devices/qemu)' --run "mips-vm result/vmlinux result/squashfs" - -If you run with `--background /path/to/some/directory` as the first -parameter, it will fork into the background and open Unix sockets in -that directory for console and monitor. Use `connect-vm` (also in the -`buildEnv` environment) to connect to either of these sockets, and ^O -to disconnect. - -### Emulated upstream connection - -In pkgs/routeros there is a derivation to install and configure -[Mikrotik RouterOS](https://mikrotik.com/software) as a PPPoE access -concentrator connected to the `access` and `world` networks, so that -Liminix PPPoE client support can be tested. - -This is made available in the `buildEnv`, so you can do something like - - mkdir ros-sockets - nix-shell -A buildEnv --arg device '(import ./devices/qemu)' - nix-shell$ routeros ros-sockets - nix-shell$ connect-vm ./ros-sockets/console - -to start it and connect to it. - -_Liminix does not provide RouterOS licences and it is your own -responsibility if you use this to ensure you're compliant with the -terms of Mikrotik's licencing._It may be supplemented or replaced in -time with configurations for RP-PPPoE and/or Accel PPP. - -## Running tests - -Assuming you have nixpkgs checked out in a peer directory of this one, -you can run all of the tests by evaluating `ci.nix`: - - nix-build --argstr liminix `pwd` --argstr nixpkgs `pwd`/../nixpkgs --argstr unstable `pwd`/../unstable-nixpkgs/ ci.nix - -or to run a named test, use the `-A` flag. For example, `-A pppoe` - - -## Hardware - -How you get the thing onto hardware will vary according to the device, -but is likely to involve U-Boot and TFTP. - -There is a rudimentary TFTP server bundled with the system which runs -from the command line, has an allowlist for client connections, and -follows symlinks, so you can have your device download images direct -from the `./result` directory without exposing `/nix/store/` to the -internet or mucking about copying files to `/tftproot`. If the -permitted device is to be given the IP address 192.168.8.251 you might -do something like this: - - nix-shell -A buildEnv --arg device '(import ./devices/qemu)' \ - --run "tufted -a 192.168.8.251 result" - - -## Troubleshooting - -### Diagnosing unexpectedly large images - -Sometimes you can add a package and it causes the image size to balloon -because it has dependencies on other things you didn't know about. Build the -`outputs.manifest` attribute, which is a json representation of the -filesystem, and you can run `nix-store --query` on it. - - NIX_PATH=nixpkgs=../nixpkgs:$NIX_PATH nix-build -I liminix-config=path/to/your/configuration.nix --arg device "import ./devices/qemu" -A outputs.manifest -o manifest - nix-store -q --tree manifest - - -## Contributing - -Contributions are welcome, though in these early days there may be a -bit of back and forth involved before patches are merged. Have a read -of [CONTRIBUTING](CONTRIBUTING.md) and [STYLE](STYLE.md) and try to -intuit the unarticulated vision :-) - -Liminix' primary repo is https://gti.telent.net/dan/liminix but that -doesn't help you much because registrations are closed: - -* There's a [mirror on Github](https://github.com/telent/liminix) for -convenience and visibility: you can open PRs against that - -* or, you can send me your patch by email using [`git send-email`](https://git-send-email.io/) - -* or in the future, some day, we will have federated Gitea using - ActivityPub. - - -## Articles of interest - -* [Build Safety of Software in 28 Popular Home Routers](https://cyber-itl.org/assets/papers/2018/build_safety_of_software_in_28_popular_home_routers.pdf): - "of the access points and routers we reviewed, not a single one -took full advantage of the basic application armoring features -provided by the operating system. Indeed, only one or two models even -came close, and no brand did well consistently across all models -tested" - -* [A PPPoE Implementation for Linux](https://static.usenix.org/publications/library/proceedings/als00/2000papers/papers/full_papers/skoll/skoll_html/index.html): "Many DSL service providers use PPPoE for residential broadband Internet access. This paper briefly describes the PPPoE protocol, presents strategies for implementing it under Linux and describes in detail a user-space implementation of a PPPoE client." - -* [PPP IPV6CP vs DHCPv6 at AAISP](https://www.revk.uk/2011/01/ppp-ipv6cp-vs-dhcpv6.html) + nix-shell -p sphinx --run "make -C doc html" diff --git a/doc/developer.rst b/doc/developer.rst new file mode 100644 index 0000000..98248bb --- /dev/null +++ b/doc/developer.rst @@ -0,0 +1,225 @@ +Developer Manual +################ + +As a developer working on Liminix, or implementing a service or +module, you probably want to test your changes more conveniently +than by building and flashing a new image every time. This manual +documents various affordances for iteration and experiments. + +In general, packages and tools that run on the "build" machine are +available in the ``buildEnv`` derivation. + +.. code-block:: console + + nix-shell -A buildEnv --arg device '(import ./devices/qemu)' + + +Emulated devices +**************** + +Liminix has a ``qemu`` device, which generates images suitable for +running on your build machine using the free `QEMU machine emulator `_. +This is useful for developing userland without needing to keep +flashing or messing with U-Boot: it also enables testing against +emulated network peers using `QEMU socket networking `_, +which may be preferable to letting Liminix loose on your actual LAN. +To build it, + +.. code-block:: console + + NIX_PATH=nixpkgs=../nixpkgs:$NIX_PATH nix-build -I liminix-config=path/to/your/configuration.nix --arg device "import ./devices/qemu" -A outputs.default + +In a ``buildEnv`` nix-shell, you can use the ``mips-vm`` command +to run Qemu with appropriate options. It connects the Liminix +serial console and the `QEMU monitor `_ to stdin/stdout. Use ^P (not ^A) to switch to the monitor. + +.. code-block:: console + + nix-shell -A buildEnv --arg device '(import ./devices/qemu)' --run "mips-vm result/vmlinux result/squashfs" + +If you run with ``--background /path/to/some/directory`` as the first +parameter, it will fork into the background and open Unix sockets in +that directory for console and monitor. Use ``connect-vm`` (also in the +``buildEnv`` environment) to connect to either of these sockets, and ^O +to disconnect. + +Networking +========== + +VMs can network with each other using QEMU +socket networking. We observe these conventions, so that we can run +multiple emulated instances and have them wired up to each other in +the right way + +* multicast 230.0.0.1:1234 : access (interconnect between router and "isp") +* multicast 230.0.0.1:1235 : lan +* multicast 230.0.0.1:1236 : world (the internet) + +A VM started with ``mips-vm`` is connected to "lan" and "access", and +the emulated border network gateway (see below) runs PPPoE and is +connected to "access" and "world". + +Border Network Gateway +---------------------- + +In pkgs/routeros there is a derivation to install and configure +`Mikrotik RouterOS `_ as a PPPoE access +concentrator connected to the ``access`` and ``world`` networks, so that +Liminix PPPoE client support can be tested without actual hardware. + +This is made available as the ``routeros`` command in ``buildEnv``, so you +can do something like:: + + mkdir ros-sockets + nix-shell -A buildEnv --arg device '(import ./devices/qemu)' + nix-shell$ routeros ros-sockets + nix-shell$ connect-vm ./ros-sockets/console + +to start it and connect to it. Note that by default it runs in the +background. It is connected to "access" and "world" virtual networks +and runs a PPPoE service on "access" - so a Liminix VM with a +PPPOE client can connect to it and thus reach the virtual internet. +[ check, but pretty sure this is not the actual internet ] + +`Liminix does not provide RouterOS licences and it is your own +responsibility if you use this to ensure you're compliant with the +terms of Mikrotik's licencing. It may be supplemented or replaced in +time with configurations for RP-PPPoE and/or Accel PPP.` + +Hardware devices +**************** + +How you get your image onto hardware will vary according to the +device, but is likely to involve taking it apart to add wires to +serial console pads/headers, then using U-Boot to fetch images over +TFTP. + +There is a rudimentary TFTP server bundled with the system which runs +from the command line, has an allowlist for client connections, and +follows symlinks, so you can have your device download images direct +from the ``./result`` directory without exposing ``/nix/store/`` to the +internet or mucking about copying files to ``/tftproot``. If the +permitted device is to be given the IP address 192.168.8.251 you might +do something like this: + +.. code-block:: console + + nix-shell -A buildEnv --arg device '(import ./devices/qemu)' \ + --run "tufted -a 192.168.8.251 result" + +and then issue appropriate U-boot commands to download and flash the +image. + +For quicker development cycle, you can build a TFTP-bootable image +instead of flashing. [ .... add this bit ....] + + +Running tests +************* + +You can run all of the tests by evaluating ``ci.nix``, which is the +input I use in Hydra. Note that it expects Nixpkgs stable `and` unstable +as inputs, because it builds the qemu device against both. + +.. code-block:: console + + nix-build --argstr liminix `pwd` --arg nixpkgs "" \ + --argstr unstable `pwd`/../unstable-nixpkgs/ ci.nix + +or to run a named test, use the ``-A`` flag. For example, ``-A pppoe`` + + + + +Troubleshooting +*************** + +Diagnosing unexpectedly large images +==================================== + +Sometimes you can add a package and it causes the image size to balloon +because it has dependencies on other things you didn't know about. Build the +``outputs.manifest`` attribute, which is a JSON representation of the +filesystem, and you can run ``nix-store --query`` on it.:: + + NIX_PATH=nixpkgs=../nixpkgs:$NIX_PATH nix-build -I liminix-config=path/to/your/configuration.nix --arg device "import ./devices/qemu" -A outputs.manifest -o manifest + nix-store -q --tree manifest + + +Contributing +************ + +Contributions are welcome, though in these early days there may be a +bit of back and forth involved before patches are merged: +Please get in touch somehow `before` you invest a lot of time into a +code contribution I haven't asked for. Just so I know it's expected +and you're not wasting time doing something I won't accept or have +already started on. + + +Nix language style +================== + +In an attempt to keep this more consistent than NixWRT ended up being, +here is a Nix language style guide for this repo. + +* favour ``callPackage`` over raw ``import`` for calling derivations + or any function that may generate one - any code that might need + ``pkgs`` or parts of it. + +* prefer ``let inherit (quark) up down strange charm`` over + ``with quark``, in any context where the scope is more than a single + expression or there is more than one reference to ``up``, ``down`` + etc. ``with pkgs; [ foo bar baz]`` is OK, + ``with lib; stdenv.mkDerivation { ... }`` is usually not. + +* ```` is defined only when running tests, so don't refer to it + in "application" code + +* the parameters to a derivation are sorted alphabetically, except for + ``lib``, ``stdenv`` and maybe other non-package "special cases" + +* indentation is whatever emacs nix-mode says it is. + +* where a ``let`` form defines multiple names, put a newline after the + token ``let``, and indent each name two characters + +* to decide whether some code should be a package or a module? + Packages are self-contained - they live in ``/nix/store/eeeeeee-name`` + and don't directly change system behaviour by their presence or + absense. modules can add to + ``/etc`` or ``/bin`` or other global state, create services, all that + side-effecty stuff. Generally it should be a package unless it + can't be. + + + +Copyright +========= + +The Nix code in Liminix is MIT-licenced (same as Nixpkgs), but the +code it combines from other places (e.g. Linux, OpenWrt) may have a +variety of licences. I have no intention of asking for copyright +assignment: just like when submitting to the Linux kernel you retain +the copyright on the code you contribute. + +Code of Conduct +=============== + +Please govern yourself in Liminix project venues according to the guidance in the `geekfeminism "Community Anti-harassment Policy" `_. + + +Where to send patches +===================== + + +Liminix' primary repo is https://gti.telent.net/dan/liminix but that +doesn't help you much, because it doesn't have open registrations. + +* There's a `mirror on Github `_ for + convenience and visibility: you can open PRs against that + +* or, you can send me your patch by email using `git send-email `_ + +* or in the future, some day, we will have federated Gitea using + ActivityPub. diff --git a/doc/etc.rst b/doc/etc.rst new file mode 100644 index 0000000..810ad61 --- /dev/null +++ b/doc/etc.rst @@ -0,0 +1,48 @@ +The Future +########## + +What about NixWRT? + +This is an in-progress rewrite of NixWRT, incorporating Lessons +Learned. That said, as of today it is not yet at feature parity. + +Liminix will eventually provide these differentiators over NixWRT: + +* a writable filesystem so that software updates or reconfiguration + (e.g. changing passwords) don't require taking the device offline to + reflash it. + +* more flexible service management with dependencies, to allow + configurations such as "route through PPPoE if it is healthy, with + fallback to LTE" + +* a spec for valid configuration options (a la NixOS module options) + to that we can detect errors at evaluation time instead of producing + a bad image. + +* a network-based mechanism for secrets management so that changes can + be pushed from a central location to several Liminix devices at once + +* send device metrics and logs to a monitoring/alerting/o11y + infrastructure + +Today though, it does approximately none of these things and certainly +not on real hardware. + + +Articles of interest +#################### + +* [Build Safety of Software in 28 Popular Home Routers](https://cyber-itl.org/assets/papers/2018/build_safety_of_software_in_28_popular_home_routers.pdf): "of the access + points and routers we reviewed, not a single one took full + advantage of the basic application armoring features provided by + the operating system. Indeed, only one or two models even came + close, and no brand did well consistently across all models tested" + +* [A PPPoE Implementation for Linux](https://static.usenix.org/publications/library/proceedings/als00/2000papers/papers/full_papers/skoll/skoll_html/index.html): + "Many DSL service providers use PPPoE for residential broadband + Internet access. This paper briefly describes the PPPoE protocol, + presents strategies for implementing it under Linux and describes in + detail a user-space implementation of a PPPoE client." + +* [PPP IPV6CP vs DHCPv6 at AAISP](https://www.revk.uk/2011/01/ppp-ipv6cp-vs-dhcpv6.html) diff --git a/doc/index.rst b/doc/index.rst index 85f3f45..8ab4c64 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -1,15 +1,14 @@ -.. Liminix documentation master file, created by - sphinx-quickstart on Wed Feb 15 23:24:26 2023. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. - -Welcome to Liminix's documentation! -=================================== +Liminix +####### .. toctree:: - :maxdepth: 2 + :maxdepth: 3 :caption: Contents: + intro + user + developer + etc Indices and tables diff --git a/doc/intro.rst b/doc/intro.rst new file mode 100644 index 0000000..04a62c5 --- /dev/null +++ b/doc/intro.rst @@ -0,0 +1,15 @@ +Introduction +############ + +Liminix is a Nix-based collection of software tailored for domestic +wifi router or IoT device devices, of the kind that OpenWrt or DD-WRT +or Gargoyle or Tomato run on. + +This is not NixOS-on-your-router: it's aimed at devices that are +underpowered for the full NixOS experience. It uses busybox tools, +musl instead of GNU libc, and s6-rc instead of systemd. + +The Liminix name comes from Liminis, in Latin the genitive declension +of "limen", or "of the threshold". Your router stands at the threshold +of your (online) home and everything you send to/receive from the +outside word goes across it. diff --git a/doc/user.rst b/doc/user.rst new file mode 100644 index 0000000..1b5b359 --- /dev/null +++ b/doc/user.rst @@ -0,0 +1,64 @@ +User Manual +########### + +This manual is an early work in progress, not least because Liminix is +not yet ready for users who are not also developers. + +Configuring for your use case +***************************** + +You need to create a ``configuration.nix`` that describes your router +and the services that you want to run on it. Start by copying +``vanilla-configuration.nix`` and adjusting it. + + +Building and flashing +********************* + +You need to set ```` to point to your +``configuration.nix``, the file for your hardware device definition as +argument ``device``, and to choose an appropriate output attribute +depending on what your device is and how you plan to install onto +it. For example: + +.. code-block:: console + + NIX_PATH=nixpkgs=../nixpkgs:$NIX_PATH nix-build -I liminix-config=./tests/smoke/configuration.nix --arg device "import ./devices/qemu" -A outputs.default + +``outputs.default`` is intended to do something appropriate for the +device, whatever that is. For the qemu device, it creates a directory +containing a squashfs root image and a kernel. + +As of Feb 2023, flashing devices is not implemented other than by +taking the covers off and connecting wires to the serial console +pads - so, check in the Developer Manual. + +Future versions of this manual will at this point refer to +device-specific instructions for installing Liminix using the router's +Web UI or other non-invasive method. + + +Updating a running device +************************* + +Feature forthcoming. + +Module options +************** + + + +Foo module +========== + +Module docs will go here. This part of the doc should be autogenerated. + + +Bar module +========== + +Baz module +========== + +Quuz net device +===============