Commit graph

48 commits

Author SHA1 Message Date
eta
4dca44ac72 fix(buildLisp): use full store path in FASL output pathnames
If you compiled dirA/test.lisp and dirB/test.lisp in the same library,
they'd both get written to /test.fasl and the second would overwrite the
first. Instead, use the whole store path (with / swapped for -) as the fasl
filename.

Change-Id: I4eb88b5d33757751e1f67e72ed328bd58079b1b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/944
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-07-06 22:01:09 +00:00
Profpatsch
2f21e0c8c0 refactor(nix): use our own writeScript(Bin)
We have this nice `runExecline` now, so we don’t need to use
`runCommand` (which spawns bash) just to write a simple script.

Change-Id: I2941ed8c1448fa1d7cc02dc18b24a8a945b2c38b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/704
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: BuildkiteCI
2020-06-30 00:19:27 +00:00
Profpatsch
4402c363b6 feat(nix/runExecline): add runExecline
runExecline is a primitive that just does not care.

It’s similar to `runCommand`, but instead of concatenating bash
scripts left and right, it actually *uses* the features of
`derivation`, passing things to `args` and making it possible to
overwrite the `builder` in a sensible manner.

Additionally, it provides a way to pass a nix string to `stdin` of the
build script.

Similar to `writeExecline`, the passed script is not a string, but a
nested list of nix lists representing execline blocks. Escaping is
done by the implementation, the user can just use normal nix strings.

Change-Id: I890d9e5d921207751cdc8cc4309381395d92742f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/701
Reviewed-by: BuildkiteCI
Reviewed-by: isomer <isomer@tvl.fyi>
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-06-29 22:58:47 +00:00
Profpatsch
7a6a007302 refactor(nix/escapeExecline): move into own directory
The escaping functions are going to be used by both `writeExecline`
and `runExecline`, so let’s move them to their own namespace.

Change-Id: Iccf69eaeca3062573e0751a17c548b7def86196d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/706
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: Kane York <rikingcoding@gmail.com>
2020-06-29 22:58:32 +00:00
Profpatsch
cdfae7de48 feat(nix/emptyDerivation) add emptyDerivation
The most trivial of all derivations. It is more useful than it looks.

Can be used to bind nix expressions (e.g. test suites) to a
derivation, so that `nix-build` does not crap itself.

Change-Id: I61c24d8c129c9505733161207f3c30e820f5b15e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/665
Reviewed-by: tazjin <mail@tazj.in>
2020-06-28 17:59:46 +00:00
Profpatsch
98a990c6a6 feat(nix/getBins): add getBins
This is a simple-stupid “unix import system” for nix, for referencing
binaries in `/bin/` by their name and lifting them to a Nix attrset.

Allows for simple aliasing of executable names.

Change-Id: Ifa23cb377201c3b08050c5026e9751e736afaf56
Reviewed-on: https://cl.tvl.fyi/c/depot/+/664
Reviewed-by: tazjin <mail@tazj.in>
2020-06-28 17:58:15 +00:00
Profpatsch
c8e888c1d2 feat(nix/runTestsuite): add runTestsuite
This is a very simple test suite for nix expressions.
It should help us set up a good suite of unit tests for our nix-based
stuff.

Since we allow import from derivation, these tests can also depend on
derivations and e.g. use `builtins.readFile` to check outputs.

This is a first PoC to get us going, we can always replace it by
something different in the future if we don’t like it.

Change-Id: I206c7b624db2b1dabd9c73ffce4f87e658919958
Reviewed-on: https://cl.tvl.fyi/c/depot/+/662
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: tazjin <mail@tazj.in>
2020-06-28 03:52:40 +00:00
Profpatsch
e2c7e38934 feat(nix/yants): add unit type
Has one less value than bool.

Change-Id: I2f2db07b1eb1f49172942bc1d8c6ceb30d987d84
Reviewed-on: https://cl.tvl.fyi/c/depot/+/661
Reviewed-by: tazjin <mail@tazj.in>
2020-06-27 20:48:15 +00:00
Profpatsch
3fd583d27c feat(nix/writeExecline): add writeExecline
This is a writer, similar to `pkgs.writeBashScript` or
`pkgs.writers.writePython3`.

The difference is that we can correctly write all execline scripts by
using nix lists of lists, so the user doesn’t have to care about
escaping arguments (like they have to in bash scripts with
`lib.escapeShellArg` for example).

Change-Id: I2f2874cf61170ddca07b89b692f762725f4a75dc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/625
Reviewed-by: Kane York <rikingcoding@gmail.com>
Reviewed-by: tazjin <mail@tazj.in>
2020-06-27 17:52:14 +00:00
Profpatsch
17eba437a7 feat(nix/binify): add binify
Create a store path where the executable `exe` is linked to
$out/bin/${name}. This is useful for e.g. including it as a “package”
in `buildInputs` of a shell.nix.

For example, if I have the exeutable /nix/store/…-hello, I can make it
into /nix/store/…-binify-hello/bin/hello with
`binify { exe = …; name = "hello" }`.

Change-Id: I600bdcd8f143bca2dd8dfbb165a9a5a8d6397622
Reviewed-on: https://cl.tvl.fyi/c/depot/+/624
Reviewed-by: tazjin <mail@tazj.in>
2020-06-27 03:42:05 +00:00
Vincent Ambo
b72d232075 fix(buildTypedGo): Pass correct arguments to .package
Change-Id: I535e084e7b3195628609bb43e382e450bc0003ba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/497
Reviewed-by: eta <eta@theta.eu.org>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-06-19 15:25:11 +00:00
Vincent Ambo
e1e1764e32 feat(nix/buildTypedGo): Add a //nix/buildGo wrapper for typed Go
These functions work like buildGo.program & buildGo.package, but run
the .go2 sources through go2go first before passing them to the
//nix/buildGo equivalents.

Change-Id: Id1ebab6085c390d6986387370181377b9f5d39e8
2020-06-18 02:29:52 +01:00
Vincent Ambo
81b6bde074 fix(nix/buildGo): Fix import of buildGo in example file
Change-Id: I56b4d19f24c5d8ec45254b7b05c9e49002d2ef85
Reviewed-on: https://cl.tvl.fyi/c/depot/+/382
Reviewed-by: edef <edef@edef.eu>
2020-06-15 16:50:54 +00:00
edef
cec109807b fix(nix/readTree): Ignore hidden files and directories
This skips any directory entries starting with a dot.

Change-Id: I95767f3d35bcb2ed9b3d6e772f3924dd57612711
Reviewed-on: https://cl.tvl.fyi/c/depot/+/123
Reviewed-by: tazjin <mail@tazj.in>
2020-06-13 20:08:05 +00:00
edef
01ddbb4397 fix(nix/buildGo/external): Properly match import path prefixes
Prior to this patch, github.com/hashicorp/terraform-svchost is
erroneously considered a sub-package of github.com/hashicorp/terraform,
breaking dependency searching:

    error: missing local dependency 'github.com.hashicorp.terraform-svchost' in 'github.com/hashicorp/terraform'

Change-Id: Ibcf0f3a9b1742ce46f84cbbf84e90127b8c1df0d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/122
Reviewed-by: tazjin <mail@tazj.in>
2020-06-13 03:04:40 +00:00
edef
f0aca6bd8a fix(nix/buildGo): Pass sane arguments to log.Fatalf
Change-Id: Ie0f2c0a50d8c0618abf6dae2242155fdf1f98e2b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/121
Reviewed-by: tazjin <mail@tazj.in>
2020-06-13 01:32:18 +00:00
Vincent Ambo
980bf5365c fix(nix/buildGo): Do not silently ignore filepath.Walk() errors 2020-05-25 23:24:57 +01:00
Vincent Ambo
6e7006c061 docs(nix/yants): Mention Yants subtree split in README
This subtree split makes it possible for people to clone only
yants (similar to the kontemplate and journaldriver branches).

The subtree continues the history of the old git repository.
2020-05-16 12:37:12 +01:00
Vincent Ambo
db74ffb4b1 fix(nix/buildGo): Remove absolute references to GOROOT from binaries
Setting the GOROOT_FINAL environment variables replaces the absolute
location of the Go standard library sources in the final build
artefacts with a fake location (in this case starting with
go/src/...).

This is despite the documentation for 'trimpath' (in 'go tool
compile') stating that it would affect all source paths: That's only
true for user code!

I figured this out by reading through the implementation of the other
'trimpath' (in 'gob build'):

https://go-review.googlesource.com/c/go/+/173345
2020-03-31 01:29:35 +01:00
Vincent Ambo
4bbbb58cb5 chore: Rename pkgs->depot in all Nix file headers 2020-02-21 13:54:53 +00:00
Vincent Ambo
b4c0292753 fix(nix/tailscale): Fix incorrect Tailscale ACL config type 2020-02-11 21:00:50 +00:00
Vincent Ambo
df1a4fef2b feat(nix/tailscale): Add function for generating tailscale ACLs
... and use it on Camden!
2020-02-11 16:36:28 +00:00
Vincent Ambo
ddc9ee4dcd docs(nix/yants): Fix screenshot URLs
The URLs served on the /about page are relative to the path's dirname,
which is tree/nix not tree/nix/yants.
2020-02-07 19:11:55 +00:00
Vincent Ambo
8e8bbbca04 fix(nix/buildLisp): Ensure SBCL uses UTF-8 encoding 2020-01-29 10:12:07 +00:00
Vincent Ambo
0001dfd7f6 docs(nix/buildLisp): Update the README with actual docs 2020-01-29 00:03:24 +00:00
Vincent Ambo
ca60eafa80 feat(nix/buildLisp): Add 'bundled' function for built-in libraries
Makes it possible to add virtual dependencies on built-in libraries,
e.g. `buildLisp.bundled "sb-posix"`.
2020-01-26 23:58:31 +00:00
edef
30286d5454 chore(buildLisp): use lib.optionalString where applicable 2020-01-24 22:18:19 +00:00
edef
ccbac831d3 chore(buildGo): use lib.optionalString where applicable 2020-01-24 22:18:19 +00:00
Vincent Ambo
21e9a65a35 fix(nix/buildLisp): Don't load binaries in sbclWith
Adds an attribute on each Lisp derivation that specifies whether it is
a binary or not. This attribute is then filtered for in sbclWith.
2020-01-17 16:41:57 +00:00
Vincent Ambo
e1cc4966b7 feat(nix/buildLisp): Support passing programs to sbclWith
Adds the necessary attributes on derivations created by
buildLisp.program for them to be passed to buildLisp.sbclWith.

This makes it possible to easily spin up Lisp environments that
contain everything needed for a given program.
2020-01-17 12:44:24 +01:00
Vincent Ambo
09720e2da2 fix(buildLisp): Wrap executables to set load paths correctly
I can not currently find a way to set the CFFI variables correctly to
get it to load libraries from Nix.

In the absence of that feature, a wrapper also does the trick.
2020-01-09 03:32:29 +00:00
Vincent Ambo
44820827d1 feat(buildLisp): Initial implementation of foreign library loading
Adds a new 'native' parameter to the buildLisp functions in which
libraries can be passed in.

This does not yet work with CFFI packages.
2020-01-09 02:57:02 +00:00
Vincent Ambo
2a170f1ed7 fix(buildLisp): Perform a topological sort of dependencies
This ensures that dependencies are loaded in the correct order in
larger dependency graphs.
2020-01-09 00:46:20 +00:00
Vincent Ambo
e3a8dc9500 fix(buildLisp): Cursed code to fix load ordering
It's not enough to compile in the right order - turns out you also
have to load the compiled objects in the right order.

To achieve this some cursed code has been added that changes the Lisp
generated by Nix to compile the other Lisp so that it also generates
some bash, which Nix can then use to concatenate the FASLs in the
right order to feed them to Lisp again.

It works but I'll replace it with a more elegant solution once one is
needed.
2020-01-08 23:57:34 +00:00
Vincent Ambo
7bc10eb9b7 feat(buildLisp): Add initial, tiny example program 2020-01-08 21:41:43 +00:00
Vincent Ambo
bdad8f6642 feat(buildLisp): Implement buildLisp.program to dump executables
Dumps the executable image from SBCL to $out/bin/$name.

Image compression is disabled.
2020-01-08 21:39:26 +00:00
Vincent Ambo
2bfe073eb2 refactor(buildLisp): Inline dependency loading in genCompileLisp 2020-01-08 21:39:06 +00:00
Vincent Ambo
b5e1e81a3d feat(buildLisp): Add function to wrap SBCL with dependencies
Adds `buildLisp.sbclWith` which creates an SBCL wrapper the contains
all the requested dependencies.
2020-01-08 19:38:29 +00:00
Vincent Ambo
ca199a57d9 feat(buildLisp): Implement dependency loading & propagation
Similar to buildGo.nix, the library derivations carry information
about their dependencies which is merged when a load file is
instantiated.

The load files are created when compiling libraries, but will in the
future also be created when wrapping SBCL and dumping images.
2020-01-08 18:40:53 +00:00
Vincent Ambo
1297afec4b fix(buildLisp): Fail the build on compilation errors
This needs to be handled explicitly in the COMPILE-FILE form.
2020-01-08 17:53:06 +00:00
Vincent Ambo
a954bd8d5e feat(nix/buildLisp): Add initial sketch including buildLisp.library
Adds a Nix function to build a Lisp library out of a specified set of
Nix files. All files are combined into a single FASL.

This is by design only compatible with SBCL (for now).
2020-01-08 02:00:54 +00:00
Vincent Ambo
eb650eb8d8 docs(nix/readTree): Add a very descriptive README for readTree 2019-12-21 05:42:52 +00:00
Vincent Ambo
4a9c6ab6a2 refactor(nix/readTree): Move readTree to its own subfolder 2019-12-21 05:42:49 +00:00
Vincent Ambo
9885036eec chore(yants): Move tests into subfolder & add to CI builds 2019-12-20 21:53:19 +00:00
Vincent Ambo
902500c4ea fix(yants): Allow extra import arguments
Required for readTree compatibility.
2019-12-20 21:49:00 +00:00
Vincent Ambo
b98c60ecca merge(yants): Integrate yants into depot at //depot/nix/yants 2019-12-20 21:47:38 +00:00
Vincent Ambo
210893ce09 chore(yants): Prepare for depot-merge
Yants is being integrated at //depot/nix/yants
2019-12-20 21:46:59 +00:00
Vincent Ambo
03bfe08e1d chore: Significantly restructure folder layout
This moves the various projects from "type-based" folders (such as
"services" or "tools") into more appropriate semantic folders (such as
"nix", "ops" or "web").

Deprecated projects (nixcon-demo & gotest) which only existed for
testing/demonstration purposes have been removed.

(Note: *all* builds are broken with this commit)
2019-12-20 20:18:41 +00:00