Commit graph

330 commits

Author SHA1 Message Date
edef
2fce356ce0 fix(nix-compat/nar/writer): use ByteSlice closer to call site
Otherwise, it'll generate unused import warnings in release mode.

Change-Id: I0399d2617a8a8b973a2b8c18f4a0a613d70c48f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9864
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 17:07:06 +00:00
edef
9253bf6632 feat(tvix/nix-compat): add narinfo parsing and serialisation
Change-Id: I72c63414794642ca8d85c3f635f49db888420c40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9852
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 16:08:06 +00:00
edef
b1ad94cc9a refactor(nix-compat/store_path): use nixbase32::decode_fixed
Change-Id: I81471ee57920aa8fa889fb00c7903cdc570af9c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9863
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 16:08:06 +00:00
edef
50aa74a59c feat(nix-compat/store_path): add StorePathRef
Change-Id: I0d888a55d93e5c22e77cb0264d09757656f731d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9862
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 16:08:06 +00:00
edef
e525272019 refactor(tvix): turn nullary enum variants into unit variants
Change-Id: Iad4f2cb4aa92b5bb29ead6050348a8cd3e7b8632
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9860
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 13:56:51 +00:00
edef
520c5a191e refactor(nix-compat/store_path): speed up validate_name fast path
Change-Id: Ie50b29145804777f7644952c65cb42519a8565e3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9859
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 13:56:51 +00:00
edef
c19c6c23cb feat(nix-compat/store_path): validate_name takes AsRef<[u8]>
Change-Id: I8819e2a7b63008a68f4f82035a08b960ac480dc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9858
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 13:56:51 +00:00
edef
b994f692d3 feat(nix-compat/store_path): validate_name over borrowed data
Change-Id: Ifeb6231f48d4ad267a7acd398b4b3b687ee4d560
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9857
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 13:56:51 +00:00
edef
6238a05868 refactor(nix-compat/store_path): don't materialise fingerprint
Change-Id: I6a88531ded05c0dfb9232a0343a465fa02fb6989
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9856
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 13:56:51 +00:00
edef
7f7c1ae7be refactor(nix-compat/store_path): make digest and name private
Change-Id: I62cbe883afcf3dd0c8d4de0e3b845069eb750c97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9855
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 13:56:51 +00:00
edef
36f2b69de5 fix(tvix/nix-compat): validate store path name length
Change-Id: I89ac0ad147a1872c021ab4235ca46ef3f51d0446
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9854
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 13:56:51 +00:00
edef
99a61def17 fix(tvix/nix-compat): don't box CAHash::Text
Change-Id: I31df3909bc21c9038f9fb831879e60e541242819
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9853
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 12:18:23 +00:00
edef
fdc2e90ef2 feat(tvix/nix-compat): add fixed-length base32 decoding
Change-Id: Iea9d51157c1e38e1734dc1ba752eba9b2372c61b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9850
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 12:18:23 +00:00
edef
3c0e42167f feat(tvix/nix-compat): use AsRef<[u8]> for base32 decode input
Change-Id: If2275dda62b852aedb64b4f9915f7dc4173cabd2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9851
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 12:18:23 +00:00
edef
6a0a75c8e1 refactor(tvix): condense long bytestrings
Change-Id: I3bea0827ec2c8db835334ce378a7bf3a39e9b1a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9849
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 07:37:26 +00:00
edef
55c37a2871 feat(tvix/nix-compat): decode base32 with a lookup table
This also takes input validation out of the loop, leaving the loop
backedge as the sole branch in the hot path.

Change-Id: Id08e6fb9cf5b074780efa09a7ad389352a601bcc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9847
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 07:31:46 +00:00
edef
67b08469db refactor(tvix/nix-compat): clean up base32 decoding
Change-Id: I8591a3e1075f2281b7dc49f7fa9e0027062a78a5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9846
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 07:31:46 +00:00
edef
4d35a56798 refactor(tvix/nix-compat): simplify encode_len
Change-Id: Idfefd3b5b0d1ffbd4e9208e634a2d9989965a291
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9845
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 07:31:46 +00:00
Florian Klink
6fe7684832 fix(nix-compat/src/derivation/validate): remove break
If the output was fixed, we broke out of the for loop too early, before
actually validating individual outputs.

Change-Id: I2259697dfa2a157764358f6d326a1f7f6610647c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9815
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-23 14:57:32 +00:00
Florian Klink
34fc4637eb refactor(tvix/nix-compat): rename NixHashWithMode -> CAHash
This specific struct is only used to represent content-addressed paths
(in case a Derivation has a fixed-output hash, for example).
Rename `Output`'s `hash_with_mode` to `ca_hash`.

We now also include `CAHash::Text`, and update the `validate` function
of the `Output` struct to reject text hashes there.

This allows cleaning up the various output path calculation functions
inside nix-compat/src/store_path/utils.rs, as they can now match on
the type.

`make_type` is renamed to `make_references_string`,
`build_regular_ca_path` is renamed to `build_ca_path`, and
`build_text_path` has a disclaimer added, because you might not actually
want to use it.

Change-Id: I674d065f2ed5c804012ddfed56e161ac49d23931
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9814
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-23 14:57:32 +00:00
Florian Klink
42d3c1a748 test(tvix/nix-compat/nar/writer/sync): add tests
Change-Id: I04c556e7b57f20dd33f71b0379e6104e1702a8ee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9785
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-18 12:31:57 +00:00
Florian Klink
05bca22121 feat(tvix/nix-compat/nar/reader): add tests
Change-Id: If0abaf74a3b7d201af8972e54e3e34b20b8c0bc4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9764
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-18 12:31:57 +00:00
edef
e64acba454 fix(tvix/nix-compat/nar/wire): correct Node::Some doc comment
It is *eventually* followed by a Node, but there is some stuff in
between.

Change-Id: Ie7c7b462828bd3e066f4a7e774895f30b82763ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9768
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-18 12:22:47 +00:00
edef
2cae6a02b9 refactor(nix-compat/nar/writer): use clone_into
Equivalent to the existing code, but a little less cryptic.

Change-Id: Ib9b2f9aedddc84d0e79840bba4cce01f92d9bc56
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9766
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-18 11:42:28 +00:00
edef
6638f4d4ea feat(tvix/nix-compat): NAR reader
Change-Id: I50d51baf62c0419eaf17f0dc262f728aaff9794d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9688
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-18 11:40:06 +00:00
edef
08b98b7503 docs(tvix/nix-compat/nar): document the wire format
Change-Id: I6c8e23bad27fa6ada1b8973482b4d99190cf050d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9767
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-18 11:40:06 +00:00
Florian Klink
57b69baf7c refactor(tvix/nix-compat): use hash_with_mode's nix_hash_string
HashWithMode already provides a to_nix_hash_string() method, giving us
exactly the string we need here.

Change-Id: Id2635bf3ea6c2514faf3c26b297866d774f4ff4a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9799
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-18 10:32:43 +00:00
Florian Klink
73a19c249e refactor(tvix/nix-compat/derivation): remove some clones
Change-Id: Ie464467edfafd2050aa9e5fd0e07ff745cb15df8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9753
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-16 16:17:41 +00:00
Florian Klink
652afd21b3 feat(nix-compat): add bench for aterm parsing
Change-Id: Ie9416f0046b6ee0a02096a7712cdba9cc84db4f6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9733
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: Gabriel Doriath Döhler <gabriel.doriath.dohler@ens.fr>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-16 12:26:36 +00:00
Florian Klink
c3fad811f4 refactor(nix-compat/derivation): drop non_unicode test
The previous CLs did already absorb all the logic into the common tests,
no need to write this here again.

Change-Id: I7ba84ba86d5445ed247e5d11d5e59b7fa815670e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9732
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-16 12:26:36 +00:00
Florian Klink
398cf8e7ce refactor(nix-compat/derivation): use ATerm for output_paths test
Due to the lack of a ATerm parser, we were previously loading the JSON
fixtures to construct our Derivation structs to run the output path
calculations with.

However, as we now have a ATerm parser, we can load the ATerm
representation directly.
This also means we can test the output path calculation for non-UTF8
Derivations.

Change-Id: I0e53f41a23566b5ad5f0fed12724e02a10b02707
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9731
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-16 12:26:36 +00:00
Florian Klink
8934b34489 fix(nix-compat/derivation): handle dups
This now properly checks for duplicate output names in input derivation
output names, and duplicate input sources.

Change-Id: I0053854bfbf504f4f511fb3fe1a77a82b3aa61dd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9738
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-16 12:23:00 +00:00
Florian Klink
2410f2292f feat(nix-compat/{aterm,derivation}): init parser
This provides a nom-based parser for Nix derivations in ATerm format,
which can be reached via `Derivation::from_aterm_bytes`.

Some of the lower-level ATerm primitives are moved into a (new) aterm
module, and some more higher-level ones that construct derivation-
specific types.

Also, move the escape_bytes function into there, this is a generic ATerm
thing.

Change-Id: I2b03b8a1461c7ea2fcb8640c2fc3d1fa3ea719fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9730
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-10-16 12:23:00 +00:00
Florian Klink
c739df3d78 refactor(nix-compat/derivation): fix function name
typo: write_enviroment -> write_environment.
Change-Id: I1a2dc826c0bf71f6ba97fe41c4d44619b5e6beff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9739
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-10-16 11:32:13 +00:00
Florian Klink
1a3c610081 refactor(tvix/nix-compat): clippy
Change-Id: I56282d9754b42e0e6cdbb08e9ae71f5700d28dd1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9747
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-10-16 11:13:33 +00:00
Florian Klink
9aafbe8d95 refactor(nix-compat/derivation): make write_environment more generic
We don't actually care if it's a BTreeMap of strings, bstrings or any of
that sort.

The only thing we want to be able to do is get a reference to the bytes
from the keys and values.

Change-Id: I21b85811a9ea47fe06afa3108836ef9295e5d89b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9737
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-10-15 15:10:19 +00:00
Florian Klink
f337601f69 refactor(nix-compat/derivation): move test fixtures to subdir
We use test_resources and globbing for some of the test cases, so adding
additional files in there will also create new test cases, which we
don't always want.

Move it down one level to make some more space.

Change-Id: I619867dc80a4ced59d45096d0703678663b559cd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9729
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-15 15:10:12 +00:00
Florian Klink
8a0fa69c6c feat(tvix/nix-compat/nixhash/NixHashWithMode): from_algo_mode_hash
Change-Id: Ieae628fab1926a498ae5c3eb27df1b722e6151d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9728
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-15 15:10:07 +00:00
Florian Klink
c3446da1c7 refactor(nix-compat/nixhash): add Result type alias
Change-Id: Id0248047e9642d38afc106629957a2e7608f8c78
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9727
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-15 02:21:34 +00:00
Florian Klink
cb1a14334a refactor(nix-compat/nixhash): box sha512
Limit the amount of memory consumed on the stack for NixHash. Sha512
isn't used that often, so it's fine if we heap-allocate it.

Change-Id: I4a9eecd20c6184610124dc130c41bfa5d0dc04c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9726
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-15 02:21:34 +00:00
Florian Klink
cc040a5ad3 refactor(tvix/nix-compat/derivation): fod_digest return [u8; 32]
This is of known length.

Change-Id: Iba48ccc486d5bf9e38ec1a2da6e7b80997d2c6ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9723
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-14 17:55:08 +00:00
Florian Klink
4ae0f428bd refactor(tvix/nix-compat): make NixHash an enum with fixed-len bytes
Less Vec<u8> passed around.

Change-Id: Ie153a6bfaa084d7490ffa38634efdf5f3c31a768
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9722
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-14 17:55:08 +00:00
Florian Klink
2d2c4322d9 feat(tvix/store/protos): add Deriver field to PathInfo
This uses the newly introduced StorePath message type to add a Deriver
field to the PathInfo message.

Support for validation is added to both the golang and rust
implementation. This includes extending unit tests.

Change-Id: Ifc3eb3263fa25b9eec260db354cd74234c40af7e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9647
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-14 13:23:07 +00:00
edef
8b35d97b4b feat(tvix/nix-compat): add nar::writer::async
Change-Id: Ib962f889836659fd278beec461ee329b00e64ea5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9615
Autosubmit: edef <edef@edef.eu>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-10 17:13:27 +00:00
edef
f38290cfb0 refactor(tvix/nix-compat): move nar::writer to nar::writer::sync
This is preparation for adding an async port.

Change-Id: Id638ec1f6f46e2f3935448184eed51e2233263fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9618
Tested-by: BuildkiteCI
Autosubmit: edef <edef@edef.eu>
Reviewed-by: flokli <flokli@flokli.de>
2023-10-10 17:13:27 +00:00
edef
bb54e04c1b fix(tvix/nix-compat): clean up the debug assertions a bit
Consistent error messages, and slightly nicer code layout. We avoid
printing the input data, since we primarily want to point out the
specific violated invariant. In the one place where we do want to,
we use BStr's Debug implementation, since byte slices don't print
nicely.

Change-Id: I3a9a0c37be270ea5f16cf124922c254608fb849e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9617
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: edef <edef@edef.eu>
2023-10-10 16:58:17 +00:00
edef
17d6f4320f fix(tvix/nix-compat): drop unnecessary reference taking
Found by Clippy, which we should probably run in CI.

Change-Id: Id79c30b63f681021ab79358e02d29454d43c0aa6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9614
Autosubmit: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-10 08:57:04 +00:00
edef
5cc7171b5f docs(tvix/nix-compat): mention Directory::close on Node::directory
Change-Id: I19e6a3b63eb1b1609b96544b2529d71234bcb13b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9613
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-10-09 18:39:57 +00:00
edef
ca1ede17cb docs(tvix/nix-compat): document the name ordering requirement
Change-Id: Ia2c5f0b357a23d4bfac1fd52cf01845c78ebfa22
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9612
Autosubmit: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-09 18:39:57 +00:00
edef
49b106b0cb fix(tvix/nix-compat): don't transiently allocate names
The point of clearing and reusing the same Vec is to avoid transiently
allocating for every directory entry. This was lost in cl/8974 when we
switched from String to Vec<u8>.

Change-Id: I65647e5c4e54e88f1fe45e9a752cb5154d98fb33
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9607
Autosubmit: edef <edef@edef.eu>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-09 18:22:54 +00:00
edef
155f53b264 fix(tvix/nix-compat): reject dotfiles
Nix has historically rejected these. The current behaviour was
accidentally introduced in Nix 2.4, and is considered a bug.

Link: https://github.com/NixOS/nix/pull/9095
Change-Id: I38ffa911f0a413086479bd972de09671dbe85121
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9507
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: edef <edef@edef.eu>
2023-10-04 17:18:11 +00:00
edef
e014c3ef70 fix(tvix/nix-compat): bits are not bytes
Change-Id: Ib5744ea240895fe84454ad271ee5640b29cab838
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9505
Autosubmit: edef <edef@edef.eu>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-03 08:09:28 +00:00
Florian Klink
dfb3d30d45 fix(tvix/nix-compat): drop first character period check
Suggested in https://cl.tvl.fyi/c/depot/+/9108/5, but this disallows
adding a .gitignore file to the store.

Remove the check, and add a testcase.

Change-Id: Ieb78c417934756b2dbeb493040fe79726d1b9079
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9447
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2023-09-23 12:54:18 +00:00
Florian Klink
09a901de5b feat(tvix/nix-compat): fix and improve error message
Similar to cl/9350. Most of the times, this is still a regular string,
so let's print it like that if we can, and resort to base64 encoding if
we can't.

We were also wrongly always outputting the second character in the
string.

Change-Id: Id0e2a9d9f1ad3d2d7b554893ecd89a7e6383e9c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9445
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-23 12:54:18 +00:00
Florian Klink
7c9a52e3ee chore(tvix): remove some unused crate dependencies
Change-Id: I31b0e7be3a2ebc268c6f4045ba282e8fb1c72a64
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9375
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 12:12:24 +00:00
Connor Brewster
37a348b4fa refactor(tvix/store): Asyncify PathInfoService and DirectoryService
We've decided to asyncify all of the services to reduce some of the
pains going back and for between sync<->async. The end goal will be for
all the tvix-store internals to be async and then expose a sync
interface for things like tvix eval io.

Change-Id: I97c71f8db1d05a38bd8f625df5087d565705d52d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9369
Autosubmit: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-09-21 17:58:23 +00:00
Florian Klink
3c340b28bd refactor(tvix/{cli,store}): move TvixStoreIO to tvix-cli crate
This trait is eval-specific, there's no point in dealing with these
things in tvix-store.

This implements the EvalIO interface for a Tvix store.
The proper place for this glue code (for now) is tvix-cli, which knows
about both tvix-store and tvix-eval.

There's one annoyance with this move: The `tvix-store import` subcommand
previously also used the TvixStoreIO implementation (because it
conveniently did what we wanted).
Some of this code had to be duplicated, mostly logic to calculate the
NAR-based output path and create the PathInfo object.

Some, but potentially more of this can be extracted into helper
functions in a shared crate, and then be used from both TvixStoreIO in
tvix-cli as well as the tvix-store CLI entrypoint.

Change-Id: Ia7515e83c1b54f95baf810fbd8414c5521382d40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9212
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-09-04 16:02:12 +00:00
Florian Klink
bba7bbf820 docs(tvix/nix-compat/nixhash): update comments
Change-Id: I46660da84065fd6938f581e14d67e231dca3c3ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9112
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-08-20 22:40:53 +00:00
Florian Klink
0193f07642 refactor(tvix/nix-compat/nixhash): validate digest lengths
There was a NixHash::new() before, which didn't perform any validation
of the digest length. We had some length validation when parsing nix
hashes or SRI hashes, but some places didn't perform validation and/or
constructed the struct directly.

Replace NixHash::new() with a
`impl TryFrom<(HashAlgo, Vec<u8>)> for NixHash`,  which does do this
validation, and update constructing code to use that, rather than
populating structs directly. In some rare cases where we're sure the
digest length is correct we still populate the struct manually.

Fixes b/291.

Change-Id: I7a323c5b18d94de0ec15e391b3e7586df42f4229
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9109
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-08-20 22:19:22 +00:00
Florian Klink
4017039595 refactor(tvix/nix-compat): cargo clippy
Change-Id: I1c1608a6e75e451940fe1c61dc5ace5f0e7d7752
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9111
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-08-20 22:19:22 +00:00
Ryan Lahfa
2d687e068a fix(tvix/nix-compat): disallow empty derivation names
Yes:

```
$ nix-build -E 'derivation { name = ""; builder = "/bin/sh"; system = "x86_64-linux"; }'
error: store path 'nr7i5pf18hw2zg487vkdyrbasdqylfcj-' has an empty name
```

Change-Id: I552f9ed1c1fe3bfceca18ca9b8e13d4b06dc6ff7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9108
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-08-20 19:34:35 +00:00
Ryan Lahfa
d504b440c2 feat(tvix/nix-compat): don't swallow hash validation errors
Previously, Output deserialization would silence validation errors and
provide `None` for `hash_with_mode` as soon as a validation error would
happen inside of the `NixHashWithMode` deserialization, e.g. invalid
hash length would not provide an validation error but a silent `None`
value.

This is problematic, we workaround a serde limitation here by writing
our own Deserializer.

As you can see, we write some boilerplate code unfortunately, as, for
example:

- `#[serde(fail_if_unparsed_as="Option::is_none")]` is not a thing,
  otherwise, we could have been able to just bubble up errors in case of
  "not fully parsed" (and not missing) values.

- `From<&serde_json::Value> for serde:🇩🇪:Unexpected` is not a thing,
  otherwise, we could just map invalid type errors and reuse the
  existing types instead of doing extremely bizarre things with
  `serde:🇩🇪:Unexpected::Other`, note: this is a not problem for
  expected, we know what we expect, we don't know what we received in
  practice.

I decided to write a `NixHashWithMode::from_map` which will eat a map
deserialized via `serde_json`, so our serde magic is not totally "data
model" agnostic.

I wanted to go for data model agnosticity and enable maximal
performance, e.g. building the structure as the map values are streamed
in the Visitor, this is needlessly painful because `Output` and
`NixHashWithMode` are in different files and this really makes sense
only if we write the full implementation in one file, indeed, otherwise,
you end up duplicating the work or having strange coupling.

So, for now, we will allocate a full map of the fields inside the
`Output`, i.e. if any "unknown field" is in that map, we will
deserialize it for no reason.

Doing it properly, as I repeat it in the code and to flokli at C3Camp
2023, requires to patch serde upstream IMHO.

Change-Id: I46fe6ccb8c390c48d6934fd3e3f02a0dfe59557b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9107
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-08-20 19:34:20 +00:00
Florian Klink
946270142e refactor(tvix/nix-compat/derivation): cargo clippy
This `.into_iter()` call is equivalent to `.iter()` and will not consume
the `BTreeMap`.

Change-Id: Ie26637ebecb0bea5b09c447cc45ed207f8b50913
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9088
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-08-19 11:47:48 +00:00
Florian Klink
311752412b docs(tvix/nix-compat/store_path): fix broken docstr reference
Change-Id: Ie95ac265f8707e138c6a7b529760650d211f259d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9087
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-08-19 11:41:26 +00:00
Florian Klink
999b2d6995 refactor(tvix/nix-compat/nixhash): use shorthand struct init
Change-Id: I1e52b752408a70fdb27e3e5c6f3bef5417b3e922
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9085
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2023-08-19 11:37:46 +00:00
Florian Klink
b5ed3a90f2 docs(tvix/nix-compat): remove disambiguity
Don't import thiserror::Error directly, so rustdoc knows what
`crate::store_path::Error` we're talking about.

Change-Id: I755c9377521a6833e9a77cb1a41b48edafb31fe0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9086
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-08-19 11:27:45 +00:00
Florian Klink
39efe50311 refactor(tvix/nix-compat/derivation/escape): move to Vec<u8>
We only need bstr::ByteSlice to be able to use replace, it doesn't need
to return a BString.

Change-Id: I811948436fb89652e880970c2c05356183f3e439
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9084
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-08-19 10:54:32 +00:00
Florian Klink
f9fbc9be85 refactor(tvix/nix-compat/derivation/tests): fix typo
Change-Id: I764232378f79f16c781870d467469db91a2f14d2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9001
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-07-31 21:59:42 +00:00
Florian Klink
a10d1e7bdf refactor(tvix/nix-compat): use local write::*;
Change-Id: I43cc1d5b3419255f815490b2e10f05db19e3bffc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9000
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-07-31 21:59:42 +00:00
Florian Klink
edde496040 refactor(tvix/nix-compat/test): rename data identifier
Make it more obvious if these are bytes pointing to JSON or ATerm, so we
don't get confused.

Change-Id: I2402c687b7ba9c05aac20ed63b0df54e4e96a9d8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8998
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-07-31 21:59:42 +00:00
Florian Klink
8258dbecdf refactor(tvix/nix-compat/derivation): drop some useless allocations
Change-Id: I85e44e24e9ec3fe4284f02623a1ac7ca9935e554
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8999
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-07-31 21:59:42 +00:00
Florian Klink
779fad29a1 refactor(tvix/nix-compat/derivation): use writer.write_all
This is more concise than a io::copy of a Cursor to bytes, and we have
everything to be written in memory.

Change-Id: I81f34666aa61aef4e16b33423ce4a69c3781efc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8997
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-07-31 21:50:48 +00:00
Florian Klink
aa1982c085 refactor(tvix/derivation): move comma writing business up
write_input_derivations shouldn't need to write a comma to separate it
from the previous output from write_outputs.
This is better placed in the function calling all of these helper
functions.

Change-Id: I9ccc440e4665b52369ef39e75151b9a29469ce48
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8995
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-07-31 21:50:44 +00:00
Florian Klink
3f856d8977 refactor(tvix/nix-compat/derivation): generic write_array_elements
We're happy with any &[S], as long as <S: AsRef<[u8]>.

This allows passing both strings and &[u8].

Change-Id: If2a80d9b1ee33ba328c9cdab4fa83ca7b98a71e2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8994
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-07-31 21:50:44 +00:00
Florian Klink
34b7620764 refactor(tvix/nix-compat/derivation): simplify
Let the escape function only take care of string escaping, not quoting.

Let write_array_elements always quote and escape strings it consumes.
Move the business of writing additional wrapping characters around it to
the caller.

Change-Id: Ib8dea69c409561b49862c531ba5a3fe6c2f061f8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8993
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-07-31 21:41:39 +00:00
Florian Klink
79531c3dab refactor(tvix/nix-compat): support non-unicode Derivations
Derivations can have non-unicode strings in their env values, so the
ATerm representations are not necessarily String anymore, but Vec<u8>.

Change-Id: Ic23839471eb7f68d9c3c30667c878830946b6607
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8990
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
2023-07-31 21:41:38 +00:00
Florian Klink
9521df708f feat(nix-compat/store_path): implement PartialOrd and Ord
This allows sorting Store Paths. We delegate the sorting business to the
PartialOrd, Ord impls for our digest fields only, as two StorePaths with
the same digest, but different names can't exist.

Change-Id: I5f81631e5f5063893b316c63a240c5266b7e5bad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8988
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-07-31 20:29:25 +00:00
Florian Klink
92e976b3a9 chore(tvix/nix-compat): drop useless clone
HashAlgo implements Copy, no need to clone here.

Change-Id: Ief11d2cfbd4fd0cd44224c7ddd575f518edbbd55
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8989
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-07-31 09:01:34 +00:00
Florian Klink
7971d7d9ff feat(tvix/nix-compat/store_path): store position in InvalidName
Add the position in the string where the name is problematic.

Change-Id: If6fd8be6100b718f8d68568eafc77ebb3cfb82d0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8979
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-07-21 19:06:30 +00:00
Florian Klink
72e82ffcb1 refactor(tvix/store): use bytes for node names and symlink targets
Some paths might use names that are not valid UTF-8. We should be able
to represent them.

We don't actually need to touch the PathInfo structures, as they need to
represent StorePaths, which come with their own harder restrictions,
which can't encode non-UTF8 data.

While this doesn't change any of the wire format of the gRPC messages,
it does however change the interface of tvix_eval::EvalIO - its
read_dir() method does now return a list of Vec<u8>, rather than
SmolStr. Maybe this should be OsString instead?

Change-Id: I821016d9a58ec441ee081b0b9f01c9240723af0b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8974
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-07-21 19:01:49 +00:00
Florian Klink
42dc18353d feat(tvix/nix-compat): have StorePath accept bytes
The primary constructor for this is now from_bytes, from_string is
simply calling .as_bytes() on the string, passing it along.

The InvalidName error now contains a Vec<u8>, to encode the invalid name
(which might not be a string anymore).

from_absolute_path now accepts a &[u8] (even though we might want to
make this a OSString of some sort).

StorePath::validate_name has been degraded to a pub(crate) function.
It's still used in src/derivation, even though it probably shouldn't at
all - that cleanup is left for cl/8412 though.

Change-Id: I6b4e62a6fa5c4bec13b535279e73444f0b83ad35
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8973
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-07-21 18:04:38 +00:00
Florian Klink
5364fcb127 feat(tvix/nix-compat): fold NameError into Error
This being a nested error makes things more complicated than necessary.

Also, this caused BuildStorePathError to only hold NameError,
so refactor these utility functions to either return Error, or
BuildStorePathError.

Change-Id: I046fb403780cc5135df8b8833a291fc2a90fd913
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8972
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-07-21 18:04:38 +00:00
Florian Klink
365937cd08 feat(tvix/nix-compat): derive Hash
This allows using a StorePath as a key in a hashmap.

Change-Id: Id3eed623da4e1fc44a970a3982c7caa21d2495c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8666
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-10 14:54:36 +00:00
Florian Klink
dcbcac8955 refactor(tvix/nix-compat): clippy
Change-Id: Ie5277a5c15d9dfe543ca41fa7c6a1eedf22a9f64
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8593
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-05-18 19:29:53 +00:00
Florian Klink
71c29d0f4c feat(tvix/nix-compat): add StorePath::from_absolute_path_full
This allows decomposing a path consisting of a store path AND a suffix
into a StorePath, and a PathBuf containing the rest.

Change-Id: I81290e2fd804cdc9d1e88c71cb22c0fb882d7936
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8567
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-05-16 10:01:25 +00:00
Florian Klink
1a2190cd3b docs(tvix/nix-compat): update docstrings
Make it cleaner that StorePath only does encode the first path component
after the STORE_DIR prefix. Also, move some of the comments around a
bit, so it makes more sense what's using what.

Change-Id: Ibb57373a13526e30c58ad561ca50e1336b091d94
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8566
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-05-16 10:01:25 +00:00
Florian Klink
46ca98a7a2 refactor(tvix/nix-compat): update expect_err strings
Make it more explicit that we expect the from_string calls to fail here.

Change-Id: Ib3d46fc0850e364125e3548670ef301eeea2e45c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8565
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-05-14 18:59:55 +00:00
Florian Klink
b46f2b52c7 feat(tvix/nix-compat/nar/writer): check for more data in reader
We already returned UnexpectedEof in case the reader stopped returning
bytes too early, but similarly we should also fail if there's still
bytes left to be read in the reader passed.

We normally use the NAR writer to produce new NAR files, so the readers
point to the blobs we actually want to render, and having some data left
in there should be an error.

If for some reason the reader points to more data than just the blob,
the `.take` method can be used to limit it to the (known) size.

Change-Id: I9e8fa0a6dd9c794492abb6dc9e55995e619cb3bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8553
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-05-10 06:15:14 +00:00
John Ericson
26c68f8e89 refactor(nix-compat): Properly encapsulate store path construction
Before there was code scattered about (e.g. text hashing module and
derivation output computation) constructing store paths from low level
building blocks --- there was some duplication and it was easy to make
nonsense store paths.

Now, we have roughly the same "safe-ish" ways of constructing them as
C++ Nix, and only those are exposed:

- Make text hashed content-addressed store paths

- Make other content-addressed store paths

- Make input-addressed fixed output hashes

Change-Id: I122a3ee0802b4f45ae386306b95b698991be89c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8411
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-04-09 15:12:21 +00:00
John Ericson
a8bd926e0f refactor(nix-compat): Avoid encoding round trip
When building store paths we can just construct the thing.

Change-Id: Ife5d461d6a440ecbb22f32a86a6d51d212a2035b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8409
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-03-31 18:46:33 +00:00
John Ericson
b932cf2d85 refactor(nix-compat) Make nixhash_* mods hierarchical
They can go under `nixhash`

Change-Id: Ia15835c57130b66d58f5df80ae9595dceee00941
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8408
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-03-31 18:46:18 +00:00
John Ericson
df78a4d24e refactor(tvix/nix-compat): Inline texthash mod
It is moved into `store_path::utils` with the other path builders.

Change-Id: I3257170e442af5d83bcf79e63fa7387dd914597c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8410
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-03-31 18:46:08 +00:00
Florian Klink
5f2c2e79e1 refactor(tvix/nix-compat): move build_store_path out of derivation
This doesn't have anything to do with ATerms, we just happen to be using
the aterm representation of a Derivation as contents.

Moving this into store_path/utils.rs makes these things much cleaner -
Have a build_store_path_from_references function, and a
build_store_path_from_fingerprint helper function that makes use of it.

build_store_path_from_references is invoked from the derivation module
which can be used to calculate the derivation path.

In the derivation module, we also invoke
build_store_path_from_fingerprint during the output path calculation.

Change-Id: Ia8d61a5e8e5d3f396f93593676ed3f5d1a3f1d66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8367
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-30 14:03:35 +00:00
Florian Klink
4ab180b1eb refactor(tvix/nix-compat): use Sha256::new_with_prefix
Change-Id: I328983008c0680e0448ccff604dbdae312f7988b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8366
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-30 14:03:34 +00:00
Florian Klink
6454769eef refactor(tvix/nix-compat): drop is_derivation in build_store_path
This only added a suffix to the input argument, if build_store_path was
building the path of a Derivation.

As we need to also add the `.drv` suffix to the name we pass into
text_hash_string inside calculate_derivation_path, we can simply add the
suffix there and drop the parameter from build_store_path.

Change-Id: Icd5343dd1458f112b9296b389e81ce2ebdd16a9f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8365
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-30 14:03:34 +00:00
Florian Klink
971080c912 refactor(tvix/nix-compat): add text_hash_string function
Use it to calculate the text_hash_string, which is then used in the
calculate_derivation_path and path_with_references functions.

Relates to b/263.

Change-Id: I7478825e2a23a11224212fea5e3fd06daa97d5e5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8364
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-30 14:03:34 +00:00
Florian Klink
14c5781389 refactor(tvix/nix-compat): move HashAlgo to separate file
… and keep the pub exports as is.

Change-Id: I2ad21660577553395f05b5ba71083626429b0dfc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8363
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-03-30 13:47:31 +00:00
Florian Klink
e53152866e refactor(tvix/nix-compat): move NixHashWithMode into separate file
… and keep the pub exports as is.

Change-Id: I9f89a738c508c478ddba61303c21ea294f01ee9f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8362
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-30 13:47:31 +00:00
Florian Klink
def2d32319 refactor(tvix/nix-compat): remove get_fixed_output() func
The behaviour of this function is a bit unintuitive, and cl/8310 already
inlined the other consumer of it.

Rewrite the last consumer of the function, so we can drop it.

Change-Id: I59c8486037ce3f777667d1d9e4f4a9316d5a0cb9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8311
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-15 15:31:55 +00:00
Florian Klink
09b30f895a refactor(tvix/nix-compat): simplify fingerprint calculation
Instead of having two very similar match branches for the FOD and non-
FOD case, detect the FOD case while looping over all outputs.

In the case of anything other than recursive sha256 FODs, the
fingerprint and output path calculation is exactly the same.

Change-Id: Ieb6995653d008766e595cf29d7cd4fb1334e33dd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8310
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-03-15 14:02:24 +00:00
Florian Klink
b55d1f97ce refactor(tvix/nix-compat): -derivation::Hash, +NixHash
This stops using our own custom Hash structure, which was mostly only
used because we had to parse the JSON representation somehow.

Since cl/8217, there's a `NixHash` struct, which is better suited to
hold this data. Converting the format requires a bit of serde labor
though, but that only really matters when interacting with JSON
representations (which we mostly don't).

Change-Id: Idc5ee511e36e6726c71f66face8300a441b0bf4c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8304
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-15 11:50:12 +00:00
Florian Klink
a14880903c refactor(tvix/nix-compat): rename helper func
This only trims the output paths from a Derivation struct, not the
output hashes.

Change-Id: I9250fec4602ed05bb64540c4a89ddb6fb052be1f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8303
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-03-14 20:53:12 +00:00
Florian Klink
84a8ed265c refactor(tvix/nix-compat): remove manual map
Change-Id: I1652e24c9be28112d98683de2d2db51dc46001ed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8302
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-03-14 20:53:12 +00:00
Florian Klink
c699ace11b refactor(tvix/nix-compat): rename test name
Change-Id: Ia6af8d4b53fbd4d9ea26f0094cd98dde66a91f78
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8301
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-14 20:53:12 +00:00
Florian Klink
075dcbf1e0 fix(tvix/store/derivation): fix comment
The digest is encoded in LOWERHEX here, not nixbase32.

Change-Id: I8e14742f0135966424073bfdb93ada8cd7bd52e2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8300
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-14 20:53:12 +00:00
Florian Klink
5f260edf7f refactor(tvix/nix-compat): replace calculate_drv_replacement_str
Call this function derivation_or_fod_hash, and return a NixHash.

This is more in line with how cppnix calls this, and allows using
to_nix_hash_string() in some places.

Change-Id: Iebf5355f08ed5c9a044844739350f829f874f0ce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8293
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-03-14 18:26:19 +00:00
Florian Klink
1f9b582239 feat(tvix/nix-compat/derivation): make use of NixPath in some places
Some of these strings are actually just the nix hash representation of
the hash calculated earlier.

There's another one passed around via calculate_drv_replacement_str, but
that's left for a followup.

Change-Id: Id99a2a926a980d679eb49c34ee6a36bf224699b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8218
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-03-04 12:26:00 +00:00
Florian Klink
c2a681eaff feat(tvix/nix-compat): add NixHash::{new, to_nix_hash_string}
This provides a way to construct a NixHash struct without parsing
strings.

Change-Id: I947d96e15e51e72d5b02929cda8c5fc31d81253a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8217
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-04 12:26:00 +00:00
Florian Klink
67c9e2c770 feat(tvix/nix-compat): introduce parsing for Nix Hash strings
We ironically didn't add support parsing for the "native" format that
Nix uses under the hood.

This extends the from_str method to peek at the prefix of the string to
determine whether to try decoding as SRI, Nix string, or whether it
should be a bare digest.

Change-Id: I33efd24968b16f86eff18305b4ca8f112c7131d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8216
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-03-04 12:26:00 +00:00
Florian Klink
4cc4b2a814 refactor(tvix/nix-compat): move digest decoding into helper function
This will be used for both Nix hash strings and hash strings without the
algo specified.

Change-Id: Iedfe5494fba5f2be00614ba0fc38bf659eafd447
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8215
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-04 12:26:00 +00:00
Florian Klink
daba4cc0e5 refactor(tvix/nix-compat): address clippy in nixbase32.rs tests
Change-Id: If8820cba4cf19bf0f7aa27e0d93b70eb7b4ee81f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8221
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-04 12:21:28 +00:00
Florian Klink
93b0766485 refactor(tvix/nix-compat): address clippy in derivation/tests/mod.rs
Change-Id: I3f3f7cb590c900abf8b39533ed73fe9135d58f0b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8220
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-03-04 12:19:24 +00:00
Florian Klink
f9dcf54b04 refactor(tvix/nix-compat): address clippy in store_path.rs
Change-Id: Ib10bd93bbb23696d7048e7ed8e405953db94693f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8219
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-03-04 12:19:24 +00:00
Vincent Ambo
248f1c2151 chore(tvix/nix-compat): include invalid hashes in SRI hash error
Change-Id: I379b18cb07da0988f8ce4934976ae7d6566d6bb5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8210
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-03-03 23:35:47 +00:00
Florian Klink
d4cbdc2beb feat(tvix/nix-compat/nixhash): support BASE64_NOPAD SRI
Nix accepts SRI hashes that are missing their padding characters
in base64, as seen in
7e49471316/pkgs/development/libraries/kerberos/krb5.nix .

It only seems to work in the SRI case, not with `sha256` being set to a
(nopad) base64 string.

Add regression tests for this, and document why we don't want to support
*additional* characters afterwards.

Reported in https://b.tvl.fyi/issues/252

Change-Id: I9ffc2b417501b426ced1894a9cbf95ff5f0e5159
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8037
Reviewed-by: Alyssa Ross <hi@alyssa.is>
Tested-by: BuildkiteCI
2023-02-08 10:39:26 +00:00
Florian Klink
5fa35d9d49 refactor(tvix/nix-compat/nixhash): fix test function name
Change-Id: I3b089758ce8c01df544064422e56a97a8402513d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8036
Tested-by: BuildkiteCI
Reviewed-by: Alyssa Ross <hi@alyssa.is>
2023-02-08 10:39:26 +00:00
Alyssa Ross
5da1f4161c fix(tvix): fix InvalidEncodedDigestLength message
Before:
	error[E997]: invalid output hash: invalid encoded digest length '43' for algo 43

After: error[E997]: invalid output hash: invalid encoded digest length '43' for algo sha256

Change-Id: Icb533839586b47f90b4e8d22083cce397f9f3655
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8035
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-07 17:40:54 +00:00
Florian Klink
f5ccb65f26 feat(tvix/nix-compat/derivation): Display -> to_aterm_string()
Instead of implementing `std::fmt::Display for Derivation` and relying
on the `to_string` method, introduce a `to_aterm_string()` method, which
does the same thing, but makes it clearer what we're producing, rather
than just calling `to_string()``.

Change-Id: I21823de9096a0f2c2eb6f4591e48c1aa9fd94161
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7998
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-01 16:31:56 +00:00
Florian Klink
265dd36b98 feat(tvix/nix-compat): add nixhash module
This module takes care of parsing various hashes and algorithms.

It will get used to modify derivation output hashes in the next CL.

Change-Id: Idc07c401dbb7510f49883ac02b8379b9a5d930c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7990
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-02-01 15:29:48 +00:00
Florian Klink
366d2c5e92 refactor(tvix/nix-compat/derivation): remove module inception
The module `src/derivation/derivation.rs` is a sign of module inception.

Move the Derivation struct definiton up into `src/derivation/mod.rs`,
and some of the helpers in a `util.rs`.

Change-Id: Ib24a5f8a27bdd45df8b1fa2b3482a79b33cab8d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7997
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-02-01 13:43:25 +00:00
Florian Klink
2551adb4f1 refactor(tvix/nix-compat): operator precedence can trip the unwary
warning: operator precedence can trip the unwary
  --> nix-compat/src/nixbase32.rs:41:23
   |
41 |                 c |= ((input[i + 1] as u16) << 8 - j as u16) as u8
   |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: consider parenthesizing your expression: `(input[i + 1] as u16) << (8 - j as u16)`
   |
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#precedence
   = note: `#[warn(clippy::precedence)]` on by default

Change-Id: I091071d649abf4ed38f5f4e39a0c5d21a0459bff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7996
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-01 13:43:24 +00:00
Florian Klink
361ffd7903 refactor(tvix/derivation): don't create deref'd immediately ref
clippy says:

> This expression creates a reference which is immediately dereferenced
> by the compiler

Change-Id: Ic2c093b043ebee9ae80912075083107e4d216cf1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7995
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-01 13:38:21 +00:00
Florian Klink
ce4d87b129 refactor(tvix/nix-compat): remove unneeded return
Change-Id: I97b9fb3ad33d2f51baf29486d4eff11f8dedcbab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7994
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-02-01 13:38:19 +00:00
Florian Klink
894d17b29b refactor(tvix/nix-compat): const is always 'static
Change-Id: If33bbf47a6cc376f2c0935ea4331a819223b2f00
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7993
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-02-01 13:38:18 +00:00
Vincent Ambo
759f9dbf39 feat(tvix/cli): implement builtins.placeholder
This doesn't require any other corresponding handling *yet*, as the
actual replacements happen in the builder logic (which we delegate to
cppnix at the moment).

Change-Id: I034147c933f05ae427c7a8794647132d108d0ede
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7972
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-02-01 10:01:40 +00:00
Florian Klink
a7e20bbef3 docs(tvix/nix-compat/store_path): fix docstrings
Change-Id: I19261233e338b406a87fe680675eb4dfd853fe11
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7970
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-01-31 15:35:45 +00:00
Florian Klink
a23b7e17c0 refactor(tvix/nix-compat): absorb nar writer
Expose it at `nix_compat::nar::writer`.

Change-Id: I98a8acfa6277074f24608d64fb6c0082b41d10f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7969
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-31 15:18:39 +00:00
Florian Klink
1e28cc7ede docs(tvix/nix-compat/derivation): fix docstrings
Change-Id: I3fd02e62c8a24fa7f27461b17cdd824b060d9bdb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7968
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-01-31 15:16:40 +00:00
Florian Klink
2d24c5f260 refactor(tvix/nix-compat): absorb //tvix/derivation
Put this in its src/derivation.

Change-Id: Ic047ab1c2da555a833ee454e10ef60c77537b617
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7967
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-01-31 15:16:39 +00:00
Florian Klink
c27bacd905 refactor(tvix): introduce nix-compat crate
Move nixbase32 and store_path into this.

This allows //tvix/cli to not pull in //tvix/store for now.

Change-Id: Id3a32867205d95794bc0d33b21d4cb3d9bafd02a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7964
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-31 13:26:18 +00:00