This is only used inside this function, in 2 of the match cases.
Change-Id: Ib361f5ee0e3b203802f7d05b9a7f332d14bbcf80
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11143
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: John Ericson <git@johnericson.me>
Tested-by: BuildkiteCI
The outer fingerprint used for store path calculation is always a sha256
digest. This includes both input and output-addressed store paths.
We used a NixHash here, which can also represent other hash types, and
that had a bunch of annoyances:
- Whenever we had the bytes, we had to wrap them in a NixHash::Sha256().
- Things like AtermWriteable had to be implemented on NixHash,
even though we then had an assertion it was only called in the
NixHash::Sha256 case.
Change-Id: Ic895503d9b071800d2e52ae057666f44bd0ab9d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11142
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: John Ericson <git@johnericson.me>
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Implement the fetchurl builtin, and lay the groundwork for implementing
the fetchTarball builtin (which works very similarly, and is implemented
using almost the same code in C++ nix).
An overview of how this works:
1. First, we check if the store path that *would* result from the
download already exists in the store - if it does, we just return
that
2. If we need to download the URL, TvixStoreIO has an `http_client:
reqwest::Client` field now which we use to make the request
3. As we're downloading the blob, we hash the data incrementally into a
SHA256 hasher
4. We compare the hash against the expected hash (if any) and bail out
if it doesn't match
5. Finally, we put the blob in the store and return the store path
Since the logic is very similar, this commit also implements a *chunk*
of `fetchTarball` (though the actual implementation will likely include
a refactor to some of the code reuse here).
The main thing that's missing here is caching of downloaded blobs when
fetchurl is called without a hash - I've opened b/381 to track the TODO
there.
Adding the `SSL_CERT_FILE` here is necessary to teach reqwest how to
load it during tests - see 1c16dee20 (feat(tvix/store): use reqwests'
rustls-native-roots feature, 2024-03-03) for more info.
Change-Id: I83c4abbc7c0c3bfe92461917e23d6d3430fbf137
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11017
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: aspen <root@gws.fyi>
Otherwise this causes conflicts when adding a more recent zstd version
to another crate.
Change-Id: I02d1b1a37f18711c0969cb04e761e967fc4655dc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11100
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
This allows peeking at the mode without matching on all enum kinds of
CAHash directly.
Change-Id: Icca147a876f7f4cff0bbaa067ac4dae40d05bee9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11069
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Currently CAHash only deserializes the hash in hex code while
the serializer outputs a nixbase32 hash. This means that you can't currently
deserialize what has been serialized.
This change makes deserialize support any digest format (so hex, nixbase32
and base64) as well as flattens the deserialize code and error handling.
It also implements serde methods of HashAlgo directly using Display and TryFrom
implementations because otherwise these would get serialized as eg. Sha256 instead
of sha256 which also broke CAHash serialize/deserialize.
Change-Id: I1941a72eaec741e4956292adaaf0115b97f260ba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11082
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This doesn't return the digest, but the internal NixHash.
To get the digest, you use `ca_hash.hash().digest_as_bytes()`
(as done in tvix/nix-compat/src/derivation/write.rs).
Change-Id: Ib07918dec63ae42ddf8c8d3f1d10510f9c4df255
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11070
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Move the the `fixed:out:[r:]{}:` generation to a helper function,
use matches! for more clarity.
Change-Id: I4e930c42aacbf5c7451d1f8c8c80ccb4c45389f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11006
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
Autosubmit: flokli <flokli@flokli.de>
...in `Derivation`.
This is more type-safe and should consume less memory.
This also removes some allocations in the potentially hot path of output hash calculation.
https: //b.tvl.fyi/issues/264
Change-Id: I6ad7d3cb868dc9f750894d449a6065608ef06e8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10957
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
...so that we can also use `StorePath`s in
derivation.input_derivations.
Towards https://b.tvl.fyi/issues/264
Change-Id: I71d296ca273979c70f277a7f4f88a5f76de3d8be
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10973
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
...so that we can parse (and at the same time
validate) store paths directly to `StorePath`s.
https: //b.tvl.fyi/issues/264
Change-Id: Ide4cb5403f8e3400f7d2a5838c652ac35dbd1a4f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10968
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Tested-by: BuildkiteCI
Necessary, if we want to use it inside of `Derivation` etc.
Change-Id: I8888060417b2ee83ac52d7ec3e7b27c393271d8b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10947
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Instead of constructing a completely new derivation for hashing,
allow to call hashing with substituted input_derivations.
This
1. reduces the number of allocations,
2. prepares substituting `String`s for store paths with proper `StorePath`s without needing a separate derivation struct that allows hashes without `/nix/store` in input_derivations, and
3. keeps the change local to the `Derivation` implementation.
Change-Id: I36732c78f98fc59f0925b65823773222782017b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10935
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Peter Kolloch <info@eigenvalue.net>
This will be used inside glue, to populate the CA field.
Change-Id: I52350d66ad09c90c4e9f3bc6ea44fba4427f1576
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10946
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Populate the input_derivations struct field in-place, using
self.input_derivations.from_iter, which seems a bit more readable.
Change-Id: I4aa12cfd66e08486ef27aedab1bb731d94b63657
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10934
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Peter Kolloch <info@eigenvalue.net>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
In cl/10468, we accepted SRI hashes of invalid padding while checking
their trailing bits.
In this commit, we accept SRI hashes of invalid padding and invalid length, as Nix does.
Real world example: `pkgs.javaPackages.openjfx11.deps`
<849e4dc5ff/pkgs/development/compilers/openjdk/openjfx/11.nix (L71)>
in nixpkgs.
Change-Id: I834437e7b94dab9fbb030163f7a2741f52bbf03a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10668
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
- Adjust to ecl 23.9.9 release
- Regenerate go protos after protoc-gen-go update
- Drop dhall fork which hasn't kept up with 1.42.*
- Address new clippy warnings:
- Variant naming of Error::ValidationError
- Simplify .try_into().unwrap()
- Drop unnecessary identity function
- Test module must be last in file
- Drop unused `pub use`
- Update agenix to 0.15.0. Current master has a installCheckPhase that
doesn't work with C++ Nix 2.3.*:
a23aa271be (commitcomment-137185861)
Change-Id: Ic29eef20d6fd1362ce1031364a5ca6b4edf195bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10615
Reviewed-by: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Rather than passing strings around, use a StorePathRef.
This makes things a bit more typesafe, and more aligned with what we
want to do in b/264.
Change-Id: Ib7080addf27e7f1a9c8da1d8aaa66744468e3b5a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10633
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Nix is quite tolerant when it comes to parsing SRI hashes and their
padding (and only for SRI hashes, it funnily is strict about that in the
non-SRI-hash case).
Nix essentially accepts any number of padding characters, no matter if
it's too much or too little. So we do the only sane thing - simply strip
all padding characters, and parse it with BASE64_NOPAD and the length
the algo uses.
Change-Id: I6a721aa289b06cc36741589792b9dd4c4f930b86
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10468
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
self_reference being set to true is only allowed for
`CAHash::Nar(NixHash::Sha256(_))`, so we can handle this in a check at
the front.
Change-Id: Ic363ade4789a7767cbe26a6959b143bb53e50e5a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10391
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
All match cases essentially construct `ty` and `hash`, which is then
passed to the `build_store_path_from_fingerprint_parts` function.
Change-Id: I01dfd219f9b0ac1afe8af7c6e361ea048117a0e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10390
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
In most case, we don't actually need an owned `StorePath` struct, but a
`StorePathRef<'_>` is sufficient. The lifetime is only due to it holding
onto the name, but that one is mostly left untouched.
`Derivation::calculate_derivation_path` still needs to return
`StorePath`, as its name has a `.drv` appended.
Change-Id: Ie0d52f369d785711bb0658ea2b0bd2617fd9f45e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10389
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
With the introduction of StorePathRef::to_absolute_path(), there's no
need to convert this StorePathRef to a StorePath first.
Change-Id: I634c977c4b63858e4f329fd21726e0611b99da4a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10388
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Keep the method around in StorePath for convenience, but move the
implementation to StorePathRef.
Change-Id: Ie1844fa01ce6529dc1a58907563c95c3112c831d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10387
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
This pushes to a Vec immediately after creation.
Change-Id: I2360b45810475d98ededc1d097fb4cbdeabc576b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10232
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
The only non-test usage was only checking for the error case, and we can
still convert this to an owned StorePath by calling to_owned() on
StorePathRef.
Change-Id: I9f67a759e580c9c429c96896bcdd295392aa5a2a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10225
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Match the naming in parse_input_derivations, call the keys
"input_derivation", and the values "output_names".
Change-Id: I7d1974819028f8ea543dc3ad78afb803ff9db865
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10224
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This makes it easy for each PubKey to check if a given Signature is
correct for a given fingerprint.
Change-Id: I56e6211d133f74f390fd1ae3ae799eef12221904
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10151
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This represents a ed25519 public key and "name".
These are normally passed in the `trusted-public-keys` Nix config option,
and consist of a name and base64-encoded ed25519 pubkey, separated by a `:`.
Change-Id: I9ab4b3e0e5821805ea6faf2499626630fc5a3f0a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10150
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Just call this Error, we can infer from the package what error this is.
Change-Id: I5df25d2873ec739c49c08804f35562c84c222e06
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10149
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
We should restrict this to alphanumeric mostly, and we definitely don't
want newlines.
Not entirely sure about the exact additionally allowed characters
outside of alphanumeric, but this can always be extended further.
Change-Id: I1357e79e553f2df2fa97792889f63f0f35d50ed5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10147
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
BASE64.decode_mut panics if we're passing data that has the wrong size.
Do the size check first and error out there.
Also update the error, and talk about b64-encoded sizes.
Change-Id: I290f80a37d48526a30bf1df9d1d9fe34865008eb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10146
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
No need to hardcode magic numbers here, we have a constant for that.
Change-Id: I67b671c0c4bb7c3bfb001e9c36499f31873ee717
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10145
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This is useful when creating a new Signature struct where the individual
elements are already parsed.
Change-Id: Ie33c66287641951e7a030aaa1e7ff0a86b2628ac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10111
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We can simply use .to_owned() on that thing afterwards if we want to
construct an owned StorePath.
Change-Id: I0f3e2e4434b99ee522f2a7dbfa391e13a987479c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10105
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
We always know this needs to end with a .drv, and fail parsing if it
doesn't, so there's no need to hang onto these 4 bytes.
This will make it much easier to synthesize a NarInfo<'_> later on from
a PathInfo proto, because we don't have to make this ".drv" appear out
of thin air.
Change-Id: Id95e7fd937d7c9a420a39b5a4bab73985640ca3b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10084
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
These were used to format to and parse from strings.
Move this to the CAHash and NixHash structs directly, and be explicit in
the name about which encoding for digests is used.
For output path calculation, nix encodes the nixpaths in hex, but for
writing out NARInfos, it's using nixbase32.
Change-Id: Ia585a76a3811b2609e7ce259fda66a29403b7e07
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10079
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
This adds support to compute the fingerprint string, which is what's
ed25519-signed in binary caches.
Change-Id: I8947239c609896acfd7261f110450014bedf465a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10080
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This adds support for verifying signatures found in NARInfo files.
This still needs to be hooked together with the nix+http[s] backend.
Change-Id: Ic1c8ddbdecfb05cefca2492808388b0f7f3f2637
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10081
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Allow taking advantage of the buffer of the underlying reader to avoid
unnecessary copies of file data.
We can't easily implement the methods of BufRead directly, since we
have some extra I/O to perform in the final consume() invocation.
That could be resolved at the cost of additional bookkeeping, but this
will suffice for now.
Change-Id: I8100cf0abd79e7469670b8596bd989be5db44a91
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10089
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We rely on being able to make small reads cheaply, so this was already
an implicit practical requirement. Requiring it explicitly removes a
performance footgun, and makes further optimisations possible.
Change-Id: I7f65880a41b1d6b5e6bf2e52dfe47d4c49b34bcd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10088
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
We don't need to validate UTF-8 separately, since valid names are
a strict subset of ASCII, and therefore a strict subset of UTF-8.
Change-Id: I3261bf0efe3480b5b315074efafcf5e47a6c5a65
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10087
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
This suggests it's cheap to convert around, but name actually does
allocate.
Move to a `to_owned(&self) -> StorePath`, to better signal that this
does allocate.
Change-Id: Ifaf7c21599e2a467d06e2b4ae1364228370275db
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10066
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This appears in the cache.nixos.org dataset.
Change-Id: I2eadafe8441e0132a448828026553da2dc7c12aa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9994
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This appears in the cache.nixos.org dataset.
Change-Id: I35921f7ef148f6681081a4e371abb8c9cc98854d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9993
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Rather than having our own error type, just make decoding errors use
the same common error type.
Change-Id: Ie2c86972f3745c695253adc3214444ac0ab8db6e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9995
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This appears in the cache.nixos.org dataset.
Change-Id: I055b60b9950a1a6a36c1b0576b957e11e1d4264b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9990
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This small tool formats A-Term in a more readable format. It's a lossy
conversion for non-valid UTF-8 environment values.
Change-Id: I65a51054d7faf528321bc2d9fc4425180a7813f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9970
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Walking a btree_map twice is more expensive than copying a string,
especially because the cloning only happens in the (non-hot) error
path.
This fixes a clippy lint, so it's related to b/321.
Change-Id: I2ccfd0bc46792a45d277f47564e595b87107d8be
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9962
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This already has the right type.
Change-Id: I8f5850a41f9e97f1bc5f2a45ca05cf7439665c9d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9954
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
We also switch the MissingField error to &'static str, since we only
parse a fixed set of fields.
Together, this makes the performance impact of error handling
negligible in batch happy-path parsing.
Change-Id: I2bd0ef2f5b35fcaced56b32d238eca75ac199ef1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9867
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: edef <edef@edef.eu>
Tested-by: BuildkiteCI
We primarily want to measure the speed of the happy path.
Change-Id: Iad0146dde86fc262e2a4b8295bde4eb297b8bf30
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9866
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: edef <edef@edef.eu>
This provides more info about where a NARInfo failed to parse, rather
than just returning None and leaving a library user to manually debug.
Change-Id: I9a28ddd8e5712101483ebe686fdc474c7bbc8e4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9831
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
