This test will otherwise obviously fail on aarch64-linux, or anything
not x86_64-linux, as it'll produce a different Derivation.
Change-Id: Iedddcb6aeff05e49fc2e6da27f08111cdc9affec
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12677
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Deduping early saves a fair bit of memory, but the extra hashing is
costly.
We switch to FxHash, since we don't need a DoS-proof hash, but we do
need it to be *fast*.
Change-Id: Ic6b7010874c417862baa9b882593208c8dd1d5e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12648
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Parsing of store-paths.xz is now handled by //users/edef/fetchroots.
Change-Id: I78be5aada0c0a321ed79d80c9b615e5f997ac3e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12670
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This vastly reduces the memory requirements, so we can run in ~40G RAM.
Change-Id: I4952a780df294bd852a8b4682ba2fd59b9bae675
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12667
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We switch to working with the underlying Arc<Bytes<u8>> type, since
Buffer<u8> is already essentially an OwningRef-esque subslice.
Because we're now working with an exposed Arc directly, we don't need
to have our own `unsafe impl StableAddress` any more.
Change-Id: I9ce2edc6899177145e15b72aa5380f708a62173c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12668
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We only care about ordering on the same variable, and we rely on the
release barrier provided by rayon's thread joining.
The comparison failing is always an error path, and we're indifferent
about which thread it fails on.
Change-Id: I592a7eaae5b6935cf424c6576a49786f39909fb6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12666
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
It seems we need more memory these days, and llama frequently ran OOM.
Decrease the number of concurrent evaluations.
Change-Id: I2648ebdedf09b80c9a231c4614004f953a646bc0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12662
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
When evaluating nixpkgs from the <nixpkgs> channel, store_path_to_node
returns an error, falling back to regular filesystem access.
This currently produces a warning in the CLI, which is misleading, so
degrade to trace level.
Change-Id: I4cb2297cc85a2c0e904a37343748f9051aa6d5c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12665
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Construct the owned signature in a separate scope, so all borrows to the
original PathInfo are already dropped again, and we can modify the
PathInfo without having to clone it.
Change-Id: I03e7390540c2cfe7a2c61850bdbe8a33d213a5d9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12663
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This pushes generating spans with pb_show up to the caller.
They usually have more context on how to present things, if at all.
Change-Id: Icfcaa64a8a57dce50c0261f2d06e7c051e3946c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12657
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
We have two places where we parse protos and want their names to be
empty:
- Receiving a root node in a nar-bridge NAR request
- Processing the CalculateNAR gRPC call
We don't have any place where we want to keep a name as bytes::Bytes
around, yet we used the `into_name_bytes_and_node` method.
It was also a bit wrongly named - it wasn't very clear the name was
not validated, and that the function may fail.
This moves the "splitting off the name as bytes::Bytes" part into a
private helper, only leaving the `try_into_name_and_node` and
`try_into_anonymous_node` methods around.
Change-Id: I2c7fd9871d49ec67450d7efa6a30d96197fb319c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12664
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Marijan Petričević <marijan.petricevic94@gmail.com>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This implements BS, DS, PS for Box'ed or Arc'ed variants of it with less
code, and less potential to accidentially forget to proxy default trait
methods for blanked impls, as fixed in cl/12658.
Change-Id: If2cdbb563a73792038ebe7bff45d6f880214855b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12661
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
If the PathInfoService is using gRPC, expose it via the
nar_calculation_service() method in the PathInfoService.
Also pass nar_calculation_service through in tha blanket impl.
This now causes a `tvix-store import` using the default config to not
fall back to `SimpleRenderer`, which will calculate the NAR hash and
size by downloading the uploaded blobs (and blobs it didn't need to
upload) locally, making such imports faster.
Change-Id: If2c3fe6584e9093cba322d2360f355a3923904ae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12658
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Clients can (and do) send an empty name in here, and we discard it
anyways.
Change-Id: Iec135d9cba48cc066464e5639319c845079787e9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12659
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
With the seekable NAR renderer, figuring out the next few blobs to
render became possible.
Change-Id: I1214302f88e6f9aba74227f84df0f964d587baf2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12652
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
This is now supported in the standard library via std::sync::LazyLock,
but requires some manual shuffling around of code.
Change-Id: Ifca792f4d2dbc36b703de4a4dfa406015ab86da7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12614
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
This is now supported in the standard library via std::sync::LazyLock,
but requires some manual shuffling around of code.
Change-Id: Ia0370ca46cb1c6122a452b1d117160536b632c7e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12612
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This is the only (remaining) occurence of it, and not really
more code than just calling store_path::build_ca_path with
`CAHash::Nar(NixHash::Sha256(…))`, especially considering we need the
CAHash in the PathInfo struct later anyways - so let's remove this
function.
Change-Id: Ia82212086062c366e0280ca0823d9e68a3f91d3a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12632
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This became obsolete, since the introduction of a stricter `Directory`
struct invalid names cannot be represented anymore.
Change-Id: I9e4b1b6cca01831d0a9735f58d8a1f59ac18676b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12615
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This is not gonna work out as-is, as we still key PathInfos by
their store path digest, and how to handle thing if we encounter a
Frankenbuild.
For now, let's keep the PathInfoService data as it is, we can record
this information (and more) in the builder structures.
Change-Id: Ic38fc3ecd8096a5fe002e681bdc812a9dbeaa7d2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12607
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
For mystifying reasons, Type=simple and CREDENTIALS_DIRECTORY in
ExecStop have stopped working (when exactly I don't know, but presumably
256). Apparently, you are supposed to use Type=exec with credentials due
to raciness (I've personally never experienced):
<https://github.com/systemd/systemd/issues/32583>.
Just changing the type did not resolve the issue of
CREDENTIALS_DIRECTORY being unset, though. It appears, though, that the
issue is merely an unset environment variable and not the credentials
being unavailable: We can work around the problem by setting an
appropriate environment variable ourselves.
Change-Id: Ifcdb1f3bce782ea1c568a9bc413f3fb29f0985c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12649
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: sterni <sternenseemann@systemli.org>
This moves the implementation from builtins.path into a helper function,
which we now call from both builtins.
Most of the Value plumbing stays inside this helper.
We also implemented handling of symlinks at the root, which was handled
in builtins.filterSource, but not builtins.path - by peeking at the
FileType using std::fs::metadata, instead of the EvalIO trait.
For now, this is fine, as our filtered_ingest also goes via the
filesystem directly. It ends up with the same semantics as before and in
Nix - symlinks at the root are followed, except if they point to an
invalid target.
In the future, we should revisit this, and then maybe get both stat and
lstat into EvalIO, though we will need to be very careful about the
semantics for following symlink inside store paths.
Change-Id: I6a941c0187db36165c2f7a338015e4e32d41b298
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12629
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
In a previous refactoring CL this into_bstring method was accidentally
kept, when we don't need it and can just to_str directly.
Change-Id: Idd531d508b8fd530611b213d0164e7aaf0e87d80
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12631
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
These nested ifs are a bit confusing, a match block makes this cleaner.
Change-Id: I256fd0bc921fbf2e60ad0f6e1ea51c2e0fb00317
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12628
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This removes all the intermediate helper functions and reorganizes the
import code to only do the calculations where/when needed, and hopefully
makes things easier to understand as well.
Change-Id: I7e4c89c742bf8569b45e303523f7f801da7127ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12627
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Jörg Thalheim <joerg@thalheim.io>
Reviewed-by: edef <edef@edef.eu>
This makes it easier to understand what the specific test is testing.
Change-Id: I34b2798841c6b9367849668451af2165dc78f997
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12626
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: Jörg Thalheim <joerg@thalheim.io>
This didn't support store paths with a subpath joined to them, while
Nix does.
Use state.path_exists, which does. This also means we can drop the
`store_path_exists` helper, which was only used here.
Change-Id: I918ccb270f64acbdc41cb4d2a9c3c5871ce15002
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12618
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: Jörg Thalheim <joerg@thalheim.io>
Autosubmit: flokli <flokli@flokli.de>
These are not necessarily strings, and making it paths allows us to stop
converting them to lossy strings.
Change-Id: I11366c721dc5da1778aafe89092a1966b5a43178
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12617
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: Jörg Thalheim <joerg@thalheim.io>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Make this generic on the StorePath<SP> that's being used, similar to the
other functions in there.
Change-Id: I453d1fd3749053d4e5aca156abc18da1f95ca264
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12616
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Jörg Thalheim <joerg@thalheim.io>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
This is now supported in the standard library via std::sync::LazyLock,
but requires some manual shuffling around of code.
Change-Id: I14bee4068dc73c948321481b5a4e1fc922a89a27
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12611
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
This is now supported in the standard library via std::sync::LazyLock,
but requires some manual shuffling around of code.
Change-Id: Ie2af74beda9fcf8aa19fca7d844bcbe732f05bf8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12610
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This is now supported in the standard library via std::sync::LazyLock, but
requires some manual shuffling around of code.
Change-Id: Ibb3be8458b8a8912ea04c9360d64c5cf914254d4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12609
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This is now supported in the standard library via std::sync::LazyLock, but
requires some manual shuffling around of code.
I found at least one dead variable along the way, which I deleted.
Change-Id: I8600c87c49078fb5ff72671994c77b919259e67b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12608
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
This is not a core Tvix tool, it's a tool that uses a Tvix component.
Change-Id: I81d2b2374da23489df0097dcabb8295c82652fc1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12606
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
