This provides a service using /dev/shm, that's deleted once the
reference is dropped.
Refactor all tests to use these, which allows getting rid of most
TempDir usage in the tests.
The only place where we still use TempDir is in the importer tests,
which work on a filesystem path.
Change-Id: I08a950aa774bf9b46d9f5c92edf5efba36053242
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8193
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
While we don't want to keep all of the data in memory, we want to
feed a reasonably-enough buffer to the chunking function, to prevent
unnecessarily trying to chunk over and over again.
Change-Id: I5bbe2d55e8c1c63f8f7ce343889d374b528b559e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8160
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This imports the contents at a given Path into the tvix store.
It doesn't register the contents at a Path in the store itself, that's up
to the PathInfoService.
Change-Id: I2c493532d65b90f199ddb7dfc90249f5c2957dee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8159
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This also uploaded HELLOWORLD_BLOB_CONTENTS before, but it's not
referred from anywhere in the fixture.
Change-Id: I823133afe0f08d18a59e2ac4e4d4bb7d34ce8a2b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8158
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This allows hiding to tests what exact implementation we're using, when
testing things that do something with a store, but don't care what's
used for underlying storage.
Change-Id: I7cdf60fd73c25d5050159cb31ec177db2bc2a7f1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8155
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This will moves the chunking-as-we-receive logic that so far only lived
in grpc_blobservice_wrapper.rs into a generic BlobWriter.
Change-Id: Ief7d1bda3c6280129f7139de3f6c4174be2ca6ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8154
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Use the FastCDC::cut function to ask fastcd for cutting points as we
receive the data. Make sure to keep the last chunk in the temporary
buffer, as we might not actually cut at the end.
Also, use rayon to calculate the blake3 hash if the input data is
> 128KiB.
Change-Id: I6195f3b74eac5516965cb12d8d026aa720c8b891
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8135
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This switches away from the less canonical "ronomon" version to the
implementation as described in the
[paper](https://ieeexplore.ieee.org/document/9055082) by Wen Xia, et
al., in 2020.
That version uses 64-bit hash values and tends to be faster than both
the ronomon and v2016 versions, and produces the same chunking as the
2016 version.
As per https://docs.rs/fastcdc/latest/fastcdc/#implementations-1, it's
the recommended choice.
The crate also gained support for streaming version of chunkers:
https://docs.rs/fastcdc/latest/fastcdc/#large-data, which might be
useful.
Change-Id: Ieabec3da54eb2b73c045cb54e51f7a216f63641e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8134
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
These were implementing against the (more complicated) gRPC interface,
for which we now have a wrapper.
Change-Id: I0a8284493718ab99618a1d21a76df4d173edb899
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8100
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This swaps out the implementation used in the main entrypoint.
We now use the non-gRPC aware Sled*Service, and wrap it with
GRPC*ServiceWrapper to spin up the gRPC server.
Change-Id: I5a1236c0612eee85e49891062040127c8bd95058
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8106
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This exposes a proto::pathinfo_service_server::PathInfoService for a
directoryservice::PathInfoService and a way to calculate NARs.
Change-Id: I30cd058562f83d063c78b84976ec97190de49400
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8097
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This adds a PathInfoService trait, and an implementation for it using
sled, and one using a HashMap.
Change-Id: I85fe4c9b6105808d7b0c095441326424ffc2d2bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8099
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This adds a NARCalculationService trait, which will take a root node,
and provide a proto::CalculateNarResponse in return.
It also adds a NonCachingNARCalculationService implementation, that will
simply always render the NAR in memory to calculate the size and sha256,
without any caching.
Change-Id: Id1ffb18559212fa6001f70f2634bbc3dfd0aa343
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8096
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Make sure the blob size in the current proto node matches what we get
back from the blob backend.
Change-Id: I939fa18f37c7bc86ada8a495c7be622e69ec47f8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8129
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
It's much easier to just unwrap the TempDir::new() result.
Change-Id: I7b05bc18f3146401e30e1cc2bb412503c5171a66
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8128
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This moves the logic rendering NARs to a struct using the
previously introduced, more granular BlobService, ChunkService and
DirectoryService.
Instead of passing them around to the helper functions, they're kept as
members of a struct.
Remove the async invocations in the nar_renderer tests, there's nothing
async in here.
Change-Id: Ic6d24aaad68a1fda46ce29f2cdb5f7b87f481d5c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8095
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This exposes a proto::directory_service_server::DirectoryService for a
directoryservice::DirectoryService.
Change-Id: I191a6e44e5bce10d904b5f10f0073d6adaf1f572
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8094
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This adds a DirectoryService trait, and an implementation for it using
sled, and one using a HashMap.
Change-Id: Ida61524b2ca949e1b3a78089a5aa7d9f9800c8d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8093
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This takes a BlobService and ChunkService in the constructor, and
provides a [proto::blob_service_server::BlobService] trait for it.
Implementing proto::blob_service_server::BlobService is a lot of surface
to cover, and providing this wrapper will make individual
implementations taking care of how to store chunks or chunking
information much simpler.
Change-Id: Ia7b46484fb3ac9104354d496ff2922dca96ff7b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8092
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This allows adding more stuff into this namespace, from different files.
Also move tests on proto-related code from src/tests to src/proto/tests.
Change-Id: I49e066fce90efbc18e16d68f94497b32ed5625c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8091
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
A BlobReader can be used to read a blob, which might consist out of
multiple chunks. Chunks are fetched from a ChunkService.
Change-Id: I1806225b0052adaa4a2320b79b744e554e524ee3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8088
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: tazjin <tazjin@tvl.su>
One normal chromium is enough. google-chrome is not needed because
Chromecasts usually don't work anymore anyways.
Change-Id: I48bed05c56bd6f450be393cf727930eaad5cd6b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8231
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
We missed renaming this as well while iterating over
https://cl.tvl.fyi/c/depot/+/7950.
Change-Id: I704d3b60bb3beb1a2148e27bdd4a49075a6649b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8230
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This just creates a text file (similar to the propagated deps file)
which lists the outpaths of all targets.
It's unclear why the previous fix didn't work, but it was done solely
based on docs not on reading code. In general we don't really need a
setup hook though and nobody remembers why that was the solution we
used anyways, so lets remove it.
Change-Id: I2a945925f4fa56b272aa44c29229b6a1aefa80e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8228
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
We've been seeing CI failures after a recent nixpkgs bump which only
occur on canon *after* a CL has been merged. In these CI failures, the
`ci.gcroot` attribute is built to "anchor" the latest canon build, but
fails because it tries to execute random non-executable files (e.g.
the sqlite database produced by //corp/russian/data-import).
From what I can tell, there has been a recent change in nixpkgs where
makeSetupHooks' arguments were restructured. What was previously
`deps` is now `propagatedBuildInputs`, and
`depsTargetTargetPropagated`.
The latter is supposed to be used for non-executable targets,
according to the docs.
This commit changes the entire set to use that flag, as we don't
actually want any executable setup hook - just a gcroot.
Change-Id: Ib258c4f3b09d746c3d000d1c7480d2b4101ab75e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8227
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This module contains the request/response types for generators
requesting actions from the VM.
For most of these, an async helper function is added that will be used
inside of generator functions to make use of these requests/responses
instead of constructing them directly.
Change-Id: I1e085f88adaf784a34867957a0e82532d3a83d7c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8148
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
As applies are thunked, there was no situation where OpCall could be
emitted. In practice, all calls were already tail calls.
Change-Id: Id0d441dcdd86f804d7cddd0cc14f589bbfc75e5b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8147
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This should avoid having two builds of Nix 2.3 in the closure of whitby
and sanduny.
Change-Id: Id4b8d34da73d3f579c97adcda44df26992290764
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8192
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
It's been a long time since we updated josh, almost 400 commits in
between. I read through the entire changelog, and here are relevant
josh commits from in between that might be interesting to us:
38eecee Fix optimisation bug for compose filter (#1159)
e1d10b6 Add :rev(...) filter
0f1a07b Initial implementation of refs locking (#929)
88cea2a Initial work on meta repo support
030ad93 Change magic refs to include "for"
28b1d75 Add split changes feature (#904)
1f908d7 Discover filters only on HEAD (#774)
a368d8f Make --require-auth only apply to push
8d80230 Add :linear filter (#741)
3460ec2 Implement redundant refs filtering (#700)
55b4e50 Implement stacked changes support (#699)
ea1f814 Handle @sha urls by creating magic ref (#690)
883a381 Run filter discovery only on changed refs (#685)
4bb004f Prepend refs/heads to base parameter as default (#664)
Of particular interest is a368d8f, which allows us to drop our
authentication patch and use the standard --require-auth flag again.
The default behaviour of dropping signatures on commits (which are
invalid after filtering) has also been changed in josh, now only
occuring when the `:unsign` filter is present. Since this breaks
commit hashes with our existing exported histories, we are opting to
set a `:unsign` filter prefix on all proxy requests to ensure that the
hashes stay consistent.
During this update we found a bug (josh#1155) which was fixed in the
commit that this CL moves josh to.
Change-Id: I3afac1619f3aa90313a0441da91f0e4a96fe0a3b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8186
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This adds a BlobService trait, and an implementation for it using sled,
and one using a HashMap.
Change-Id: Id6bc1b629195d0b26fc503bd7d2dc9e43c41c317
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8087
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This adds the simpler ChunkService trait, and an implementation for it
using sled, and one using a HashMap.
Change-Id: Icb0fdc41b37b44e9e9e4f548d0f4acae1d83b71e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8086
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This came up recently again, it makes sense to document the reasoning
behind the decision.
Change-Id: Ic51d5bc7998c70e8b070b6f42877d8e88613935b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8223
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Because they do not use it, and it can not be passed with the coming
generator refactoring.
Change-Id: I0d96f2357a7ee79cd8a0f401583d4286230d4a6b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8146
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Instead of using a suspended native thunk, calculate and optionally
insert the storeDir builtin when the VM is constructed.
We already have the IO handle available at this point and can just
check whether a storeDir is present, and insert its absolute value as
a builtin.
Change-Id: If966eee6ff26dc888b6e888e7c46170c0c346b05
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8145
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
