Commit graph

71 commits

Author SHA1 Message Date
Florian Klink
c7845f3c88 refactor(tvix/castore): move *Node and Directory to crate root
*Node and Directory are types of the tvix-castore model, not the tvix
DirectoryService model. A DirectoryService only happens to send
Directories.

Move types into individual files in a nodes/ subdirectory, as it's
gotten too cluttered in a single file, and (re-)export all types from
the crate root.

This has the effect that we now cannot poke at private fields directly
from other files inside `crate::directoryservice` (as it's not all in
the same file anymore), but that's a good thing, it now forces us to go
through the proper accessors.

For the same reasons, we currently also need to introduce the `rename`
functions on each *Node directly.

A followup is gonna move the names out of the individual enum kinds, so
we can better represent "unnamed nodes".

Change-Id: Icdb34dcfe454c41c94f2396e8e99973d27db8418
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12199
Reviewed-by: yuka <yuka@yuka.dev>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-08-13 18:39:49 +00:00
Yureka
3ca0b53840 refactor(tvix/castore): use Directory struct separate from proto one
This uses our own data type to deal with Directories in the castore model.

It makes some undesired states unrepresentable, removing the need for conversions and checking in various places:

 - In the protobuf, blake3 digests could have a wrong length, as proto doesn't know fixed-size fields. We now use `B3Digest`, which makes cloning cheaper, and removes the need to do size-checking everywhere.
 - In the protobuf, we had three different lists for `files`, `symlinks` and `directories`. This was mostly a protobuf size optimization, but made interacting with them a bit awkward. This has now been replaced with a list of enums, and convenience iterators to get various nodes, and add new ones.

Change-Id: I7b92691bb06d77ff3f58a5ccea94a22c16f84f04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12057
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-13 12:17:01 +00:00
Ilan Joselevich
f648f17ec3 feat(tvix): Jemalloc -> MiMalloc
Use the faster and newer MiMalloc memory allocator for all endpoints in
the workspace.

Change-Id: Ic60237284ed168e46ec6e8f28e2710bae4385c6f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12149
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
2024-08-08 08:06:23 +00:00
Yureka
67335c41b7 refactor(tvix): move service addrs into shared clap struct
Change-Id: I7cab29ecfa1823c2103b4c47b7d784bc31459d55
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12008
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
2024-07-22 13:36:08 +00:00
Florian Klink
ba38883a4f fix(tvix/nar-bridge): remove leftover debug statement
Change-Id: Ie6fb822e1a4b76f2f6aef15bbe846e0a89c644fb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11999
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
2024-07-21 11:26:13 +00:00
Florian Klink
98f59cb828 feat(tvix/nar-bridge): change default port
Having `tvix-store daemon` and `nar-bridge` listen on the same port by
default is silly.
Use the same port that nar-bridge-go was using previously.

Change-Id: Id374a12e52ba9dac4318e43d0ca0853866eadabc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11998
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-07-21 11:26:13 +00:00
Yureka
ca8e2b9fbf chore(tvix): upgrade to tonic 0.12 / hyper 1.0
Change-Id: Idd8ce48869ddd869d51a10959b920f1290a8a9b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11991
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-21 11:01:38 +00:00
Simon Hauser
1515a970be feat(tvix/tracing): http propagation for axum
It introduces a new accept_trace function for axum0.7 which can be used
to accept a header trace from a received request. This function can be
used for tonic 0.12 once that version is released, and the specific
`accept_trace` function within `tvix_tracing::propagate::tonic` can then
be removed.

This also integrates http propagation into the nar_bridge crate.

Change-Id: I46dcc797d494bb3977c2633753e7060d88d29129
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11925
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Reviewed-by: Simon Hauser <simon.hauser@helsinki-systems.de>
Reviewed-by: flokli <flokli@flokli.de>
2024-07-21 05:45:19 +00:00
Florian Klink
d136f2c881 fix(tvix/nar-bridge): fix root node decoding and validation
This got broken while moving things around. We need to parse the
b64-decoded bytes.

Since we're now validating the root node, we also need to rename the
root node to get past the node name validation.

There probably should be some tests for this.

Co-Authored-By: sinavir@sinavir.fr
Change-Id: I8f24a4a0ac107b1ea5b94c0e0ed872a34eb7b587
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11996
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
2024-07-21 00:06:23 +00:00
Yureka
8b77c7fcd7 refactor(tvix/store): use composition in tvix_store crate
Change-Id: Ie6290b296baba2b987f1a61c9bb4c78549ac11f1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11983
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
2024-07-20 19:37:27 +00:00
Florian Klink
6180a7cecf feat(tvix/nar-bridge): implement PUT $outhash.narinfo
This adds support to upload NARInfo files. We lookup the root node from
the LRU cache, rename it appropriately and then put it into the
PathInfoService.

Change-Id: I5479032b51cd855363bc016dee63cf84b3304a36
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11988
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-07-20 17:38:58 +00:00
Florian Klink
5bd48de418 feat(tvix/nar-bridge): add 404 handler for GET/HEAD /nar/…
We currently send 405, and that confuses `nix copy`.
Send a 404 for now, and add a futurework, as we can actually at least do
something more meaningful in case we still have that nar hash in our
LRU, which would avoid some unnecessary uploads in some cases.

Change-Id: If625e9bd0fd6506cb73b88962d889aa08315fcea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11987
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-07-20 17:38:58 +00:00
Florian Klink
5d906054da feat(tvix/nar-bridge): support uploading NAR files
This ingests NAR files into the {Blob,Directory}Service, which are
already part of the AppState.

As we then need to correlate the root node to the uploaded PathInfo, we
need to keep a (short-lived) lookup table from NARHash to root node
around. We insert it into a `LruCache` after the NAR is uploaded, and
use `peek()` to do the lookup, which doesn't update the LRU list.

Change-Id: I48a4c6246bacf76559c5a4ccad2a0bc25c1b7900
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11986
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-07-20 17:23:18 +00:00
Florian Klink
861cc1f341 feat(tvix/nar-bridge): init
This adds an implementation of nar-bridge in Rust.
Currently, only the GET parts are implemented.

Contrary to the Go variant, this doesn't try to keep a mapping from nar
hashes to root node in memory, it simply encodes the root node itself
(stripped by its basename) into the URL.

This pulls in a more recent version of axum than what we use in
tonic, causing two versions of http and hyper, however dealing with
`Body::from_stream` in axum 0.6 is much more annoying, and
https://github.com/hyperium/tonic/pull/1740 suggests this will be fixed
soon.

Change-Id: Ia4c2dbda7cd3fdbe47a75f3e33544d19eac6e44e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11898
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
2024-07-20 16:53:23 +00:00
Florian Klink
1392913e98 chore(tvix/nar-bridge): move to nar-bridge-go
Make some space for the rust implementation.

Change-Id: I924dc1657be10abe5a11951c3b9de50bae06db19
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11662
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: yuka <yuka@yuka.dev>
2024-05-15 21:31:36 +00:00
Edwin Mackenzie-Owen
465370c11f feat(tvix/nar-bridge): support HTTP HEAD for narinfo
Change-Id: I4ec8e9108f52d0a84bc1af01a27bd51de4ab493f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11456
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-01 13:11:51 +00:00
Florian Klink
50c81d7838 feat(tvix/nar-bridge): support listening on unix sockets
This simply checks for the address to contain slashes, and if so, opens
a unix socket, rather than a tcp one. We'll use this in //tvix/boot
tests to simplify waiting for nar-bridge to be up.

Change-Id: I7184f548d57142b1c5f698a1f0c30343489373a5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11184
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-18 14:57:30 +00:00
Connor Brewster
85421b7f97 feat(tvix/store/nar-info): Support async blob upload
Right now all blob uploads are performed synchronously, this means if a
NAR contains many small files, the import time is dominated by round
trip time to the blob service. For small files, we can buffer them in
memory and upload them asynchronously to the blob service. Before
returning we make sure to join all the uploads to make sure they
complete successfully before responding OK.

This reduces time to import a bash-interactive closure on my machine
from 1m19s to 7s.

Change-Id: Ica3695c159e6c8ad8769281ac20d037e3143e856
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10679
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Connor Brewster <cbrewster@hey.com>
2024-01-23 17:01:16 +00:00
Connor Brewster
d056329412 feat(tvix/store/nar-bridge): Setup OpenTelemetry
Sets up OpenTelemetry integration for nar-bridge. Right now it will
export spans for HTTP server requests and all gRPC client requests.
Having the spans available will make performance work significantly
easier as it provides a high level overview of where time is being
spent.

In the future we can add application-specifc metrics and
integrate logrus.

Change-Id: Ie3860675d7ffc626a95673ba062c3c798d8bb2a7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10678
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
2024-01-23 15:50:53 +00:00
Adam Joseph
2132bbcb47 fix(tvix/nar-bridge): fix FOD hash
The golang mothership seems to be monkeying with hashes again.

Change-Id: I7430b4cde84fa51be2b572fba02e3567864bb87a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10209
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: flokli <flokli@flokli.de>
2023-12-09 09:24:53 +00:00
Florian Klink
dfb48dcade fix(tvix/nar-bridge): drop pathinfoservice
This now exists in tvix-store directly, as NixHTTPPathInfoService, and
contrary to this version, also validates signatures.

Change-Id: Ib6ca161e40d627b7d9741839fc849f2392f422da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10155
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-11-28 16:08:59 +00:00
Florian Klink
c86de1b2c4 chore(tvix/nar-bridge): bump castore-go
Change-Id: I81814ae326fb90ab50201cd06e3d9c0a3b15aec5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9958
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-06 08:34:14 +00:00
Florian Klink
718af22dbb chore(tvix/nar-bridge): update to latest go mod renames
- code.tvl.fyi/tvix/castore/protos -> code.tvl.fyi/tvix/castore-go
 - code.tvl.fyi/tvix/store/protos -> code.tvl.fyi/tvix/store-go

 See cl/9791, cl/9792 for context.

Change-Id: I44614c6ed40b9f52d9dcdea8e61fe2c3c830ce78
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9793
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-17 20:46:38 +00:00
Florian Klink
1c5bf366be refactor(tvix/nar-bridge): panic on inval PathInfo.NarInfo.NarSha256
Bump code.tvl.fyi/tvix/store/protos past cl/9649, where Validate()
already ensures the NarSha256 has the correct size.

Change-Id: I774668822f4d9dbd4dea47dde6e4745dc95e8e7f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9665
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-10-12 06:24:07 +00:00
Florian Klink
6dab608198 docs(tvix/nar-bridge): remove redundant comments
These really only describe what the code is doing, remove.

Pointed out in
https://cl.tvl.fyi/c/depot/+/9652/comment/987b3b7f_c7b5e47b/

Change-Id: I512a274295244f96744a489871861fed9523bb37
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9654
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Tested-by: BuildkiteCI
2023-10-11 15:22:22 +00:00
Florian Klink
155ab77cff refactor(tvix/nar-bridge): use mh.SHA2_256
As correctly mentioned in
https://cl.tvl.fyi/c/depot/+/9652/comment/03b9b96e_bbb337fd/,
we shouldn't be using these magic constants, but pull them from where
they're defined.

This already is a dependency of go-nix, and pkg/pathinfosvc/server.go,
so no changes in go.mod.

Change-Id: I0cc41ce040fcbddf4b6171417bc9b0de55af4991
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9653
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
2023-10-11 15:22:22 +00:00
Florian Klink
673f5febbb feat(tvix/nar-bridge): stop parsing nixbase32 manually, validate
We have nixhash.FromHashTypeAndDigest now.

Also, run Validate() on the PathInfo received from the remote
PathInfoService.

Change-Id: I14db0d9356c539c084afc9dd712314b56da2587e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9652
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
2023-10-11 11:41:50 +00:00
Florian Klink
98c17147c6 refactor(tvix/nar-bridge): have Export return root node
… and nar size / sha256 digest.

Instead of producing sparse PathInfo messages when NARs are sent to
nar-bridge, the nar-bridge http server now keeps a lookup table
(narsha256) -> (rootNode, narSize)

This removes a whole bunch of noise, because we don't need to keep
sparse fields around.

A convenience function
`GenPathInfo(rootNode *castorev1pb.Node, narInfo *narinfo.NarInfo)` is
added, which is used to produce PathInfo messages, either when receiving
a NAR file over http and uploading it to a remote PathInfoService, or to
synthesize the PathInfoMessage to return to the client, if nar-bridge is
acting as a PathInfoService for a remove Nix HTTP Binary cache.

Change-Id: Ibba1ab6238a050816c4fab29cb21ae88877d8613
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9651
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
2023-10-11 11:41:50 +00:00
Florian Klink
ceb1674e9f refactor(tvix/nar-bridge): pass root node to storev1pb.Export
Bumps the go module past cl/9604 and update the consumer side.

Change-Id: Id44245017f1dc2f8aac28051cdbb45b83bdc5be3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9650
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Tested-by: BuildkiteCI
2023-10-11 11:41:50 +00:00
Florian Klink
fe963ae0a3 chore(tvix/nar-bridge): move to Export from storev1pb
This removes the Export method in nar-bridge, and updates all users to
the version now in storev1pb.

It moves the roundtrip test to the importer crate, and some of the
utility functions into a separate util_test.go file.

Change-Id: I81d9e0b35dfd78ef1042bed307281eecd2aaa2a8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9603
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Tested-by: BuildkiteCI
2023-10-10 19:08:29 +00:00
Florian Klink
e6ba84ea50 feat(tvix/store/protos): add Export
Export will traverse a given PathInfo structure, and write the contents
in NAR format to the passed Writer.
It uses directoryLookupFn and blobLookupFn to resolve references.

This is being moved over from nar-bridge. We need to keep the code there
around until we can bump go.mod to storev1 with this merged, but the
tests can already be moved entirely.

Change-Id: Ie0de3077b09344cafa00ff1e2ddb8b52e9e631bc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9602
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Autosubmit: flokli <flokli@flokli.de>
2023-10-10 18:55:49 +00:00
Florian Klink
6fe34b7ba0 refactor(tvix/nar-bridge): use castorev1pb.RenamedNode
We can use the helper to rename the node.

Change-Id: Id8defea7e5ebbd43d7b7a9b2992c62084e1828ec
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9601
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Tested-by: BuildkiteCI
2023-10-10 18:02:38 +00:00
Florian Klink
2699b61052 chore(tvix/nar-bridge): bump google.golang.org/protobuf
… to 1.28.1 as well.

Change-Id: Ieaaf9363a8dd63c2437c34d78a0ab369e7492f19
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9600
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-10 18:02:38 +00:00
Florian Klink
78abeaa920 refactor(tvix/nar-bridge): use storepath for store paths
Remove the handwritten parsing and formatting of store paths.

Change-Id: Ia4ba486b4363c33b98937bcbf6f5f7bcda289b82
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9588
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-09 22:05:02 +00:00
Florian Klink
28cd4b1a2f feat(tvix/nar-bridge): add ToNixNarInfo()
Convenience function, moves all code converting from a PathInfo struct
to to go-nix's NarInfo.

Change-Id: Idf0dcc38675674563f2dfd3286a4a55fa2a24a82
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9593
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
2023-10-09 22:05:02 +00:00
Florian Klink
92481825b3 test(tvix/store/protos): add go tests for Validate()
We already have validation tests for Rust, let's add the missing ones
for golang too.

Change-Id: Iaf3a3e1ee72d5647da3f2aa977d6e0d0379b2ce5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9595
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-09 22:01:59 +00:00
Florian Klink
87396d29fd chore(tvix/nar-bridge): bump go-nix
Change-Id: I72d139c47ede56d63d896bb5d5ed385e22a32264
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9592
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-09 19:08:28 +00:00
Florian Klink
9c0d66197b feat(tvix/nar-bridge): do pathInfo.Validate() as additional check
This should make it quite quick to spot writing code breaking some of
the assumptions we have on PathInfo messages ourselves.

Change-Id: I480caaec41f8ea5246c3c3081460c7ad12e78569
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9554
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-07 02:06:55 +00:00
Florian Klink
7e9da4ddef fix(tvix/nar-bridge): properly check error during HTTP shutdown
We didn't bind err := here.

Change-Id: I779d7ba5660ba4b126a4a0db7fe044bf94425855
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9547
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-05 19:15:00 +00:00
Florian Klink
a2c81de4cb docs(tvix/nar-bridge): fix comment
We run narInfo.Check to ensure this parses to a StorePath, not
nixpath.Check.

Change-Id: Id91183128df74a60d98fa2a31174cd879194c34d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9550
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-05 18:51:53 +00:00
Florian Klink
61e5a63fe1 chore(tvix/nar-bridge): bump go-nix dependency
nixpath.FromString -> storepath.FromAbsolutePath.

See https://github.com/nix-community/go-nix/pull/113 for details.

Closes: https://b.tvl.fyi/issues/314
Change-Id: I25277fb6006cbbb2a323ffb5809a1be500822a97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9551
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 18:47:52 +00:00
Florian Klink
cb807ad79b refactor(tvix/nar-bridge): do root directory dgst check in uploader
This check makes more sense there, and gives stronger semantics - Done()
only succeeds if the other side successfully received everything, *and*
came up with the same hashes as we did.

Change-Id: I20b706961053fd00d22cc70e1c8cc859705587e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9542
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 15:02:12 +00:00
Florian Klink
c04041a001 feat(tvix/nar-bridge): add nar-bridge-pathinfo command
This adds an additional nar-bridge-pathinfo command.

It exposes a PathInfoService for a HTTP Binary Cache, ingesting data
into a BlobService/DirectoryService as it goes through the NAR file.

It does this whenever it receives a Get request for a specific output
path, and waits returning with the PathInfo response until it ingested
the data.
It does not do any sort of caching - this means it re-downloads NAR
files again whenever the PathInfo is requested again, so you most likely
do not want to use this currently.

It's one building component as soon as we have store composition (which
we currently don't, so don't use this).

It can be used as an alternative mechanism to ingest data (Blobs and
Directories) of a given store path from a binary cache into tvix-store.

```
❯ nix-build -A third_party.nixpkgs.hello
/nix/store/mdi7lvrn2mx7rfzv3fdq3v5yw8swiks6-hello-2.12.1

❯ nix hash to-sri --type sha1 mdi7lvrn2mx7rfzv3fdq3v5yw8swiks6
sha1-Rs/INeK+7IGbG/u7fHoVNm96Yqs=

❯ out=$(mg build //tvix/nar-bridge)
$out/bin/nar-bridge-pathinfo --log-level debug &
INFO[0000] Starting nar-bridge-pathinfosvc at [::]:8001

❯ mg run //tvix:store -- daemon &
[mg] building target //tvix:store
[mg] running target //tvix:store
  2023-10-03T16:21:57.433739Z  INFO tvix_store: tvix-store listening on [::]:8000
    at src/bin/tvix-store.rs:229

❯ evans --host localhost --port 8001 -r repl
[…]
tvix.store.v1.PathInfoService@localhost:8001> call Get
✔ by_output_hash
by_output_hash (TYPE_BYTES) => Rs/INeK+7IGbG/u7fHoVNm96Yqs=
{
  "narinfo": {
    "narSha256": "sXrPtjqhSoc2u0YfM1HVZThknkSYuRuHdtKCB6wkDFo=",
    "narSize": "226552",
    "referenceNames": [
      "aw2fw9ag10wr9pf0qk4nk5sxi0q0bn56-glibc-2.37-8",
      "mdi7lvrn2mx7rfzv3fdq3v5yw8swiks6-hello-2.12.1"
    ],
    "signatures": [
      {
        "data": "7guDbfaF2Q29HY0c5axhtuacfxN6uxuEqeUfncDiSvMSAWvfHVMppB89ILqV8FE58pEQ04tSbMnRhR3FGPV0AA==",
        "name": "cache.nixos.org-1"
      }
    ]
  },
  "node": {
    "directory": {
      "digest": "xvo6BYbYaDw76IibLu5sr+VZoj9iM0ET2RUuYSYLwKE=",
      "name": "bWRpN2x2cm4ybXg3cmZ6djNmZHEzdjV5dzhzd2lrczYtaGVsbG8tMi4xMi4x",
      "size": 141
    }
  },
  "references": [
    "ptgFMIhdl2nJxMDdlDkITyXuBFc=",
    "Rs/INeK+7IGbG/u7fHoVNm96Yqs="
  ]
}
```

Change-Id: I50167d0ac081c91adf5cf2733bbc4dc0993bd46e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9539
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: Brian Olsen <me@griff.name>
2023-10-05 15:02:12 +00:00
Florian Klink
8239f32b62 refactor(tvix/nar-bridge): mv nar-bridge{,-http}
Rename the nar-bridge CLI to nar-bridge-http, because it's the one
spinning up an http server.

Change-Id: I0fb75c50e4299272a128dd5ecaa4be8f06fa3dbe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9538
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-05 06:17:10 +00:00
Florian Klink
c18ff1a270 refactor(tvix/nar-bridge): deduplicate NAR HEAD and GET
Use a genNarHandler() function accepting a boolean to construct the
HTTP handler.

Change-Id: I17c054826d91a9dbed8b1f53945a51f27fa60ace
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9537
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-10-05 06:17:10 +00:00
Florian Klink
45511004df docs(tvix/nar-bridge): fix typo in comment
Change-Id: I74f6892e089ed306ef93e78a39a25cf3738bb09f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9536
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-05 06:17:10 +00:00
Florian Klink
28d1b9c01d refactor(tvix/nar-bridge): move pkg/server to pkg/http
This is only dealing with the HTTP interface.

Change-Id: I011b624fd9f11ea96231b92fea1166c118a219f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9535
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 06:17:10 +00:00
Florian Klink
0353108e99 refactor(tvix/nar-bridge): move DirectoriesUploader to pkg/importer
This is useful outside a HTTP server scenario.

Change-Id: I0042a6e773906a15a254d850520e6f841035bf20
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9533
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 06:17:10 +00:00
Florian Klink
b78b8d8338 refactor(tvix/nar-bridge): move blob cb function to pkg/importer
This is useful outside a HTTP server scenario.

Change-Id: If35f1ab245855378fd01f16ad7b5774d0cf590ba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9532
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-10-05 06:17:10 +00:00
Florian Klink
259269482c docs(tvix/nar-bridge): add more comments for DirectoriesUploader.
Change-Id: I6684186f3120ea42036239baa82e0ca37d3e0e6c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9531
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-05 06:17:10 +00:00