Commit graph

29 commits

Author SHA1 Message Date
Florian Klink
9c0d66197b feat(tvix/nar-bridge): do pathInfo.Validate() as additional check
This should make it quite quick to spot writing code breaking some of
the assumptions we have on PathInfo messages ourselves.

Change-Id: I480caaec41f8ea5246c3c3081460c7ad12e78569
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9554
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-07 02:06:55 +00:00
Florian Klink
a2c81de4cb docs(tvix/nar-bridge): fix comment
We run narInfo.Check to ensure this parses to a StorePath, not
nixpath.Check.

Change-Id: Id91183128df74a60d98fa2a31174cd879194c34d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9550
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-05 18:51:53 +00:00
Florian Klink
61e5a63fe1 chore(tvix/nar-bridge): bump go-nix dependency
nixpath.FromString -> storepath.FromAbsolutePath.

See https://github.com/nix-community/go-nix/pull/113 for details.

Closes: https://b.tvl.fyi/issues/314
Change-Id: I25277fb6006cbbb2a323ffb5809a1be500822a97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9551
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 18:47:52 +00:00
Florian Klink
cb807ad79b refactor(tvix/nar-bridge): do root directory dgst check in uploader
This check makes more sense there, and gives stronger semantics - Done()
only succeeds if the other side successfully received everything, *and*
came up with the same hashes as we did.

Change-Id: I20b706961053fd00d22cc70e1c8cc859705587e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9542
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 15:02:12 +00:00
Florian Klink
c04041a001 feat(tvix/nar-bridge): add nar-bridge-pathinfo command
This adds an additional nar-bridge-pathinfo command.

It exposes a PathInfoService for a HTTP Binary Cache, ingesting data
into a BlobService/DirectoryService as it goes through the NAR file.

It does this whenever it receives a Get request for a specific output
path, and waits returning with the PathInfo response until it ingested
the data.
It does not do any sort of caching - this means it re-downloads NAR
files again whenever the PathInfo is requested again, so you most likely
do not want to use this currently.

It's one building component as soon as we have store composition (which
we currently don't, so don't use this).

It can be used as an alternative mechanism to ingest data (Blobs and
Directories) of a given store path from a binary cache into tvix-store.

```
❯ nix-build -A third_party.nixpkgs.hello
/nix/store/mdi7lvrn2mx7rfzv3fdq3v5yw8swiks6-hello-2.12.1

❯ nix hash to-sri --type sha1 mdi7lvrn2mx7rfzv3fdq3v5yw8swiks6
sha1-Rs/INeK+7IGbG/u7fHoVNm96Yqs=

❯ out=$(mg build //tvix/nar-bridge)
$out/bin/nar-bridge-pathinfo --log-level debug &
INFO[0000] Starting nar-bridge-pathinfosvc at [::]:8001

❯ mg run //tvix:store -- daemon &
[mg] building target //tvix:store
[mg] running target //tvix:store
  2023-10-03T16:21:57.433739Z  INFO tvix_store: tvix-store listening on [::]:8000
    at src/bin/tvix-store.rs:229

❯ evans --host localhost --port 8001 -r repl
[…]
tvix.store.v1.PathInfoService@localhost:8001> call Get
✔ by_output_hash
by_output_hash (TYPE_BYTES) => Rs/INeK+7IGbG/u7fHoVNm96Yqs=
{
  "narinfo": {
    "narSha256": "sXrPtjqhSoc2u0YfM1HVZThknkSYuRuHdtKCB6wkDFo=",
    "narSize": "226552",
    "referenceNames": [
      "aw2fw9ag10wr9pf0qk4nk5sxi0q0bn56-glibc-2.37-8",
      "mdi7lvrn2mx7rfzv3fdq3v5yw8swiks6-hello-2.12.1"
    ],
    "signatures": [
      {
        "data": "7guDbfaF2Q29HY0c5axhtuacfxN6uxuEqeUfncDiSvMSAWvfHVMppB89ILqV8FE58pEQ04tSbMnRhR3FGPV0AA==",
        "name": "cache.nixos.org-1"
      }
    ]
  },
  "node": {
    "directory": {
      "digest": "xvo6BYbYaDw76IibLu5sr+VZoj9iM0ET2RUuYSYLwKE=",
      "name": "bWRpN2x2cm4ybXg3cmZ6djNmZHEzdjV5dzhzd2lrczYtaGVsbG8tMi4xMi4x",
      "size": 141
    }
  },
  "references": [
    "ptgFMIhdl2nJxMDdlDkITyXuBFc=",
    "Rs/INeK+7IGbG/u7fHoVNm96Yqs="
  ]
}
```

Change-Id: I50167d0ac081c91adf5cf2733bbc4dc0993bd46e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9539
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: Brian Olsen <me@griff.name>
2023-10-05 15:02:12 +00:00
Florian Klink
c18ff1a270 refactor(tvix/nar-bridge): deduplicate NAR HEAD and GET
Use a genNarHandler() function accepting a boolean to construct the
HTTP handler.

Change-Id: I17c054826d91a9dbed8b1f53945a51f27fa60ace
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9537
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-10-05 06:17:10 +00:00
Florian Klink
45511004df docs(tvix/nar-bridge): fix typo in comment
Change-Id: I74f6892e089ed306ef93e78a39a25cf3738bb09f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9536
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-05 06:17:10 +00:00
Florian Klink
28d1b9c01d refactor(tvix/nar-bridge): move pkg/server to pkg/http
This is only dealing with the HTTP interface.

Change-Id: I011b624fd9f11ea96231b92fea1166c118a219f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9535
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 06:17:10 +00:00
Florian Klink
0353108e99 refactor(tvix/nar-bridge): move DirectoriesUploader to pkg/importer
This is useful outside a HTTP server scenario.

Change-Id: I0042a6e773906a15a254d850520e6f841035bf20
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9533
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 06:17:10 +00:00
Florian Klink
b78b8d8338 refactor(tvix/nar-bridge): move blob cb function to pkg/importer
This is useful outside a HTTP server scenario.

Change-Id: If35f1ab245855378fd01f16ad7b5774d0cf590ba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9532
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-10-05 06:17:10 +00:00
Florian Klink
259269482c docs(tvix/nar-bridge): add more comments for DirectoriesUploader.
Change-Id: I6684186f3120ea42036239baa82e0ca37d3e0e6c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9531
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-05 06:17:10 +00:00
Florian Klink
d3c44ab03d refactor(tvix/nar-bridge): rename writer package to exporter
Change-Id: I7cf6a3ed0dca4c450fadd9228dc0154ad6254f51
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9530
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-10-05 06:17:10 +00:00
Florian Klink
49b427d773 refactor(tvix/nar-bridge): combine writers/readers
We can drop most of Hasher if we use a MultiWriter writing to the hash
function and a minimal CountingWriter.

This should make things a bit more understandable.

Change-Id: I37ee72d9a5c73f253aecc1ad761cb723389b89fc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9529
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-05 06:17:10 +00:00
Florian Klink
f92b0ef933 refactor(tvix/nar-bridge): let callbaks return calculated digests
This aligns behaviour more with how it should be - it's the
responsibility of the callback functions to return digests of the things
they consume(d). It allows further cleaning up the hasher struct.

Change-Id: I9cbfc87e6abd4ff17fadf39eb6563ec3cb7fcc6f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9528
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 06:17:10 +00:00
Florian Klink
b1ff1267be refactor(tvix/nar-bridge): drop reader package
Make the import function usable on any reader.

Change-Id: I84d2004cb73cdd7a11fe8efb0f2efb6335d5e6b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9527
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-05 06:11:27 +00:00
Florian Klink
6e9a5dcd59 refactor(tvix/nar-bridge): rename item to stackItem
Change-Id: I76c8db3b04d9f97e51e91f7d36a4dd8c6500d1a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9526
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 06:11:23 +00:00
Florian Klink
174485038c refactor(tvix/nar-bridge): inline assemblePathInfo
This is only called once.

Change-Id: I342443b8d04050929733fc84d5f36cd64060afe3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9525
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-05 06:11:23 +00:00
Florian Klink
732dc68727 refactor(tvix/nar-bridge): update to new separated protos
Change-Id: I5eaadc837a4d3a7f635574437127a22de88f556b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9407
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-09-22 17:41:32 +00:00
Florian Klink
6c586bc2a7 feat(tvix/nar-bridge): graceful shutdown
This gives existing clients 30s to finish their requests after receiving
an interrupt.

Change-Id: Ia9b0e662fd1ffbbb6c2d03f3dd6548b13cf3d241
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9365
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-18 14:34:12 +00:00
Florian Klink
02aed32bf2 refactor(tvix/nar-bridge): rename contentReader to blobReader
More aligned with how it's called in other places

Change-Id: I759ac7ca3b5b69c1101d2d51a569d76c183a6330
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9362
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-09-18 14:02:28 +00:00
Florian Klink
a9c47ffc50 refactor(tvix/nar-bridge): don't buffer blob in memory
Create a pipe, pass the read end, and have a goroutine write to the
write end.

Change-Id: I301c273355705e60113b018e7e84b76972200e8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9361
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-18 14:02:28 +00:00
Florian Klink
07af692ecb refactor(tvix/nar-bridge): simplify CLI interface
Only keep the `serve` subcommand, and make it appear at the root.
Introduce a --log-level argument, and be a bit less noisy in normal
operation.

Change-Id: I86b8abde1869a5c0c947508bcc29f845222aac09
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9360
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-09-18 14:02:28 +00:00
Florian Klink
dd7cc6ed68 fix(tvix/nar-bridge): don't log error on simple 404s
`nix copy` checks if NARs and NARInfo files are present, before
uploading them. That's not an error, but normal behaviour, so no need to
log with level info for these cases.

We only want to log if the error is not a 404, and log with Warn level.

Change-Id: I762de3b862d070a0f18bc62e324e94ca5c7c3693
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9359
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-09-18 14:02:28 +00:00
Florian Klink
749ab67216 feat(tvix/nar-bridge): buffer request body a bit more
Let's make sure we don't end up blocking a client too much when
inserting very small blobs.

Change-Id: I640dda92efae538c70d32a40e6e85a23e9749e20
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9358
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-18 14:02:28 +00:00
Florian Klink
0c031461c3 fix(tvix/nar-bridge): chunk blobs
Instead of creating one big BlobChunk containing all data, and creating
way too large proto messages, chunk blobs up to a reasonable (1MiB)
chunk size, and send them to the server like that.

Change-Id: Ia45a53956a6d7c0599cc59ac516ba37e9fb1b30e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9357
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-09-18 14:02:28 +00:00
Connor Brewster
4ac6423b26 refactor(tvix/nar-bridge): Clean up directory popping loop
This change got lost in the rebases in cl/9348. There's unnecessary
`break`/`continues` that can be replaced by moving the conditional into
the for loop condition.

Change-Id: I559e21087630b05e483f768ab59f8067961a2eae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9352
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-09-17 13:55:22 +00:00
Connor Brewster
84aa07a736 fix(tvix/nar-bridge): Fix directory stack tracking
Previously, nar-bridge, had a couple of bugs with tracking the current
directory when traversing a NAR file.

The included test case looks like:
```
/ (dir)
/test (dir)
/test/tested (file)
/tested (file)
```

Previously, we would do a string prefix match between the current node
and the top of the directory stack to determine if the node is in the
directory. In this case `/test` is a substring of `/tested`; however,
`/tested` is not in the `/test` directory. The fix is to append a `/` to
the directory name when doing the prefix match, so `/test/` is not a
prefix of `/tested`.

Additionally, when popping the stack, we need to continuously pop the
stack until the new node is in the directory at the top of the stack
(stopping before we pop the root directory)

Example:
```
/ (dir)
/a (dir)
/a/b (dir)
/a/b/c (file)
/z (file)
```

Previously, `z` would end up in directory `/a` because we only the pop
the stack once.

The included test case requires both of these issues to be fixed for it
to pass, so I think it is sufficient.

Change-Id: I22f601babf04d39d85535ba7ad585d3970757211
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9348
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Connor Brewster <cbrewster@hey.com>
2023-09-17 13:34:00 +00:00
Florian Klink
5841047c38 feat(tvix/nar-bridge): increase timeouts
In some cases, Nix is not able to stream the NAR file fast enough. Bump the
timeouts for now. We might want to get a better understanding in what's
happening here long-term, and/or make the timeouts configurable.

Change-Id: Ieaa9c8f04bc73c6ce0679a058d07eaf87126634e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9340
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-09-17 13:24:53 +00:00
Florian Klink
0ecd10bf30 feat(tvix/nar-bridge): init
This provides a Nix HTTP Binary Cache interface in front of a tvix-store
that's reachable via gRPC.

TODOs:

 - remove import command, move serve up to toplevel. We have nix-copy-
   closure and tvix-store commands.
 - loop into CI. We should be able to fetch the protos as a third-party
   dependency.
 - Check if we can test nar-bridge slightly easier in an integration
   test.
 - Ensure we support connecting to unix sockets and grpc+http at least,
   using the same syntax as tvix-store.
 - Don't buffer the entire blob when rendering NAR

Co-Authored-By: Connor Brewster <cbrewster@hey.com>
Co-Authored-By: Márton Boros <martonboros@gmail.com>
Co-Authored-By: Vo Minh Thu <noteed@gmail.com>

Change-Id: I6064474e49dfe78cea67676957462d9f28658d4a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9339
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-09-17 13:24:53 +00:00