Commit graph

18878 commits

Author SHA1 Message Date
Vincent Ambo
f4787355a4 chore(ops/modules): enable passwordless sudo in users module
Change-Id: I8522a106bbadacf1b5720b4cd1102052aa360ff0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9575
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-08 19:45:25 +00:00
Vincent Ambo
5f32f7610a chore(users): remove inactive users
Change-Id: I3cfb425e4dac0a467e3917df996e9800a3ebe875
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9576
Reviewed-by: isomer <isomer@tvl.fyi>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-10-08 18:18:08 +00:00
Vincent Ambo
8b2884de09 chore(ops): move yandex-base-image to //ops and bake in keys
Change-Id: I607af1fc41c1f6ee24eed1386a23663346c3acc2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9574
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-08 18:13:49 +00:00
Vincent Ambo
713a70d154 fix(ops/modules): remove cloud-init from yandex-cloud module
cloud-init stopped working for unknown reasons, enabling it will break
DHCP and SSH, and make the image inaccessible.

This means that access needs to be provided by baking keys into the
image instead.

Change-Id: Ib8d32a02d0a8ea61d75921f147349d73a27ef751
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9572
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-08 18:13:49 +00:00
Florian Klink
1629f3064f docs(tvix/castore): remove TODO
This probably was about passing around directory_putter at some point,
which we do, so whatever this meant, it's not actionable anymore.

Change-Id: I1b4e0cdd2119bf2b2a9cf06d186a3b476b0ff367
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9573
Reviewed-by: Linus Heckemann <git@sphalerite.org>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-08 16:00:44 +00:00
Florian Klink
b172c804b0 refactor(tvix/castore): use DuplexStream instead of unix socket
We can use DuplexStream to create to bidirectional pairs, which avoids
manually waiting for unix sockets to pop up and connect, and creating
temporary directoires to create the unix sockets in.

Turns out, we also don't actually need to spawn the server in a separate
runtime, it works just fine these days. This might be due to all the
sync barriers in between being gone.

Change-Id: I6b79823bc6209cbcb343b7a498c64a2ba6e0aee7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9562
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-10-08 11:25:23 +00:00
Florian Klink
d1adefc9f9 feat(third_party/overlays): support unix domain sockets in evans
This applies a patch adding support for unix domain sockets to evans,
which was also sent upstream at
https://github.com/ktr0731/evans/pull/680.

Change-Id: I6e483ace45428a7526a200f885aa92c2de88b650
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9563
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-10-08 11:11:52 +00:00
Florian Klink
09328c79d5 feat(tvix/store/listener): implement connect_info()
For Unix and TCP streams, we can actually give more info than just ().

Change-Id: I80ca261b9e7a880389ec2029ea5893180314f0b2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9561
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-08 07:44:13 +00:00
Florian Klink
3b620e6d0c refactor(tvix/castore): move gRPC directorysvc tests off tonic_mock
The tests were using the gRPC Service/Server interface, not the gRPC
clients.

Use our new gen_directorysvc_grpc_client() function to spin up a server
in a separate thread, and test via that client, rather than *just* the
server implementation.

This is more correct, because we're now really exercising the gRPC
stack, and some of the stream implementations are also only implemented
on the client.

Change-Id: I99434a2399856a44c9e6988d0b720f7a429d0ad1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9560
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-08 07:44:13 +00:00
Florian Klink
a77914db73 refactor(tvix/castore/directorysvc): factor out gRPC client gen
Move this code into a helper function, which we'll use in other places
in a bit.

Change-Id: Icae6f6dd2d4b2fa86fd2b836ddd7a4ca0e0354e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9559
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-08 07:30:34 +00:00
Florian Klink
31f28b6105 chore(tvix/store): drop tonic-mock from dependencies
It's not used in here.

Change-Id: If1dbf360ec37b649c27d3a9bf11f7bf0ca947631
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9558
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-10-08 07:17:54 +00:00
Florian Klink
8b8a344647 fix(view/tvix): remove other default.nix files
These only work in depot usecases anyways.

Change-Id: I463ffb4e3ac2d3a20db3cace749e357c893e7188
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9557
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-07 05:47:32 +00:00
Florian Klink
03fec0b393 refactor(tvix,views/tvix): move shell into separate file
So far, we provided a custom `default.nix` in the root of the tvix josh
workspace, which re-defined the shell attribute from `tvix/default.nix`.
Some of the recent fixes, e.g. the MacOS-specific additions to the list
of dependencies however didn't get ported over to this file, and in
general, it's quite annoying to have two different places for these
things.

Initially I explored the idea of moving this default.nix file to a
default-depot.nix file in the josh worktree only, and then "polyfill"
some of the dependencies, or set up readTree in the josh workspace too,
but it turned out to pull in too many dependencies to be worth
the effort (nix.sparseTree, tools.depotfmt, crate2nix overlay,
third_party.gitignoreSource).

I now took a different approach - moving the definition of the `shell`
attribute from `tvix/default.nix` to its own `shell.nix` file, which is
imported from `tvix/default.nix` in regular depot usecases.

Josh workspace consumers only see the `shell.nix`, which can be used
in a self-contained fashion, the other `default.nix` is gone entirely,
and we update the workspace file to also not show `tvix/default.nix` at
the root either, so running `nix-shell` and then `cargo build` should
still work.

Change-Id: I6cb54d45d150c597612530ba44bc578f9d7f9120
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9556
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-10-07 05:47:31 +00:00
Connor Brewster
9019d8568e fix(tvix/store): Remove virtiofs from default features
vhost-user-backend doesn't support macOS yet, so the virtiofs features
will not work on macOS. This removes it as a default feature which makes
`cargo build` work out of the box on macOS.

The `virtiofs` feature is enabled for Linux when building via Nix, but
if being built by cargo directly, the feature must be enabled via a
cargo flag.

Change-Id: I2aaca9582f8e3dbcf9ee5f1b9831d614909f3799
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9555
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
2023-10-07 02:34:00 +00:00
Florian Klink
9c0d66197b feat(tvix/nar-bridge): do pathInfo.Validate() as additional check
This should make it quite quick to spot writing code breaking some of
the assumptions we have on PathInfo messages ourselves.

Change-Id: I480caaec41f8ea5246c3c3081460c7ad12e78569
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9554
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-07 02:06:55 +00:00
Vincent Ambo
4bf541109a chore(whitby): remove Nixery configuration
nixery.dev is running on a separate host now, it's not required here anymore.

Change-Id: Ie03d5847f8313fdfcf56fa43bb03651b3e4925f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9552
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
2023-10-06 09:20:15 +00:00
Vincent Ambo
053643c66f chore(ops): remove images.tvl.fyi
I don't even know what this is/was.

Change-Id: I743efa88258bbc13b7a3d4b8de8df222325b00ed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9553
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-10-06 09:17:22 +00:00
Florian Klink
7e9da4ddef fix(tvix/nar-bridge): properly check error during HTTP shutdown
We didn't bind err := here.

Change-Id: I779d7ba5660ba4b126a4a0db7fe044bf94425855
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9547
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-05 19:15:00 +00:00
Florian Klink
96c0b3f069 refactor(tvix/store/proto): use NamedNode trait
This saves us writing the name parsing code three times. We can also
delay parsing until we did other (cheaper) checks.

Change-Id: I1abe3f20dba4215b38839cf7466297e028d64656
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9548
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-10-05 18:58:55 +00:00
Florian Klink
a2c81de4cb docs(tvix/nar-bridge): fix comment
We run narInfo.Check to ensure this parses to a StorePath, not
nixpath.Check.

Change-Id: Id91183128df74a60d98fa2a31174cd879194c34d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9550
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-05 18:51:53 +00:00
Florian Klink
61e5a63fe1 chore(tvix/nar-bridge): bump go-nix dependency
nixpath.FromString -> storepath.FromAbsolutePath.

See https://github.com/nix-community/go-nix/pull/113 for details.

Closes: https://b.tvl.fyi/issues/314
Change-Id: I25277fb6006cbbb2a323ffb5809a1be500822a97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9551
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 18:47:52 +00:00
Florian Klink
600815c168 feat(tvix/store/protos): add Validate() method to pathinfo.go
This is very similar to the Rust counterpart.

Change-Id: I40d51aaac3fcf7f52e5896587e561bc2377f6269
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9549
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-05 18:42:58 +00:00
Vincent Ambo
618853d6b0 chore(web/tvl): graph updates
* removed mdjnsn (not seen him for a long time, and not on IRC!)
* added K900

Change-Id: I21e7db61d0a360244f95f51eebca6df94d703cff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9503
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-10-05 17:20:47 +00:00
Vincent Ambo
ddbb25e145 feat(tvix/website): add structured data for search engines
This closes b/303.

Change-Id: Icaeccba23a2590c22106bed6e859ed3188c87a02
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9540
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-10-05 17:14:13 +00:00
Florian Klink
cb807ad79b refactor(tvix/nar-bridge): do root directory dgst check in uploader
This check makes more sense there, and gives stronger semantics - Done()
only succeeds if the other side successfully received everything, *and*
came up with the same hashes as we did.

Change-Id: I20b706961053fd00d22cc70e1c8cc859705587e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9542
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 15:02:12 +00:00
Florian Klink
c04041a001 feat(tvix/nar-bridge): add nar-bridge-pathinfo command
This adds an additional nar-bridge-pathinfo command.

It exposes a PathInfoService for a HTTP Binary Cache, ingesting data
into a BlobService/DirectoryService as it goes through the NAR file.

It does this whenever it receives a Get request for a specific output
path, and waits returning with the PathInfo response until it ingested
the data.
It does not do any sort of caching - this means it re-downloads NAR
files again whenever the PathInfo is requested again, so you most likely
do not want to use this currently.

It's one building component as soon as we have store composition (which
we currently don't, so don't use this).

It can be used as an alternative mechanism to ingest data (Blobs and
Directories) of a given store path from a binary cache into tvix-store.

```
❯ nix-build -A third_party.nixpkgs.hello
/nix/store/mdi7lvrn2mx7rfzv3fdq3v5yw8swiks6-hello-2.12.1

❯ nix hash to-sri --type sha1 mdi7lvrn2mx7rfzv3fdq3v5yw8swiks6
sha1-Rs/INeK+7IGbG/u7fHoVNm96Yqs=

❯ out=$(mg build //tvix/nar-bridge)
$out/bin/nar-bridge-pathinfo --log-level debug &
INFO[0000] Starting nar-bridge-pathinfosvc at [::]:8001

❯ mg run //tvix:store -- daemon &
[mg] building target //tvix:store
[mg] running target //tvix:store
  2023-10-03T16:21:57.433739Z  INFO tvix_store: tvix-store listening on [::]:8000
    at src/bin/tvix-store.rs:229

❯ evans --host localhost --port 8001 -r repl
[…]
tvix.store.v1.PathInfoService@localhost:8001> call Get
✔ by_output_hash
by_output_hash (TYPE_BYTES) => Rs/INeK+7IGbG/u7fHoVNm96Yqs=
{
  "narinfo": {
    "narSha256": "sXrPtjqhSoc2u0YfM1HVZThknkSYuRuHdtKCB6wkDFo=",
    "narSize": "226552",
    "referenceNames": [
      "aw2fw9ag10wr9pf0qk4nk5sxi0q0bn56-glibc-2.37-8",
      "mdi7lvrn2mx7rfzv3fdq3v5yw8swiks6-hello-2.12.1"
    ],
    "signatures": [
      {
        "data": "7guDbfaF2Q29HY0c5axhtuacfxN6uxuEqeUfncDiSvMSAWvfHVMppB89ILqV8FE58pEQ04tSbMnRhR3FGPV0AA==",
        "name": "cache.nixos.org-1"
      }
    ]
  },
  "node": {
    "directory": {
      "digest": "xvo6BYbYaDw76IibLu5sr+VZoj9iM0ET2RUuYSYLwKE=",
      "name": "bWRpN2x2cm4ybXg3cmZ6djNmZHEzdjV5dzhzd2lrczYtaGVsbG8tMi4xMi4x",
      "size": 141
    }
  },
  "references": [
    "ptgFMIhdl2nJxMDdlDkITyXuBFc=",
    "Rs/INeK+7IGbG/u7fHoVNm96Yqs="
  ]
}
```

Change-Id: I50167d0ac081c91adf5cf2733bbc4dc0993bd46e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9539
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: Brian Olsen <me@griff.name>
2023-10-05 15:02:12 +00:00
Florian Klink
d45d6de561 test(tvix/store/pathinfo): add more tests for references
This should cover all error cases produced.

Change-Id: If31816d9b087551d86d7913df55df8f9f44bb554
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9546
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-10-05 10:25:16 +00:00
Florian Klink
1f03a520a9 feat(tvix/store): add validation for references
This validates the size of reference digests in the PathInfo message,
as well as inside the narinfo submessage. If narinfo is set, they need
to parse to StorePath, and have the same digest there as in the PathInfo
message.

`proto::tests::pathinfo::validate_references` needed to be updated,
because we actually did not populate the proper references before.

Change-Id: I9545b2487aab9fe0d229c26aceba5ddc5e6daafd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9545
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-10-05 09:52:19 +00:00
Florian Klink
7706a8f224 refactor(tvix/store): mv *Err*::Invalid{Node,}DigestLen
There's other digests in the PathInfo structure, that also might have
wrong digest lengths. Rename this to give some room for them, and update
the error message a bit as we go.

Change-Id: I06562664721156e658f2ed14ba1de907377d284b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9543
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-10-05 09:52:13 +00:00
Florian Klink
c67ab911eb refactor(tvix/castore): move magic number to B3_LEN const
… and export it.

Change-Id: I47d2dc2f5a8174da65c614b43801d648506e2d73
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9544
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
2023-10-05 09:50:31 +00:00
Florian Klink
8239f32b62 refactor(tvix/nar-bridge): mv nar-bridge{,-http}
Rename the nar-bridge CLI to nar-bridge-http, because it's the one
spinning up an http server.

Change-Id: I0fb75c50e4299272a128dd5ecaa4be8f06fa3dbe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9538
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-05 06:17:10 +00:00
Florian Klink
c18ff1a270 refactor(tvix/nar-bridge): deduplicate NAR HEAD and GET
Use a genNarHandler() function accepting a boolean to construct the
HTTP handler.

Change-Id: I17c054826d91a9dbed8b1f53945a51f27fa60ace
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9537
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-10-05 06:17:10 +00:00
Florian Klink
45511004df docs(tvix/nar-bridge): fix typo in comment
Change-Id: I74f6892e089ed306ef93e78a39a25cf3738bb09f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9536
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-05 06:17:10 +00:00
Florian Klink
28d1b9c01d refactor(tvix/nar-bridge): move pkg/server to pkg/http
This is only dealing with the HTTP interface.

Change-Id: I011b624fd9f11ea96231b92fea1166c118a219f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9535
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 06:17:10 +00:00
Florian Klink
0353108e99 refactor(tvix/nar-bridge): move DirectoriesUploader to pkg/importer
This is useful outside a HTTP server scenario.

Change-Id: I0042a6e773906a15a254d850520e6f841035bf20
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9533
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 06:17:10 +00:00
Florian Klink
b78b8d8338 refactor(tvix/nar-bridge): move blob cb function to pkg/importer
This is useful outside a HTTP server scenario.

Change-Id: If35f1ab245855378fd01f16ad7b5774d0cf590ba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9532
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-10-05 06:17:10 +00:00
Florian Klink
259269482c docs(tvix/nar-bridge): add more comments for DirectoriesUploader.
Change-Id: I6684186f3120ea42036239baa82e0ca37d3e0e6c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9531
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-05 06:17:10 +00:00
Florian Klink
d3c44ab03d refactor(tvix/nar-bridge): rename writer package to exporter
Change-Id: I7cf6a3ed0dca4c450fadd9228dc0154ad6254f51
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9530
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-10-05 06:17:10 +00:00
Florian Klink
49b427d773 refactor(tvix/nar-bridge): combine writers/readers
We can drop most of Hasher if we use a MultiWriter writing to the hash
function and a minimal CountingWriter.

This should make things a bit more understandable.

Change-Id: I37ee72d9a5c73f253aecc1ad761cb723389b89fc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9529
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-05 06:17:10 +00:00
Florian Klink
f92b0ef933 refactor(tvix/nar-bridge): let callbaks return calculated digests
This aligns behaviour more with how it should be - it's the
responsibility of the callback functions to return digests of the things
they consume(d). It allows further cleaning up the hasher struct.

Change-Id: I9cbfc87e6abd4ff17fadf39eb6563ec3cb7fcc6f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9528
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 06:17:10 +00:00
Florian Klink
b1ff1267be refactor(tvix/nar-bridge): drop reader package
Make the import function usable on any reader.

Change-Id: I84d2004cb73cdd7a11fe8efb0f2efb6335d5e6b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9527
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-05 06:11:27 +00:00
Florian Klink
6e9a5dcd59 refactor(tvix/nar-bridge): rename item to stackItem
Change-Id: I76c8db3b04d9f97e51e91f7d36a4dd8c6500d1a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9526
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-05 06:11:23 +00:00
Florian Klink
174485038c refactor(tvix/nar-bridge): inline assemblePathInfo
This is only called once.

Change-Id: I342443b8d04050929733fc84d5f36cd64060afe3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9525
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-05 06:11:23 +00:00
edef
155f53b264 fix(tvix/nix-compat): reject dotfiles
Nix has historically rejected these. The current behaviour was
accidentally introduced in Nix 2.4, and is considered a bug.

Link: https://github.com/NixOS/nix/pull/9095
Change-Id: I38ffa911f0a413086479bd972de09671dbe85121
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9507
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: edef <edef@edef.eu>
2023-10-04 17:18:11 +00:00
edef
9a7c078a69 fix(tvix/castore): explicitly name lifetimes in process_entry
Otherwise this produces absolutely inscrutable errors:

    note: hidden type `[async fn body@castore/src/import.rs:63:1: 63:94]` captures lifetime '_#24r

Change-Id: If5d9626c9edf400de5bcec038bcaa5a3117561f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9506
Tested-by: BuildkiteCI
Autosubmit: edef <edef@edef.eu>
Reviewed-by: flokli <flokli@flokli.de>
2023-10-04 08:31:40 +00:00
edef
e014c3ef70 fix(tvix/nix-compat): bits are not bytes
Change-Id: Ib5744ea240895fe84454ad271ee5640b29cab838
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9505
Autosubmit: edef <edef@edef.eu>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-03 08:09:28 +00:00
Brian Olsen
cfb810d81a fix(tvix/store): Fix FUSE support on MacOS
This partially fixes b/312 and gets FUSE to work again on MacOS.

It is mostly small type changes and an update to fuse-backend-rs because
upstream currently doesn't work with MacFuse. It also sets the default
FUSE thread count on MacOS to 1 because otherwise the mount command will
hang when shutting down as only one thread gets ENODEV and all the others
just keep blocking.

Change-Id: Ifb3c4268caf296c487049c1dc4618acb32497f44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9490
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: flokli <flokli@flokli.de>
2023-10-02 15:46:47 +00:00
Brian Olsen
5c2cad0ac4 fix(tvix/store): FuseDaemon unmount is blocking async runtime
The unmount method in FuseDaemon calls join on a bunch of threads and that is
a blocking call but it is called from an async context in the tvix-store
binary.

This change wraps the call to unmount in a spawn_blocking.

Change-Id: If89183b4a3f890874e75f5faf90cd24cb18da1e1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9489
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-02 12:50:16 +00:00
Ryan Lahfa
f5a33ab15f feat(web/tvl): add raitobezarius on the TVL graph
I think this may be representative of reality, but I forgot :D

Change-Id: I6aaeb959a44b3a5cdbbaea1d99b3034a7db05ba6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9498
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-10-02 11:19:49 +00:00
Vincent Ambo
63d3d8cc35 fix(tazjin/koptevo): let gonic access /etc/resolv.conf
Without this line, accessing external APIs from Gonic (e.g. scrobbling
to Listenbrainz) does not work (unless a local resolver exists). This
is because in the pure Go DNS implementation used in gonic, only
/etc/resolv.conf can provide resolvers.

Change-Id: I26dd9a845b0a70c4cfb983c68da2b76b7e57dfec
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9502
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-10-01 22:15:34 +00:00