Commit graph

75 commits

Author SHA1 Message Date
Yureka
6774d9c59c chore(tvix/tracing): switch tracing-opentelemetry from git to release
Change-Id: Ib830c8b642496a6cdf3d4b9093f0343f5fb80622
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12018
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-22 13:36:08 +00:00
Yureka
ca8e2b9fbf chore(tvix): upgrade to tonic 0.12 / hyper 1.0
Change-Id: Idd8ce48869ddd869d51a10959b920f1290a8a9b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11991
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-07-21 11:01:38 +00:00
Florian Klink
fca8462530 fix(tvix): fix outputHashes calculation for crates with versions
In case two crate versions are present, the key in the Cargo.nix file
includes the version number too.

To be able to set a specific hash for "tracing-opentelemetry 0.25.0" for
example, this needs to account for keys with the version included.

Access `crateName`, `version` and `src.outputHash` individually.

Change-Id: Ib9800691a445ac403ff646cb32e85a7a4cbef9d8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11994
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
2024-07-20 22:17:15 +00:00
Ilan Joselevich
6a7069904e feat(tvix/utils): Add defaultCrateOverridesForPkgs function
This function can be reused across the rest of the repo to make use of
our overriden defaultCrateOverrides with support for tvix crates.

Change-Id: I8c554dece052bd9dd32acac13dab8114933272a7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11951
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-05 20:19:14 +00:00
Vincent Ambo
4b2f3c5454 chore(3p/sources): bump to OpenSSH vulnerability hotfix
See https://github.com/NixOS/nixpkgs/pull/323753 for details.

Changes:

* git: temporarily comment out dottime patch (it doesn't apply, but it's not critical)
* third-party/cgit: use an older git version where dottime patch still applies
* 3p/crate2nix: remove crate2nix patches included in latest release
* tvix: remove unneeded defaultCrateOverrides (upstreamed to nixpkgs)
* tvix: regenerate Cargo.nix
* tvix/nix-compat: remove unnused AtermWriteable::aterm_bytes pub(crate) function
* tvix/nix-compat: remove redundant trait bounds
* tvix/glue: use clone_into() to set drv.{builder,system}
* tools/crate2nix: apply workaround for https://github.com/numtide/treefmt/issues/327
* toold/depotfmt: expose treefmt config as passthru
* tools/crate2nix: undo some more hacks in the crate2nix-check drv

Change-Id: Ifbcedeb3e8f81b2f6ec1dbf10189bfa6dfd9c75c
Co-Authored-By: Florian Klink <flokli@flokli.de>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11907
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-07-01 17:42:30 +00:00
Florian Klink
f6c759de58 chore(tvix): bump bigtable_rs to latest release
https://github.com/liufuyang/bigtable_rs/pull/72 has been merged for a
while, no need to use our own checkout here.

Change-Id: Ide5acd9b7e0f5a46b1c795178e29a037206b2448
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11906
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-07-01 15:53:53 +00:00
Ilan Joselevich
92c5433304 chore(tvix): move filterRustCrateSrc to utils.nix
Change-Id: Ib6bae1ea0457d3309a6ec1e08b9e4b320523c161
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11857
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-17 15:26:19 +00:00
Ilan Joselevich
ac51803ad8 chore(tvix): Remove no longer needed libiconv addition for darwin
Since https://github.com/NixOS/nixpkgs/pull/220212 libiconv is now
automatically added for Darwin.

Change-Id: I2aaa266bcbc4d5a8256131aed36b4c2d040a8758
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11856
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
2024-06-17 15:08:10 +00:00
Ilan Joselevich
210b4277b6 chore(tvix): Use callPackage instead of import on Cargo.nix
Since crate2nix v0.11 the new preferred way of importing Cargo.nix is
with callPackage instead of import.

Change-Id: Ifcc5c21e6c101cb9d0b3f1e8025b5ddb8cf99f20
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11855
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-17 14:34:46 +00:00
Ilan Joselevich
fbe29615df chore(tvix): Remove no longer needed rlib stripping fix for darwin
This issue was fixed in https://github.com/NixOS/nixpkgs/pull/255900

Change-Id: I4f569099e62c4284bdea18a512387735f139e702
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11854
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-06-17 14:34:04 +00:00
Simon Hauser
825d498908 feat(tvix/tracing): introduce common tvix-tracing crate
Introduce a new common crate that contains tracing boilerplate which then
can be used in the cli, tvix-store and tvix-build crates.
It has otlp as an optional feature, which is currently only used by
tvix-store.

Change-Id: I41468ac4d9c65174515d721513b96fea463d6ed2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11758
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Simon Hauser <simon.hauser@helsinki-systems.de>
2024-06-10 16:35:08 +00:00
Ilan Joselevich
9b77ce9f8f fix(tvix/crate2nix-check): make drv less likely to be cached
The derivation name for the check will now be calculated from the hash
of all Cargo related files (including in subdirs), this makes it less
likely for the drv to be cached and for CI to miss an outdated
Cargo.lock.

Change-Id: I900e9355be3f8a9d6f01162e8ef0da4d8901af30
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11753
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-05 19:05:33 +00:00
Ilan Joselevich
1b39d5868a feat(tvix): add CI targets for checking crate features powerset
Closes: https://b.tvl.fyi/issues/401

With this change all crate features (and their combinations) will be built and
tested in CI.

From now on, when adding/removing a Cargo feature for a crate,
you will want to add it to the features power set that gets tested in CI.
For each crate there's a default.nix with a `mkFeaturePowerset` invocation,
modify the list to include/remove the feature.
Note that you don't want to add "collection" features,
such as `fs` for tvix-[ca]store or `default`.

Change-Id: I966dde1413d057770787da3296cce9c1924570e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11717
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-06-03 16:35:51 +00:00
Ilan Joselevich
3ebcf30b64 feat(tvix): filter src via lib.fileset
Previously changing any file (including default.nix or README.md) would
cause Nix to compute a different derivation path, resulting in needing
to (very often redundantly) rebuild the crate/package. With this change
you can expect less rebuilds.

Crate2nix currently does its own insufficient src filtering and it does
not expose an API to override the filtering, so instead I created a
function that we can use to override the src to have stricter filtering.

I implemented the filtering for all of the workspace members, if you
want to modify any of them, please read the lib.fileset docs
https://nixos.org/manual/nixpkgs/unstable/#sec-functions-library-fileset

Change-Id: I7ab5a0064a76938d14f7f65801be9f3a5c3bad04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11714
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-26 14:27:50 +00:00
Florian Klink
afcbc1d86d chore(tvix/clippy): configure clippy allow block_in_conditions
… in Cargo.toml.

This gets an imperative `cargo clippy` run to pick up that config,
so `-A clippy::blocks_in_conditions` doesn't need to be explicitly
specified anymore.

Change-Id: I32b6cc50c77c22cba0d816d0db508c2f94b2c383
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11659
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2024-05-13 16:57:03 +00:00
Florian Klink
cd820d07d7 chore(nix-compat): remove unused test-case and test-generator
This is especially nice, as it allows us getting rid of the other
versions of proc-macro2 and quote.

Change-Id: I9fdd012ee6c0ded3e18ec30504b83ac2032d1390
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11474
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-04-19 19:22:10 +00:00
Florian Klink
17849c5c00 feat(tvix/castore/directory): add bigtable backend
This adds a Directory service using
https://cloud.google.com/bigtable/docs/ as a K/V store.

Directory (closures) are put in individual keys.

We don't do any bucketed upload of directory closures (yet), as castore/
fs does query individually, does not request recursively (and buffers).
This will be addressed by store composition at some point.

Change-Id: I7fada45bf386a78b7ec93be38c5f03879a2a6e22
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11212
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-04-09 15:50:34 +00:00
sterni
e220d80727 chore(3p/sources): Bump channels & overlays
- agenix has not been updated (https://github.com/ryantm/agenix/pull/241).

- //tvix: regenerate protobuf files

- //tvix:clippy: work around https://github.com/rust-lang/rust-clippy/issues/12281
  which exclusively causes false positives in our code at the moment.

Change-Id: I38d2f4c0e6d1abc92be360b06f58e6d40e7732a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11127
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
2024-03-17 10:56:07 +00:00
Nikita Voloboev
70c068df51 fix(tvix): fix build on darwin
on macOS additional framework was needed to build tvix build

Change-Id: I0e327378d1bd4837f62496c4c6e66bc489ddedb4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10747
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-02-06 19:10:59 +00:00
Florian Klink
4497ac41ab refactor(tools): move crate2nix generate here
Having something running the depot crate2nix and formatting it with
depotfmt is useful outside of tvix too.

Change-Id: Iecc8f207da38cc6995747c5ea48d3911433fd416
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10658
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-01-18 15:02:15 +00:00
Florian Klink
3297843bf1 chore(tvix): run clippy with --{benches,examples}
While doing the obvious thing, ensuring clippy doesn't complain on our
benchmarks and examples, this will also flag build failures on them,
and contrary to cl/10301 not pollute the /nix/store with stuff we can't
execute anyways.

Change-Id: I7ee1c2babbc67dac9794274d05bb9d6e1251ed01
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10544
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-01-05 16:37:30 +00:00
Florian Klink
3307791855 feat(tvix/store): add opentelemetry
This is behind the otlp feature flag (currently enabled by default).

By default, this will try to push traces to a OTLP collector running at
https://localhost:4317.

You can make one available by running:

```
docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  -p 14250:14250 \
  -p 14268:14268 \
  -p 14269:14269 \
  -p 9411:9411 --rm \
  jaegertracing/all-in-one:1.42
```

Started like that, jaeger brings a web interface at
http://localhost:16686/search

As documented in
https://docs.rs/opentelemetry-otlp/latest/opentelemetry_otlp/, you can
point this to another location by setting `OTEL_EXPORTER_OTLP_ENDPOINT`.

Change-Id: Id1dca367d70027b2ea98bb70bcf99a68363ec2be
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8194
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: aaqaishtyaq <aaqaishtyaq@gmail.com>
2024-01-01 17:27:39 +00:00
Florian Klink
acbb613e61 chore(tvix): switch to upstream futures 0.3.30
The bugs have been fixed,
https://github.com/rust-lang/futures-rs/pull/2801 and
https://github.com/rust-lang/futures-rs/pull/2812 were merged and ended
up in that release.

Change-Id: Iefd990d2d1719b884504093343e54e9c5258e2e2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10414
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-24 21:45:04 +00:00
Florian Klink
43ab1543e2 chore(tvix/store): switch back to fuse-backend-rs upstream
The fix mentioned in Cargo.toml did land in
https://github.com/cloud-hypervisor/fuse-backend-rs/pull/162.

Change-Id: I4451f910806f05400f7e0f83581ae786f170b9e2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10375
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-12-16 21:46:30 +00:00
Florian Klink
1cc3e9c1a8 refactor(tvix): use granular proto files
Only pass in the proto files that are actually needed to build that
crate. They are already constructed in depot.tvix.$crate.protos.protos.

Change-Id: If4381e6c3350e420ee4ddce1e0513bfe970678a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10240
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-12-11 22:35:39 +00:00
Florian Klink
fd27d8ddc3 feat(tvix/build): init
This adds the tvix-build crate, currently only containing a
`tvix_build::proto` module, exposing the data structures defined in
tvix/build/protos.

Change-Id: I75f5d9196969ed0877b1fe640cacfecba0fb2e03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10235
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-12-11 21:43:38 +00:00
Florian Klink
efaff79584 feat(tvix): make clippy stricter
Also run clippy on tests, and enable all features.

Change-Id: Ide9f1bc9f565333072afb918c391c7930b658f41
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10234
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-09 12:58:39 +00:00
Adam Joseph
0e2633048e feat(tvix): run crate2nix generate in CI
This runs `crate2nix generate` in CI and then runs `depotfmt` on the
result to ensure that our machine-generated code is really, really
readable and pretty.  Then it checks that the result of all that
is identical to the committed Cargo.nix.

A self-hashing FOD is used to allow network access.

No magic hashes are involved.

Co-Authored-By: Florian Klink <flokli@flokli.de>
Change-Id: I68ec5003dbc6a40894a5a4d6e902f138c99f6719
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10194
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-12-07 18:43:46 +00:00
Florian Klink
41235bf908 refactor(tvix): move src into let binding
Change-Id: Ida2a3ac722fb2445745759323975884dfeef3e87
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10193
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-12-05 21:55:09 +00:00
edef
9ad7cc629e fix(tvix): patch futures::AsyncBufReadExt::fill_buf
This fixes EOF handling for buffered readers.

Link: https://github.com/rust-lang/futures-rs/pull/2801
Change-Id: Ie98ca6a3e1de38500b0195e9b62511501acb1d2c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10086
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-11-19 08:35:38 +00:00
Florian Klink
a9329ff811 feat(tvix): add clippy target in CI
This will invoke `cargo clippy` on the codebase, ensuring we spot things
early on.

Relates to b/321.

Change-Id: Ifba13e5bef995d33b24346c6cee134e84d6566c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9106
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-05 20:28:53 +00:00
Florian Klink
45d8a78a8b chore(tvix): move regenerate scripts to passthru
This keeps things isolated a bit more.

Change-Id: I437f2f63ee1567c0cbc02298514ad4d89f1dce85
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9790
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-17 20:25:45 +00:00
Florian Klink
1b26bf21e3 chore(tvix): move store golang bindings to tvix/store-go
Similar to the castore-go CL before, this also updates the store-go
bindings to the new layout.

Change-Id: Id73d7ad43f7d70171ab021728e303300c5db71f0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9788
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-10-17 19:51:24 +00:00
Florian Klink
e38733a955 chore(tvix): move castore golang bindings to tvix/castore-go
Have `tvix/castore/protos` only contain the protos, no go noise.

Make the `.pb.go` file generation a pure Nix build
at `//tvix/castore/protos:go-bindings`, and have a script at
`//tvix:castore-go-generate` (TBD) that copies the results to
`tvix/castore-go`.

`//tvix:castore-go`, with sources in `tvix/castore-go` now contains the
tooling around the generated bindings, and the generated bindings
themselves (So go mod replace workflows still work).

An additional CI step is added from there to ensure idempotenty of
the .pb.go files.

The code.tvl.fyi webserver config is updated to the new source code
path. I'm still unsure if we want to also update the go.mod name. While
being a backwards-incompatible change, it'll probbaly make it easier
where to find these files, and the amount of external consumers is still
low enough.

Part of b/323.

Change-Id: I2edadd118c22ec08e57c693f6cc2ef3261c62489
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9787
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-17 19:51:24 +00:00
Florian Klink
e0e3dc08a9 fix(tvix/cli): don't use protobuf from pkgs, but pkgs.buildPackages
We need to invoke protoc at build time (only).

Change-Id: Ib4c101d2ccdbdbb078725fc98c30d6c00086667d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9754
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-16 16:08:04 +00:00
Florian Klink
0f44451919 fix(tvix): fix binary building on Darwin
On Darwin, some crates producing binaries need to be able to link
against security.

Change-Id: I5bdd69247c12729b9efd5c4f18527d361ef99e87
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9758
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-16 15:14:21 +00:00
Florian Klink
5ce1a03a3a refactor(tvix): push rarely-used darwin-specific hacks down
Also, make some of them buildInputs, not nativeBuildInputs.

We likely need to link against libiconv, so it shouldn't be a native
build input.

Change-Id: I757bc7f5d8c864ecaa3db26988d00b496c085c8f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9757
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-10-16 15:09:28 +00:00
Florian Klink
786b0324a9 feat(tvix/store/protos): use Validate() function on root node
This updates the code to make use of the new Validate() function defined
on a Node.

Change-Id: I9b6ed694661f41e700f19cc78d53d2224b61852d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9718
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-10-14 14:21:32 +00:00
Florian Klink
e6ba84ea50 feat(tvix/store/protos): add Export
Export will traverse a given PathInfo structure, and write the contents
in NAR format to the passed Writer.
It uses directoryLookupFn and blobLookupFn to resolve references.

This is being moved over from nar-bridge. We need to keep the code there
around until we can bump go.mod to storev1 with this merged, but the
tests can already be moved entirely.

Change-Id: Ie0de3077b09344cafa00ff1e2ddb8b52e9e631bc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9602
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Autosubmit: flokli <flokli@flokli.de>
2023-10-10 18:55:49 +00:00
Florian Klink
d1bccfe82f chore(tvix/*store): use google.golang.org/protobuf
The other one is deprecated, according to
https://pkg.go.dev/github.com/golang/protobuf/proto.

Change-Id: I502d5bfa7431df0dea3d564f1e0ecfea14c90e10
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9597
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
2023-10-09 22:05:07 +00:00
Florian Klink
28cd4b1a2f feat(tvix/nar-bridge): add ToNixNarInfo()
Convenience function, moves all code converting from a PathInfo struct
to to go-nix's NarInfo.

Change-Id: Idf0dcc38675674563f2dfd3286a4a55fa2a24a82
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9593
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Brian McGee <brian@bmcgee.ie>
2023-10-09 22:05:02 +00:00
Florian Klink
92481825b3 test(tvix/store/protos): add go tests for Validate()
We already have validation tests for Rust, let's add the missing ones
for golang too.

Change-Id: Iaf3a3e1ee72d5647da3f2aa977d6e0d0379b2ce5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9595
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-09 22:01:59 +00:00
Florian Klink
67ccbf8c94 fix(tvix/castore-protos-go): fix src
This was unfortunately using ./castore/protos all the time, so not
testing castore at all.

Update src to make sure it does get built.

Change-Id: I563910c259ffca6aee7f964532bd39c4bcf7aed8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9596
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian McGee <brian@bmcgee.ie>
Tested-by: BuildkiteCI
2023-10-09 22:01:58 +00:00
Florian Klink
c847cc32d9 refactor(tvix/castore): move tests to grpc client, rm tonic-mock
Similar to gen_directorysvc_grpc_client, introduce a
gen_blobsvc_grpc_client function that provides a gRPC client connected
to a blobservice.

The test is update to use that client to test against, rather than the
server trait, removing the last usage of tonic_mock, so it's removed
as well.

Fixes b/243.

Change-Id: If746e8600588da247eb53a63b70fe72f139e9e77
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9564
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-10-08 21:53:20 +00:00
Florian Klink
03fec0b393 refactor(tvix,views/tvix): move shell into separate file
So far, we provided a custom `default.nix` in the root of the tvix josh
workspace, which re-defined the shell attribute from `tvix/default.nix`.
Some of the recent fixes, e.g. the MacOS-specific additions to the list
of dependencies however didn't get ported over to this file, and in
general, it's quite annoying to have two different places for these
things.

Initially I explored the idea of moving this default.nix file to a
default-depot.nix file in the josh worktree only, and then "polyfill"
some of the dependencies, or set up readTree in the josh workspace too,
but it turned out to pull in too many dependencies to be worth
the effort (nix.sparseTree, tools.depotfmt, crate2nix overlay,
third_party.gitignoreSource).

I now took a different approach - moving the definition of the `shell`
attribute from `tvix/default.nix` to its own `shell.nix` file, which is
imported from `tvix/default.nix` in regular depot usecases.

Josh workspace consumers only see the `shell.nix`, which can be used
in a self-contained fashion, the other `default.nix` is gone entirely,
and we update the workspace file to also not show `tvix/default.nix` at
the root either, so running `nix-shell` and then `cargo build` should
still work.

Change-Id: I6cb54d45d150c597612530ba44bc578f9d7f9120
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9556
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-10-07 05:47:31 +00:00
Connor Brewster
9019d8568e fix(tvix/store): Remove virtiofs from default features
vhost-user-backend doesn't support macOS yet, so the virtiofs features
will not work on macOS. This removes it as a default feature which makes
`cargo build` work out of the box on macOS.

The `virtiofs` feature is enabled for Linux when building via Nix, but
if being built by cargo directly, the feature must be enabled via a
cargo flag.

Change-Id: I2aaca9582f8e3dbcf9ee5f1b9831d614909f3799
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9555
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
2023-10-07 02:34:00 +00:00
Florian Klink
600815c168 feat(tvix/store/protos): add Validate() method to pathinfo.go
This is very similar to the Rust counterpart.

Change-Id: I40d51aaac3fcf7f52e5896587e561bc2377f6269
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9549
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-05 18:42:58 +00:00
Florian Klink
10717605a2 chore(third_party/crate2nix): bump crate2nix to current HEAD
This will bring conditional features support.

Also invoke crate2nixgenerate with the --all-features argument, so all
dependencies, including the ones for optional features are included in
the Cargo.nix file.

Change-Id: I3bbcb200c9b481f660db89efba650ea4f7418a63
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9470
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-09-26 10:03:36 +00:00
Florian Klink
e13c86d916 chore(tvix): re-enable {ca,}store-protos-go
This couldn't be enabled before until go get was able to fetch the
modules again.

Change-Id: I82db7d22e497661ea2a0c8a45c0c865fcc0a1f18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9406
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 17:36:05 +00:00
Florian Klink
32f41458c0 refactor(tvix): move castore into tvix-castore crate
This splits the pure content-addressed layers from tvix-store into a
`castore` crate, and only leaves PathInfo related things, as well as the
CLI entrypoint in the tvix-store crate.

Notable changes:
 - `fixtures` and `utils` had to be moved out of the `test` cfg, so they
   can be imported from tvix-store.
 - Some ad-hoc fixtures in the test were moved to proper fixtures in the
   same step.
 - The protos are now created by a (more static) recipe in the protos/
   directory.

The (now two) golang targets are commented out, as it's not possible to
update them properly in the same CL. This will be done by a followup CL
once this is merged (and whitby deployed)

Bug: https://b.tvl.fyi/issues/301

Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 12:51:21 +00:00