Commit graph

2827 commits

Author SHA1 Message Date
Yureka
1c24d483d5 chore(tvix/castore/fuse): impl Layer for TvixStoreFs
Allows using a TvixStoreFs in overlay filesystems

Change-Id: I10e63c92776b1c783947d92070f360865ef2883c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12539
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-09-28 13:30:36 +00:00
Yureka
1034cc7774 chore(tvix/castore/fuse): update fuse-backend-rs
This release includes support for overlay filesystems

Change-Id: I946cbf346df045209afaea2d720bb57fc2f2659f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12538
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
2024-09-28 13:24:00 +00:00
Yureka
caf597db81 refactor(tvix): move refscan module to castore
This is required to add the optional refscanner parameter to the
ingest functions.

Change-Id: Ib40a7287cf857eb55e31e0df309a79474fefb518
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12527
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-09-27 20:39:58 +00:00
Florian Klink
2180e6b820 fix(tvix/castore): don't return object_store::has with INFO level
This otherwise spams the logs quite a bit, for example when uploading
to nar-bridge with the new check from cl/12497.

Change-Id: Idc2bcc513caea6fae38ae04489e4e3ee7be64bce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12510
Reviewed-by: yuka <yuka@yuka.dev>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-09-26 15:03:40 +00:00
sinavir
9f36632509 feat(tvix/store): Add a signing PathInfoService
- Add a new PathInfoService implementation that wraps transparently
around another except that it dynamically signs all the incoming
path-infos with the provided signer.

- Add a ServiceBuilder for this PathInfoService that provides a
SigningPathInfoService with a keyfile signer

Change-Id: I845ddfdf01d14c503c796b2b80c720dab98be091
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12032
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: sinavir <tvix@sinavir.fr>
Tested-by: BuildkiteCI
2024-09-26 12:29:03 +00:00
Yureka
e4378f0143 feat(tvix/store): seekable nar renderer
Co-authored-by: edef <edef@edef.eu>
Change-Id: I233206e8aae35504ca0519ac88178dfc5596bedb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12439
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
2024-09-25 19:23:45 +00:00
Yureka
6deff4d8e9 refactor(tvix/store/tests): combine tests into one parametrized function
Change-Id: I9ff43b29be68b9840c58286da96fa52927691804
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12507
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-09-25 19:07:54 +00:00
Ilan Joselevich
c1e69e260d feat(tvix/eval): Use thiserror for ErrorKind and CatchableErrorKind
thiserror is much more easier to maintain than manually implementing Error and Display.

Change-Id: Ibf13e2d8a96fba69c8acb362b7515274a593dfd6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12452
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
2024-09-25 18:18:37 +00:00
sinavir
0f92400112 fix(tvix/nar-bridge): Remove name check for root node in nar generation
Nar-bridge tried to parse the name of the protobuf node encoded in the
URL into a PathComponent but this name was empty, leading to an error
when the user tried to retrieve the nar file.

This was an oversight from the conversion to stricter types (some of the
CLs in the serious containing cl/12217).

We need a version converting a protobuf without a name to our stricter
types, but an empty PathComponent cannot be constructed.

So we need a into_name_and_node() version that returns the name as
Bytes, not PathComponent.

Change-Id: I2996cdd2e0107133e502748947298f512f1cc521
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12504
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-09-24 09:45:33 +00:00
Yureka
cd0c3a96ab feat(tvix/nix-compat/nar/writer/sync): add file_manual_write
This is useful for building other NAR writers which use custom (async or optimized) I/O to write the blob parts of the NAR.

Change-Id: I447c09914fb0c99044e2fa910d4213660dc51c64
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12437
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-09-23 12:13:00 +00:00
Florian Klink
21e5fc024d fix(tvix/castore/import): check small blobs first
ConcurrentBlobUploader buffers small blobs in memory, and then uploads
them to the BlobService in the background.

In these cases, we know the hash of the whole blob, so we could check if
it exists first before, uploading it.

We were however not, and this caused rate limiting issues in GCS, as it
has an update limit of one write per second on the same key, which we
ran into especially frequently with the empty blob.

This reduces the amount of writes of the same blob considerably.

In the future, we might be able to drop this, as our chunked blob
uploading protocol gets smarter and covers these cases.

Change-Id: Icf482df815812f80a0b65cec0426f8e686308abb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12497
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-09-19 12:51:09 +00:00
edef
e573f8389c docs(tvix/nix-compat/wire/bytes/reader): None case doesn't exist
We always read the length before returning a BytesReader, so len()
cannot be called before the length is read.

Change-Id: Ifa1cbc2c0923dd24d59e0af5e135ab8ed7314aa9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12453
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-09-08 02:32:57 +00:00
Yureka
863146295b feat(tvix/nix-compat/nar/writer/sync): trait objects -> generics
This will later be used to allow the user to access the original writer.

Change-Id: I1734600c5d580d18b16727f892cd2f6335cc9459
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12442
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-09-07 12:23:06 +00:00
Yureka
cec28377d8 feat(tvix/castore): expose ValidatedDirectoryGraph type
Change-Id: Id994258e74f75df8790159ed3180ce360874d0de
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12438
Reviewed-by: benjaminedwardwebb <benjaminedwardwebb@gmail.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-09-07 12:22:58 +00:00
Vincent Ambo
609b68031b chore(3p/sources): bump channels & overlays (2024-09-01)
Included changes:

* users/aspen: explicitly use open-source nvidia driver

  This now has to be specified explicitly, otherwise evaluation fails with an
  error.
* users/aspen: nixfmt -> nixfmt-classic
* users/aspen: fixes for renamed packages & options
* users/tazjin: fixes for renamed packages & options
* 3p/overlays: remove cbtemulator patch (merged upstream)
* tvix/shell: remove unnecessary patches (merged upstream)
* 3p/rust-crates: mark libgit2_sys as broken
* users/Profpatsch: mark git-db as broken
* 3p/overlays: pick `mypaint` from stable channel
* tvix: fix comments that clippy doesn't like anymore
* tvix/glue: disable a misfiring clippy lint (applying its suggestion breaks
  code below)

Change-Id: I6d3fc027694bbe7425a2d25dc53d65467a44f3b0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12403
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Autosubmit: tazjin <tazjin@tvl.su>
2024-09-05 20:56:31 +00:00
Florian Klink
c60f4ccfda docs(tvix/contributing): remove reference to b/201
This should be fixed with the switch to a "GitHub App" for
authentication.

Change-Id: I0bd8b4b9c9f53a754a12ce07f193d19f61f56014
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12414
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-09-03 15:17:30 +00:00
Ilan Joselevich
9803712ab8 fix(tvix): Follow-up fixing no-default-features and comments
In https://cl.tvl.fyi/c/depot/+/12389 I accidentally deleted some
comments. Also fixed some useless no-default-features with futures crate, making it match the previous feature combination.

Change-Id: I72bb2cfb88719ff2f8812d90193de2bd49149cce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12395
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-08-30 13:55:11 +00:00
Ilan Joselevich
0979379980 fix(tvix/utils): Add missing src filtering for nix-compat-derive[-tests]
Change-Id: I2beed2cdcb5423d3594562e0011b1cb889add07f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12390
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
2024-08-30 12:09:41 +00:00
Florian Klink
348998e1fa docs(tvix): update account creation instructions
This is tvix/docs, so asking to be a member of #tvl here is confusing.
Being a member of #tvix-dev is sufficient, we can still mention #tvl
though.

Change-Id: If9fbf7d098c0a8d77a45a95e321fc0a553f4c63b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12396
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
2024-08-30 10:36:23 +00:00
Ilan Joselevich
5a97888d8b chore(tvix): Migrate members to inherit deps from workspace
From now on we will add the dependencies and their version in the root
Cargo.toml and in order to enable the dependency for a workspace member
we set `workspace = true` in the member's Cargo.toml.

Change-Id: I9738c1cf99810b7ace87ca712c3ea965ba846e25
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12389
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-29 14:37:22 +00:00
Ilan Joselevich
2945a359b4 chore(tvix): Bump versions of all compatible deps via cargo upgrade
This updates all the dependencies and their "minimum" versions in
Cargo.{lock,toml} to the latest compatible version using `cargo-edit`'s
`cargo upgrade` command that will eventually be merged into `cargo
update`.

Change-Id: Iccb2aa4a1c84a0465222244a0bd0cafe2a82e781
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12388
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
2024-08-29 11:26:15 +00:00
Florian Klink
02ee441626 chore(tvix/[ca]store): bump bigtable_rs
This bumps bigtable_rs to
https://github.com/liufuyang/bigtable_rs/pull/86, allowing us to drop
our second set of prost/tonic/http/axum crates.

Change-Id: I70f9150289c3e8611ebe8a7d99490e3dfd085a6e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12384
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-08-28 20:15:55 +00:00
Florian Klink
b6dd9d2d5b chore(tvix/[ca]store): bump hyper-util to 0.1.7
Change-Id: Ia5eaf7f4614701ad4dd06114476a73e1a041d830
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12383
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-08-28 20:15:55 +00:00
Florian Klink
b2b38f23d0 chore(tvix): bump tonic[-build] from 0.12.1 to 0.12.2
Change-Id: If7d1ae4491d3c70f323cf872d1a70afe6238f78a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12382
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-28 19:01:04 +00:00
Vincent Ambo
c622af481e feat(tvix/eval): allow external users to construct native thunks
This interface is the only way to construct lazy builtins outside of eval, which
is actually a useful feature to have.

Change-Id: I386323af20aa3134bb4f669fa66fbb21e9b05fd4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12386
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: Yury Shvedov <yury.shvedov@kaspersky.com>
2024-08-28 18:05:30 +00:00
Vincent Ambo
82429f0661 refactor(tvix/eval): remove useless impl block
There's no need to duplicate this.

Change-Id: If3d930211a1d625d6c7ef129b05034e7a915da83
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12385
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: Yury Shvedov <yury.shvedov@kaspersky.com>
2024-08-28 18:04:58 +00:00
Brian Olsen
a5fcfd80a1 fix(tvix/nix-compat-derive): Get rid of external feature flag
The external feature flag was there because I couldn't find a way to
refer to crate and nix-compat with the same name so that the generated
code could be the same.

In essence `use nix_compat::nix_daemon:🇩🇪:NixDeserialize` is an error
when used inside nix_compat crate.

So my best fix was the external feature flag until I found the solution
used here which also removes the flag completely.

Change-Id: Ia3e89c6c350c3fb22ca87f974a39c21542aae152
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12376
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Brian Olsen <me@griff.name>
2024-08-28 14:43:00 +00:00
Brian Olsen
743407e7ab fix(tvix/nix-compat-derive): Fix doctest for Rust older than 1.80
Exclusive range patterns were stabilized in Rust version 1.80 but Tvix
still uses Rust 1.79 and so would fail this one test when you ran
doctests from `mg shell //tvix:shell`.

It was not caught by CI because that does not currently run doctests.

The fix is just to use an inclusive pattern instead.

Fixes: b/417
Change-Id: Ifea7a3b84bb8f6f8c76e277979833713bdf51f46
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12375
Autosubmit: Brian Olsen <me@griff.name>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-08-28 14:08:39 +00:00
Florian Klink
2fa5e71d5e fix(tvix/store): restore v1alpha reflection endpoint
tonic-reflection 0.12.x moved from the v1alpha to v1 of the reflection
protocol.

However, most clients, like Postman, Kreya and evans don't support that
one yet.

Bump tonic-reflection to 0.12.2, which re-introduces v1alpha support
alongside the v1 version of it, registering both services.

This fixes the example documented in tvix/store/README.md, it was
previously failing as evans couldn't find the v1alpha reflection
service.

See https://github.com/hyperium/tonic/pull/1888 for details.

Change-Id: I55438877317f82dc39face13afeb9594cda07a4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12353
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-27 10:34:43 +00:00
Brian Olsen
228bf55646 fix(tvix/nix-compat): Feature flag code only used in async
When only the wire feature was enabled two methods were unused and so
would faild `cargo check --no-default-features --features wire`.

This feature flags those two methods on async feature since that is the
only place they are used.

Change-Id: I6ec18a670e3c6e3ecee8d1417c99f1a5084e0ae7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12346
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
2024-08-25 15:05:53 +00:00
Brian Olsen
ced05a2bb6 feat(tvix/nix-compat-derive): Add deriver for NixDeserialize
This adds a nix-compat-derive derive crate that implements a deriver
for NixDeserialize implementations. This is to reduce the amount of
code needed to implement deserialization for all the types used by
the Nix daemon protocol.

Change-Id: I484724b550e8a1d5e9adad9555d9dc1374ae95c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12022
Autosubmit: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-25 15:05:50 +00:00
Brian Olsen
9af6920478 feat(nix-compat): Add NixDeserialize and NixRead traits
Add a trait for deserializing a type from a daemon worker connection.
This adds the NixDeserialize trait which is kind of like the serde
Deserialize trait in that individual types are meant to implement it
and it can potentially be derived in the future.

The NixDeserialize trait takes something that implements NixRead as
input so that you can among other things mock the reader.

Change-Id: Ibb59e3562dfc822652f7d18039f00a1c0d422997
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11990
Autosubmit: Brian Olsen <me@griff.name>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-08-25 15:05:50 +00:00
Florian Klink
a742fafdbf chore(tvix/tools/turbofetch): bump deps
h2:
 - https://rustsec.org/advisories/RUSTSEC-2024-0332.html
 - https://rustsec.org/advisories/RUSTSEC-2024-0003.html

mio:
 - https://rustsec.org/advisories/RUSTSEC-2024-0019.html

shlex:
 - https://rustsec.org/advisories/RUSTSEC-2024-0006.html

zerocopy:
 - https://rustsec.org/advisories/RUSTSEC-2024-0006.html

Change-Id: I7ce3cab410b9c2b6c28474ca81c3fa5b5283f3fd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12341
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-24 18:08:08 +00:00
Florian Klink
25f8ddca6d chore(tvix/tools/crunch-v2): bump deps
h2:
 - https://rustsec.org/advisories/RUSTSEC-2024-0332.html
 - https://rustsec.org/advisories/RUSTSEC-2024-0003.html

mio:
 - https://rustsec.org/advisories/RUSTSEC-2024-0019.html

shlex:
 - https://rustsec.org/advisories/RUSTSEC-2024-0006.html

zerocopy:
 - https://rustsec.org/advisories/RUSTSEC-2024-0006.html

Change-Id: Ia5f6f96378d5f7e77e178dc054f47bda8b766a30
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12340
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-24 18:08:08 +00:00
Florian Klink
478a746a92 chore(tvix/tools/weave): bump mio
https: //rustsec.org/advisories/RUSTSEC-2024-0019.html
Change-Id: I03543105b35da6407a4930893b1514b379a89fbc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12339
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-24 18:08:08 +00:00
Ilan Joselevich
e7ff04b5f1 feat(tvix/tools/weave): Add crate2nix-check
Change-Id: I4e02f56678e6efa6e2656b2f7398b6ad92581c57
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12303
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-23 19:55:10 +00:00
Ilan Joselevich
beb6713983 feat(tvix/tools/turbofetch): Add crate2nix-check
Change-Id: Ie615d824032c85af6f8f40419bc22d7e3f38bb4c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12302
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
2024-08-23 19:55:10 +00:00
Ilan Joselevich
424b1ab5ad feat(tvix/tools/narinfo2parquet): Add crate2nix-check
Change-Id: I22bc6e7711e5e2c0a76d4eea4dd6edaf724e4cef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12301
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-23 19:55:10 +00:00
Ilan Joselevich
a8d2d6e97e feat(tvix/tools/crunch-v2): Add crate2nix-check
Change-Id: I922caae2ec3f87a7f3a5dcc8e9d366f0e71ce900
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12300
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
2024-08-23 19:55:10 +00:00
Ilan Joselevich
6da55dc1a6 feat(tvix/utils): Add mkCrate2nixCheck
This adds a function which can be used across the monorepo to create a
an extra CI step that checks whether the Cargo.nix file is up-to-date.

Change-Id: Idb8298b29ddc2ca5dff1facb1b9ed86a236ee66d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12227
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-23 17:03:02 +00:00
Florian Klink
a4ebc8da7c feat(tvix/nar-bridge): send content-type headers
This prevents browsers from treating NARInfo and nix-cache-info paths as
a separate "Download", but just show it in plaintext.

Change-Id: If99abe20ef1d24e4fa86c055160861ca47aa81ce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12267
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-23 15:02:14 +00:00
Florian Klink
35d5811eec feat(nix-compat): add constants for mime types
This adds the MIME types Nix uses when uploading files to a S3 bucket /
HTTP endpoint. It ignores them when retrieving.

Change-Id: I75fa96d5e53349f0470bbe8cbba93f961fbd5f05
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12266
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-08-23 15:02:14 +00:00
Ben Webb
565c0fd24c fix(tvix/castore): u32 -> u64 in DirectoryError::SizeOverflow message
Fix a discrepancy in the error message for DirectoryError::SizeOverflow.

The message indicates that the SizeOverflow error occurs when total size
exceeds u32::MAX, but that's not true. All size fields within the
castore's internal Directory ADT are u64, and the SizeOverflow error is
only returned after a call to the checked_add implementation on u64.

See tvix/castore/nodes/directory.rs +111
and tvix/castore/nodes/directory.rs +88
as of this commit.

Change-Id: I74d161ea8927362e1cb601ba163489aa96fb91b1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12259
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-21 22:05:57 +00:00
Florian Klink
e03ea11bad feat(nix-compat/nix_http): init parse_nar[info]_str
This moves the URL component parsing code we had in nar-bridge to
nix-compat.

We change the function signature to return an Option, not a
Result<_, StatusCode>.

This allows returning more appropriate error codes, as we can
ok_or(…) at the callsite, which we now do: on an upload to an
invalid path, we now return "unauthorized", while on a GET/HEAD, we
return "not found".

This also adds support to parse compression suffixes. While not
supported in nar-bridge, other users of nix-compat might very well want
to parse these paths.

Also fix the error message when parsing NAR urls, it mentioned 32, not
52, which is a copypasta error from the narinfo URL parsing code.

Change-Id: Id1be9a8044814b54ce68b125c52dfe933c9c4f74
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12260
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-08-21 09:40:32 +00:00
Florian Klink
2beabe968c refactor(nix-compat/store_path): make StorePath generic on S
Similar to how cl/12253 already did this for `Signature`, we apply the
same logic to `StorePath`.

`StorePathRef<'a>'` is now a `StorePath<&'a str>`, and there's less
redundant code for the two different implementation.

`.as_ref()` returns a `StorePathRef<'_>`, `.to_owned()` gives a
`StorePath<String>` (for now).

I briefly thought about only publicly exporting `StorePath<String>`
as `StorePath`, but the diff is not too large and this will make it
easier to gradually introduce more flexibility in which store paths to
accept.

Also, remove some silliness in `StorePath::from_absolute_path_full`,
which now doesn't allocate anymore.

Change-Id: Ife8843857a1a0a3a99177ca997649fd45b8198e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12258
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-20 15:14:27 +00:00
Florian Klink
413135b925 refactor(nix-compat/aterm): update function names a bit
Don't call functions bstr or str when they return BString or String,
it's confusing.

Rename them to `string` and `bytes`. We might be able to generalize over
this being BString or Vec<u8> later.

Change-Id: I8198551ed3ba1cfc479bf7e3cbbc13a426faf4c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12257
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-20 15:14:27 +00:00
Ryan Lahfa
a4e40d1dfa feat(tvix/nix-compat): entry locator in listing structures
This implements a simple DFS locator in listing structures.

It is interoperable with the Rust standard library paths, we also build our
own errors to restrict path values to reasonable secure defaults, e.g.
relative paths with no `..` component.

Tests are added for this new feature for a positive and a negative
check.

In addition, a path validation test was added. The Windows-style prefix
is gated on the Windows platform as UNIX does not parse `C:\\` as a
`Component::Prefix(_)` but as a `Component::Normal(_)`.

Change-Id: Iae2a80bebd8138e41af94aa7d09f2842c3c5a786
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12255
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-08-19 22:32:09 +00:00
Florian Klink
a259613c76 feat(nix-compat/narinfo/signature): generalize name field
Requiring `name` to be a `&str` means it'll get annoying to pass around
`Signature`, but being able to pass them around in an owned fashion is
kinda a requirement for a stronger typed `PathInfo` struct, where we
want to have full ownership.

Rework the `Signature` struct to become generic over the type of the
`name` field. This means, it becomes possible to have owned versions
of it.

We don't want to impose `String` or `SmolStr` for example, but want to
leave it up to the nix-compat user to decide.

Provide a type alias for the existing `&str` variant (`SignatureRef`),
and use it where we previously used the non-generic `Signature` one.

Add some tests to ensure it's possible to *use* `Signature` with both
`String` and `SmolStr` (but only pull in `smol_str` as dev dependency
for the tests).

Also, add some more docstrings, these were a bit sparse.

Change-Id: I3f75691498c6bda9cd072d2d9dac83c4f6c57287
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12253
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-08-19 19:46:34 +00:00
Ryan Lahfa
7612cb4c31 feat(tvix/nix-compat): add a basic listing deserializer
.ls files are useful to seek in a NAR without parsing it entirely.

The responsibility of validating the files is on the caller.

Change-Id: I5d1da28b5479c38f20ca5babe60e362a2217c9ea
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12196
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-19 19:18:42 +00:00
Ilan Joselevich
73e16c1855 feat(tvix): drop usage of sparseTree in favor of lib.sourceByRegex
We can avoid depending on things outside //tvix by just using a similar
util from nixpkgs.

Change-Id: I9ea3e1f0a8a059ea10caaec173569ba9f316aec6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12247
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-19 16:56:19 +00:00