Commit graph

20321 commits

Author SHA1 Message Date
Profpatsch
3068cfd866 refactor(users/Profpatsch/whatcd-resolver): handler response struct
Change-Id: I3224ccc5ccaea9eb26c60a65f048ca64024a7b9b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11641
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
2024-05-13 14:45:13 +00:00
Profpatsch
0a9b5efac9 fix(users/Profpatsch/MonadPostgres): fix query logging msg
Change-Id: Ic499a59002fa4860067589ff2d187795e6013066
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11640
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2024-05-13 14:40:09 +00:00
Profpatsch
15054715d6 fix(users/Profpatsch/whatcd-resolver): always use good html renderer
The pretty renderer would add extra newlines, making the html
different.

Change-Id: I28496cbec61d4c9c63f657a499a1990f891949ee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11639
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
2024-05-13 14:38:38 +00:00
Profpatsch
58156967b8 fix(users/Profpatsch/whatcd-resolver): prepend / for route logging
Change-Id: I46e77540cb47ae42dc44ec950a3ad97cb408974c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11638
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2024-05-13 14:38:38 +00:00
Profpatsch
5d57ca2dc7 fix(users/Profpatsch/whatcd-resolver): use threaded runtime (again?)
Might have made a mistake when refactoring the cabal file.

Change-Id: Ic45ac9bf0272784ef77a2dfcc59ce2dbd421113c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11637
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2024-05-13 14:38:38 +00:00
Florian Klink
ed2c0be920 feat(tvix/store/pathinfo/*): add more instrumentation
Add instrumentation to the get() and put() implementations of all
PathInfoService.

Use the nixbase32 representation of the digest, not the base64 one.

Change-Id: Iea79bbd363bf20f23985e877c6fc1793bbee6a7e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11630
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-12 22:03:15 +00:00
Florian Klink
8324913045 chore(users/flokli/kb/dilemma): bump qmk to latest releasewt
Change-Id: Ia0b4784384b0f1dda0f849f14e5619274dec1dd3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11631
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-12 11:36:59 +00:00
Florian Klink
b22fc8cd9a refactor(tvix/castore/error): drop From<PoisonError<T>> for Error
We don't produce these erorrs anymore, no need to provide a conversion
to it.

Change-Id: I37933e436ad15c5d90b3ac270c4ef5742980513d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11614
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-05-11 15:09:33 +00:00
Florian Klink
25d97c5b55 refactor(tvix/castore/directoryservice/memory): use tokio RwLock
We don't want to block here, and this also means there's no poisoning to
deal with.

Change-Id: Ic375571970c48beace0005ae2c012135086a4d67
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11613
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-11 15:09:33 +00:00
Florian Klink
84114cf02c refactor(tvix/castore/blobservice/memory): use parking_lot RwLock
This one doesn't require us to deal with poisoning, is upgradeable and
the right thing to use when locking access to data, not IO resources.

Change-Id: I78634953a73404500d28f51f1d93a87e215c8149
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11612
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-11 15:09:33 +00:00
Florian Klink
96b8b1a205 refactor(tvix/store/pathinfo/memory): tokio RwLock, improve list()
We don't want to use the std::sync::RwLock here, as it blocks.

This also means we don't need to deal with the error cases anymore.

The list() implementation is updated to use try_stream, which means we
can now avoid collecting everything into a Vec before returning from it.

Change-Id: I9057dcc410dc553e6b1be3f20d5ee830569e8218
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11611
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-11 15:07:31 +00:00
Florian Klink
11850249c8 docs(tvix/docs): async nar reader is done
Change-Id: I4e8fd03d521b7d30904f84a6afcf842f1505ec7b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11628
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-05-11 15:07:30 +00:00
Florian Klink
728ceda8a6 docs(tvix/todo): add section on [ca]store errors
Change-Id: I832eed8abb62b7e13972ed58649a96e31b17f033
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11610
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-11 14:59:57 +00:00
Florian Klink
fd49dfa6ca docs(tvix/todo): update test suite section
nix_oracle is quite misleading, and fooled me into thinking this
validates all .exp files in the repo to match Nix output (which it
doesn't, that's done by verify-lang-tests).

Also, the whole test suite structure thing can use a bit more
explanation.

Change-Id: I2fadcc871843143270ad2ed9ac98de8287280e6f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11609
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-11 14:59:57 +00:00
Florian Klink
bdf4df1ac7 chore(tvix/narinfo2parquet): bump dependencies
Get rid of the RUSTSEC warnings.

Also upgrade tokio to drop the Sized requirement, similar to cl/11608.

Change-Id: Idc6fece23c79eb30fd1dfc5fe64fa2e4c08cd412
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11624
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-05-11 14:18:30 +00:00
Florian Klink
d9c2a1994c chore(tvix): update dependencies
Get rid of the RUSTSEC warnings.

Change-Id: I949f5173eb3e807f2de497f2eb94493f7a331e18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11623
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-05-11 13:55:21 +00:00
Florian Klink
fe20ba5ffc feat(tvix/store): add LruPathInfoService
This provides an implementation of PathInfoService storing PathInfo in
memory up to a certain capacity, then evicting these that have been used
the least recently.

Change-Id: I9d738687caf4f181a957f72245f26b92832313cd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11622
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-05-11 13:55:21 +00:00
Florian Klink
a49c32ef42 refactor(tvix/store/pathinfo/sled): drop {blob,directory}_service
These are not used anymore.

Change-Id: I9c348391c9600e9319f171faf3eda7175ebf7076
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11621
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-11 13:55:21 +00:00
Florian Klink
03af6ab725 refactor(tvix/store/pathinfo/memory): drop {blob,directory}_service
These are not used anymore.

Change-Id: I6c16b4d80ddaabcb75fec3ea3e32b923b7719485
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11620
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-05-11 13:55:21 +00:00
Florian Klink
14766cfe1d refactor(tvix/store): drop calculate_nar from PathInfoService
This shouldn't be part of the PathInfoService trait.

Pretty much none of the PathInfoServices do implement it, and requiring
them to implement it means they also cannot make use of this calculation
already being done by other PathInfoServices.

Move it out into its own NarCalculationService trait, defined somewhere
at tvix_store::nar, and have everyone who wants to trigger nar
calculation use nar_calculation_service directly, which now is an
additional field in TvixStoreIO for example.

It being moved outside the PathInfoService trait doesn't prohibit
specific implementations to implement it (like the GRPC client for the
`PathInfoService` does.

This is currently wired together in a bit of a hacky fashion - as of
now, everything uses the naive implementation that traverses blob and
directoryservice, rather than composing it properly. I want to leave
that up to a later CL, dealing with other parts of store composition
too.

Change-Id: I18d07ea4301d4a07651b8218bc5fe95e4e307208
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11619
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-11 13:33:59 +00:00
Florian Klink
944a781354 fix(tvix/store/pathinfo/sled): use spawn_blocking
This does IO, which might take a longer amount of time than what we want
to be blocking the normal executor.

Use spawn_blocking instead. I didn't add it for the constructors, as we
only call these once.

Change-Id: I9a1063099bac9582ca9681043c58c1edc780c5ff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11618
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-05-11 13:33:59 +00:00
Florian Klink
717081ae37 fix(tvix/castore/directory/sled): use spawn_blocking
This does IO, which might take a longer amount of time than what we want
to be blocking the normal executor.

Use spawn_blocking instead. I didn't add it for the constructors, as we
only call these once.

Change-Id: I96231fcff8d10abe90cafde25a099a2db6ea9414
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11617
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-05-11 13:33:59 +00:00
Florian Klink
ebb7b32a2c chore(tvix/castore/blob): remove sled implementation
This never did any chunking, and sled (rightfully) performs really bad
if values get too large.

We switched the default to using the objectstore backend with the local
filesystem a while ago, no need to keep this footgun around anymore.

Change-Id: I2c12672f2ea6a22e40d0cbf9161560baddd73d4a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11616
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-11 13:28:51 +00:00
Florian Klink
73c848a18f refactor(tvix/store/pathinfo/grpc): simplify pingpong
We already have the same code in make_grpc_path_info_service_client.

Change-Id: Ibcd60831af8a061a2c3bb2f960f52a43d06cf6fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11615
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-11 13:28:51 +00:00
Florian Klink
2cdf8346a9 fix(users/flokli/keyboards/*): fix flash command
`qmk flash` disappeared from the `qmk` command itself.

See `qmk_cli/script_qmk.py`:

```
This program can be run from anywhere, with or without a qmk_firmware
repository. If a qmk_firmware repository can be located we will use that
to augment our available subcommands.
```

By pointing QMK_HOME to our firmware sources, we can make it re-appear.

Change-Id: I5348fff546e864773063fb1f712bf2abd198972e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11627
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
2024-05-11 12:36:01 +00:00
Adam Joseph
37ddddc2e3 feat(users/amjoseph): init
Change-Id: Icfc73faa951b2ad041f2cecdae0614a3b6b82426
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11625
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
2024-05-11 08:45:13 +00:00
Aspen Smith
cff903caff feat(tvix/cli): Use the same TvixStoreIO for the full runtime
This specifically allows subsequent fetch calls in the REPL to avoid
fetching the same path twice (because they share the same
PathInfoService).

Change-Id: Ieda089080174913a8c014f13d6852cac599a7e17
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11484
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
2024-05-09 13:57:02 +00:00
edef
31d73cd443 refactor(nix-compat/nar/reader): reuse prev_name allocation
We reuse the prev_name allocation for Entry, instead of allocating and
returning a separate Vec.

We encode the `prev_name: None` case as an empty vector, since we don't
allow empty names anyway, and the sorting is equivalent.

Change-Id: I975b37ff873805f5ff099bc82128706891052247
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11607
Reviewed-by: Brian Olsen <me@griff.name>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-08 15:30:46 +00:00
edef
17a7dac94f feat(nix-compat/wire/bytes): read bytes into an existing buffer
For small bytestrings (like NAR names), we can read into a preallocated
fixed-size buffer, instead of allocating a Vec every time.

Change-Id: Id8da9e9cea99c814361230c0ec02606b731c79a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11606
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-08 15:30:46 +00:00
Vincent Ambo
b68ef475ee feat(tools/when): support more timestamps with better merging logic
Adds a FieldSet type which defines which parts of a timestamp to merge into
another in a new `mergeTime` function.

With this two timestamps can be merged granularly, enabling more queries like
`Sep 03 18:00` to return useful results (e.g., in this case, in the current year
and location).

Change-Id: I8db161608c5f68c8f9c61de8c6fa46067eced39b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11601
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-05-08 13:26:34 +00:00
Vincent Ambo
4bac58e10a feat(tools/when): try to parse input as one segment first
Before getting clever about durations & stuff, try to parse the whole thing at
once. This is useful for plain timestamps that contain spaces, like `2024-01-01
15:00:01` for example.

Change-Id: I50b0ee8488c153b4e6db75abaea409d55c0b92d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11600
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-05-08 13:26:34 +00:00
Vincent Ambo
51ce40c599 feat(tazjin/nixos): install when everywhere
Change-Id: I49f201250cbcbba98e0641c2868e1cb2230d6b3b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11599
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2024-05-08 13:26:34 +00:00
Vincent Ambo
00135d1c28 feat(tools/when): simple time-conversion tool
I often need unix timestamps, or have unix timestamps, or need timestamps at
certain relative offsets etc.

This adds a tool called `when` which can do all of this. It has basically no
user interface, you just call it with a time query and it does it if it
understands what you meant. This will get smarter over time as I find more uses.

Example:

    ~> when yesterday 5PM
    Local: Mon 06 May 2024 at 17:00:00 MSK
    UTC:   2024-05-06T14:00:00Z
    UNIX:  1715004000

It supports all kinds of queries already, see the usage for details.

Change-Id: I694ffef7608586acfb1ff8010ac0fac4d9951e2e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11598
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
2024-05-08 13:26:34 +00:00
edef
368a11ee0a chore(tvix/tools/crunch-v2): upgrade tokio
This drops the Sized requirement on AsyncReadExt.

Change-Id: I2a89c708922ca528cb493aefd0613f477f648e83
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11608
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2024-05-08 10:19:17 +00:00
edef
1eedb88939 refactor(nix-compat/wire/bytes): style fixes
Change-Id: I65c3c43df83e0c364a4b7f1f3054c5b676bd07d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11605
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-08 08:48:59 +00:00
edef
ca10a8726f fix(nix-compat/store_path): use Box<str>
We don't actually build up names in place here, so we don't need a
capacity field. Saves 8 bytes.

Change-Id: Icb01b45561e28fd525b726612f56d4640bc834c7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11604
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-08 08:48:59 +00:00
edef
51e0f78e93 feat(nix-compat/wire/bytes/reader): support buffered reading
If our underlying reader supports AsyncBufRead, then we can too.

Change-Id: If4b948c983400ca591c1c475bbcf7dc00d562040
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11545
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-08 06:03:37 +00:00
edef
ebad318ab3 fix(nix-compat/wire/bytes/reader): always read trailer tag data
Even if we have an aligned (or zero) size (and thus have no padding),
we may still have a non-padding trailer.

Since an exact read of only the user data bytes must always read the
trailer bytes as well, this has to happen for aligned sizes too.

For zero-sized trailers, we solve this by reading immediately, since no
read is required of the user at all.

user_len thus becomes a NonZeroU64, and the definition of "body data"
is adjusted to always exclude the final block.

Change-Id: I2c6bb51270fb22ad72a65076a3468673ab56ef68
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11597
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-08 04:44:01 +00:00
edef
0472b55b20 fix(nix-compat/wire/bytes/reader): drop allow(dead_code)
We're using this in the NAR reader now.

Change-Id: I28f17b1ccedd62ffcaf2fa32b517f16bcd036d94
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11603
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-08 04:07:46 +00:00
edef
d8a6cc862d refactor(nix-compat/wire/bytes/reader): drop prev_filled
We specifically structured the code this way to allow using
`this.filled` as-is, so we should use it.

Change-Id: I7e11bddceb4d5f37b1dd4c453a9d53b85fc1f6c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11602
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-05-08 04:07:46 +00:00
Connor Brewster
da9bc274f3 refactor(tvix): remove usage of async-recursion
Rust 1.77 supports async recursion as long as there is some form of
indirection (ie. `Box::pin`). This removes the need to use the
async-recursion crate.

Change-Id: Ic9613ab7f32016f0103032a861edff92e2fb8b41
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11596
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-05-06 16:05:09 +00:00
Florian Klink
01a4a2399c fix(tvix/store/pathinfo/nix_http): update error message
Autosubmit was too fast and submitted this before my push went through.

Flagged in https://cl.tvl.fyi/c/depot/+/11580/comment/40a56824_7be73334/

Change-Id: I1f835ae60c2c6fd2db6654c1b1c71d90bee141af
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11595
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-05-06 15:35:24 +00:00
Florian Klink
aaf258f61e feat(tvix/store): use async nar reader for ingest_nar
Rename read_nar to ingest_nar, and have it use the async nar reader
version, and the ingest_entries machinery.

This means we can now drop all code dealing with manually assembling
castore nodes.

Update our consumer, NixHTTPPathInfoService to use the new API.
As we now accept an AsyncRead, we don't need to do any blocking here
anymore, and can use the same async-compression crate as in the fetching
logic (and support some more compression formats out of the box).

Change-Id: I8646d20bd8603f8da47b5c84bc9e4ac236eb7f1a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11580
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-06 15:24:17 +00:00
Florian Klink
bc92f4188e test(tvix-castore/import): add tests for ingest_entries
Change-Id: Ia7906533868fd948509419e0d64b64582575a7fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11591
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-06 15:11:39 +00:00
Florian Klink
4aff40fff8 fix(tvix/castore/import): assert end of stream
Once we break out with the root node, there may be no more elements in
the stream.

Change-Id: I6f5fc5662095aa2b2a56bcad506d25520d9ad00c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11592
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-05-06 15:11:39 +00:00
Florian Klink
75f2a1f97d fix(tvix/castore/import): deal with entry.path() not having a parent
We got away with not properly dealing with this for the archive case,
where everything is contained inside a toplevel dir, but NARs can encode
a single file/symlink.

Properly break if the IngestionEntry path has the ROOT as parent, and
only create filling directories in the other case.

Change-Id: Ib378d0d1040de7c3fe310912a0b0488c55afee83
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11590
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-06 15:11:39 +00:00
Florian Klink
281bd46a43 feat(tvix-castore/import) have IngestionEntry.path() return &Path
There's no need for this to be a &PathBuf.

Change-Id: I2d4126d57cfd8ddaad5dd327943b70b83d45c749
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11589
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-05 14:54:19 +00:00
edef
08feea4817 feat(nix-compat/nar/reader): async support
This is a first cut at the async NAR reader, with some rough edges.
Poisoning is left unimplemented for now, pending future work.

Change-Id: Ifaafe0581a5e0e165a13357b909fb441f7bd8bab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11524
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-04 21:45:39 +00:00
edef
343e176bec feat(nix-compat/wire/bytes/reader): parametrise on trailer tag
This allows using BytesReader with a custom tag, eg the closing parens
for the NAR reader.

No public constructor is provided for custom-tagged readers, since this
feature isn't public API.

Change-Id: I82e73d064edc4b6783ead1d6fe46a5b35f45c844
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11543
Reviewed-by: Brian Olsen <me@griff.name>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-05-04 21:45:39 +00:00
Florian Klink
ba00f0c695 refactor(tvix/*store): use DS: DirectoryService
We implement DirectoryService for Arc<DirectoryService> and
Box<DirectoryService>, this is sufficient.

Change-Id: I0a5a81cbc4782764406b5bca57f908ace6090737
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11586
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-05-04 21:27:26 +00:00