This is a slightly less annoying version of `join`, which does not
consume self. It's more consistent with HashSet::extend().
Change-Id: Ifd0872da36fe8e7b2aa6948674cb8e4023abe9d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11703
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
In order to be compatible with the nix XML generator, it’s easier to
generate the XML directly, instead of going through a library which we
have to bend to do what we need.
Removes dependency on `xml-rs`, which came with a full XML parser that
we didn’t use. Only takes a tiny bit of code for the XML escaping,
somewhat simplified.
I add a little escaping value, to make sure we have the same behaviour
as nix proper.
Interestingly enough, we never need to escape XML attribute names,
because the `builtins.toXML` format encodes user-defined values as
attribute keys only. So we only escape attribute values.
Fixes: https://b.tvl.fyi/issues/399
Change-Id: If4d407d324864b3bb9aa3160e2ec6889f7727127
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11697
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
Currently all blobs are uploaded serially when ingesting NARs. If a NAR
contains many, small blobs, ingestion may become slow if there is a lot
of round-trip latency to the blob service.
This makes the NAR ingester use the ConcurrentBlobUploader which allows
for buffering small blobs in memory so they can be uploaded concurrently
to the blob service without blocking further deserialization.
Change-Id: I093a73770232df12d9a11e5d901b99c08505c3cb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11694
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
The archive ingester has a mechanism for concurrently uploading small
blobs to the blob service in order to hide round trip latency with the
blob service when ingesting many small blobs.
Other ingestion sources like NARs also need a similar mechanism, this
extracts the concurrent blob uploading mechanism into its own struct to
make it more reusable.
Change-Id: I05020419ff4b9ad5829fbfb5cd08d36db983b8c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11693
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Since cl/…, a PathInfoService doesn't need to implement `calculate_nar`
anymore, so most of them don't actually have a handle to a
{Blob,Directory}Service anymore.
This means, we can simplify the construction of them for test cases
a lot.
Change-Id: I100e9e1c9b00a049b4d6136c57aad4cdb04461c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11691
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Have this return the same values as builtins.unsafeGetAttrsPos, rather
than returning a CatchableErrorKind, which crashes the CLI if it bubbles
up.
The environment we're in doesn't allow emitting a warning, as we don't
have `co` in scope, but that's probably OK as a stopgap solution.
Alternative to cl/11665.
Change-Id: I5b2c2530842547c93b6533ed9601ee9b2923b1bf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11685
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This adds rough notes documenting the history of the Nix
daemon protocol, how logging works as well as begins
documenting inputs and outputs for all operations.
Change-Id: Id24a9a658c3e4e7c350ca1e4622f63ed96ccef5a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11594
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Using remove_node messed up the extraction of nodes from the graph. Use
into_nodes_edges() instead, to remove the nodes without cloning.
Change-Id: Id76c7935d082d6f26192cc3cd490483594f1d1e2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11684
Tested-by: BuildkiteCI
Autosubmit: yuka <yuka@yuka.dev>
Reviewed-by: flokli <flokli@flokli.de>
Drop the (unused) feature flags here, and get rid of some crate
dependencies.
Change-Id: Id64852b498725467d56abb12b548301cfba6a760
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11678
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This has already been dropped for tvix-store, drop it here as well.
Change-Id: Ib3aa37bbe9fd4c498b5ba1213f7d922d0c64ffc7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11677
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
This gives multi-listener support, as well as listening on named socket
FDs.
Change-Id: I92b441f854e2faf80074463d0ca6bdc23cbd890a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11663
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
Autosubmit: flokli <flokli@flokli.de>
This aligns the tonic version we use directly in Tvix to the one pulled
in by the OTLP stack.
Change-Id: I658528c8dabb7cd6948f1207ddcdef1984e82037
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11666
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
Make some space for the rust implementation.
Change-Id: I924dc1657be10abe5a11951c3b9de50bae06db19
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11662
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: yuka <yuka@yuka.dev>
This allows querying two PathInfoService implementations sequentially,
and inserts into the "near" one if it's not there yet.
There is no negative cache, and put / listing is not implemented (for
now).
Change-Id: I24c3d0e0c3c2f0524a6cc7b2f3cbc33eb20cf92b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11636
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
We can just use the `BoxStream` directly, or a `once` with the single
`Directory`.
In the recursive case, we also did not properly close the channel after
the first error.
Change-Id: Ifad56d307fc7861107b6d3cffd28d35631d526e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11635
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Necessary to directly use this in the GRPC DirectoryService wrapper
directly.
Change-Id: Ic6a0038a40dc30071d145af5035345fcd93288ae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11634
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Use try_stream! rather than stream!, and a bit more map_err and ok_err
to make things a bit more concise. Once we have proper error types here,
and impl Froms, a lot of the error mapping would disappear entirely.
Change-Id: I5240a6b0ff7818b94c151322774242b2c142e33b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11633
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Replace the loop manually driving the iterator with a for … in, and some
of the match with ok_or_else.
Change-Id: I6d7b3ef1bf1c7aa128bd6adef09390b54f79479e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11632
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
… in Cargo.toml.
This gets an imperative `cargo clippy` run to pick up that config,
so `-A clippy::blocks_in_conditions` doesn't need to be explicitly
specified anymore.
Change-Id: I32b6cc50c77c22cba0d816d0db508c2f94b2c383
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11659
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
The pool library would always take out the most recently used perl
resource again, and since that is the one that we just spawned, we’d
be back at square one. Instead, we try to find an older one (or up to
200ms old) and use that instead, because that should be the one with
the fastest response time.
Okay, that was enough bullshit lol.
Change-Id: I6b999e682d02ab03206a9d1b707edf16daa04a0d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11657
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
Instead of opening a separate span, we just add events for start and
end of formatting.
Change-Id: I26f6792dfdcd23c01cff415fa0f436d6a22d93fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11655
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
It does chip of the init overhead of like 50–100ms, even though the
formatting still takes quite some time (up to 200ms for more complex
expressions).
Maybe we need some simplistic formatter in the future that just splits
on parens? It’s not an easy problem …
Change-Id: I2ce951e6b3c2dc56294b1bdab913480727b50f0b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11654
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
When showing the best torrents table, we want to be able to filter for
“only downloaded”.
Change-Id: Ibfe9212f3d422d305f858fbef023ee985c1183d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11650
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
The pretty renderer would add extra newlines, making the html
different.
Change-Id: I28496cbec61d4c9c63f657a499a1990f891949ee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11639
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
Might have made a mistake when refactoring the cabal file.
Change-Id: Ic45ac9bf0272784ef77a2dfcc59ce2dbd421113c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11637
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Add instrumentation to the get() and put() implementations of all
PathInfoService.
Use the nixbase32 representation of the digest, not the base64 one.
Change-Id: Iea79bbd363bf20f23985e877c6fc1793bbee6a7e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11630
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We don't produce these erorrs anymore, no need to provide a conversion
to it.
Change-Id: I37933e436ad15c5d90b3ac270c4ef5742980513d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11614
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
We don't want to block here, and this also means there's no poisoning to
deal with.
Change-Id: Ic375571970c48beace0005ae2c012135086a4d67
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11613
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
This one doesn't require us to deal with poisoning, is upgradeable and
the right thing to use when locking access to data, not IO resources.
Change-Id: I78634953a73404500d28f51f1d93a87e215c8149
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11612
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
