This makes it easier to see what's being communicated, by setting
`RUST_LOG=nix_daemon=debug`.
Change-Id: Ifcd89ff6c5c1727e97569e29d4f63993cc37ed8e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12884
Tested-by: BuildkiteCI
Reviewed-by: Vladimir Kryachko <v.kryachko@gmail.com>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: flokli <flokli@flokli.de>
rush is like GNU parallel, but in Go, and most importantly, without
the annoying citation.
Change-Id: Id4737e6dee43037c1c2bc814738416410a603e07
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12887
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
mkBootTest is not limited to only listing files, it can also be used (is
used) to boot init.
Change-Id: Iaa0d2b5bad3be856aa8a0172450efe166620ba41
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12882
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
This makes it a bit less verbose.
Change-Id: I41835f43628d7a10855b9d89816e8d20eb7546d2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12881
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: Domen Kožar <domen@cachix.org>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
There's more than one result in there.
Change-Id: I5d519db51fda050ed293bfb52215a643882e0116
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12879
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Move back to a proper release containing the opentelemetry bump. Also
enable the `axum` feature, which will give us a per-route accounting.
Change-Id: Icdf4dc73588ef45b6596b320c14d9f44946327b3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12865
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Use the crate name in the user-agent, similar to tvix-[ca]store.
Change-Id: I10527fb1f29006dbfd8630d8bb1f00d7905efdd3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12851
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Vladimir Kryachko <v.kryachko@gmail.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
GitHub does not allow downloads by clients without a User-Agent set.
This changes sets the client User-Agent to "tvix".
Change-Id: I829b37e63bcedb5ea785b55eff5b10175f5caaa6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12845
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This operation is particularly used when invoking the following
nix commands:
```
nix-store --add-fixed some-path
nix-store --add-fixed --recursive some-path
```
Change-Id: I0f9b129c838c00e10415881f1e6e0d7bc1d7a3a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12800
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Validation is done inside ingest_nar_and_hash and
is used by Fetch::NAR and the nar-bridge.
Change-Id: I7e2be4cc13d2447035f1e5a444f44b62339988bf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12836
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
We previously didn't handle compression in nar-bridge, and left it up to
a fronting reverse proxy.
However, at least nginx with http2 enabled pins each connection to a
single core, causing compression to be limited by the throughput of a
single CPU.
Change-Id: Ia11c2ff5c012192b25eb8ad05dae5542a2d2f777
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12834
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: Jörg Thalheim <joerg@thalheim.io>
With more implementing NixDeserialize, this error message changed.
Fixtures were regenerated by running:
```
TRYBUILD=overwrite cargo test -p nix-compat-derive-tests --all-features
```
Unfortunately, it's not possible to loop this into CI, as trybuild
invokes cargo during the build.
Change-Id: Ia0ab07d0907d21366845fe06e01df9fb1fe3e7cb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12831
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
We got some double-quoted strings at narinfo_str, and it didn't align
well with our other field names.
Change-Id: I5c08786d2c4435542bf39ff44b9d4ada5400550d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12826
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
We should be able to distinguish tvix-store and nar-bridge.
Change-Id: I616c8e0c1ce2dbacab92975582dd36141b673aa2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12824
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
This provides some global HTTP statistics.
Change-Id: I8bd3e034123154a49d94720b0c8d0c3babde5ae3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12557
Reviewed-by: Jonas Chevalier <zimbatm@zimbatm.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This creates and registers a global meter provider, which uses the same
mechanism to get notified of flushes.
Change-Id: I856a67f0b282d494de3b2c2a1b79c06ae8ffe252
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12556
Reviewed-by: Jonas Chevalier <zimbatm@zimbatm.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This can be used to transitively enable the `xp-composition-cli` feature
flag in the `tvix-store` crate, which is unnecessarily hard to do with
crate2nix.
Change-Id: I3f7c505a3fd505561c9a7b2f063b6540532cdfd4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12809
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
When sending nars over the wire to the nix-daemon, nix protocol versions
>= 1.23 use this framing protocol.
This change implements an AsyncRead for this protocol, to be used in
AddToStoreNar and any other operations when necessary.
Change-Id: I5f7972fe1c9ea145780bf449321bd3efeb833d18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12814
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
When sending nars over the wire to the nix-daemon, nix protocol versions
1.21 to 1.23 use this framing protocol.
This change implements an AsyncRead for this protocol, to be used in
AddToStoreNar and any other operations when necessary.
Change-Id: I571f1adbb2343c14c98503d1a2c12eea4c783ec9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12813
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
The TracingHandle::flush function allowed a user to pass in their
own (optional) oneshot::Sender<()> to get notified once the flush is
completed, but that's making things unnecessary complicated.
By simply having the flush() function await the flush, we make its
interface more intuitive, and callsites (only inside tvix-tracing itself
so far) simpler.
We can also remove the Option around the oneshot::Sender entirely, as we
now always call it with that.
For some more clarity, we can remove the channel from the struct fields
entirely if otlp support isn't compiled in.
Change-Id: I0870b9e8e88c6be6494a9c201c1c70b87e0f0810
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12801
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Marijan Petričević <marijan.petricevic94@gmail.com>
Tested-by: BuildkiteCI
* //3p/overlays/tvl: build nixos-option with latest Nix version (2.24)
as is required now. It would be nice to avoid this somehow to prevent
NixOS machines in depot having to carry around two versions of Nix.
Maybe we can at least use a statically linked nixos-option?
* //3p/{gerrit,gerrit_plugins}: update deps hash
* //tvix/eval: adjust our nixVersion “user agent” so that it'll pass the
new 2.3.17 minimum version nixpkgs prescribes (to check for zstd
support when substituting from the binary cache).
Change-Id: I4eb715afdc3dbb857340839f08ce86612aa7f117
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12805
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
These are required to support certain nix's local-overlay store
operations, it's safer to return empty results for these
operations than failing with "operation not implemented" errors.
Change-Id: Ic9b69d75dd52af5a826bfb6a8b283b082a0f6bcf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12766
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This change contains 2 improvements to the tvix-store copy command:
1. Allows reading the reference graph from stdin, using `-` argument
2. Supports json representation produced by `nix path-info --json`
command.
In general it makes is easier and faster to import arbitrary closures
from an existing nix store with e.g the following command:
```
nix path-info ./result --json --closure-size --recursive | \
jq -s '{closure: add}' | \
tvix-store copy -
```
Change-Id: Id6eea2993da233ecfbdc186f1a8c37735b686264
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12765
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
By manually writing out the async function, and moving the owned String
we can avoid for S to be Sync in the NixSerialize for StorePath<S> impl.
Co-Authored-By: edef@edef.eu
Change-Id: I8427b38d2bd61de2818088728cbad79cda69d17d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12764
Reviewed-by: Vladimir Kryachko <v.kryachko@gmail.com>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
This describes the current composition system used for BlobService /
DirectoryService / PathInfoService, why it's hidden, how to expose it,
and adds some common examples to explain it.
Change-Id: I2ce7da40992cc988947c3e924499f8157c5e4937
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12749
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
Implement PartialEq/Eq ourselves instead of deriving, by proxying to
name.as_ref() (and digest of course).
Also implement Hash on our own, clippy doesn't like this to be derived,
while Eq/PartialEq is not.
Change-Id: Idbe289a23ba3bc8dabf893d4d8752792ae2778c3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12744
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
NixString doesn't contain a `HashSet<BuildReference>` anymore, there's a
more packed internal data structure. We don't need to be overly specific
in the docs, just say we expose an API.
Change-Id: I13380c49293f9a86d1916909fdfeefbe64d9024b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12755
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
This conversion was a bit too magic, and we can just use
`NixString::new_context_from` without having to worry about the
distinction between an empty context or no context, as
NixString::new_context_from already deals with that internally.
Change-Id: I3e5d57ecfa0f7456aa6c526863e49f2523afaec3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12754
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
This change includes only the basic nix handshake protocol handling and
sets up a client session. The only supported operation at this point is
SetOptions.
Additional operations will be implemented in subsequent cls.
Change-Id: I3eccd9e0ceb270c3865929543c702f1491768852
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12743
Autosubmit: Vladimir Kryachko <v.kryachko@gmail.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: Brian Olsen <me@griff.name>
This prevents the field from getting printed to stderr, apparently.
Change-Id: Ia9860e4ff37224003154db88ee5f83103060e626
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12756
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Allow this to be overridden via the CLI, also getting rid of the unsafe
because this doesn't need to be const anymore.
Change-Id: I5e51b52e42522a21f59ef69628b464477c0764d1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12753
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
When uploading a Store Path to a Nix HTTP Binary Cache, Nix first does a
HEAD request for $outhash.narinfo, and if that's not found, for
`{narhash}.nar[.compression_suffix]`.
If the NAR is already present, only the NARInfo is uploaded.
Even though we don't have a service allowing to globally look up from
NARHash to root node, `root_nodes` in `AppState` at least contains
recently uploaded NARHashes.
If we find it in there, we can prevent Nix unnecessarily uploading NARs
if the same contents have already been recently uploaded.
We also promote this key, chances are high Nix will subsequently upload
a NARInfo referring to this NARHash.
Change-Id: I34e3fd9b334b695abe945e64cd291e30f303c2a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12752
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Autosubmit: flokli <flokli@flokli.de>
This being in combinators makes it harder to find.
Change-Id: If7984bdbd43f164c670548639bb4846d859f6695
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12750
Reviewed-by: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
Extend the docstrings of `add_default_services`, and add one for
`addrs_to_configs` as well as the module-wide one at
`tvix_store::composition`.
Change-Id: Ie9b449988eb210cd65b19b174094bbe0c4af2fd6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12748
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
tvix-castore already supports composition without any additional feature
flags, the only thing that can be explicitly enabled is referring to
other stores via an anonymous url. Rename that feature flag to
"xp-composition-url-refs".
tvix-store effectively only controls the CLI surface, so rename this to
"xp-composition-cli".
The "store" in the feature name was dropped, as it's already apparent
from being in the tvix-[ca]store crate.
Change-Id: I1175dc6280cbba4cbcdfb7fd4b35fce713b45fc4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12747
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
This becomes the root of the composition. `default` implies we can
directly access anything else, which we cannot. `root` makes this more
understandable, and it's all internal only anyways.
Change-Id: I297511bc05a7c32c59510b9d192b40d1bd937b5f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12746
Reviewed-by: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
This groups most `wire` feature gated logic into a single module.
The nix_daemon module will be gated by a feature that adds
nix-compat-derive as a dependency.
All of this is a way to break the crate2nix dependency cycle between
nix-compat and nix-compat-derive(which depends on nix-compat for its
doctests).
Change-Id: I95938a6f280c11967371ff21f8b5a19e6d3d3805
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12761
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Currently it is not possible to distinguish between tracing of the same
*Service type whenever there are multiple of them. Now the instance_name
of ServiceBuilder is passed into the *Service and used in the existing
instrument as the `instance_name` field.
Places that did not already have a instance_name in its context use
`"default"`. In tests I used `"test"`.
Change-Id: Ia20bf2a7bb849a781e370d087ba7ddb3be79f654
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12739
Tested-by: BuildkiteCI
Autosubmit: Bob van der Linden <bobvanderlinden@gmail.com>
Reviewed-by: flokli <flokli@flokli.de>
