GitHub does not allow downloads by clients without a User-Agent set.
This changes sets the client User-Agent to "tvix".
Change-Id: I829b37e63bcedb5ea785b55eff5b10175f5caaa6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12845
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This operation is particularly used when invoking the following
nix commands:
```
nix-store --add-fixed some-path
nix-store --add-fixed --recursive some-path
```
Change-Id: I0f9b129c838c00e10415881f1e6e0d7bc1d7a3a6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12800
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Validation is done inside ingest_nar_and_hash and
is used by Fetch::NAR and the nar-bridge.
Change-Id: I7e2be4cc13d2447035f1e5a444f44b62339988bf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12836
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
We previously didn't handle compression in nar-bridge, and left it up to
a fronting reverse proxy.
However, at least nginx with http2 enabled pins each connection to a
single core, causing compression to be limited by the throughput of a
single CPU.
Change-Id: Ia11c2ff5c012192b25eb8ad05dae5542a2d2f777
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12834
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: Jörg Thalheim <joerg@thalheim.io>
With more implementing NixDeserialize, this error message changed.
Fixtures were regenerated by running:
```
TRYBUILD=overwrite cargo test -p nix-compat-derive-tests --all-features
```
Unfortunately, it's not possible to loop this into CI, as trybuild
invokes cargo during the build.
Change-Id: Ia0ab07d0907d21366845fe06e01df9fb1fe3e7cb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12831
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
We got some double-quoted strings at narinfo_str, and it didn't align
well with our other field names.
Change-Id: I5c08786d2c4435542bf39ff44b9d4ada5400550d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12826
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
We should be able to distinguish tvix-store and nar-bridge.
Change-Id: I616c8e0c1ce2dbacab92975582dd36141b673aa2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12824
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
This provides some global HTTP statistics.
Change-Id: I8bd3e034123154a49d94720b0c8d0c3babde5ae3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12557
Reviewed-by: Jonas Chevalier <zimbatm@zimbatm.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This creates and registers a global meter provider, which uses the same
mechanism to get notified of flushes.
Change-Id: I856a67f0b282d494de3b2c2a1b79c06ae8ffe252
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12556
Reviewed-by: Jonas Chevalier <zimbatm@zimbatm.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This can be used to transitively enable the `xp-composition-cli` feature
flag in the `tvix-store` crate, which is unnecessarily hard to do with
crate2nix.
Change-Id: I3f7c505a3fd505561c9a7b2f063b6540532cdfd4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12809
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
When sending nars over the wire to the nix-daemon, nix protocol versions
>= 1.23 use this framing protocol.
This change implements an AsyncRead for this protocol, to be used in
AddToStoreNar and any other operations when necessary.
Change-Id: I5f7972fe1c9ea145780bf449321bd3efeb833d18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12814
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
When sending nars over the wire to the nix-daemon, nix protocol versions
1.21 to 1.23 use this framing protocol.
This change implements an AsyncRead for this protocol, to be used in
AddToStoreNar and any other operations when necessary.
Change-Id: I571f1adbb2343c14c98503d1a2c12eea4c783ec9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12813
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
The TracingHandle::flush function allowed a user to pass in their
own (optional) oneshot::Sender<()> to get notified once the flush is
completed, but that's making things unnecessary complicated.
By simply having the flush() function await the flush, we make its
interface more intuitive, and callsites (only inside tvix-tracing itself
so far) simpler.
We can also remove the Option around the oneshot::Sender entirely, as we
now always call it with that.
For some more clarity, we can remove the channel from the struct fields
entirely if otlp support isn't compiled in.
Change-Id: I0870b9e8e88c6be6494a9c201c1c70b87e0f0810
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12801
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Marijan Petričević <marijan.petricevic94@gmail.com>
Tested-by: BuildkiteCI
* //3p/overlays/tvl: build nixos-option with latest Nix version (2.24)
as is required now. It would be nice to avoid this somehow to prevent
NixOS machines in depot having to carry around two versions of Nix.
Maybe we can at least use a statically linked nixos-option?
* //3p/{gerrit,gerrit_plugins}: update deps hash
* //tvix/eval: adjust our nixVersion “user agent” so that it'll pass the
new 2.3.17 minimum version nixpkgs prescribes (to check for zstd
support when substituting from the binary cache).
Change-Id: I4eb715afdc3dbb857340839f08ce86612aa7f117
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12805
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
These are required to support certain nix's local-overlay store
operations, it's safer to return empty results for these
operations than failing with "operation not implemented" errors.
Change-Id: Ic9b69d75dd52af5a826bfb6a8b283b082a0f6bcf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12766
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This change contains 2 improvements to the tvix-store copy command:
1. Allows reading the reference graph from stdin, using `-` argument
2. Supports json representation produced by `nix path-info --json`
command.
In general it makes is easier and faster to import arbitrary closures
from an existing nix store with e.g the following command:
```
nix path-info ./result --json --closure-size --recursive | \
jq -s '{closure: add}' | \
tvix-store copy -
```
Change-Id: Id6eea2993da233ecfbdc186f1a8c37735b686264
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12765
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
By manually writing out the async function, and moving the owned String
we can avoid for S to be Sync in the NixSerialize for StorePath<S> impl.
Co-Authored-By: edef@edef.eu
Change-Id: I8427b38d2bd61de2818088728cbad79cda69d17d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12764
Reviewed-by: Vladimir Kryachko <v.kryachko@gmail.com>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
This describes the current composition system used for BlobService /
DirectoryService / PathInfoService, why it's hidden, how to expose it,
and adds some common examples to explain it.
Change-Id: I2ce7da40992cc988947c3e924499f8157c5e4937
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12749
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
Implement PartialEq/Eq ourselves instead of deriving, by proxying to
name.as_ref() (and digest of course).
Also implement Hash on our own, clippy doesn't like this to be derived,
while Eq/PartialEq is not.
Change-Id: Idbe289a23ba3bc8dabf893d4d8752792ae2778c3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12744
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
NixString doesn't contain a `HashSet<BuildReference>` anymore, there's a
more packed internal data structure. We don't need to be overly specific
in the docs, just say we expose an API.
Change-Id: I13380c49293f9a86d1916909fdfeefbe64d9024b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12755
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
This conversion was a bit too magic, and we can just use
`NixString::new_context_from` without having to worry about the
distinction between an empty context or no context, as
NixString::new_context_from already deals with that internally.
Change-Id: I3e5d57ecfa0f7456aa6c526863e49f2523afaec3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12754
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
This change includes only the basic nix handshake protocol handling and
sets up a client session. The only supported operation at this point is
SetOptions.
Additional operations will be implemented in subsequent cls.
Change-Id: I3eccd9e0ceb270c3865929543c702f1491768852
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12743
Autosubmit: Vladimir Kryachko <v.kryachko@gmail.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: Brian Olsen <me@griff.name>
This prevents the field from getting printed to stderr, apparently.
Change-Id: Ia9860e4ff37224003154db88ee5f83103060e626
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12756
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Allow this to be overridden via the CLI, also getting rid of the unsafe
because this doesn't need to be const anymore.
Change-Id: I5e51b52e42522a21f59ef69628b464477c0764d1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12753
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
When uploading a Store Path to a Nix HTTP Binary Cache, Nix first does a
HEAD request for $outhash.narinfo, and if that's not found, for
`{narhash}.nar[.compression_suffix]`.
If the NAR is already present, only the NARInfo is uploaded.
Even though we don't have a service allowing to globally look up from
NARHash to root node, `root_nodes` in `AppState` at least contains
recently uploaded NARHashes.
If we find it in there, we can prevent Nix unnecessarily uploading NARs
if the same contents have already been recently uploaded.
We also promote this key, chances are high Nix will subsequently upload
a NARInfo referring to this NARHash.
Change-Id: I34e3fd9b334b695abe945e64cd291e30f303c2a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12752
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Autosubmit: flokli <flokli@flokli.de>
This being in combinators makes it harder to find.
Change-Id: If7984bdbd43f164c670548639bb4846d859f6695
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12750
Reviewed-by: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
Extend the docstrings of `add_default_services`, and add one for
`addrs_to_configs` as well as the module-wide one at
`tvix_store::composition`.
Change-Id: Ie9b449988eb210cd65b19b174094bbe0c4af2fd6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12748
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
tvix-castore already supports composition without any additional feature
flags, the only thing that can be explicitly enabled is referring to
other stores via an anonymous url. Rename that feature flag to
"xp-composition-url-refs".
tvix-store effectively only controls the CLI surface, so rename this to
"xp-composition-cli".
The "store" in the feature name was dropped, as it's already apparent
from being in the tvix-[ca]store crate.
Change-Id: I1175dc6280cbba4cbcdfb7fd4b35fce713b45fc4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12747
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
This becomes the root of the composition. `default` implies we can
directly access anything else, which we cannot. `root` makes this more
understandable, and it's all internal only anyways.
Change-Id: I297511bc05a7c32c59510b9d192b40d1bd937b5f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12746
Reviewed-by: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
This groups most `wire` feature gated logic into a single module.
The nix_daemon module will be gated by a feature that adds
nix-compat-derive as a dependency.
All of this is a way to break the crate2nix dependency cycle between
nix-compat and nix-compat-derive(which depends on nix-compat for its
doctests).
Change-Id: I95938a6f280c11967371ff21f8b5a19e6d3d3805
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12761
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Currently it is not possible to distinguish between tracing of the same
*Service type whenever there are multiple of them. Now the instance_name
of ServiceBuilder is passed into the *Service and used in the existing
instrument as the `instance_name` field.
Places that did not already have a instance_name in its context use
`"default"`. In tests I used `"test"`.
Change-Id: Ia20bf2a7bb849a781e370d087ba7ddb3be79f654
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12739
Tested-by: BuildkiteCI
Autosubmit: Bob van der Linden <bobvanderlinden@gmail.com>
Reviewed-by: flokli <flokli@flokli.de>
This will be used by tvix-daemon to write tracing data into the active
client's connection socket.
Change-Id: I8889dd0a638e004ee2c8cb312946b029c9779313
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12734
Tested-by: BuildkiteCI
Autosubmit: Vladimir Kryachko <v.kryachko@gmail.com>
Reviewed-by: flokli <flokli@flokli.de>
This change implements the serialization part that is needed to
implement the nix daemon protocol. Previously was add deserialization
and derivers for that and this then adds the other part of that equation
so that you can write types that can then be read using deserialization.
Change-Id: I2917de634980a93822a4f5a8ad38897b9ce16d89
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12729
Autosubmit: Brian Olsen <me@griff.name>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This adds a span to default handling so that any error message
gives a more precise location as a hint for why the error occurred
instead of just pointing to the type.
Change-Id: I7bf6cf38e0284f9726d670ea50a94a6b1edd8a94
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12728
Reviewed-by: Vladimir Kryachko <v.kryachko@gmail.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Brian Olsen <me@griff.name>
Currently Value::Json is used in combination with VMRequest::ToJson to
recursively convert tvix Value to serde_json::Value. This functionality
is used in builtins.toJSON as well as derivation __structuredAttrs.
Both Value::Json and VMRequest::ToJson were removed in this commit.
Related functionality in vm.rs is also removed: vm.rs does not know
about JSON anymore.
Recursively converting to serde_json now happens without going through
the VM.
Thrown errors that are part of the value of toJSON are now directly
propagated as ErrorKind, were-as previously there was a split between
CatchableErrorKind and ErrorKind, where eventually CatchableErrorKind
would be converted to ErrorKind::Catchable.
Change-Id: I066f064926c491e4c087a984f07af43d19124cfe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12732
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Some references to nixrs had sneaked in while writing the original
CL so this just removes those.
Change-Id: I635adbd2de46c7c1f31e1d449396253a78e1b762
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12727
Reviewed-by: Vladimir Kryachko <v.kryachko@gmail.com>
Autosubmit: Brian Olsen <me@griff.name>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
When running tests for buildservice on MacOS it would fail
because of the oci test. I also got some clippy warnings on
MacOS because of disabled code.
Change-Id: I235739fa4270a4ef46e54d3e2b8cbb55eb20bdda
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12726
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Before this commit only deriving of tuple enums where tested.
This adds a few tests to cover the other types of enums.
Change-Id: I62f311e4db98cab84bd4ac164d50051e9aab0d4d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12725
Autosubmit: Brian Olsen <me@griff.name>
Reviewed-by: Vladimir Kryachko <v.kryachko@gmail.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
