Since https://github.com/curl/curl/commit/9c845be2797e20475
(presumably), libcurl will overwrite our previously set request
method to POST if we set .post_field_size(…). The fix is to use
the proper option for PUT/upload, .in_filesize(…). While we're
at it, switch to using .upload(…) instead of the deprecated
.put(…) which should be the same for HTTP.
Change-Id: I393c1a02c70d5b99dff5901cd6e9d9434f68c15b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11132
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Add a new AsyncRead wrapper, DecompressedReader, that wraps an
underlying AsyncRead, but sniffs the magic bytes at the start of the
stream to determine which compression format is being used out of the
three that are supported by builtins.fetchTarball, and switches to the
correct decompression algorithm adapter dynamically.
This will be used in the implementation of builtins.fetchTarball
Change-Id: I892a4683d5c93e67d4c173f3d21199bdc6605922
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11019
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Implement the fetchurl builtin, and lay the groundwork for implementing
the fetchTarball builtin (which works very similarly, and is implemented
using almost the same code in C++ nix).
An overview of how this works:
1. First, we check if the store path that *would* result from the
download already exists in the store - if it does, we just return
that
2. If we need to download the URL, TvixStoreIO has an `http_client:
reqwest::Client` field now which we use to make the request
3. As we're downloading the blob, we hash the data incrementally into a
SHA256 hasher
4. We compare the hash against the expected hash (if any) and bail out
if it doesn't match
5. Finally, we put the blob in the store and return the store path
Since the logic is very similar, this commit also implements a *chunk*
of `fetchTarball` (though the actual implementation will likely include
a refactor to some of the code reuse here).
The main thing that's missing here is caching of downloaded blobs when
fetchurl is called without a hash - I've opened b/381 to track the TODO
there.
Adding the `SSL_CERT_FILE` here is necessary to teach reqwest how to
load it during tests - see 1c16dee20 (feat(tvix/store): use reqwests'
rustls-native-roots feature, 2024-03-03) for more info.
Change-Id: I83c4abbc7c0c3bfe92461917e23d6d3430fbf137
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11017
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: aspen <root@gws.fyi>
This is like orderless but a million times better (no need for spaces!)
Change-Id: Id8d2382e1d83792959580bcbe807cdd9aebdde88
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11130
Reviewed-by: aspen <root@gws.fyi>
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Similar to YAML, but with more weird keys and with sections.
Change-Id: I94d69d2f0e53366855ccb5d99a432c6a0361a910
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11126
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Fairly similar to YAML, except with only one way of naming keys and
arrays.
Change-Id: Ic2fa539466d69af55b36401bb8c03f2c7edcd91d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11112
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Supports namespaces, functions, fields (including trailing return type
syntax) and so on.
One notable issue about this is that the tree-sitter parser for C++
returns the node *following* point if point is on whitespace, which
means that at the top-level of a namespace the crumbs will often show
the *next* function.
I'm against adding workarounds for stuff like that, so I'll just keep
it as is.
Change-Id: If7e71525c4e86e128157dd4eb17c130297ed1e0a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11109
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Adds a macro that provides a more convenient syntax for defining new
languages for treecrumbs (and that can also be easily used outside of
the core treecrumbs code).
This macro automatically takes care of compiling tree-sitter queries
for reuse.
Change-Id: I6a0b892a083c3f243e8b8f0e1c865a9a8a1a5cf5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11107
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Makes crumbs work correctly for weird stuff like:
```yaml
foo:
- { "bar": [ { baz: 1 }]}
```
Would be nice to have tests for this at some point.
Change-Id: I034e83318435404ef8613e439313dbb08865f228
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11105
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This adds a separate owners file that can be imported in projects with
FSF copyright assignment, so that only people with copyright paperwork
on file can commit (or verify commits before approving).
Change-Id: Ifbe07792572b9d6bf5f7d47c41135892bddea46b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11108
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Adds a new 'treecrumbs' Emacs package that allows displaying fast,
tree-sitter based breadcrumbs. The initial version only supports
YAML (which is what I needed this for!).
The package is documented, please go read the code and comments for
more information.
I assign copyright over this package to the FSF, paperwork is on file.
This means that I can only accept contributions from people with the
right paperwork. If this package ends up being useful, I intend to
eventually submit it to GNU Emacs.
Change-Id: If1297a080f63f402f83b6cc57fa7c970d32d3695
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11104
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
This function is inspired by BQN's [⍉] though it is much less elegant
since Nix lacks multi-dimensional arrays. I thought this would be useful
to to avoid multiple `map`s over a single list if we want to return
multiple, separate values from it:
transpose (builtins.map (x: [ (calcA x) (calcB x) ]) myList)
# => [ [ (calcA a) … ] [ (calcB a) … ] ]
While this is quite elegant, it turns out that it is faster to write out
multiple maps:
[ (builtins.map calcA myList) (builtins.map calcB myList) ]
[⍉]: https://mlochbaum.github.io/BQN/doc/transpose.html
Change-Id: Ic333c33af38ab03573b215c9696d75caf2ee18e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11113
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
`#tvl` is the general TVL community channel, `#tvix-dev` is dedicated to
Tvix development discussion.
Change-Id: If899d9ae70dffb4acf6a41ded54f80a1f0551c5d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11110
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
request only contains the outer metadata wrapping, and that's not too
interesting:
> Request { metadata: MetadataMap { headers: {"content-type":
> "application/grpc", "user-agent": "grpc-go/1.60.1", "te": "trailers",
> "grpc-accept-encoding": "gzip"} }, message: Streaming, extensions:
> Extensions }
Drop these fields for now, and rely on the underlying implementations to
add instrumentation for the application-specific fields.
Also, ensure we handle all error cases properly, and log them. We
don't use `err` from instrument, as that'd also log an error on
`Status::not_found`.
Change-Id: Id1b983cb8b059c148c8a376f8802a1d28c59ba97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11103
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
This only contains the outer metadata wrapping, and that's not too interesting:
> Request { metadata: MetadataMap { headers: {"content-type":
> "application/grpc", "user-agent": "grpc-go/1.60.1", "te": "trailers",
> "grpc-accept-encoding": "gzip"} }, message: Streaming, extensions:
> Extensions }
Drop these fields for now, and rely on the underlying implementations to
add instrumentation for the application-specific fields.
Clean up the error logging a bit.
Change-Id: Ife1090ed411766a61e1fa60fd4c9570f38de1e98
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11102
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
This only contains the outer metadata wrapping, and that's not too interesting:
> Request { metadata: MetadataMap { headers: {"content-type":
> "application/grpc", "user-agent": "grpc-go/1.60.1", "te": "trailers",
> "grpc-accept-encoding": "gzip"} }, message: Streaming, extensions:
> Extensions }
Drop these fields for now, and rely on the underlying implementations to
add instrumentation for the application-specific fields.
Log errors in some places where we didn't so far.
Change-Id: Ia68d6c526987d3716be62a0809195401cf28512b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11101
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Otherwise this causes conflicts when adding a more recent zstd version
to another crate.
Change-Id: I02d1b1a37f18711c0969cb04e761e967fc4655dc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11100
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
In principle we don't want to build any (later) pipeline target during
pipeline evaluation insofar they appear in extraSteps. For this reason,
we have the needsOutput mechanism which prevents the parent target of an
extraStep from being built in 🦙.
Unfortunately, this mechanism is not general purpose enough, as we use
other (i.e. non parent) targets from depot in extraSteps. As a
consequence, kind of expensive builds need to happen during pipeline
construction at the moment. The solution is to use the fact that the
command script we want to run is exposed via the readTree interface to
depot and build the script proper only when the extra step is executed.
To facilitate this, some prerequisite changes need to be made:
- We need to use a symlink different to result in case needsOutput is
true which needs support in mkBuildCommand. We also need to avoid this
symlink being picked up by git, as many extra steps check whether the
tree is dirty or not. (Is there a way to have it outside the depot
tree?)
- Since we rely on the build command printing a single store path we
store in $command_script, we need to avoid it printing two paths
in cases where nix-store(1) is used (nix-store(1) prints the symlink
and readlink(1) would print the store path in a separate line).
Future work would be to remove/deprecate the needsOutput mechanism:
After this change the parent target wouldn't be built right away even if
it appeared in the script via string interpolation. Thus we could,
instead of expecting the target being available as `./result`, make our
extra steps nix-ier.
Change-Id: Idd2e88a865eadabe229ce1e05406e8cc4cb63f94
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10850
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
- Set LIBGIT2_NO_VENDOR so that libgit2-sys won't start silently
building the vendored libgit2 version.
- Update to match version distributed by nixpkgs.
- Update related dependencies:
- git2 (reverse dependency)
- bitflags (dependency) which necessitates updating:
- inotify*
- epoll
- libc
Change-Id: I1d9059a16a44b5e02d04f034c8f02bbfdc55b6ac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10845
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
This introduces a NixOS configuration using config.nix as a NixOS
module, and also checks the firmware is present in the location it's
expected to.
Change-Id: I3ec4333d73fe1b28e10c589c1cf351c59372bb99
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11099
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Adds SVG versions of the logo in two colours (using the .ru flag
colours, and using the standard Nix colours).
The flag colours are difficult to use in some contexts because of the
white background of the top, but flokli says the Nix colour version is
"too similar to NixCon". Not yet decided what should go on to swag etc
...
Change-Id: I1e1655e2c7ed4ed6113ddf6b5df76157de5e32cb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11093
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
For everything using reqwest here during test cases, we also need to
set SSL_CERT_FILE.
Change-Id: If8aeda65f3d75cb9ac5c9bc64e37a0cb7dffc17c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11092
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
implement From<&nix_compat::nixhash::CAHash> for nar_info::ca::Hash
individually, and make
From<&nix_compat::nixhash::CAHash> for nar_info::Ca a small wrapper that
uses it, as well as the .hash().digest_as_bytes() for the digest.
Change-Id: I7e9b6edd1e3f149eb270faf2928cd846d74e77ad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11071
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
If there's an unexpected test failure, print it out, rather than just
saying something is false even though it should be true.
Use .expect() for this, which displays the error if it failed.
We can't use expect_err(), as our stores are not display'able, so use an
assertion with a message there.
Change-Id: I2d88861d979d107edc0717fbdb3cdac9a6bfc5e4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11091
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Reviewed-by: flokli <flokli@flokli.de>
This was true while we didn't do string contexts, but drvPath should not
differ anymore.
Change-Id: I58e6ed439519bd4d68259d1577946f3c65110dc4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11074
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Some of the docs are still outdated (like architecture and drv path
inconsistencies).
Change-Id: I7a6afceb008ef4cd19a764dd6c637b39fa842a2e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11072
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
The start/end offsets are not necessarily coterminous with the underlying
values array, so even if the stride is fixed, we still we need to slice
the chunks down to match the start/end offsets.
This bug shouldn't affect the correctness of any existing code, since
we're always working with unsliced arrays read directly from Parquet.
Change-Id: I2f7ddc4e66d4d3b2317a44bd436a35bff36bac79
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11081
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
- agenix has not been updated (waiting for
https://github.com/ryantm/agenix/pull/241).
- libgit2_1_5 can be removed (no longer used by cargo-audit).
Change-Id: I96c6a1a4175dc4f2a32b9b2e4ed71caa826a9c42
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11077
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: flokli <flokli@flokli.de>
This makes reqwest honor `SSL_CERT_FILE` - previously it was using the
chain bundled in webpki-roots.
`object_store` pulls in `reqwest` with this feature, and the cargo
solver will enable that feature globally as soon as we pull it in, as it
assumes features are additive.
This requires setting `SSL_CERT_FILE` when running tests, otherwise
they'll fail with the unhelpful "NotFound" error.
This was quite some fun to debug, why adding `object_store` to
tvix-castore suddenly made tvix-store tests fail!
Change-Id: I64fc82b4d994715480efdb1ffecb279716456ab9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11090
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
HashingReader wraps an existing AsyncRead, and allows querying for the
digest of all data read "through" it.
The hash function is configurable by type parameter, and we define
B3HashingReader.
Change-Id: Ic08142077566fc08836662218f5ec8c3aff80be5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11087
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This allows calling .into() to get a B3Digest.
Change-Id: I6e63b496413cd00d84acfcd15c7de0f64c79721f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11086
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
I haven’t thought about how to migrate this when the postgres version
changes, so for now we just pin to 14 and worry about it when we reach
EOL lol
Change-Id: Ib72ec7835c660db636a95fb5af8c9ce40897b1ce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11085
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
they switched to TOML & I had a typo in `scolling`
Change-Id: If68408162e7772fbd3b06944ffd9d51dd2134efa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11083
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
This allows peeking at the mode without matching on all enum kinds of
CAHash directly.
Change-Id: Icca147a876f7f4cff0bbaa067ac4dae40d05bee9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11069
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Currently CAHash only deserializes the hash in hex code while
the serializer outputs a nixbase32 hash. This means that you can't currently
deserialize what has been serialized.
This change makes deserialize support any digest format (so hex, nixbase32
and base64) as well as flattens the deserialize code and error handling.
It also implements serde methods of HashAlgo directly using Display and TryFrom
implementations because otherwise these would get serialized as eg. Sha256 instead
of sha256 which also broke CAHash serialize/deserialize.
Change-Id: I1941a72eaec741e4956292adaaf0115b97f260ba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11082
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This doesn't return the digest, but the internal NixHash.
To get the digest, you use `ca_hash.hash().digest_as_bytes()`
(as done in tvix/nix-compat/src/derivation/write.rs).
Change-Id: Ib07918dec63ae42ddf8c8d3f1d10510f9c4df255
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11070
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
The public-consumable thing here is ChunkedReader, not ChunkedBlob.
ChunkedBlob is a helper that can be used to get a new AsyncRead, but
not AsyncSeek. It is used internally by ChunkedReader whenever the
client seeks.
Make this more obvious, by extending the documentation, and putting
ChunkedReader at the top of this file.
Also make ChunkedBlob and its methods private, and give ChunkedReader a
more useful constructor (from_chunks, instead of from_chunked_blob).
Change-Id: I2399867591df923faa73927b924e7c116ad98dc0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11079
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
