Commit graph

19045 commits

Author SHA1 Message Date
Florian Klink
e13c86d916 chore(tvix): re-enable {ca,}store-protos-go
This couldn't be enabled before until go get was able to fetch the
modules again.

Change-Id: I82db7d22e497661ea2a0c8a45c0c865fcc0a1f18
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9406
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 17:36:05 +00:00
Florian Klink
2aa0c70245 chore(ops/modules/www/code.tvl.fyi): add missing go get redirect
This was missing in cl/9370.

Change-Id: I02048b0e65d1192e9e300160bb8f78fe30a70da1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9405
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-09-22 17:36:05 +00:00
Florian Klink
80a02f96ce chore(tvix/castore): remove dependency on anyhow
We don't use this here, except in a test, where we don't really use
it either.

Change-Id: Ia6c45fccf663fe328942b1e1a2cd1c3a1b7f9ae5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9376
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-09-22 12:51:21 +00:00
Florian Klink
32f41458c0 refactor(tvix): move castore into tvix-castore crate
This splits the pure content-addressed layers from tvix-store into a
`castore` crate, and only leaves PathInfo related things, as well as the
CLI entrypoint in the tvix-store crate.

Notable changes:
 - `fixtures` and `utils` had to be moved out of the `test` cfg, so they
   can be imported from tvix-store.
 - Some ad-hoc fixtures in the test were moved to proper fixtures in the
   same step.
 - The protos are now created by a (more static) recipe in the protos/
   directory.

The (now two) golang targets are commented out, as it's not possible to
update them properly in the same CL. This will be done by a followup CL
once this is merged (and whitby deployed)

Bug: https://b.tvl.fyi/issues/301

Change-Id: I8d675d4bf1fb697eb7d479747c1b1e3635718107
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9370
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 12:51:21 +00:00
Florian Klink
d8ef0cfb4a docs(tvix/castore): rename traverse_to -> descend_to
With the move of this code out into castore it has become apparent this
is a general descent inside the castore. Concerns like making sure the
whole Directory closure has been fetched/is fetched initially is nothing
this layer needs to worry about. We can handle this during substitution
of a new PathInfo, once there's store composition.

Closes b/270.

Change-Id: I661ed08e54bc81478e032cfb9abeb23e5b337fbe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9373
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 12:12:25 +00:00
Florian Klink
7c9a52e3ee chore(tvix): remove some unused crate dependencies
Change-Id: I31b0e7be3a2ebc268c6f4045ba282e8fb1c72a64
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9375
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 12:12:24 +00:00
Florian Klink
8f56e71cba feat(tvix/shell): add cargo-machete
This helps spotting unused crate dependencies.

It's preferred over cargo-udeps for the reasons described in
https://blog.benj.me/2022/04/27/cargo-machete/

Change-Id: Ie801004485858741f2fa5ae6f33bd0ddfc292cd0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9374
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-22 12:12:24 +00:00
Florian Klink
3d064f714c docs(tvix/store/pathinfosvc): fix copypasta error
This is not a simple get and put returning digest.

You currently can get by output hash, and a put gives you back the
(possibly modified) PathInfo message.

Change-Id: I43174d87a764417b8620c4d5bdac058cc25b0373
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9371
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-09-22 09:14:33 +00:00
Florian Klink
731996fbfe docs(tvix/eval): fix some broken docstr references
There's some more left, but they've been renamed/refactored out of
sight.

Change-Id: I41579dedc74342b4c5f8cb39d2995b5b0c90b0f4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9372
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2023-09-22 09:14:33 +00:00
Connor Brewster
37a348b4fa refactor(tvix/store): Asyncify PathInfoService and DirectoryService
We've decided to asyncify all of the services to reduce some of the
pains going back and for between sync<->async. The end goal will be for
all the tvix-store internals to be async and then expose a sync
interface for things like tvix eval io.

Change-Id: I97c71f8db1d05a38bd8f625df5087d565705d52d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9369
Autosubmit: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-09-21 17:58:23 +00:00
Connor Brewster
7e737fde34 refactor(tvix/store/fs): Separate FUSE and filesystem code
In prepration for adding virtiofs support, I thought it would make sense
to split out the filesystem implementation from FUSE itself.

The `fs` module holds the tvix-store filesystem implemetation and the
`fuse` module holds the code to spawn a FUSE daemon backed by multiple
threads.

Change-Id: I8c58447b8c3aa016a613068f8e7ec166554e237c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9343
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Connor Brewster <cbrewster@hey.com>
2023-09-20 14:27:53 +00:00
Connor Brewster
6b7c936bc5 refactor(tvix/store/fuse): Switch from fuser to fuse-backend-rs
This switches the FUSE implementation from fuser to fuse-backend-rs.
fuse-backend-rs is designed to work with both FUSE and virtiofs.
Virtiofs support will make it possible to plug the tvix-store into a
microvm and have `/nix/store` access without having to setup FUSE inside
the guest.

Additionally fuse-backend-rs has nice support for running multiple FUSE
threads and has some async support.

The goal of this commit is to mechanically switch over to
fuse-backend-rs with minimal changes. I did have to add some locks here
and there because fuse-backend-rs uses `&self` on all methods whereas
fuser uses `&mut self`. `&self` is required for concurrent access to the
FUSE server, so this makes sense.

We can consider switching to concurrent maps and use some other
techniques to reduce lock contention and critical section size.

Issue: https://b.tvl.fyi/issues/305

Change-Id: Icde5a58c6eef98f8984c1e04e980b756dfb76b47
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9341
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-09-20 14:21:32 +00:00
Vincent Ambo
237c0eb415 chore(3p/sources): bump channels and overlays (2023-09-17)
* 3p/cgit: import fixes for git v2.42.0

  Upstream patch: https://git.zx2c4.com/cgit/patch/?id=2f50b47c72cbc4270bbd12ae7f520486d5f42736

Change-Id: I7549f909b145becaccb983936dc5c3e17941122b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9353
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-09-19 21:24:07 +00:00
Vincent Ambo
b87af6ccaa chore(3p/overlays): remove unnecessary nixpkgs workarounds
These things were fixed upstream.

Change-Id: I027862aabd8239111c8e94aa5390af192386c4a1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9368
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: tazjin <tazjin@tvl.su>
2023-09-19 20:58:02 +00:00
Vincent Ambo
1c3b6341cf chore(users/wpcarro): drop 'exa' from NixOS systems
This is unmaintained and has been removed from nixpkgs. As an
alternative, 'eza' can be installed.

This is the last instance of 'exa' in depot, so the workaround from
the channel bump has been removed.

Change-Id: Id915260b6c969a6b2a8ecae49d3ce92285f09f71
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9330
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: wpcarro <wpcarro@gmail.com>
2023-09-19 20:38:27 +00:00
guangwu
3e5279aeff docs(nixery): occurrences typo
Change-Id: I3798e1c23d6b0580b99b14bb4aae1c7cfc81fb6e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9366
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-09-19 08:58:13 +00:00
Florian Klink
6c586bc2a7 feat(tvix/nar-bridge): graceful shutdown
This gives existing clients 30s to finish their requests after receiving
an interrupt.

Change-Id: Ia9b0e662fd1ffbbb6c2d03f3dd6548b13cf3d241
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9365
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-18 14:34:12 +00:00
Florian Klink
02aed32bf2 refactor(tvix/nar-bridge): rename contentReader to blobReader
More aligned with how it's called in other places

Change-Id: I759ac7ca3b5b69c1101d2d51a569d76c183a6330
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9362
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-09-18 14:02:28 +00:00
Florian Klink
a9c47ffc50 refactor(tvix/nar-bridge): don't buffer blob in memory
Create a pipe, pass the read end, and have a goroutine write to the
write end.

Change-Id: I301c273355705e60113b018e7e84b76972200e8c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9361
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-18 14:02:28 +00:00
Florian Klink
07af692ecb refactor(tvix/nar-bridge): simplify CLI interface
Only keep the `serve` subcommand, and make it appear at the root.
Introduce a --log-level argument, and be a bit less noisy in normal
operation.

Change-Id: I86b8abde1869a5c0c947508bcc29f845222aac09
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9360
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-09-18 14:02:28 +00:00
Florian Klink
dd7cc6ed68 fix(tvix/nar-bridge): don't log error on simple 404s
`nix copy` checks if NARs and NARInfo files are present, before
uploading them. That's not an error, but normal behaviour, so no need to
log with level info for these cases.

We only want to log if the error is not a 404, and log with Warn level.

Change-Id: I762de3b862d070a0f18bc62e324e94ca5c7c3693
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9359
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-09-18 14:02:28 +00:00
Florian Klink
749ab67216 feat(tvix/nar-bridge): buffer request body a bit more
Let's make sure we don't end up blocking a client too much when
inserting very small blobs.

Change-Id: I640dda92efae538c70d32a40e6e85a23e9749e20
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9358
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-18 14:02:28 +00:00
Florian Klink
0c031461c3 fix(tvix/nar-bridge): chunk blobs
Instead of creating one big BlobChunk containing all data, and creating
way too large proto messages, chunk blobs up to a reasonable (1MiB)
chunk size, and send them to the server like that.

Change-Id: Ia45a53956a6d7c0599cc59ac516ba37e9fb1b30e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9357
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-09-18 14:02:28 +00:00
Florian Klink
136b12ddd7 chore(tvix/nar-bridge): bump tvix/store/protos
Include the changes from cl/9351

Change-Id: Ie60c9dddcafaeee190439fa19fa7704917600fdb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9363
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-09-18 14:02:28 +00:00
Florian Klink
ca25acf11b chore(tvix/store/protos): drop unused fields for now
This wasn't removed yet, and no code is using/populating it so far.

It's confusing, let's update it to the state of things now, and re-
introduce it once we get there.

Change-Id: I68f5ba17a8eee604d8ccd82749da7c8be094cb99
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9351
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-09-18 10:33:30 +00:00
Florian Klink
bf2fe88a5c fix(tvix/store/fuse): fix executable bit
We were blindly returning 0o444 for all regular files, but regular files
with executable bit need to be 0o555.

This wasn't spotted because stat'ing executable files was not part of
the test suite, it's now added.

Change-Id: I04c69784053e7e43d838c01bb288f2df48f40b4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9345
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-18 10:33:30 +00:00
Florian Klink
da6cbb4a45 refactor(tvix/store/blobsvc): make BlobStore async
We previously kept the trait of a BlobService sync.

This however had some annoying consequences:

 - It became more and more complicated to track when we're in a context
   with an async runtime in the context or not, producing bugs like
   https://b.tvl.fyi/issues/304
 - The sync trait shielded away async clients from async worloads,
   requiring manual block_on code inside the gRPC client code, and
   spawn_blocking calls in consumers of the trait, even if they were
   async (like the gRPC server)
 - We had to write our own custom glue code (SyncReadIntoAsyncRead)
   to convert a sync io::Read into a tokio::io::AsyncRead, which already
   existed in tokio internally, but upstream ia hesitant to expose.

This now makes the BlobService trait async (via the async_trait macro,
like we already do in various gRPC parts), and replaces the sync readers
and writers with their async counterparts.

Tests interacting with a BlobService now need to have an async runtime
available, the easiest way for this is to mark the test functions
with the tokio::test macro, allowing us to directly .await in the test
function.

In places where we don't have an async runtime available from context
(like tvix-cli), we can pass one down explicitly.

Now that we don't provide a sync interface anymore, the (sync) FUSE
library now holds a pointer to a tokio runtime handle, and needs to at
least have 2 threads available when talking to a blob service (which is
why some of the tests now use the multi_thread flavor).

The FUSE tests got a bit more verbose, as we couldn't use the
setup_and_mount function accepting a callback anymore. We can hopefully
move some of the test fixture setup to rstest in the future to make this
less repetitive.

Co-Authored-By: Connor Brewster <cbrewster@hey.com>
Change-Id: Ia0501b606e32c852d0108de9c9016b21c94a3c05
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9329
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-18 10:33:30 +00:00
Profpatsch
3de9601764 feat(users/Profpatsch/whatcd-resolver): add executable
Change-Id: I28f5c26443b9a878f8f374af2c7935831d17ab12
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9355
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
2023-09-17 18:05:42 +00:00
Profpatsch
07b0dff209 refactor(users/Profpatsch/whatcd-resolver): rename main function
Change-Id: I8ce8ef1cc8a35358d20610be7c8bdadefd3aa30f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9354
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2023-09-17 18:05:42 +00:00
Connor Brewster
4ac6423b26 refactor(tvix/nar-bridge): Clean up directory popping loop
This change got lost in the rebases in cl/9348. There's unnecessary
`break`/`continues` that can be replaced by moving the conditional into
the for loop condition.

Change-Id: I559e21087630b05e483f768ab59f8067961a2eae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9352
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-09-17 13:55:22 +00:00
Connor Brewster
84aa07a736 fix(tvix/nar-bridge): Fix directory stack tracking
Previously, nar-bridge, had a couple of bugs with tracking the current
directory when traversing a NAR file.

The included test case looks like:
```
/ (dir)
/test (dir)
/test/tested (file)
/tested (file)
```

Previously, we would do a string prefix match between the current node
and the top of the directory stack to determine if the node is in the
directory. In this case `/test` is a substring of `/tested`; however,
`/tested` is not in the `/test` directory. The fix is to append a `/` to
the directory name when doing the prefix match, so `/test/` is not a
prefix of `/tested`.

Additionally, when popping the stack, we need to continuously pop the
stack until the new node is in the directory at the top of the stack
(stopping before we pop the root directory)

Example:
```
/ (dir)
/a (dir)
/a/b (dir)
/a/b/c (file)
/z (file)
```

Previously, `z` would end up in directory `/a` because we only the pop
the stack once.

The included test case requires both of these issues to be fixed for it
to pass, so I think it is sufficient.

Change-Id: I22f601babf04d39d85535ba7ad585d3970757211
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9348
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Connor Brewster <cbrewster@hey.com>
2023-09-17 13:34:00 +00:00
Florian Klink
5841047c38 feat(tvix/nar-bridge): increase timeouts
In some cases, Nix is not able to stream the NAR file fast enough. Bump the
timeouts for now. We might want to get a better understanding in what's
happening here long-term, and/or make the timeouts configurable.

Change-Id: Ieaa9c8f04bc73c6ce0679a058d07eaf87126634e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9340
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-09-17 13:24:53 +00:00
Florian Klink
0ecd10bf30 feat(tvix/nar-bridge): init
This provides a Nix HTTP Binary Cache interface in front of a tvix-store
that's reachable via gRPC.

TODOs:

 - remove import command, move serve up to toplevel. We have nix-copy-
   closure and tvix-store commands.
 - loop into CI. We should be able to fetch the protos as a third-party
   dependency.
 - Check if we can test nar-bridge slightly easier in an integration
   test.
 - Ensure we support connecting to unix sockets and grpc+http at least,
   using the same syntax as tvix-store.
 - Don't buffer the entire blob when rendering NAR

Co-Authored-By: Connor Brewster <cbrewster@hey.com>
Co-Authored-By: Márton Boros <martonboros@gmail.com>
Co-Authored-By: Vo Minh Thu <noteed@gmail.com>

Change-Id: I6064474e49dfe78cea67676957462d9f28658d4a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9339
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-09-17 13:24:53 +00:00
Florian Klink
683d3e0d2d chore(tvix): upgrade workspace.resolver to "2"
This shuts off the following warning:

```
warning: some crates are on edition 2021 which defaults to `resolver = "2"`, but virtual workspaces default to `resolver = "1"`
note: to keep the current resolver, specify `workspace.resolver = "1"` in the workspace root's manifest
note: to use the edition 2021 resolver, specify `workspace.resolver = "2"` in the workspace root's manifest
```

Stuff still seems to work with it, so no need to stick with "1".

See https://doc.rust-lang.org/edition-guide/rust-2021/default-cargo-resolver.html
for more details.

Change-Id: I6056d95cd5cb793f37ef843ed43009a27ad36367
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9342
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-09-17 13:14:03 +00:00
Florian Klink
51b8571c48 feat(tvix/store): improve error message formatting
WrongSorting(Vec<u8>) actually encodes the name of a node, so if we can,
we want this to display it as a string (and fall back to the base64
encoding if we can't).

Before:

> rpc error: code = InvalidArgument desc = directory b3:yfwNlpPm8MkrRvshGHfgMtKLoSDtX2pKliVuVWmUt5g= failed validation: [108, 111, 99, 97, 108, 101] is not sorted

After:

> rpc error: code = InvalidArgument desc = directory b3:yfwNlpPm8MkrRvshGHfgMtKLoSDtX2pKliVuVWmUt5g= failed validation: locale is not sorted

Change-Id: I68420c53a89cb1aa96e4bdce414366cebcb7915f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9350
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-09-17 13:14:03 +00:00
Florian Klink
54bb63dea3 chore(tvix/store): allow non_snake_case
Some of the tonic-generated types create new warnings.

Change-Id: Ib538bfd7d90d857d80a129fbbdf3a38c89cc04ab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9349
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-09-17 13:14:03 +00:00
Florian Klink
3816eaa5e9 feat(tvix/store): add simple integration test
This imports the docs folder into the tvix store, and ensures it comes
up with the same store path as Nix.

While we validate hashes in general through tvix-cli output path checks
already, it doesn't include the tvix-store CLI itself.

See https://cl.tvl.fyi/c/depot/+/9329/comment/339f0720_524f0104/
for context.

Change-Id: I239ce5b6a07cb962b242142ab716693359b8674c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9338
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-15 21:00:54 +00:00
Florian Klink
32f2f8b618 feat(tvix/store/import): print store path to stdout
This allows comparing the output in an integration test.

Change-Id: I8bb2254e18e90005a4f1b30fd47ef69642e3732e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9337
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2023-09-15 21:00:54 +00:00
Florian Klink
b52bb6e791 fix(tvix/store): log to stderr, not stdout
This leaves some space in stdout to provide actual meaningful output.

Also, rename print_node to log_node because that's what it does, it's
using the logger to log out more detailed info.

Change-Id: Ic64a6330dbfcdc63eb4198067a5c5e47b841b9a5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9336
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
2023-09-15 19:38:23 +00:00
Profpatsch
d111a0fda8 feat(users/Profpatsch/whatcd-resolver): misc improvements
* run on port 9092 (transmission runs on 9091)
* run postgres on port 5431 instead of 5432 (to not interfere)
* only search for albums for now
* correctly handle missing torrent file in SELECT

Change-Id: I20125f7731c9b80a9e8ea05b726adfb1244a24bc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9335
Reviewed-by: Profpatsch <mail@profpatsch.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2023-09-15 12:44:06 +00:00
Profpatsch
8cfe6bc99b feat(users/Profpatsch/whatcd-resolver): query all pages in search
If more than one page is returned by the search, query all of them.

Also add an ON CONFLICT clause in case the torrent group already
exists, to update it.

This function is getting a bit unwieldy (plus it suffers from an n+1
problem), but ok.

Change-Id: Ib505a2be8286d658ae44a3fe124a4fb42d0fc0c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9334
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2023-09-15 12:34:04 +00:00
Vincent Ambo
99f2ba1f56 chore(3p/sources): bump channels & overlays (2023-09-15)
* 3p/overlays: add shell script warning about 'exa' removal

  This has been removed in nixpkgs, leading to eval errors. This trick
  is a nicer way to work around the hard deprecation without requiring
  approval from inactive users.

  This will be cleaned up in cl/9330.

Change-Id: I10be59feff8f658dc566456acfc2c9ebbd78b59d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9328
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-09-15 08:32:32 +00:00
Vincent Ambo
0c3af32d1c chore(users/tazjin): drop 'exa' from my systems
I don't use it, and it's deprecated.

Change-Id: I8e8c505ba309d5228b54f155463af4ed007f5452
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9331
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-09-15 08:32:31 +00:00
Adam Joseph
9256621009 fix(tvix/eval): update identifier quoting to match cppnix 2.17
In cppnix 2.17, commit b72bc4a972fe568744d98b89d63adcd504cb586c, the
libexpr pretty-printing routine was fixed so that it would no longer
pretty-print attrsets with keywords in their attrnames incorrectly.

This commit implements the corresponding fix for tvix, fixes our
tests to work with cppnix>=2.17 oracles, and expands our test cases
to cover all the keywords.

Change-Id: I4b51389cd3a9c44babc8ab2a84b383b7b0b116ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9283
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2023-09-15 00:00:59 +00:00
Eugene Lomov
95ee688f03 feat(ops/users): add totikom to users
Change-Id: Id2577449ec0a52f8c16f13150896ec0680f02051
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9325
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-09-12 22:11:57 +00:00
Vincent Ambo
cec2baa6ca chore(tazjin/emacs): remove unused or duplicated packages
Some of these I don't use, some of these have been replaced with
tree-sitter, some of these have moved into core.

Change-Id: Id0a8aac4189607a89db861bbc09b95c75c45dc0f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9310
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-09-12 16:40:15 +00:00
Vincent Ambo
b479725181 fix(tazjin/emacs): manually hook up tree-sitter grammars
The mechanism in nixpkgs does not work, but doing it myself is not
that difficult.

Change-Id: I91b12e9c1f5d381c9990fd46e02169f5275f88da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9309
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-09-12 15:09:49 +00:00
Vincent Ambo
4d0ea1a317 feat(tazjin/emacs): enable a bunch of tree-sitter modes
Especially interested in the C++ mode, as I am regularly dealing with
extremely large C++ files where the existing mode is having trouble.

Change-Id: I8c17a35011a23b67043de5de58c4269c636de68b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9308
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-09-12 15:07:46 +00:00
Vincent Ambo
2b1eb01124 feat(tazjin/emacs): add a bunch of tree-sitter grammars
I want to experiment with the new tree-sitter modes in Emacs, maybe
they're good.

Change-Id: I706fc8a889fcfa61d0f11d50ecbc67a2899b5bbf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9307
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-09-12 15:07:46 +00:00
Vincent Ambo
6e1ea80e5b docs(REVIEWS): point people to signup.tvl.fyi for account creation
Rather than invoking the old shell script, people can now use the web
UI to create the entry.

Change-Id: Ic1b065d219fde9977bb1d8ee59ce7ac5f27e2e29
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9306
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-09-12 13:59:34 +00:00