This took a while to figure out, but essentially sorting a thing with
one million queries takes a long time, as compared to doing it on a
small subset of things and then joining against the final ordering.
The generated column helps, too.
Change-Id: I1bf283e2be060748eebda92576e3d062c51a6777
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11681
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Generated columns!!
This reduces the query time for distinct on somewhat!
Change-Id: Ic004d327b719a9f44a224ec87709992ce3d8f160
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11676
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Simple artist album page that only shows albums the artist was
involved with.
Change-Id: Icff34afc6d1b39b6fb17765c1b3ea500dd4b4d95
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11675
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
The seeding weight would slow down the query quite a bit, so let’s
move it into a procedure and add an index onto the torrents table that
caches the result.
Baba’s first pl/SQL function!
Change-Id: I3bc6919b115c02b9c9aa74702fac0a8bbc66d2c1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11674
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
The logic around transmission session handling was f*cked, this fixes
that.
We use an IORef instead of an MVar, since we want to unconditionally
write the new value. Even if multiple requests race, I *hope* that
transmission returns the same session id, otherwise we might get a
request loop. But it should be fine. (The semantics is not nicely
documented in the RPC docs.)
Additionally, log the session ids in the requests.
Change-Id: Id7d33f8cb74cb349e502331cad5eb5abe8a624cd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11673
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
apparently the `torrent_group` json only contains one artist, while
the torrent itself contains a list of them.
This is important, because we need the artist id and the
`torrent_group` does not provide it, only the torrent one.
Change-Id: I3db45f454f14e89ea8c8dafba2065ecd55f5bcda
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11670
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
We want the user thingy to see which error happened; it also gets
logged in the traces as before.
There’s another function which we should replace as well at one point.
Change-Id: I3d49edccd0e2088a45ac0138af9536b40dfa6848
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11660
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Add a command, :p, to evaluate an expression and recursively print the
result, as if `--strict` had been passed on the command line.
Demonstration of this working:
❯ cargo r --bin tvix
Finished dev [unoptimized + debuginfo] target(s) in 0.27s
Running `target/debug/tvix`
tvix-repl> { x = (x: x) 1; }
=> { x = <CODE>; } :: set
tvix-repl> :p { x = (x: x) 1; }
=> { x = 1; } :: set
Change-Id: I1a81d7481160c30d2a4483c6308e25fa45f2dfdf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11738
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Add a command to display help for the REPL, which can be either :? or
:h.
Change-Id: Ifdfd8c31130ca5afcde05a4c4276b768eb54c06f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11737
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Prepare for introducing additional REPL commands by splitting out
the *parsing* of the repl commands, which returns a new ReplCommand
type, from the actual *handling* of the commands.
Change-Id: If81a53c1e2d90204d26ce3bb2ea9eebf7bb3fd51
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11734
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
In preparation for adding some new functionality to the tvix REPL, break
it out into its own module.
Change-Id: I4fb78320e92562e3474a3724536cb22c1d893e57
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11733
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Transparently support multiline input in the Tvix REPL, by handling the
UnexpectedEOF error returned by the parser and using it to progressively
build up an input expr over multiple iterations of the REPL's outer
loop.
This works quite nicely:
❯ cargo r --bin tvix
Compiling tvix-cli v0.1.0 (/home/aspen/code/depot/tvix/cli)
Finished dev [unoptimized + debuginfo] target(s) in 1.72s
Running `target/debug/tvix`
tvix-repl> { foo
> =
> 1;
> }
=> { foo = 1; } :: set
<press up arrow>
tvix-repl> { foo
=
1;
}
=> { foo = 1; } :: set
Change-Id: Ib0ed4766b13e8231d696cdc27281ac158e20a777
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11732
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This fixes the following command:
'cargo test --no-default-features --features nix_tests'
Change-Id: I9883c39e1e428c72a0e7e0b75a73c8ed734abd3b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11740
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
`cargo test --no-default-features` fails, if we don't conditionalize
this on the `arbitrary` feature too.
Change-Id: I81a277810119fed0cfc37c942c422f731aa14b2e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11726
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
`Evaluation::new_impure()` would require the test to be impure, but
there's nothing in this test specifically requiring us to make use of
impure features.
Change-Id: Idb24981195d1a94f51053ae04403eb5f0e27f3d9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11725
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
This uses StdIO, which is only available when the `impure` feature is
enabled.
Change-Id: I039b1f45f6619dd099fa943e58322ff521482dfa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11724
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Gate this behind the nix_tests feature flag (which wasn't applied
consistently).
Also, at least right now all of these use StdIO internally, either by
creating it on their own, or through the `eval_test` helper.
Once we have some proper classification system for tests we can probably
split this up further, but for now only run them if the impure feature
is enabled.
Change-Id: I3236cf09b3391585df99073367c4e4832c5e7898
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11723
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
These VM requests are only emitted by code gated behind the impure feature.
To prevent warning from popping up when building without default
features, allow these to be unused if `impure` is not enabled.
Change-Id: Id871a5215e9a0f09aa78edecdd111369ee7ffe34
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11722
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
This contains a fix for an issue where the bridge would forward
messages from the wrong channel, because the source channel was not
checked.
tvlbot runs in 3 channels, so this caused issues.
Change-Id: Icc85406b273f375ac90287364df83fb76f028b59
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11731
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
%q is only defined for maps containing booleans, not strings. It should
be OK to use %v here.
Change-Id: I7475eb89032653a588ecdf70809eb35bbe7e04ab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11730
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
These use StdIO, which is only available if the impure feature is
enabled.
Change-Id: I18d8e191a7eba6ba5bd59f43631973eaa796c7bb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11721
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
These are only needed when building with the impure feature enabled.
This removes some warnings when building with --no-default-features.
Change-Id: I3139d9133d4846aeb1b1b5f3830c0d078d047292
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11720
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
If building tvix-build with the tonic-reflection feature, it needs
to import `tvix_castore::proto::FILE_DESCRIPTOR_SET`, which is only
available if tvix-castore is built with the `tonic-reflection` feature.
Change-Id: I355b4c5b4c1333d5cc56335de47ad5d2f1db6337
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11716
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
These tests only interact with the FUSE layer, and
import super::fuse to do its work.
However, this only works if the `fuse` feature is enabled, which we
don't do if we enable the `virtiofs` feature only, causing the tests
to fail:
```
❯ cargo test --no-default-features --features virtiofs
Compiling tvix-castore v0.1.0 (/home/flokli/dev/nixos/code.tvl.fyi-submit2/tvix/castore)
error[E0432]: unresolved import `super::fuse`
--> castore/src/fs/tests.rs:14:13
|
14 | use super::{fuse::FuseDaemon, TvixStoreFs};
| ^^^^ could not find `fuse` in `super`
```
We move src/fs/tests.rs to src/fs/fuse/tests.rs
(and src/fs/fuse.rs to src/fs/fuse/mod.rs) to better structure this,
which will automatically cause both tests and code to only be built if
we have the `fuse` feature enabled.
Change-Id: I8fbbad3e4457e326bdfd171aa5c43d25d3187b5b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11715
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
For the duration of the sprint, this bot will take care of
synchronising the IRC channel with the Telegram group.
After the sprint, it will be removed again.
Change-Id: I6d5b1316fc85ddd26adf55e31f6bff742907fc24
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11727
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We need this for the Volga Sprint channel.
Change-Id: I77ff3172e3a8eca55de51e45cb824ec0f40d4065
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11718
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Previously changing any file (including default.nix or README.md) would
cause Nix to compute a different derivation path, resulting in needing
to (very often redundantly) rebuild the crate/package. With this change
you can expect less rebuilds.
Crate2nix currently does its own insufficient src filtering and it does
not expose an API to override the filtering, so instead I created a
function that we can use to override the src to have stricter filtering.
I implemented the filtering for all of the workspace members, if you
want to modify any of them, please read the lib.fileset docs
https://nixos.org/manual/nixpkgs/unstable/#sec-functions-library-fileset
Change-Id: I7ab5a0064a76938d14f7f65801be9f3a5c3bad04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11714
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
cl/11712 simultaneously introduced this check and made it unnecessary,
since NixString::context should never return `Some` for empty contexts
now.
Change-Id: I41a655ff33910e8326cbb7d7526eb91bd19e9585
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11713
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Both `Some(NixContext::new())` and `None` represent empty contexts,
but the former trips up `NixString::has_context`, and seems likely
to trip up other things.
We could hide the difference in the accessors, but we don't really
*want* the distinction to exist, since heap-allocating a null value
is pretty much always a mistake.
Change-Id: Ie84d26fb0d4b59e68354891ba13bde3bae40ab6e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11712
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
In places where we want to extend context with that from another
NixString, use take_context() to split it off, then call .extend(),
making use of IntoIterator to avoid a bunch of clones.
Change-Id: I2460141a3ed776c64c36132b2203b6a1d710b922
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11705
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
XmlEmitter gains a NixContext field, and `write_typed_value` extends it
with all context elements present in the passed value.
Once all serialization is done, a into_context() function returns the
collected context, so we can construct a NixString with context.
Tests for this live in tvix-glue, as we use builtins.derivation, which
is not present in the tvix-eval crate.
Fixes b/398.
Change-Id: I85feaaa17b753885f8a017a54e419ec4e602af21
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11704
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Alyssa Ross <hi@alyssa.is>
This is a slightly less annoying version of `join`, which does not
consume self. It's more consistent with HashSet::extend().
Change-Id: Ifd0872da36fe8e7b2aa6948674cb8e4023abe9d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11703
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
In order to be compatible with the nix XML generator, it’s easier to
generate the XML directly, instead of going through a library which we
have to bend to do what we need.
Removes dependency on `xml-rs`, which came with a full XML parser that
we didn’t use. Only takes a tiny bit of code for the XML escaping,
somewhat simplified.
I add a little escaping value, to make sure we have the same behaviour
as nix proper.
Interestingly enough, we never need to escape XML attribute names,
because the `builtins.toXML` format encodes user-defined values as
attribute keys only. So we only escape attribute values.
Fixes: https://b.tvl.fyi/issues/399
Change-Id: If4d407d324864b3bb9aa3160e2ec6889f7727127
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11697
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
Currently all blobs are uploaded serially when ingesting NARs. If a NAR
contains many, small blobs, ingestion may become slow if there is a lot
of round-trip latency to the blob service.
This makes the NAR ingester use the ConcurrentBlobUploader which allows
for buffering small blobs in memory so they can be uploaded concurrently
to the blob service without blocking further deserialization.
Change-Id: I093a73770232df12d9a11e5d901b99c08505c3cb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11694
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
The archive ingester has a mechanism for concurrently uploading small
blobs to the blob service in order to hide round trip latency with the
blob service when ingesting many small blobs.
Other ingestion sources like NARs also need a similar mechanism, this
extracts the concurrent blob uploading mechanism into its own struct to
make it more reusable.
Change-Id: I05020419ff4b9ad5829fbfb5cd08d36db983b8c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11693
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
