Commit graph

20701 commits

Author SHA1 Message Date
Aspen Smith
83e25fdd19 fix(aspen/system): Remove duplicate email
Change-Id: I7b30ca4940bcc65488b2fbb190b3f51dc4d76c15
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12204
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
2024-08-15 01:28:10 +00:00
sterni
9d8832474c chore(tvix/verify-lang-tests): update latest Nix version
Change-Id: I656c183d970806565d45ffd31f99f73c804dda1d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12203
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-14 15:01:03 +00:00
sterni
41706c8764 chore(3p/sources): Bump channels & overlays
* //users/sterni/machines/ingeborg: adjust to yet another API
  change in the fcgiwrap module

Change-Id: Ic601bb7161887dec5cfbe68205be816cf9b92d17
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12202
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: sterni <sternenseemann@systemli.org>
2024-08-14 14:10:44 +00:00
Florian Klink
97c60d9857 docs(tvix/TODO): castore Directory struct is done
There's some followup work on making tvix-build use the stricter structs
natively, but that should probably be combined with the overall trait
changes there.

Add an item there so we won't forget, but this isn't really castore
territory anymore.

Change-Id: I6b1f9fa02d5c87c821d4ddc4b1dcc6a98c4eeaa5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12200
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-13 18:52:25 +00:00
Florian Klink
c7845f3c88 refactor(tvix/castore): move *Node and Directory to crate root
*Node and Directory are types of the tvix-castore model, not the tvix
DirectoryService model. A DirectoryService only happens to send
Directories.

Move types into individual files in a nodes/ subdirectory, as it's
gotten too cluttered in a single file, and (re-)export all types from
the crate root.

This has the effect that we now cannot poke at private fields directly
from other files inside `crate::directoryservice` (as it's not all in
the same file anymore), but that's a good thing, it now forces us to go
through the proper accessors.

For the same reasons, we currently also need to introduce the `rename`
functions on each *Node directly.

A followup is gonna move the names out of the individual enum kinds, so
we can better represent "unnamed nodes".

Change-Id: Icdb34dcfe454c41c94f2396e8e99973d27db8418
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12199
Reviewed-by: yuka <yuka@yuka.dev>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-08-13 18:39:49 +00:00
Florian Klink
2f4185ff1a docs(tvix/TODO): DirectoryService with bucketed directories is done
We have this (`ObjectStoreDirectoryService`), as well as some store
composition.

Change-Id: I876f85da1d5b183d80b148c378f825033ec080e4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12198
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
2024-08-13 17:50:27 +00:00
Yureka
3ca0b53840 refactor(tvix/castore): use Directory struct separate from proto one
This uses our own data type to deal with Directories in the castore model.

It makes some undesired states unrepresentable, removing the need for conversions and checking in various places:

 - In the protobuf, blake3 digests could have a wrong length, as proto doesn't know fixed-size fields. We now use `B3Digest`, which makes cloning cheaper, and removes the need to do size-checking everywhere.
 - In the protobuf, we had three different lists for `files`, `symlinks` and `directories`. This was mostly a protobuf size optimization, but made interacting with them a bit awkward. This has now been replaced with a list of enums, and convenience iterators to get various nodes, and add new ones.

Change-Id: I7b92691bb06d77ff3f58a5ccea94a22c16f84f04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12057
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-13 12:17:01 +00:00
Florian Klink
5d3f3158d6 docs(tvix/store/utils): add docstrings for ServiceUrls*
Describe what these structs are used for, and for each of it, explain
which usecases it's used for.

Change-Id: I8b7857bc68ec2b37df9f5163e06d028a64a12c79
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12195
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-12 15:26:31 +00:00
Ilan Joselevich
d2a80dda88 feat(tvix/cli): Add derivation file dumping functionality
Provides a derivation file dumping functionality for tvix-cli that can
be used when passing the --drv-dumpdir CLI arg to tvix-cli.

This will dump all the known derivation files into the specified
directory, making it easier to debug derivation divergences between Tvix
generated drvs and the drvs generated by Nix.

Supersedes: https://cl.tvl.fyi/c/depot/+/11265

Change-Id: I0e10b26eba22032b84ac543af0d4150ad87aed3e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12192
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-12 14:50:17 +00:00
Florian Klink
fd64df0511 fix(tvix/cli): add toplevel docstring to Args struct
Otherwise the flattened `ServiceUrlsMemory` docstring becomes the main
binary description, which doesn't make an awful lot of sense.

The help now describes what this CLI does currently (an interface to use
the different Tvix component.

Also, describe what it does not, so people running it without looking
too much other documentation might not get confused about why this
doesn't do anything in `/nix/store`, or doesn't provide the same CLI
surface as Nix.

Change-Id: Ia4838b444f03a10821801a6171d3e956b3cdfdaf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12194
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-08-12 13:41:50 +00:00
Aspen Smith
1f5a85740e chore(tvix/eval): Update module comment for value::string
This sentence is a little stale; let's just link to NixString directly
for the authoritative source of truth.

Change-Id: I64e065c4148d29702b09820a0e7724a65fae7c67
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12181
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-11 16:41:34 +00:00
Aspen Smith
e176e1f11a feat(tvix/value): Print derivation values specially
Just like tvix-repl does (except we don't force values when printing
them, so... not entirely like tvix-repl does). But it's something.

Change-Id: I2e69b08d7d82b0b2d337f1d4c5d87ed28475fa84
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12180
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
2024-08-11 16:41:34 +00:00
Elle Najt
a781cfcc49 feat(ops/users): Add elle to users
Change-Id: I18492d6e6167f3c010e8f66670a127807ac7d99c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12183
Reviewed-by: aspen <root@gws.fyi>
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
2024-08-11 03:32:13 +00:00
Aspen Smith
12e6c0bfc0 chore(mailmap): Add my name and email
Change-Id: I109da892dc01368ceb38d34e9095559895cbd60c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12182
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
Autosubmit: aspen <root@gws.fyi>
2024-08-11 01:23:46 +00:00
Aspen Smith
f5c6acbbeb fix(tvix/cli): always configure nix path
Configure the nix path even if globals is already set.

Change-Id: I6598c92ab40ff952f73da04d9e7d3aeb13c16b53
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12176
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-11 01:12:42 +00:00
Aspen Smith
7c2ac040db chore(tvix/eval): Drop obsolete todo
the answer is at https://cl.tvl.fyi/c/depot/+/10798

Change-Id: I5f0ed51a3954c7241ef15a8268e0e51695e994c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12175
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-10 19:18:40 +00:00
Aspen Smith
d378111d77 feat(tvix/eval): Store hash in key of interner
Rather than storing the leaked allocation for the string as the key in
the interner, store the hash (using NoHashHashBuilder). I thought this
would improve performance, but it doesn't:

hello outpath           time:   [736.85 ms 748.42 ms 760.42 ms]
                        change: [-2.0754% +0.4798% +2.7096%] (p = 0.72 > 0.05)
                        No change in performance detected.

but it at least doesn't *hurt* performance, and it *does* avoid an
`unsafe`, so it's probably net good.

Change-Id: Ie413955bdb6f04b1f468f511e5ebce56e329fa37
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12049
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: aspen <root@gws.fyi>
2024-08-10 15:28:30 +00:00
Vincent Ambo
a6d6fc418d chore(tazjin/khamovnik): enable ADB
Change-Id: I1c754fdc54465f93dfb10e9d903a66a90447ab85
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12070
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-08-10 11:37:45 +00:00
Vincent Ambo
c06e26f1cc chore(tazjin/emacs): add tree-sitter grammar for typescript
Had to fix some stuff in a typescript project, it's better with highlighting.

Change-Id: I984540a791167427acde3494182521b8839aed6d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12069
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-08-10 11:37:45 +00:00
Florian Klink
3ea69235a4 fix(web/tvl/2024-08-tvix-update): small fixes
The `Fetch` is an enum, not a struct, and "Type" probably makes more
sense here. Slightly reword the "generic" to not be confused with
generics.

Also fix the link to Tvixbolt, which got eaten by the line wrapping.

Change-Id: I677229f0365523620a640c2333e76790573f7318
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12174
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
2024-08-09 19:18:01 +00:00
Profpatsch
8e13838451 fix(web/tvl): fix 24-08 update typo
Change-Id: I0dc69e527a258c298275f688e728b0f951e3071d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12173
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-09 18:29:12 +00:00
Florian Klink
c9a2783d2d feat(web/tvl): add Tvix August Update
Change-Id: I97caa794a2573f7b34f25030c84b5531a9136c44
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12170
Tested-by: BuildkiteCI
Reviewed-by: benjaminedwardwebb <benjaminedwardwebb@gmail.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2024-08-09 16:24:30 +00:00
edef
7f8997fc80 test(tvix/nix-compat/nar/wire): cover TOK_PAD_PAR
Change-Id: Ib4da612a4144d062f55a22b437c725b8566bee70
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12172
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-09 15:28:46 +00:00
Ilan Joselevich
e505880fde feat(tvix/eval): Re-enable testing for eval-okay-readDir
When we moved to building tvix with crate2nix we had to disable the
readDir tests for the same reasons mentioned in
https://cl.tvl.fyi/c/depot/+/12131. But now with the 12131 CL, we can
re-enable the readDir tests.

In this change I re-enabled the readDir tests in nix_tests by moving
them out of notyetpassing and I also deleted them from tvix_tests
because we don't need them in both places.

Change-Id: I82ac39605299a8b22d80f8b51fc8ec2476d21dc9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12133
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-09 14:56:57 +00:00
Ilan Joselevich
3511e328ec feat(tvix/eval): Implement builtins.readFileType
builtins.readFileType was added to Nix back in version 2.14.

The tests were also moved out of notyetpassing in addition to the
readDir fixtures they depend on.

I caught a bug where we previously used std::fs::metadata (via the
.metadata() method on File) which follows symlinks so it would always
return false for is_symlink(). Instead we now use
std::fs::symlink_metadata directly which does not follow symlinks, so
tests now pass. This wasn't an issue for builtins.readDir as it uses
walkdir and walkdir doesn't follow symlinks either.

Change-Id: I58eb97bdb5ec95df4f6882f495f8c572fe7c6793
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12130
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
2024-08-09 14:35:57 +00:00
Ilan Joselevich
9c4b57ac63 fix(third_party/overlays): Patch crate2nix to use mkDerivation for tests
The problem with using runCommand and recreating the src directory with
lndir is that it changes the file types of individual files, they will
now be a symlink instead of a regular file. If you have a crate that tests
that a file is of regular type then it will fail inside the crate2nix
derivation.

Also regenerate Cargo.nix for //tvix as it will be needed in the next
commit.

Change-Id: I9275602cc17a428f9fdf0e55daf12cd673bbc030
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12131
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-08-09 14:35:57 +00:00
Ilan Joselevich
549e86d95b feat(tvix/verify-lang-tests): Test nixVersions.nix_{2_3,2_23}
With this change the latest Nix version being checked is 2.23
(previously was 2.18). A warning will now be printed when the
latest verified Nix version is out of date.

I added a test to the skip-list because the behavior of
builtins.dirOf has changed in Nix 2.22

Change-Id: Iaf7fa98dbfa599a2d2723df0e42bb459711e2413
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12105
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
2024-08-09 14:19:25 +00:00
Ilan Joselevich
c554c1c1c0 feat(tvix/eval): Implement Display for io::FileType
In newer versions of Nix there's a builtins.readFileType builtin, we
should try to avoid duplicating the enum -> string conversion by
implementating Display before we implement builtins.readFileType.

Change-Id: I579e95949a76eb33d2e7bda0000ed84859df765e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12129
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
2024-08-09 14:03:46 +00:00
edef
591c5ffe53 docs(tvix/nix-compat/nar/wire): fix TOK_EXE grammar
Change-Id: I12b640fe1cbe7a4c69efe4660040486e510deadc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12171
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-08-09 08:33:01 +00:00
Ilan Joselevich
f648f17ec3 feat(tvix): Jemalloc -> MiMalloc
Use the faster and newer MiMalloc memory allocator for all endpoints in
the workspace.

Change-Id: Ic60237284ed168e46ec6e8f28e2710bae4385c6f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12149
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
2024-08-08 08:06:23 +00:00
Aspen Smith
000fb856be chore(aspen/web): some tweaks
Change-Id: I268728af52785efb5e6b3df7d4bfe0e0d54c13a1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12169
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
2024-08-08 03:22:53 +00:00
Aspen Smith
3536a7c7fa feat(aspen/web): Link to some more music i released this year
Change-Id: Ib55b5b02d85257e2d9c30697df7dbb303a6b3c27
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12168
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: aspen <root@gws.fyi>
2024-08-08 03:16:18 +00:00
Aspen Smith
6bd2760aa7 feat(aspen/web): freshen up, mention tvix in projects list
Change-Id: I34d1c3efd32882271ca553f180deb6249014fb32
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12165
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
2024-08-08 00:59:17 +00:00
Aspen Smith
d33fd78202 fix(web/tvl/tvl.dot): Update my name
it's about time

Change-Id: Ib93dd8313a0006da1190ca371e83662746333d34
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12062
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
2024-08-08 00:39:18 +00:00
Aspen Smith
172405c40c fix(aspen/system): Drop a broken remote builder
bye bye mugwump

Change-Id: Ie30771caaf23dd10872c33ad60f904f089fa6b6c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12150
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
2024-08-08 00:24:53 +00:00
Aspen Smith
6559ab4cf5 feat(tvix/eval): Put interner in a thread-local RefCell
Rather than making the interner be a global lazy_static mutex, put it in
a thread-local RefCell. This doesn't change anything in terms of
sharing (since we're currently actually just single threaded), but
avoids the overhead of a mutex, for a nice performance boost (compared
to the mutex version):

hello outpath           time:   [726.71 ms 729.79 ms 735.69 ms]
                        change: [-5.7277% -3.9733% -2.1144%] (p = 0.00 < 0.05)
                        Performance has improved.

Change-Id: I240b238dcbaf854ebafc3017b4425fb7d7b91b03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12048
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: aspen <root@gws.fyi>
2024-08-08 00:02:36 +00:00
Aspen Smith
6366cee717 feat(tvix/eval): Intern (and leak) small strings, behind a mutex
This is the most naive version of string interning possible - we store a
map from the string itself to the pointer behind a global mutex, and
memoize the allocation of all strings below a threshold length (16
bytes, for now) into that map. This requires leaking /all/ strings,
since it's not easy to know just from the pointer that a string has been
interned - so interning is disabled if string leaking is also disabled.

In the case where we're leaking strings (the default), even the naive
version of this gets us a pretty nice perfomance boost:

hello outpath           time:   [742.54 ms 745.89 ms 749.14 ms]
                        change: [-2.8722% -2.0135% -1.0654%] (p = 0.00 < 0.05)
                        Performance has improved.

However, in the case where we're not leaking strings, we have to keep
track of which strings have and haven't been interned, which makes this
a little worse:

hello outpath           time:   [779.30 ms 792.82 ms 808.74 ms]
                        change: [+2.5258% +4.0884% +5.8931%] (p = 0.00 < 0.05)
                        Performance has regressed.

Hopefully we can close the gap here a bit with some clever
tricks (coming next).

Change-Id: If08cb48ede703c7fe3bdd8d617443f8a561ad09b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12047
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: aspen <root@gws.fyi>
2024-08-08 00:02:36 +00:00
Aspen Smith
b8f92a6d53 feat(tvix/eval): Forbid Hash{Map,Set}, use Fx instead
Per https://nnethercote.github.io/perf-book/hashing.html, we have
basically no reason to use the default hasher over a faster,
non-DoS-resistant hasher. This gives a nice perf boost basically for
free:

hello outpath           time:   [704.76 ms 714.91 ms 725.63 ms]
                        change: [-7.2391% -6.1018% -4.9189%] (p = 0.00 < 0.05)
                        Performance has improved.

Change-Id: If5587f444ed3af69f8af4eead6af3ea303b4ae68
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12046
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Autosubmit: aspen <root@gws.fyi>
2024-08-07 12:38:40 +00:00
Yureka
1d7ba89c19 chore(tvix): change deprecated default_features to default-features
Change-Id: Ida31fe9a1205cdfc852d2ecbcbda32933b634562
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12143
Reviewed-by: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
2024-08-07 11:36:04 +00:00
Florian Klink
5b0264223f docs(tvix/TODO): document blob prefetching dependency
This depends on the ChunkReader work.

Change-Id: I38878d0f822c312151131e55baee4db6ef1c3650
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12142
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Autosubmit: flokli <flokli@flokli.de>
2024-08-07 10:50:46 +00:00
Florian Klink
c2c4169e49 docs(tvix/TODO): add Correctness > Performance section
Explain the current caveats as far as performance tuning is concerned.

Change-Id: I1a9c11c81de09350240fb61e3c130fc401ef6618
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12141
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: yuka <yuka@yuka.dev>
Tested-by: BuildkiteCI
2024-08-07 10:50:46 +00:00
Profpatsch
f9703a9af5 fix(users/Profpatsch/whatcd-resolver): reduce json data from db
We’d transfer the full json data for each torrent from the db instead
of just the 2 or 3 fields we need.

Adds some more helpers for parsing database values.

Adds some better logging events & traces.

Change-Id: I5db386c4ea247febf5f9fc3815da2e7f11286d41
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12140
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2024-08-06 09:59:40 +00:00
Profpatsch
13d79e04d8 fix(users/Profpatsch/whatcd-resolver): fix postgres query log
The queries would not be interpolated anymore, because we didn’t pass
the thing down deep enough.

Also only init the `pgFormatPool` if we want to use the formatter,
this saves on a bunch of subprocesses.

Change-Id: I8d69ef5aab4d8eac1cbfb1c3991d4edaacba254f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12139
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2024-08-06 09:59:40 +00:00
Profpatsch
2510cd6a5c fix(users/Profpatsch/whatcd-resolver): SQL formatting off
It turns out the pg_format thing is just too slow for my use-cases
most of the time, even when pooling the mf. Most queries stay 90%+ in
the perl script, even though they are very fast to execute on their
own, screwing up the traces a lot.

So instead I replace the `postgres-simple` quasi-quoter that strips
whitespace (and tends to screw up queries anyway) with a simple one
that just removes the outer indentation up to the first line.

Why did I spend so much time on pg_format haha

Change-Id: I911cd869deec68aa5cf430ff4d111b0662ec6d28
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12138
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2024-08-06 09:59:40 +00:00
Profpatsch
3202d008d5 chore(users/Profpatsch): park ical-smolify
Parked projects: I might pick these up again, but don’t want to
maintain them right now. Might delete later, or reinstate.

Change-Id: Ieb4f51dfaeba33c84820ca804b57c016e05bf566
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12137
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2024-08-06 09:59:40 +00:00
Profpatsch
37b2c2ce78 refactor(users/Profpatsch/whatcd-resolver): continue http
Ideally there’d be a better generic abstraction of doing basic http
calls (with tracing) in the future, but for now just reexport.

Change-Id: Id7548739ea62e9172f2773f8db79fe726096b7f1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12136
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2024-08-06 09:59:40 +00:00
Profpatsch
8908fd18ca refactor(users/Profpatsch/whatcd-resolver): start moving http stuff
There’s a bunch of duplication in how http client things are done,
let’s move that all to a single module.

Change-Id: Ic08c9bce49d562e4fa640a5bdfc15973a28a7bcb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12135
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2024-08-06 09:59:40 +00:00
Aspen Smith
59056cf705 feat(tvix/eval): Leak strings (with flag to disable)
Default to always leaking strings, and copying strings by copying
pointers rather than cloning the underlying allocation. This (somewhat
bafflingly) doesn't seem to affect any benchmarks, but paves the way for
some tricks around string allocation that do.

Unfortunately, we can't do this (yet?) for contextful strings, for
reasons I don't currently understand but which I will address later,
when I address contextful strings more holistically.

I've left a flag in here to disable this, both to test the cloning logic
for strings for when/if we decide to bring this back, and to allow
people who care more about memory usage than perf to disable leaking.

Change-Id: Iec44bcbfe9b3d20389d2450b9a551792a79b9b26
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12045
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-05 10:49:27 +00:00
Ilan Joselevich
480a8106cf feat(tvix/store): Sled{PathInfo,Directory}Service -> Redb…
Use redb instead of sled for the default filesystem implementation of
PathInfoService and DirectoryService. In the future we'll also drop sled
support completely.

Change-Id: I513ff0c2ff953d59714aa50b9aa1301b02f53d40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12085
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-08-04 13:58:07 +00:00
Ilan Joselevich
26129c1761 fix(tvix/crate2nix-check): Revert use of depotfmt.check wrapper
This is a partial revert of https://cl.tvl.fyi/c/depot/+/12068 where I
changed tvix/crate2nix-check to use depotfmt.check. It turns out that we
don't actually want to be using it in this case as the wrapper sets
`--fail-on-change` which would always fail because the Cargo.nix
generated by crate2nix will always need to be changed (It's not
formatted properly).

Change-Id: Ife35c812ca69c90459ce4445f4252d0a24c218fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12132
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
2024-08-04 13:47:31 +00:00