The tvix-cli tests now include evaluating the NixOS GNOME installer ISO
image and making sure that its drvPath and outPath matches the one
evaluated by Nix. (This required extending the helper function a bit and
adding docs).
NixOS docs generation is disabled for now, see comments in diff.
Change-Id: Ia510f209b1ec3ef9a823f1e5ac0ef2f5f193976f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12177
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This splits the existing ReferenceScanner into a ReferenceScanner and
ReferencePattern as well as adds an AsyncRead implementation that can
do a scan while you read from it.
The reason to split the scanner in two is that generating the pattern
is expensive and when ingesting build results with multiple outputs you
want to do several independant scans that look for the same pattern.
The reader is for scanning files without having to load the entire file
into memory.
Change-Id: I993f5a32308c12d9035840f8e04fe82e8dc1d962
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12052
Autosubmit: Brian Olsen <me@griff.name>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This comment is a bit misleading - everything that's not in this field
is referenced in a content-addressed fashion (by their digest).
Change-Id: I5097131530fd188173393063643c057f588ea2c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12219
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
These were out of date.
Change-Id: Ideaf888c2851bb9ec36ae01b11d93165b62cd7a5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12209
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
When using the runTests feature of crate2nix the derivation that runs the
tests is put into passthru.test but all default.nix files for Rust crates
in Tvix threw that away.
This commit retains passthru so that you can get access to the test
derivation.
Change-Id: I8b7b7db57a49069348f08c12c00a3b1a41a0c05b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12215
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
These were out of date.
Change-Id: I89df37f088ad6c53b676d965f0dc1023f40481f4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12208
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
This was out of date.
Change-Id: Icea6ee865d389d56bc46941f6049721db293ef7e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12207
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Autosubmit: flokli <flokli@flokli.de>
This apparently was out of date.
Change-Id: I9768d5e62f200169d7815ad85aa0f12eadc56a8e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12206
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
There's some followup work on making tvix-build use the stricter structs
natively, but that should probably be combined with the overall trait
changes there.
Add an item there so we won't forget, but this isn't really castore
territory anymore.
Change-Id: I6b1f9fa02d5c87c821d4ddc4b1dcc6a98c4eeaa5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12200
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
*Node and Directory are types of the tvix-castore model, not the tvix
DirectoryService model. A DirectoryService only happens to send
Directories.
Move types into individual files in a nodes/ subdirectory, as it's
gotten too cluttered in a single file, and (re-)export all types from
the crate root.
This has the effect that we now cannot poke at private fields directly
from other files inside `crate::directoryservice` (as it's not all in
the same file anymore), but that's a good thing, it now forces us to go
through the proper accessors.
For the same reasons, we currently also need to introduce the `rename`
functions on each *Node directly.
A followup is gonna move the names out of the individual enum kinds, so
we can better represent "unnamed nodes".
Change-Id: Icdb34dcfe454c41c94f2396e8e99973d27db8418
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12199
Reviewed-by: yuka <yuka@yuka.dev>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
We have this (`ObjectStoreDirectoryService`), as well as some store
composition.
Change-Id: I876f85da1d5b183d80b148c378f825033ec080e4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12198
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
This uses our own data type to deal with Directories in the castore model.
It makes some undesired states unrepresentable, removing the need for conversions and checking in various places:
- In the protobuf, blake3 digests could have a wrong length, as proto doesn't know fixed-size fields. We now use `B3Digest`, which makes cloning cheaper, and removes the need to do size-checking everywhere.
- In the protobuf, we had three different lists for `files`, `symlinks` and `directories`. This was mostly a protobuf size optimization, but made interacting with them a bit awkward. This has now been replaced with a list of enums, and convenience iterators to get various nodes, and add new ones.
Change-Id: I7b92691bb06d77ff3f58a5ccea94a22c16f84f04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12057
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Describe what these structs are used for, and for each of it, explain
which usecases it's used for.
Change-Id: I8b7857bc68ec2b37df9f5163e06d028a64a12c79
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12195
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Provides a derivation file dumping functionality for tvix-cli that can
be used when passing the --drv-dumpdir CLI arg to tvix-cli.
This will dump all the known derivation files into the specified
directory, making it easier to debug derivation divergences between Tvix
generated drvs and the drvs generated by Nix.
Supersedes: https://cl.tvl.fyi/c/depot/+/11265
Change-Id: I0e10b26eba22032b84ac543af0d4150ad87aed3e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12192
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Otherwise the flattened `ServiceUrlsMemory` docstring becomes the main
binary description, which doesn't make an awful lot of sense.
The help now describes what this CLI does currently (an interface to use
the different Tvix component.
Also, describe what it does not, so people running it without looking
too much other documentation might not get confused about why this
doesn't do anything in `/nix/store`, or doesn't provide the same CLI
surface as Nix.
Change-Id: Ia4838b444f03a10821801a6171d3e956b3cdfdaf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12194
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
This sentence is a little stale; let's just link to NixString directly
for the authoritative source of truth.
Change-Id: I64e065c4148d29702b09820a0e7724a65fae7c67
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12181
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Just like tvix-repl does (except we don't force values when printing
them, so... not entirely like tvix-repl does). But it's something.
Change-Id: I2e69b08d7d82b0b2d337f1d4c5d87ed28475fa84
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12180
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Configure the nix path even if globals is already set.
Change-Id: I6598c92ab40ff952f73da04d9e7d3aeb13c16b53
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12176
Tested-by: BuildkiteCI
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Rather than storing the leaked allocation for the string as the key in
the interner, store the hash (using NoHashHashBuilder). I thought this
would improve performance, but it doesn't:
hello outpath time: [736.85 ms 748.42 ms 760.42 ms]
change: [-2.0754% +0.4798% +2.7096%] (p = 0.72 > 0.05)
No change in performance detected.
but it at least doesn't *hurt* performance, and it *does* avoid an
`unsafe`, so it's probably net good.
Change-Id: Ie413955bdb6f04b1f468f511e5ebce56e329fa37
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12049
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: aspen <root@gws.fyi>
Had to fix some stuff in a typescript project, it's better with highlighting.
Change-Id: I984540a791167427acde3494182521b8839aed6d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12069
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
The `Fetch` is an enum, not a struct, and "Type" probably makes more
sense here. Slightly reword the "generic" to not be confused with
generics.
Also fix the link to Tvixbolt, which got eaten by the line wrapping.
Change-Id: I677229f0365523620a640c2333e76790573f7318
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12174
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
When we moved to building tvix with crate2nix we had to disable the
readDir tests for the same reasons mentioned in
https://cl.tvl.fyi/c/depot/+/12131. But now with the 12131 CL, we can
re-enable the readDir tests.
In this change I re-enabled the readDir tests in nix_tests by moving
them out of notyetpassing and I also deleted them from tvix_tests
because we don't need them in both places.
Change-Id: I82ac39605299a8b22d80f8b51fc8ec2476d21dc9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12133
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
builtins.readFileType was added to Nix back in version 2.14.
The tests were also moved out of notyetpassing in addition to the
readDir fixtures they depend on.
I caught a bug where we previously used std::fs::metadata (via the
.metadata() method on File) which follows symlinks so it would always
return false for is_symlink(). Instead we now use
std::fs::symlink_metadata directly which does not follow symlinks, so
tests now pass. This wasn't an issue for builtins.readDir as it uses
walkdir and walkdir doesn't follow symlinks either.
Change-Id: I58eb97bdb5ec95df4f6882f495f8c572fe7c6793
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12130
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
The problem with using runCommand and recreating the src directory with
lndir is that it changes the file types of individual files, they will
now be a symlink instead of a regular file. If you have a crate that tests
that a file is of regular type then it will fail inside the crate2nix
derivation.
Also regenerate Cargo.nix for //tvix as it will be needed in the next
commit.
Change-Id: I9275602cc17a428f9fdf0e55daf12cd673bbc030
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12131
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
With this change the latest Nix version being checked is 2.23
(previously was 2.18). A warning will now be printed when the
latest verified Nix version is out of date.
I added a test to the skip-list because the behavior of
builtins.dirOf has changed in Nix 2.22
Change-Id: Iaf7fa98dbfa599a2d2723df0e42bb459711e2413
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12105
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
In newer versions of Nix there's a builtins.readFileType builtin, we
should try to avoid duplicating the enum -> string conversion by
implementating Display before we implement builtins.readFileType.
Change-Id: I579e95949a76eb33d2e7bda0000ed84859df765e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12129
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
Use the faster and newer MiMalloc memory allocator for all endpoints in
the workspace.
Change-Id: Ic60237284ed168e46ec6e8f28e2710bae4385c6f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12149
Tested-by: BuildkiteCI
Reviewed-by: aspen <root@gws.fyi>
Rather than making the interner be a global lazy_static mutex, put it in
a thread-local RefCell. This doesn't change anything in terms of
sharing (since we're currently actually just single threaded), but
avoids the overhead of a mutex, for a nice performance boost (compared
to the mutex version):
hello outpath time: [726.71 ms 729.79 ms 735.69 ms]
change: [-5.7277% -3.9733% -2.1144%] (p = 0.00 < 0.05)
Performance has improved.
Change-Id: I240b238dcbaf854ebafc3017b4425fb7d7b91b03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12048
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: aspen <root@gws.fyi>
This is the most naive version of string interning possible - we store a
map from the string itself to the pointer behind a global mutex, and
memoize the allocation of all strings below a threshold length (16
bytes, for now) into that map. This requires leaking /all/ strings,
since it's not easy to know just from the pointer that a string has been
interned - so interning is disabled if string leaking is also disabled.
In the case where we're leaking strings (the default), even the naive
version of this gets us a pretty nice perfomance boost:
hello outpath time: [742.54 ms 745.89 ms 749.14 ms]
change: [-2.8722% -2.0135% -1.0654%] (p = 0.00 < 0.05)
Performance has improved.
However, in the case where we're not leaking strings, we have to keep
track of which strings have and haven't been interned, which makes this
a little worse:
hello outpath time: [779.30 ms 792.82 ms 808.74 ms]
change: [+2.5258% +4.0884% +5.8931%] (p = 0.00 < 0.05)
Performance has regressed.
Hopefully we can close the gap here a bit with some clever
tricks (coming next).
Change-Id: If08cb48ede703c7fe3bdd8d617443f8a561ad09b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12047
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: aspen <root@gws.fyi>
