With the recent introduction of the RootNodes trait, there's nothing in
the fs module pulling in tvix-store dependencies, so it can live in
tvix-castore.
This allows other crates to make use of TvixStoreFS, without having to
pull in tvix-store.
For example, a tvix-build using a fuse mountpoint at /nix/store doesn't
need a PathInfoService to hold the root nodes that should be present,
but just a list.
tvix-store now has a pathinfoservice/fs module, which contains the
necessary glue logic to implement the RootNodes trait for a
PathInfoService.
To satisfy Rust orphan rules for trait implementations, we had to add a
small wrapper struct. It's mostly hidden away by the make_fs helper
function returning a TvixStoreFs.
It can't be entirely private, as its still leaking into the concrete
type of TvixStoreFS.
tvix-store still has `fuse` and `virtiofs` features, but they now simply
enable these features in the `tvix-castore` crate they depend on.
The tests for the fuse functionality stay in tvix-store for now, as
they populate the root nodes through a PathInfoService.
Once above mentioned "list of root nodes" implementation exists, we
might want to shuffle this around one more time.
Fixes b/341.
Change-Id: I989f664827a5a361b23b34368d242d10c157c756
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10378
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This makes PathInfoService::from_addr return a Box<dyn PathInfoService>,
rather than an Arc<dyn …>, and leaves it up to the consumers to rewrap
it into an Arc where needed.
This allows us to drop the Arc for the tvix-store daemon subcommand.
Change-Id: Ic83aa2ade6c51912281bd17c7eef7252e152b2d1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10409
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
This is not gonna end up as a interlinked docstring.
Change-Id: I2b0ca106aa75bae0156c0b411da5931da60c725d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10406
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
This is not an unclosed <html> tag.
Change-Id: I2bd2426fc600de2d96dbab47743f1c7bd5fed35e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10398
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
self_reference being set to true is only allowed for
`CAHash::Nar(NixHash::Sha256(_))`, so we can handle this in a check at
the front.
Change-Id: Ic363ade4789a7767cbe26a6959b143bb53e50e5a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10391
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
All match cases essentially construct `ty` and `hash`, which is then
passed to the `build_store_path_from_fingerprint_parts` function.
Change-Id: I01dfd219f9b0ac1afe8af7c6e361ea048117a0e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10390
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
In most case, we don't actually need an owned `StorePath` struct, but a
`StorePathRef<'_>` is sufficient. The lifetime is only due to it holding
onto the name, but that one is mostly left untouched.
`Derivation::calculate_derivation_path` still needs to return
`StorePath`, as its name has a `.drv` appended.
Change-Id: Ie0d52f369d785711bb0658ea2b0bd2617fd9f45e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10389
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
With the introduction of StorePathRef::to_absolute_path(), there's no
need to convert this StorePathRef to a StorePath first.
Change-Id: I634c977c4b63858e4f329fd21726e0611b99da4a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10388
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
Keep the method around in StorePath for convenience, but move the
implementation to StorePathRef.
Change-Id: Ie1844fa01ce6529dc1a58907563c95c3112c831d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10387
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Run criterion benchmarks (via `cargo bench`) for the *whole tvix
workspace*, not just `//tvix/eval`, and report the results to
windtunnel.
Change-Id: I9235c3c166ed9121f35c5bb4c46d59dc1f4c4055
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10382
Tested-by: BuildkiteCI
Autosubmit: grfn <grfn@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
These are still leftovers from before the traits being async, where we
had to clone before moving into an async closure.
Change-Id: I1b3937edf61ce3e23bb07803306622c37a3572c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10381
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
The simple filesystem `BlobService` enable a user to write blob store
on an existing filesystem using a prefix-style layout in the provided root directory,
e.g. the two first bytes of the blake3 hashes are used as directories prefixes.
Change-Id: I3451a688a6f39027b9c6517d853b95a87adb3a52
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10071
Autosubmit: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This is only used in the gRPC version (GRPCPutter), during the test
automation.
So define it as a method there, behind #[cfg(test)], and remove from
the trait.
Change-Id: Idf170884e3a10be0e96c75d946d9c431171e5e88
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10340
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
There's nothing store-path specific here anymore, it's just a name in
the mountpoint root.
Change-Id: I0f8004491baa03ba560d390053a42678ee81154a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10377
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
The PR has since been replaced with another version, using a bit of
unsafe, as rustc doesn't like the lifetimes.
We're still waiting on a backport of that to 0.3.
Change-Id: I8dd344e78162cd956f6e53ff05dc4c02763fbb04
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10376
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
To support tvix builds, we need to be able to use the `TvixStoreFs` to
materialize the sandbox's /nix/store filesystem with just the list of
inputs needed for the build. Currently we'd need to seed an in-memory
`PathInfoService`, which includes more functionality than what is
required for `TvixStoreFs`. Additionally, the `PathInfoService` is
specific to Nix. By decoupling `TvixStoreFs` and `PathInfoService`,
we allow for usage of `TvixStoreFs` with `tvix-castore` without needing
a `PathInfoService`.
This introduces a new `RootNodes` trait which provides a way for the
filesystem to look up CA nodes via their basename in the root directory
of the filesystem. We then implement `RootNodes` for any
`PathInfoService`. Additionally, the filesystem root inode tracker now
stores basenames rather than `StorePath`s since `StorePath`s are
specific to Nix.
As a followup we can rename `TvixStoreFs` to `TvixCaStoreFs` and move
it to the `castore` crate (or its own crate).
b/341
Change-Id: I928372955017c23b1bf2b37190cbc508a4ed10d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10363
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Fix the nixpkgs eval hyperfine benchmark in tvix/scripts.bench.sh to:
- Eval both a pinned (never-changing) nixpkgs commit and the nixpkgs
commit used by the depot
- Not depend on fetchTarball, which is not yet implemented.
- Be called bench-windtunnel.sh
Change-Id: Id2ae18f983ab7327625320b5b16c082ecc369a49
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10364
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
This moves the failing example from cl/10285 into its separate test
case.
There were multiple complications: tvix-[ca]store was panicking in some
places, rather than returning an error. This is now fixed.
It needs to live in tvix-glue, so we actually have a "proper" EvalIO
interface doing something.
> toString ({ line = 42; col = 42; file = /deep/thought; }.file)
Should not cause an error, because it shouldn't trigger an import, but
leave the path as-is, and not care about it not being present.
Change-Id: I76f70b3cb1f73a0fb05870375710fd9f67d5603c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10342
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: sterni <sternenseemann@systemli.org>
The default behavior of string coercion in C++ Nix is to weakly coerce
and import to store if necessary. There is a flag to make it strongly
coerce (coerceMore) and a flag that controls whether path values have
the corresponding file/directory imported into the store before
returning the (store) path as a string (copyToStore). We need to
implement our equivalent to the copyToStore (import_paths) flag for the
benefit of weak coercions that don't import into the store (dirOf,
baseNameOf, readFile, ...) and strong coercions that don't import into
the store (toString).
This makes coerce_to_string as well as CoercionKind weirder and more
versatile, but prevents us from reimplementing parts of the coercion
logic constantly as can be seen in the case of baseNameOf.
Note that it is not possible to test this properly in //tvix/eval tests
due to the lack of an appropriate EvalIO implementation being available.
Tests should be added to //tvix/glue down the line.
Change-Id: I8fb8ab99c7fe08e311d2ba1c36960746bf22f566
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10361
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: Adam Joseph <adam@westernsemico.com>
This matches the behavior of C++ Nix more closely where the decision is
made based on the first type based to ExprConcatStrings:
1f93fa2ed2/src/libexpr/eval.cc (L1967-L2025)
Note that this doesn't make a difference in any successful
evaluation (at least to my knowledge), but ensures that our error
messages will match C++ Nix more closely, e.g. in the case of
`1 + "string"`.
Change-Id: I8059930788f9c8d98baf98e3d93d8a060ef961f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10360
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: Adam Joseph <adam@westernsemico.com>
This comment didn't make a lot of sense before.
Change-Id: Ie057a133ca4b1a099ed3c885e32316b0d87c5eb0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10339
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Namely, all trait implementations should reject invalid data being fed,
and detect invalid data being returned.
b/355 tracks writing some more tests for this, to ensure we're compliant
with this.
Change-Id: I3b05752932837ce208785efb21ffc21508b4b33a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10338
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Autosubmit: flokli <flokli@flokli.de>
This ensures importing these paths also behave the same way as Nix.
Change-Id: Icaa507bbe3d9867a301fc7a300c5d2b3f9feb911
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10355
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
This creates a directory with a .keep file inside, and uses
"${path/to/there}" to coerce it to a string (and import it into the
store), ensuring it calculates the same store paths as Nix does.
Change-Id: Ie14ae075104ce278bc4f2cce93aab5762a2734d1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10343
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
If the path specified doesn't exist, construct a proper error instead
of panicking.
Part of b/344.
Change-Id: Id5c6a91248b0a387f3e8f138f8e686e402009e8f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10330
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This will emit a log event / trace in case this function returns an
error-y type.
Change-Id: I48db6807f3e42304357c422a2b6e177cb8b95228
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10329
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
For all these calls, the caller has enough context about what it did, so
it should be fine to use io::Result here.
We pretty much only constructed crate::Error::StorageError before
anyways, so this conveys *more* information.
Change-Id: I5cabb3769c9c2314bab926d34dda748fda9d3ccc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10328
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
This commit fixes out builtins.length so it propagates catchables
like cppnix does.
Change-Id: I7670bec5eee1d4cd3f67a04c9a6808979fb56a8d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10315
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
This commit fixes builtins.filter so it propagates catchables
correctly.
Change-Id: Ib23a383bc5e272e42052205ffd1e94649a0ebc47
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10313
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This adds an unimplemented placeholder for builtins.hashString.
Change-Id: Ibc770103acf5dbc3ea7589ab5ca23fe6e07bd91a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10311
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
This commit fixes builtins.elemAt so it propagates catchables like
cppnix does.
Change-Id: Ieca5e128da17e78af0b14dae4a28a1ff8796e4f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10308
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This commit adjusts the error produced by STRUCTURED_ATTRS so that
it is catchable. This way we are able to enumerate the release
packageset, and the enumeration process will simply treat the few
derivations using structured attributes as being broken, rather than
killing the whole eval session.
Change-Id: I2e17638b8e3227f88543c3718aaf505deaec22ae
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10306
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This fixes our implementation of builtins.splitVersion so it
propagates catchables like cppnix does.
Change-Id: Id5d83ea76229f8c8f202aa42353cb609e67de43f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10305
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Currently this just `throw`s a message explaining that it is not
implemented. This is necessary in order to allow enumerating the
nixpkgs release attrset (afaict only one package uses this builtin).
Change-Id: I45266d46af579ddb5856b192b6be4b481369543c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10302
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
... since this may import them to the store which changes their
basename.
Fixes b/350.
Change-Id: Iabd08ff4d6a424c66d6d7784d7a96b0c078f0a91
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10298
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
This commit adds __curPos (to the global scope, yuck) and
builtins.filterSource. These are not implemented; forcing them will
produce the same result as `throw "message"`.
Unfortunately these two post-2.3 features are used throughout
nixpkgs. Since an unresolved indentifier is a catchable error, this
breaks the entire release eval. With this commit, it simply causes
those broken packages that use these features to appear as they are:
broken.
Change-Id: Ib43dea571f6a9fab4d54869349f80ee4ec5424c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10297
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
This commit fixes our handling of `throw` within an `assert`
condition.
Fixes: b/340
Change-Id: I40a383639ec266da50a853f16216b1b7868495da
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10318
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This commit fixes out `?` operator so it correctly propagates
catchables.
Change-Id: Iebaa153a8492101ee3ddd29893c98730ff331547
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10317
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Previously, using a catchable as either argument of OpAttrsSelect
would result in an unrecoverable error. This commit matches cppnix
behavior by propagating the catchable.
Change-Id: I4877f4068ec2b823225f185290693c101d0b9c9e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10303
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
When attempting to call a Value, if it is a Value::Catchable we must
not cause an uncatchable failure. This commit simply reuses the
Value::Catchable as the result of attempting to call it. This is
safe because nix is designed so that nix code cannot distinguish
between different catchable failures -- they all look the same to
the interpreted code.
This fixes b/351.
Change-Id: Ibf763a08753e541843626182ff59fdbf15ea2959
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10300
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
error[E006]: expected value of type 'bool', but found a 'internal[catchable]'
--> src/tests/tvix_tests/notyetpassing/eval-okay-test-catchables-in-implications.nix:1:43
|
1 | (builtins.tryEval (({ foo ? throw "up" }: foo -> true) { })).success
| ^^^^^^^^^^^
Relates to b/345
Change-Id: Ic331c32ea59bf67ae775f485b444dc6804ca13d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10289
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
This commit adds Opcode::OpJumpIfCatchable, which can be inserted
ahead of most VM operations which expect a boolean on the stack, in
order to handle catchables in branching position properly.
Other than remembering to patch the jump, no other changes should be
required.
This commit also fixes b/343 by emitting this new opcode when
compiling if-then-else. There are probably other places where we
need to do the same thing.
Change-Id: I48de3010014c0bbeba15d34fc0d4800e0bb5a1ef
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10288
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
This is a test case for b/343, wherein tvix dies if you try to
branch on an argument whose defaulted value is a catchable.
Change-Id: I891ca825e39ad14dda9f220f06d9591874fcd45d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10287
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
After this commit, the only non-builtins uses of generators are:
- coerce_to_string() uses generators::request_enter_lambda()
- Thunk::force() uses generators::request_enter_lambda()
That's it! Once those two are taken care of, GenCo can become an
implementation detail of `builtins::BuiltinGen`. No more crazy
nonlocal flow control within the interpreter: if you've got a GenCo
floating around in your code it's because you're writing a builtin,
which isn't part of the core interpreter. The interpreter won't
need GenCos to talk to itself anymore.
Technically generators::request_path_import() is also used by
coerce_to_string(), but that's just because the io_handle happens to
be part of the VM. There's no recursion-depth issue there, so the
call doesn't need to go through the generator mechanism
(request_path_import() doesn't call back to the interpreter!)
Change-Id: I83ce5774d49b88fdafdd61160975b4937a435bb0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10256
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
This commit implements deep_force() nonrecursively, by maintaining
an explicit stack rather than using the call stack for recursion.
As an added bonus, we don't need to pass around the SharedThunkSet
anymore, and can in fact completely eliminate SharedThunkSet.
Change-Id: I7c4f59f37834d451a28bf6be317eb0a90eac4ee6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10252
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
The comment explaining ThunkSet makes it seem like it does the same
think as ThunkRepr::Blackhole. In fact neither one is a substitute
for the other. Let's explain the difference.
Change-Id: I89ceaaa9d3c499edbc7d48f70ca5d11f97666c43
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10250
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
LightSpan::Delayed was introduced in commit
bf286a54bc which claimed that "This
reduces the eval time for `builtins.length (builtins.attrNames
(import <nixpkgs> {}))` by *one third*!"
I am unable to reproduce this result. In fact, dropping the
LightSpan::Delayed variant of the enum makes eval of the same
expression slightly faster! I also tried a large evaluation
(pkgsCross...hello) and got similar results: slightly faster,
slightly less memory. See git footers.
I suspect that there was some unrelated horrific inefficiency that
has since been fixed. The avoided computation in `get_span()` is
nothing more than a binary search! If this were in fact a major
performance issue we could simply precompute the mapping from
CodeIdx to Span when the Chunk becomes immutable (i.e. at the end of
the compilation process, when compiler backtracking is no longer a
concern). Since a Span is just 64 bits this is not a space issue,
and since binary search is much simpler than compiling Nix
expressions it isn't a performance issue either.
Technically there is no longer any reason to have LightSpan since it
is now a single-variant enum. However there is no rush to remove
it, since Rust will optimize its representation into the same thing
you'd get if you replaced LightSpan by Span.
Prev-Benchmark: {"nixpkgs-attrnames":{"kbytes":"233824","system":"0.32","user":"2.02"}}
This-Benchmark: {"nixpkgs-attrnames":{"kbytes":"230192","system":"0.29","user":"2.00"}}
Prev-Benchmark: {"pkgsCross.aarch64-multiplatform.hello.outPath":{"kbytes":"458936","system":"0.73","user":"5.36"}}
This-Benchmark: {"pkgsCross.aarch64-multiplatform.hello.outPath":{"kbytes":"451808","system":"0.53","user":"5.10"}}
Change-Id: Ib9e04806850aa1fc4e66e2a042703986440a7b4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10254
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: Adam Joseph <adam@westernsemico.com>
Tested-by: BuildkiteCI
cl/7558 used this expression as a benchmark to justify the introduction
of LightSpan::Delayed:
builtins.length (builtins.attrNames (import ${pkgs.path} {}))
Let's add it as a benchmark case so it can be referenced easily.
Benchmark: {"nixpkgs-attrnames":{"kbytes":"233824","system":"0.32","user":"2.02"}}
Change-Id: Idb6c69ddd284605dd3b5fd9ac5c79a69b9a470b7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10253
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This is just a alias for Result<_, io::Error>, but shorter.
Change-Id: I7c22f61b85e3014885a747b5c1e5abd11b0ef17d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10327
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
This commit fixes b/338 by properly propagating catchables through
comparison operations.
Change-Id: I6b0283a40f228ecf9a6398d24c060bdacb1077cf
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10221
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
Commit 05f42519b5 fixed b/281 by
establishing a hygiene regimen to partition *catchable* errors
(i.e. those which tryEval can detect) from all other errors, like
internal VM failures or I/O errors (which Nix must not be allowed to
detect, since these errors are fundamentally impure).
Unfotunately there are still cases where tvix assumes that anything
other than Value::Bool means it should panic!(). I found another
one, and added a test case for it in:
eval_okay_src_tests_tvix_tests_eval_okay_compare_ordering_catchable_nix
Not yet passing.
Change-Id: I69c62ed9ea5c8f81870e8de5c5fe12dcde849763
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10220
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This commit rewrites Value::nix_cmp_ordering() into an equivalent
nonrecursive form. Except for calls to Thunk::force(), the new form
no longer uses generators, and is async only because of the fact
that it calls Thunk::force().
I originally believed that this commit would make evaluation faster.
In fact it is slightly slower. I believe this is due to the added
vec![] allocation. I am investigating.
Prev-Nixpkgs-Benchmark: {"attrpath":"pkgsCross.aarch64-multiplatform.hello.outPath","peak-kbytes":"460048","system-seconds":"0.68","user-seconds":"5.73"}
This-Nixpkgs-Benchmark: {"attrpath":"pkgsCross.aarch64-multiplatform.hello.outPath","peak-kbytes":"460224","system-seconds":"0.67","user-seconds":"5.84"}
Change-Id: Ic627bc220d9c5aa3c5e68b9b8bf199837cd55af5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10212
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
This is part of a fix for b/338.
We should never use PartialOrd::partial_cmp().
All Nix types except floats are obviously totally-ordered. In
addition, it turns out that because Nix treats division by zero
rather than producing a NaN, and because it does not support
"negative zero", even floats are in fact totally ordered in Nix.
Therefore, every call to PartialOrd::partial_cmp() in tvix is an
error. We have to *implement* this function, but we should never
call it on built-in types.
Moreover, nix_cmp_ordering() currently returns an Option<Ordering>.
I'm not sure what was going on there, since it's impossible for it
to return None. This commit fixes it to return simply Ordering
rather than Option<Ordering>.
Change-Id: If5c084164cf19cfb38c5a15554c0422faa5f895d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10218
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This commit rewrites Value::nix_eq() into an equivalent. Except for
calls to Thunk::force(), the new form no longer uses generators, and
is async only because of the fact that it calls Thunk::force().
I believed that the nonrecursive form would be faster. It is, in
fact, slightly slower. I believe this is due to the vec![]
allocation; I am investigating.
Prev-Nixpkgs-Benchmark: {"attrpath":"pkgsCross.aarch64-multiplatform.hello.outPath","peak-kbytes":"459068","system-seconds":"0.71","user-seconds":"5.39"}
This-Nixpkgs-Benchmark: {"attrpath":"pkgsCross.aarch64-multiplatform.hello.outPath","peak-kbytes":"460048","system-seconds":"0.68","user-seconds":"5.73"}
Change-Id: I10f4868891e4b7475df13f0cbc41ec78dd985dd8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10118
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: Adam Joseph <adam@westernsemico.com>
This commit adds a simple MVP benchmark, built on our nix
infrastructure instead of cargo. It simply runs `tvix-eval` inside
of GNU time, and prints the three essential statistics in a short
JSON blob.
You can run the benchmark with a simple `nix run`, like:
nix run -f . tvix.cli.benchmark-hello
nix run -f . tvix.cli.benchmark-firefox
nix run -f . tvix.cli.benchmark-cross-firefox
Currently these blobs are stored only in the CI logs, which I'm sure
get garbage-collected at some point. We should be putting them in
the git trailers, but that can wait for a future CL.
I tried using `cargo bench` for this but found it incredibly
frustrating. Maybe I'm doing it wrong. It seems to be designed for
microbenchmarks only, and very hard to control. It kept building
all sorts of unnecessary stuff (like the tests), and unlike
crate2nix it was doing all the builds on only a single machine
instead of using more than one machine. Worse, for that single
machine it kept picking my laptop instead of my fast servers! It
seems excessively cargo-flavored for such a straightforward task.
Benchmark: {"hello.outPath":{"kbytes":"244736","system":"0.36","user":"2.76"}}
Benchmark: {"firefox.outPath":{"kbytes":"1506736","system":"2.38","user":"32.01"}}
Benchmark: {"pkgsCross.aarch64-multiplatform.firefox.outPath":{"kbytes":"11334548","system":"10.70","user":"107.07"}}
Change-Id: I85bc046ec551360284d7ecfc81a03914f0085909
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10216
Autosubmit: Adam Joseph <adam@westernsemico.com>
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This adds a criterion.rs-based testbench into tvix-glue.
It can be invoked by running `cargo bench` from inside the `tvix-glue`
crate.
`target/criterion/report/index.html` contains nice graphs.
It's able to diff against the previous run, so you can invoke `cargo
bench` before and after a certain change to reason about the impact in
evaluation performance.
Currently, we need to create a bunch of Evaluator resources inside the
benchmark loop itself, which is a bit annoying, as it leaks into the
things we benchmark.
This should become better with b/262.
Fixes b/322.
Change-Id: I91656a308887baa1d459ed54d58baae919a4aaf2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10245
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
cl/9364 did introduce a warning here, which is visible when building in
release mode - or invoking `cargo bench` in tvix-glue.
Change-Id: Ia82082a58543f0fdd32866fdfcd37d0a5fdfda9c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10261
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
We don't really require the Path to be a PathBuf, we don't even require
it to be a Path, we only need it to be AsRef<Path>>.
This removes some conversion in the from_addr cases, which can just
reuse `url.path()` (a `&str`).
Change-Id: I38d536dbaf0b44421e41f211a9ad2b13605179e9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10258
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
It's been a while since the last sled release, and that one binds to a
pretty old version of zstd, requiring workarounds like cl/10090.
Upstream sled main branch currently has zstd halfway patched out (it's
a no-op, but the feature flag and options are still there), and it's in
that state for a year.
Rather than maintaining our own fork of sled, let's just stop using the
compression feature in sled, dropping the version pin to zstd that way,
removing the need for cl/10090.
This doesn't mean we won't reintroduce per-blob compression - but we
probably just won't let sled take care of the compression, but do it
ourselves - which is necessary for more chunked blob storage anyways.
Even though we do drop the feature flag, we still need to explicitly use
use_compression(false).
Change-Id: I0e4892d29e41c76653272dc1a3625180da6fee12
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10257
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
This only contained the (unused) evaluator.proto file.
Considering we're less likely to have the CLI talk to a long-running
evaluator, but instead embed the evaluator inside the CLI, remove this.
If we add a RPC to speak to an evaluator, we can resurrect this from
git history.
Change-Id: I2196aade55221660330dfd32dc3e52c39ec6ed43
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10241
Reviewed-by: Adam Joseph <adam@westernsemico.com>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Only pass in the proto files that are actually needed to build that
crate. They are already constructed in depot.tvix.$crate.protos.protos.
Change-Id: If4381e6c3350e420ee4ddce1e0513bfe970678a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10240
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Break up the go-bindings derivation. Keep "protos" containing all proto
files (well, and the buf config), and use it for a check phase running
linter and formatter, as well as the existing "go-bindings" attribute
Change-Id: I52cb9d08570bb76452acb831eb711c5b6c0eacfb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10239
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This function converts from a nix_compat::derivation::Derivation to
a BuildRequest.
In addition to the Derivation itself, it needs two lookup functions to
map input paths to their castore nodes.
Change-Id: I0332982f0bc7933a5fda137fe39d5a850639d929
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10236
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
