Commit graph

20814 commits

Author SHA1 Message Date
Vincent Ambo
73896eb075 feat(ops/www): redirect cs.tvl.fyi to livegrep & cgit
Goodbye, Sourcegraph.

Relates to b/290.

Change-Id: Ic1cf3c1cf52ae17cdcc18c675b4c01d477644a3c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12285
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-23 11:07:15 +00:00
Vincent Ambo
8041ce7cbd chore(web/atward): remove sourcegraph support
We're moving away from sourcegraph to livegrep and so on, as Sourcegraph has
gone fully proprietary.

This removes support for redirecting to Sourcegraph.

Relates to b/290

Change-Id: I04ccf8dfef72113cd49d444151cb0c3eb834845d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12268
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2024-08-23 11:07:15 +00:00
Vincent Ambo
8f6f45097e feat(tazjin/german-string): add constructor that leaks a Vec<u8>
Change-Id: Id0b6b4794128573ddb62e82fd77d080e564978be
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12264
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
2024-08-22 14:02:15 +00:00
Vincent Ambo
012da7d0e1 feat(tazjin/german-string): add persistent construction from statics
Makes it possible to construct persistent German Strings from data with 'static
lifetime. This doesn't leak any additional data.

Change-Id: Ifb3a36ff72b106724883302e4c2195ac54acd352
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12263
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
2024-08-22 14:02:15 +00:00
Vincent Ambo
95ebcc24b0 refactor(tazjin/german-string): add type for encoding storage class
Adds a StorageClassPtr type that is set up to be able to steal bits from an
aligned pointer to encode the storage class of a German String.

Change-Id: I64591174eac1ebcb73e624a59bd107ba1e02c69d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12262
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-08-22 14:02:15 +00:00
Ben Webb
565c0fd24c fix(tvix/castore): u32 -> u64 in DirectoryError::SizeOverflow message
Fix a discrepancy in the error message for DirectoryError::SizeOverflow.

The message indicates that the SizeOverflow error occurs when total size
exceeds u32::MAX, but that's not true. All size fields within the
castore's internal Directory ADT are u64, and the SizeOverflow error is
only returned after a call to the checked_add implementation on u64.

See tvix/castore/nodes/directory.rs +111
and tvix/castore/nodes/directory.rs +88
as of this commit.

Change-Id: I74d161ea8927362e1cb601ba163489aa96fb91b1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12259
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-21 22:05:57 +00:00
Florian Klink
e03ea11bad feat(nix-compat/nix_http): init parse_nar[info]_str
This moves the URL component parsing code we had in nar-bridge to
nix-compat.

We change the function signature to return an Option, not a
Result<_, StatusCode>.

This allows returning more appropriate error codes, as we can
ok_or(…) at the callsite, which we now do: on an upload to an
invalid path, we now return "unauthorized", while on a GET/HEAD, we
return "not found".

This also adds support to parse compression suffixes. While not
supported in nar-bridge, other users of nix-compat might very well want
to parse these paths.

Also fix the error message when parsing NAR urls, it mentioned 32, not
52, which is a copypasta error from the narinfo URL parsing code.

Change-Id: Id1be9a8044814b54ce68b125c52dfe933c9c4f74
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12260
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-08-21 09:40:32 +00:00
Armin Schlegel
2357079891 feat(kontemplate): defaults can now have nested values
See https://b.tvl.fyi/issues/409 for details.

Change-Id: Ibb54fab7a78e0e5f708c2a7dc8bb26ac0b2b4689
Signed-off-by: Armin Schlegel <a.schlegel@gridx.de>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11972
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-08-20 17:09:57 +00:00
Florian Klink
2beabe968c refactor(nix-compat/store_path): make StorePath generic on S
Similar to how cl/12253 already did this for `Signature`, we apply the
same logic to `StorePath`.

`StorePathRef<'a>'` is now a `StorePath<&'a str>`, and there's less
redundant code for the two different implementation.

`.as_ref()` returns a `StorePathRef<'_>`, `.to_owned()` gives a
`StorePath<String>` (for now).

I briefly thought about only publicly exporting `StorePath<String>`
as `StorePath`, but the diff is not too large and this will make it
easier to gradually introduce more flexibility in which store paths to
accept.

Also, remove some silliness in `StorePath::from_absolute_path_full`,
which now doesn't allocate anymore.

Change-Id: Ife8843857a1a0a3a99177ca997649fd45b8198e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12258
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-20 15:14:27 +00:00
Florian Klink
413135b925 refactor(nix-compat/aterm): update function names a bit
Don't call functions bstr or str when they return BString or String,
it's confusing.

Rename them to `string` and `bytes`. We might be able to generalize over
this being BString or Vec<u8> later.

Change-Id: I8198551ed3ba1cfc479bf7e3cbbc13a426faf4c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12257
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-20 15:14:27 +00:00
Ilan Joselevich
e579d3869d feat(tools/depotfmt): Get gofmt from pkgs.go instead of buildGo
This allows importing just //tools/depotfmt.nix when using a josh
workspace.

Change-Id: Idf7eb438be1cc1e60445aace982fdca1f4367595
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12254
Tested-by: BuildkiteCI
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
Reviewed-by: flokli <flokli@flokli.de>
2024-08-20 05:13:39 +00:00
Aspen Smith
01af37c532 chore(aspen/system): Turn off disableWhileTyping for libinput
I... actually really hate this, it turns out. Plus it breaks video
games, lol.

Change-Id: I4b86e38ee78f5f3b6ffec5e7cc4f1dc054d91c56
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12256
Reviewed-by: aspen <root@gws.fyi>
Autosubmit: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
2024-08-20 02:27:13 +00:00
Ryan Lahfa
a4e40d1dfa feat(tvix/nix-compat): entry locator in listing structures
This implements a simple DFS locator in listing structures.

It is interoperable with the Rust standard library paths, we also build our
own errors to restrict path values to reasonable secure defaults, e.g.
relative paths with no `..` component.

Tests are added for this new feature for a positive and a negative
check.

In addition, a path validation test was added. The Windows-style prefix
is gated on the Windows platform as UNIX does not parse `C:\\` as a
`Component::Prefix(_)` but as a `Component::Normal(_)`.

Change-Id: Iae2a80bebd8138e41af94aa7d09f2842c3c5a786
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12255
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-08-19 22:32:09 +00:00
Florian Klink
a259613c76 feat(nix-compat/narinfo/signature): generalize name field
Requiring `name` to be a `&str` means it'll get annoying to pass around
`Signature`, but being able to pass them around in an owned fashion is
kinda a requirement for a stronger typed `PathInfo` struct, where we
want to have full ownership.

Rework the `Signature` struct to become generic over the type of the
`name` field. This means, it becomes possible to have owned versions
of it.

We don't want to impose `String` or `SmolStr` for example, but want to
leave it up to the nix-compat user to decide.

Provide a type alias for the existing `&str` variant (`SignatureRef`),
and use it where we previously used the non-generic `Signature` one.

Add some tests to ensure it's possible to *use* `Signature` with both
`String` and `SmolStr` (but only pull in `smol_str` as dev dependency
for the tests).

Also, add some more docstrings, these were a bit sparse.

Change-Id: I3f75691498c6bda9cd072d2d9dac83c4f6c57287
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12253
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-08-19 19:46:34 +00:00
Ryan Lahfa
7612cb4c31 feat(tvix/nix-compat): add a basic listing deserializer
.ls files are useful to seek in a NAR without parsing it entirely.

The responsibility of validating the files is on the caller.

Change-Id: I5d1da28b5479c38f20ca5babe60e362a2217c9ea
Signed-off-by: Ryan Lahfa <tvl@lahfa.xyz>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12196
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-19 19:18:42 +00:00
Ilan Joselevich
73e16c1855 feat(tvix): drop usage of sparseTree in favor of lib.sourceByRegex
We can avoid depending on things outside //tvix by just using a similar
util from nixpkgs.

Change-Id: I9ea3e1f0a8a059ea10caaec173569ba9f316aec6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12247
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-19 16:56:19 +00:00
Ilan Joselevich
bc967c538d feat(tvix/boot/tests): use pkgs.nixos instead of nixosFor
Using pkgs.nixos directly allows us to create a smaller nixos closure
for the tests and also not depend on things in depot.ops which can be
beneficial for extending the tvix josh workspace.

Change-Id: Ic6ad2122733418114b43aa692d6e42ac1e308eeb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12251
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: Ilan Joselevich <personal@ilanjoselevich.com>
2024-08-19 16:55:22 +00:00
Florian Klink
70d199ea44 refactor(nix-compat/narinfo/signature): use ed25519::SignatureBytes
It's a `[u8; SIGNATURE_LENGTH]` type alias, and conveys what we're
accepting or returning a bit nicer.

Change-Id: I974cd97d56d383e51417eb0f26e1431a05711922
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12252
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2024-08-19 14:34:35 +00:00
Vincent Ambo
ed4f68b1d3 feat(tazjin/german-string): add constructor method from owned data
Makes it possible to construct a GermanString from an owned byte vector, without
having to clone the data.

This is done by "disowning" the vector using ManuallyDrop to access its internal
pointer. For transient strings, this memory is then owned (and freed) by the
GermanString instance.

Small strings are copied out of the heap and stored inline as before, to avoid
any dereferencing operations.

Change-Id: I754736099f71d646d430aed73e558a5a7626c394
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12249
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-08-19 13:19:40 +00:00
Vincent Ambo
0d6d31bc66 feat(tazjin/german-string): add pointer equality check in Eq
This can short-circuit two large string comparisons.

Change-Id: If45e7cf33921fe571482dc710c27ef8cc7c70885
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12245
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-08-19 13:19:40 +00:00
Vincent Ambo
a521f9e5f8 chore(tazjin/german-string): add Nix build
Change-Id: I1e8c500c9bb4ba365ba32f53f60bcf657f62ed62
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12243
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-08-19 13:19:40 +00:00
Vincent Ambo
56d4f01e9f feat(tazjin/german-string): add [Partial]Ord implementation
Change-Id: I8002ec63be45a15160387c21ed1fa8721a7c3eb4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12242
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
2024-08-19 13:19:40 +00:00
Vincent Ambo
86f467cee6 test(tazjin/german-string): add roundtrip proptests
Change-Id: Iee392368252d5c6e96d879b213aee34a301d13d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12241
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-08-19 13:19:40 +00:00
Vincent Ambo
64a085cf52 feat(tazjin/german-string): add Eq impl & corresponding proptests
Change-Id: I66a258fad5d4e249268b9d2743d46b30c5ac1dac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12240
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2024-08-19 13:19:40 +00:00
Vincent Ambo
ab6a4815ff feat(tazjin/german-string): add Debug implementation
Change-Id: I9a4a8a4815ac210c402cfc2feb1fe0606affb327
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12239
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-08-19 13:19:40 +00:00
Vincent Ambo
2284c00417 feat(tazjin/german-string): PartialEq implementation for GermanString
This is where one of the advantages of this string representation starts to
shine: For small strings there's no derefencing any heap memory at all, and for
the long representation we can compare the prefix before moving on.

Change-Id: Iac333a52e8d7c9dd09e33dbcf51754e321c880e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12238
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-08-19 13:19:25 +00:00
Ilan Joselevich
0714184b1f feat(tvix/boot): Use buildGoModule instead of buildGo for tvix-init
//tvix depending less on other parts of depot is prefered as it will
help with extending the josh workspace of tvix.

Change-Id: Ifcac3af1782dfd82e7543cb4c3ae57fbd186edff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12250
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-08-19 12:42:44 +00:00
Ilan Joselevich
7b1acc5c16 feat(third_party/gitignoreSource): Get gitignore-nix from sources (niv)
It's easier to implement readTree/depot polyfills for gitignoreSource
when it's imported from third_party.sources, rather than in a file at
//third_party.gitignoreSource.

Change-Id: I1323f932bd0feeb2c50ccc76397a80e035842992
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12248
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-08-19 12:14:08 +00:00
Vincent Ambo
9aa1c2dd19 fix(tazjin/german-string): add maximum length check
Change-Id: I4fd9b2a31749d65632b662807c4e703a9df9ec9e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12237
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-08-19 11:50:11 +00:00
Vincent Ambo
d8640b6e67 feat(tazjin/german-string): add data accessors for &str and &[u8]
Change-Id: I992e625861f79ef6d9993e8caee4e02d3fc5557e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12236
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
2024-08-19 11:50:11 +00:00
Vincent Ambo
4d5abbe232 feat(tazjin/german-string): add Drop impl for transient strings
All of these strings are currently transient (the storage class is not yet
represented anywhere), and the ones that are heap allocated need to be
deallocated when the transient string dies.

Change-Id: Iba0ca926df5db7594f304c5d5318db9d69c6f42c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12235
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-08-19 11:50:11 +00:00
Vincent Ambo
ef75a6300b feat(tazjin/german-string): init initial representation
This adds an initial implementation of the so-called "German Strings" in Rust.

https://cedardb.com/blog/german_strings/

This implementation is *far from* complete, the only thing that can be done
right now is construct a string, and even that I'm not fully happy with.

Change-Id: I2697932a0ef373be76ffd14d59677493a5783b58
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12234
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-08-19 11:42:56 +00:00
Vincent Ambo
abff828ccc refactor(tvix/eval): remove use of imbl::OrdMap
Removes imbl::OrdMap in favour of an Rc over the standard library's BTreeMap,
which allows us to drop the imbl dependency completely.

In my local tests this is actually slightly faster for `hello` and `firefox`.

Change-Id: Ic9597ead4e98bf9530f290c6a94a3c5c3efd0acc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12201
Reviewed-by: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
2024-08-19 11:02:50 +00:00
Vincent Ambo
adf9b4c54a refactor(tvix/eval): remove use of imbl::Vector
This vector type has served us well for now, but it contains internal refcounts
which are incompatible with upcoming changes related to garbage collection.

The performance impact of this change within all benchmarks I ran was within the
margin of error:

[nix-shell:/tmp/perf]$ hyperfine "./before -E '(import <nixpkgs> {}).firefox.outPath' --log-level ERROR --no-warnings"
Benchmark 1: ./u64 -E '(import <nixpkgs> {}).firefox.outPath' --log-level ERROR --no-warnings
  Time (mean ± σ):      7.528 s ±  0.272 s    [User: 6.578 s, System: 0.631 s]
  Range (min … max):    7.160 s …  8.012 s    10 runs

nix-shell:/tmp/perf]$ hyperfine "./std-vec -E '(import <nixpkgs> {}).firefox.outPath' --log-level ERROR --no-warnings"
Benchmark 1: ./std-vec -E '(import <nixpkgs> {}).firefox.outPath' --log-level ERROR --no-warnings
  Time (mean ± σ):      7.515 s ±  0.178 s    [User: 6.508 s, System: 0.652 s]
  Range (min … max):    7.276 s …  7.861 s    10 runs

Change-Id: Ib95f871956e336a1e5771f6293583854b1efb276
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12197
Reviewed-by: aspen <root@gws.fyi>
Tested-by: BuildkiteCI
2024-08-19 11:02:50 +00:00
Vincent Ambo
d6c57eb957 refactor(tvix/eval): ensure VM operations fit in a single byte
This replaces the OpCode enum with a new Op enum which is guaranteed to fit in a
single byte. Instead of carrying enum variants with data, every variant that has
runtime data encodes it into the `Vec<u8>` that a `Chunk` now carries.

This has several advantages:

* Less stack space is required at runtime, and fewer allocations are required
  while compiling.
* The OpCode doesn't need to carry "weird" special-cased data variants anymore.
* It is faster (albeit, not by much). On my laptop, results consistently look
  approximately like this:

  Benchmark 1: ./before -E '(import <nixpkgs> {}).firefox.outPath' --log-level ERROR --no-warnings
  Time (mean ± σ):      8.224 s ±  0.272 s    [User: 7.149 s, System: 0.688 s]
  Range (min … max):    7.759 s …  8.583 s    10 runs

  Benchmark 2: ./after -E '(import <nixpkgs> {}).firefox.outPath' --log-level ERROR --no-warnings
  Time (mean ± σ):      8.000 s ±  0.198 s    [User: 7.036 s, System: 0.633 s]
  Range (min … max):    7.718 s …  8.334 s    10 runs

  See notes below for why the performance impact might be less than expected.
* It is faster while at the same time dropping some optimisations we previously
  performed.

This has several disadvantages:

* The code is closer to how one would write it in C or Go.
* Bit shifting!
* There is (for now) slightly more code than before.

On performance I have the following thoughts at the moment:

In order to prepare for adding GC, there's a couple of places in Tvix where I'd
like to fence off certain kinds of complexity (such as mutating bytecode, which,
for various reaons, also has to be part of data that is subject to GC). With
this change, we can drop optimisations like retroactively modifying existing
bytecode and *still* achieve better performance than before.

I believe that this is currently worth it to pave the way for changes that are
more significant for performance.

In general this also opens other avenues of optimisation: For example, we can
profile which argument sizes actually exist and remove the copy overhead of
varint decoding (which does show up in profiles) by using more adequately sized
types for, e.g., constant indices.

Known regressions:

* Op::Constant is no longer printing its values in disassembly (this can be
  fixed, I just didn't get around to it, will do separately).

Change-Id: Id9b3a4254623a45de03069dbdb70b8349e976743
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12191
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-08-19 11:02:50 +00:00
Florian Klink
ddca074886 docs(tvix/glue/tvix_io): update doc-comments
Since we switched from reference scanning to string context, this only
handles the `__corepkgs__` hack. Update the docstrings.

Change-Id: Ie857c8c99ae1cdb4697323ec738f88be0580df3e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12246
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2024-08-19 10:24:37 +00:00
Florian Klink
bb5d7c9678 feat(ops/pipelines): support buildkite retries
cl/12228 did enable automatic retries for some flaky tests, which
generally did work, as can be seen in
https://buildkite.com/tvl/depot/builds/35893

However, "🦆" still reports as failing, because we check the number
of steps to be nonzero, which is not the case if retries have happened.

We cannot check for the overall status of the build, as it's still
"RUNNING", but instead of counting all failed steps so far, we can query
all failed jobs and then filter out the ones that were already retried.

Change-Id: Ib9d27587c8a8ba7970850812c4302fecdc4482e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12233
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-08-19 10:07:08 +00:00
Florian Klink
98863e7312 docs(views): update where josh-filter is coming from
We started using josh from nixpkgs since cl/11457, but forgot to update
this documentation.

Change-Id: I6e07227bcd3e955076d46146024edd89b69f08f7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12244
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
2024-08-18 20:44:39 +00:00
Aspen Smith
5382cbb93a refactor(tvix/eval): Pull context out into its own module
I'm gonna be doing some poking around in the internals of Context, so in
preparation this pulls it out into its own module.

Change-Id: I72ea7df80b5f36f838934ee07bdba66874c334c9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12189
Autosubmit: aspen <root@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-08-18 19:49:14 +00:00
Florian Klink
e086c76ee9 refactor(tvix/castore): have SymlinkTarget-specific errors
Don't use ValidateNodeError, but SymlinkTargetError.

Also, add checks for too long symlink targets.

Change-Id: I4b533325d494232ff9d0b3f4f695f5a1a0a36199
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12230
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
2024-08-18 17:22:21 +00:00
Florian Klink
56fa533e43 refactor(tvix/castore): have PathComponent-specific errors
Don't use DirectoryError, but PathComponentError.

Also add checks for too long path components.

Change-Id: Ia9deb9dd0351138baadb2e9c9454c3e019d5a45e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12229
Tested-by: BuildkiteCI
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
2024-08-18 17:22:21 +00:00
Florian Klink
0cfe2aaf6a feat(tvix/castore/proto): add owned conv to castore::Directory
Replace the hand-rolled code comparing names with a try_fold. Also, make
it slightly stricter here, detecting duplicates in the same fields
earlier.

Change-Id: I9c560838ece88c3b8b339249a8ecbf3b05969538
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12226
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2024-08-18 17:22:21 +00:00
Florian Klink
96832c0411 feat(tvix/castore): add Directory::try_from_iter()
This provides a batched variant to construct a Directory, which reuses
the previously calculated size.

Checking and inserting code is factored out into a check_insert_node
function, taking the current size as a parameter and returning the new
size.

Change-Id: Ia6c2970a0c12181b7c40e63cf7ce8c93298ea37c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12225
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2024-08-18 16:49:58 +00:00
Florian Klink
76839683a7 fix(tvix/boot): workaround test flakiness
The boot tests are sometimes flaky, and we don't want them to
periodically fail other build. Have Buildkite auto-retry them 2 times
on failure.
Logs for individual failures are still available, so it won't hinder
flakiness debuggability.

See https://buildkite.com/docs/pipelines/command-step#retry-attributes
for a documentation of a parameter, and cl/8983 for the introduction of
that feature to //nix/buildkite.

Change-Id: I1c0d25fa1d0ca940b3bdcd145ede87154b0c28eb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12228
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Ilan Joselevich <personal@ilanjoselevich.com>
Tested-by: BuildkiteCI
2024-08-17 20:36:05 +00:00
Florian Klink
c75b0d08ce chore(tvix/glue): drop some explicit allow(clippy::mutable_key_type)
This is covered by clippy.toml these days.

Change-Id: I2330af5781844d5f9d975793d770efcea48d371b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12223
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-17 15:59:10 +00:00
Florian Klink
5d01892274 refactor(tvix/castore): add name back to edge weights
Make this a proper struct with named fields. We apparently never access
B3Digest in there, so it can be removed.

Change-Id: Ifc07310393e1afb0a835778eae137a19b54070b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12224
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-08-17 15:59:10 +00:00
Florian Klink
21ceef4934 refactor(tvix/castore): add into_nodes(), implement consuming proto conv
Provide a into_nodes() function on a Directory, which consumes self and
returns owned PathComponent and Node.

Use it to provide a proper conversion from Directory to the proto
variant that doesn't clone.

There's no need for the one taking only &Directory, we don't use it
anywhere, and once someone needs that they might as well clone Directory
before converting it.

Update all other users of the `.nodes()` function to use `.into_nodes()`
where applicable, and avoid some more cloning there.

Change-Id: Id4577b9eb173c012e225337458898d3937112bcb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12218
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-17 15:59:10 +00:00
Florian Klink
5ec93b57e6 refactor(tvix/castore): add PathComponent type for checked components
This encodes a verified component on the type level. Internally, it
contains a bytes::Bytes.

The castore Path/PathBuf component() and file_name() methods now
return this type, the old ones returning bytes were renamed to
component_bytes() and component_file_name() respectively.

We can drop the directory_reject_invalid_name test - it's not possible
anymore to pass an invalid name to Directories::add.
Invalid names in the Directory proto are still being tested to be
rejected in the validate_invalid_names tests.

Change-Id: Ide4d16415dfd50b7e2d7e0c36d42a3bbeeb9b6c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12217
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-08-17 15:59:10 +00:00
Florian Klink
8ea7d2b60e refactor(tvix/castore): drop {Directory,File,Symlink}Node
Add a `SymlinkTarget` type to represent validated symlink targets.
With this, no invalid states are representable, so we can make `Node` be
just an enum of all three kind of types, and allow access to these
fields directly.

Change-Id: I20bdd480c8d5e64a827649f303c97023b7e390f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12216
Reviewed-by: benjaminedwardwebb <benjaminedwardwebb@gmail.com>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-08-17 09:46:01 +00:00
Florian Klink
49b173786c refactor(tvix/castore): remove name from Nodes
Nodes only have names if they're contained inside a Directory, or if
they're a root node and have something else possibly giving them a name
externally.

This removes all `name` fields in the three different Nodes, and instead
maintains it inside a BTreeMap inside the Directory.

It also removes the NamedNode trait (they don't have a get_name()), as
well as Node::rename(self, name), and all [Partial]Ord implementations
for Node (as they don't have names to use for sorting).

The `nodes()`, `directories()`, `files()` iterators inside a `Directory`
now return a tuple of Name and Node, as does the RootNodesProvider.

The different {Directory,File,Symlink}Node struct constructors got
simpler, and the {Directory,File}Node ones became infallible - as
there's no more possibility to represent invalid state.

The proto structs stayed the same - there's now from_name_and_node and
into_name_and_node to convert back and forth between the two `Node`
structs.

Some further cleanups:

The error types for Node validation were renamed. Everything related to
names is now in the DirectoryError (not yet happy about the naming)

There's some leftover cleanups to do:
 - There should be a from_(sorted_)iter and into_iter in Directory, so
   we can construct and deconstruct in one go.
   That should also enable us to implement conversions from and to the
   proto representation that moves, rather than clones.

 - The BuildRequest and PathInfo structs are still proto-based, so we
   still do a bunch of conversions back and forth there (and have some
   ugly expect there). There's not much point for error handling here,
   this will be moved to stricter types in a followup CL.

Change-Id: I7369a8e3a426f44419c349077cb4fcab2044ebb6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/12205
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: benjaminedwardwebb <benjaminedwardwebb@gmail.com>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-08-17 09:45:58 +00:00