Commit graph

19237 commits

Author SHA1 Message Date
Florian Klink
ceca00a866 chore(third_party/crate2nix): apply patches to fix cross
Together with temporarily pointing nixpkgs past
b10994c38c61038970a19fa60bfbec21a61755cc, this now fixes cross-
compilation for tvix.

This incorporates the changes proposed in https://cl.tvl.fyi/9888 and
https://cl.tvl.fyi/9889, but by fixing it in crate2nix, and using the
(re-)generated version of it.

Changes were sent upstream at
https://github.com/nix-community/crate2nix/pull/309,
this pulls in a minimal patch for now.

Change-Id: I70bb6f003bbc3e89de9c4eb4985ea4708ac3a9fd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9890
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-11-02 09:32:54 +00:00
Florian Klink
5a0ae8312c docs(tvix/castore): add notes on verified streaming
These are still a bit WIP, but should already explain the idea better
than not having any documentation about how we intend to implement this
at all.

Let's put this into the repo, as a working document.

Co-Authored-By: edef <edef@edef.eu>
Change-Id: Ic0c5195cdf0a33c8382af1b5cff9064fe6ef9ffb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9894
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-11-02 09:08:20 +00:00
Florian Klink
beae3a4bf1 chore(tvix/castore): move data model docs to here
These describe the castore data model, so it should live in the castore
crate.
Also, some minor edits to //tvix/store/docs/api.md, to honor the move of
the castore bits to tvix-castore.

Change-Id: I1836556b652ac0592336eac95a8d0647599f4aec
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9893
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-11-02 09:08:20 +00:00
Florian Klink
d545f11819 chore(tvix): regenerate Cargo.nix
Change-Id: I96e01f938a46d12a94da85968caaf190d041b9ab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9887
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-31 12:11:19 +00:00
Florian Klink
ea92ba788c chore(third_party/overlay): use crate2nix upstream
Apply the patch that didn't land using `patches`.

Change-Id: Id87675cd44123eea8cc2175efe04f61421f26f37
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9886
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-10-31 12:11:13 +00:00
Vincent Ambo
54def82825 chore(3p/sources): spooky channel and overlay bump (halloween '23)
Change-Id: I88542b467c0ff22e4238a9df5b5645cc6aeb4cc0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9885
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-31 06:19:50 +00:00
Vincent Ambo
67db631db8 chore(3p/sources): bump channels & overlays (2023-10-29)
* //3p/nixpkgs: use py3status from stable for //users/grfn

* //3p/nixpkgs: use electrum from stable for //users/tazjin

* //3p/overlays: update crate2nix vendor hash
  not sure why this hash is not stable if the package isn't updated,
  but whatever

Change-Id: Ia7a13ac35b9c5534eefa53eff66e47ea1c32c62a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9830
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-30 16:33:22 +00:00
Vincent Ambo
67fed3d76d chore(tools/depotfmt): use Go version from buildGo
This is required because Go 1.18 is actually being deleted. I've
applied the formatting breakage that it introduces (such as breaking
comment formatting), because I can't be bothered to try and work
around broken Go stuff.

Change-Id: Ica7cee0d01228845d6a766079fef36df99a3da96
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9832
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-30 16:33:21 +00:00
Florian Klink
c37c9cc770 feat(users/flokli/nixos): add direnv support
Expose `deps` separately, add a direnv with PATH_add for it to bring
tooling into $PATH.

Change-Id: I432cd2b082cad89e08bef78dc4653e10e137cd6b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9842
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-30 14:32:18 +00:00
Florian Klink
5b03bebce8 feat(users/flokli/nixos): use lazydeps for shell
Avoid having to re-enter the shell whenever the config is changed.

Change-Id: Ib9f6bb4075e29acaeb4863d64c017695ca85b60b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9841
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-30 12:15:36 +00:00
Florian Klink
5d2789d1ad feat(users/flokli/archeology): add awscli, htop, kitty terminfo
Change-Id: Ib7ae1871a5d0b16a68c79b68e7e79fd302da79bd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9840
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-30 12:15:36 +00:00
Florian Klink
f9323d5aa5 feat(users/flokli): use nix-copy-closure instead of nix copy
nix copy seems to stall on the EC2 box for unknown reasons.

Change-Id: I30639a52758814968d3b54d716522fb88db80cfe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9839
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: edef <edef@edef.eu>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-30 12:15:35 +00:00
Florian Klink
d83b574be7 feat(users/flokli): expose readTree targets to CI
Change-Id: I3ea801b47267f4c985c2ab5cb1b79b2659894307
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9838
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-30 12:15:35 +00:00
Florian Klink
9e2f1f4583 refactor(users/flokli): move common stuff to archeology profile
Change-Id: I8470c0a2416c0c397e009affb44f8c7a852cd526
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9837
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-30 12:15:35 +00:00
Florian Klink
18d582f96b feat(third_party/nixpkgs): allow setting crossSystem
This allows cross-compiling depot targets to other architectures, so
`nix-build --argstr crossSystem aarch64-linux -A tvix.nar-bridge`
will emit a cross-compiled aarch64-linux binary.

There's still some other cross-related issues in crate2nix to sort out
for crate2nix builds, but this CL can already land.

Co-Authored-By: raitobezarius <tvl@lahfa.xyz>
Change-Id: I467d49d125dd707a4142bfde18eea4a1c3afaf70
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9755
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-10-30 10:43:52 +00:00
Florian Klink
71fa4110fa feat(users/flokli): add archeology-ec2
This add the EC2 box config to the repo.

Change-Id: Id7a888a2cfbf1454cd9f9465018df377e14b4e9f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9836
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-30 09:31:49 +00:00
Florian Klink
9a1e5cf4c7 chore(third_party/gerrit-queue): move to tvl overlay
Bump to a version including https://github.com/flokli/gerrit-queue/pull/15

Change-Id: Ie316498ca2c608e5489901c5705ce5f2dc047f29
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9808
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-10-30 09:07:13 +00:00
Florian Klink
2513ddd2b7 feat(users/flokli/nixos): add deploy script
This adds a deploy-archeology script.

I tried getting morph to work first, but passing it a
depot.ops.nixos.nixosFor seems to be very hard - the NixOS module system
doesn't like the arguments it's called with.

Replace morph with a 3 line bash script, which assumes your ssh_config
contains config for an `archeology` host.

Change-Id: I2bf694c60ded39c201efbbb899f3b5512aa4d0f2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9835
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-10-29 17:49:25 +00:00
Florian Klink
85cd7d64fe refactor(users/flokli): drop some optional args and complexity
Change-Id: Ifdcac829ede4ec469a7ce1b608e78bae11f2766b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9834
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-10-29 17:49:24 +00:00
Florian Klink
48b6242313 feat(users/flokli/nixos): init archeology
Change-Id: Ic31cb8030179ff37b1cc3d3d9241e2582cfe3e5e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9833
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
Autosubmit: flokli <flokli@flokli.de>
2023-10-29 16:31:35 +00:00
edef
12fb5004d9 fix(nix-compat/narinfo): don't allocate errors early
We also switch the MissingField error to &'static str, since we only 
parse a fixed set of fields.

Together, this makes the performance impact of error handling 
negligible in batch happy-path parsing.

Change-Id: I2bd0ef2f5b35fcaced56b32d238eca75ac199ef1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9867
Reviewed-by: flokli <flokli@flokli.de>
Autosubmit: edef <edef@edef.eu>
Tested-by: BuildkiteCI
2023-10-28 17:11:22 +00:00
edef
ae83998381 fix(nix-compat/narinfo): unwrap in the benchmark
We primarily want to measure the speed of the happy path.

Change-Id: Iad0146dde86fc262e2a4b8295bde4eb297b8bf30
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9866
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: edef <edef@edef.eu>
2023-10-28 17:11:22 +00:00
Florian Klink
8dfd8bce04 refactor(tvix/nix-compat/narinfo): return errors
This provides more info about where a NARInfo failed to parse, rather
than just returning None and leaving a library user to manually debug.

Change-Id: I9a28ddd8e5712101483ebe686fdc474c7bbc8e4e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9831
Tested-by: BuildkiteCI
Reviewed-by: edef <edef@edef.eu>
2023-10-28 17:06:49 +00:00
edef
3a77e55d2c refactor(nix-compat/nixbase32): clean up encode
Change-Id: I63da5bd47cd9033e1cc13fbe4b4864514a5343cb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9865
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-28 15:41:46 +00:00
edef
2fce356ce0 fix(nix-compat/nar/writer): use ByteSlice closer to call site
Otherwise, it'll generate unused import warnings in release mode.

Change-Id: I0399d2617a8a8b973a2b8c18f4a0a613d70c48f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9864
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 17:07:06 +00:00
edef
9253bf6632 feat(tvix/nix-compat): add narinfo parsing and serialisation
Change-Id: I72c63414794642ca8d85c3f635f49db888420c40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9852
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 16:08:06 +00:00
edef
b1ad94cc9a refactor(nix-compat/store_path): use nixbase32::decode_fixed
Change-Id: I81471ee57920aa8fa889fb00c7903cdc570af9c5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9863
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 16:08:06 +00:00
edef
50aa74a59c feat(nix-compat/store_path): add StorePathRef
Change-Id: I0d888a55d93e5c22e77cb0264d09757656f731d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9862
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 16:08:06 +00:00
edef
621739037f feat(tvix/castore): carry name in ValidateNodeError::InvalidName
Change-Id: Ica288e94f3f6025d98ef7d56dc5d6f874ec921b7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9861
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 13:56:51 +00:00
edef
e525272019 refactor(tvix): turn nullary enum variants into unit variants
Change-Id: Iad4f2cb4aa92b5bb29ead6050348a8cd3e7b8632
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9860
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 13:56:51 +00:00
edef
520c5a191e refactor(nix-compat/store_path): speed up validate_name fast path
Change-Id: Ie50b29145804777f7644952c65cb42519a8565e3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9859
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 13:56:51 +00:00
edef
c19c6c23cb feat(nix-compat/store_path): validate_name takes AsRef<[u8]>
Change-Id: I8819e2a7b63008a68f4f82035a08b960ac480dc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9858
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 13:56:51 +00:00
edef
b994f692d3 feat(nix-compat/store_path): validate_name over borrowed data
Change-Id: Ifeb6231f48d4ad267a7acd398b4b3b687ee4d560
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9857
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 13:56:51 +00:00
edef
6238a05868 refactor(nix-compat/store_path): don't materialise fingerprint
Change-Id: I6a88531ded05c0dfb9232a0343a465fa02fb6989
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9856
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 13:56:51 +00:00
edef
7f7c1ae7be refactor(nix-compat/store_path): make digest and name private
Change-Id: I62cbe883afcf3dd0c8d4de0e3b845069eb750c97
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9855
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 13:56:51 +00:00
edef
36f2b69de5 fix(tvix/nix-compat): validate store path name length
Change-Id: I89ac0ad147a1872c021ab4235ca46ef3f51d0446
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9854
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 13:56:51 +00:00
edef
99a61def17 fix(tvix/nix-compat): don't box CAHash::Text
Change-Id: I31df3909bc21c9038f9fb831879e60e541242819
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9853
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 12:18:23 +00:00
edef
fdc2e90ef2 feat(tvix/nix-compat): add fixed-length base32 decoding
Change-Id: Iea9d51157c1e38e1734dc1ba752eba9b2372c61b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9850
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 12:18:23 +00:00
edef
3c0e42167f feat(tvix/nix-compat): use AsRef<[u8]> for base32 decode input
Change-Id: If2275dda62b852aedb64b4f9915f7dc4173cabd2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9851
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 12:18:23 +00:00
edef
6a0a75c8e1 refactor(tvix): condense long bytestrings
Change-Id: I3bea0827ec2c8db835334ce378a7bf3a39e9b1a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9849
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 07:37:26 +00:00
edef
55c37a2871 feat(tvix/nix-compat): decode base32 with a lookup table
This also takes input validation out of the loop, leaving the loop
backedge as the sole branch in the hot path.

Change-Id: Id08e6fb9cf5b074780efa09a7ad389352a601bcc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9847
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-10-27 07:31:46 +00:00
edef
67b08469db refactor(tvix/nix-compat): clean up base32 decoding
Change-Id: I8591a3e1075f2281b7dc49f7fa9e0027062a78a5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9846
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 07:31:46 +00:00
edef
4d35a56798 refactor(tvix/nix-compat): simplify encode_len
Change-Id: Idfefd3b5b0d1ffbd4e9208e634a2d9989965a291
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9845
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-27 07:31:46 +00:00
Vincent Ambo
b916c2d0b1 fix(tazjin/emacs): always list ttys locally
Change-Id: Ic05378013aa327616a822336881d9a76c8a22d58
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9829
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-10-25 11:29:28 +00:00
Florian Klink
a89695c4c5 chore(tvix): move buildkite.yml from views/tvix to tvix/
This should remove the ambiguity on where buildkite.yml should show up
at least. We might still want to move workspace.josh into tvix/ too at
some point, but that's blocked on
https://github.com/josh-project/josh/issues/1287

This commit doesn't cause any change in views/tvix, with a previous
version of this commit added to my HEAD, I ran josh-filter to verify it
still points to ce0df88561fee8a4d6052ca9f2a1792f899a202a.

Change-Id: Ic03e92260dc7725bc2fcf01b7c72a097d92c028e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9828
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-10-25 08:22:57 +00:00
Florian Klink
5c91ddbeef chore(nix/buildGo): bump to go 1.19
Reading from lukegb's tea leaves.

Change-Id: I5705fcb3212943e62de6e74cf8460171383121d5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9796
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2023-10-24 20:36:39 +00:00
Florian Klink
6a04ba6bc5 fix(tvix/cli/derivation): fix populate_output_configuration
The `if let` wasn't matching `outputHashAlgo` being unset, and didn't
populate it in that case.

Port the remaining commented-out testcases over to nix-lang based tests.

Change-Id: I140b5643b9ed9d29f9522ec65d98d0b12262d728
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9825
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-10-23 14:57:32 +00:00
Florian Klink
2d99bfc7b7 refactor(tvix/cli): rename a to algo_str
Change-Id: I65d307126647841b50db008c21c155441a6b6cb5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9818
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
Tested-by: BuildkiteCI
2023-10-23 14:57:32 +00:00
Florian Klink
6800f1cdb9 test(tvix/cli/derivation): add test cases for output path calc
Get some test coverage with some calls to builtins.derivation.

The expected output paths were calculated with Nix.

Change-Id: I2ce205ea5244e8ef939d9cacb033283fc6f15d17
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9817
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-23 14:57:32 +00:00
Florian Klink
077bf06c1d refactor(tvix/cli): move evaluator instantiation to helper
Have a private `eval` function in the test module that returns an
EvaluationResult, and migrate the existing tests over to use it, rather
than repeating itself.

Change-Id: I879987700c8507248c644ef03b62a8cb8e308139
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9816
Tested-by: BuildkiteCI
Reviewed-by: raitobezarius <tvl@lahfa.xyz>
2023-10-23 14:57:32 +00:00