Commit graph

20032 commits

Author SHA1 Message Date
Florian Klink
3dd708690f refactor(tvix/store/import): use B3Digest in log_node
Change-Id: I2347bbae8e7d4e19eeed4a3fb13729d0a94feedd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11195
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-18 16:10:15 +00:00
Florian Klink
a65c49fbc3 refactor(tvix/store/pathinfo/nix_http): NixHash to display NARHash
This will give us the sha256: prefix, and hashes we're more used to, in
that context.

Change-Id: I72e42fe685e365ba9baa7cd81001387d239fa7c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11194
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-03-18 16:10:05 +00:00
Florian Klink
d62c6e642d refactor(tvix/store/nar): use B3Digest in RenderError
Ensure consistent formatting of blake3 digests.

It looks like we don't actually construct these error types anywhere, so
no code needs to be refactored to it currently, but still good to be
consistent.

Change-Id: I49dc8a7f3cb4245ac06b9a6a44b72060434a3d32
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11193
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-03-18 16:10:05 +00:00
Florian Klink
499bc2f7ee refactor(tvix/castore/blobsvc): use B3Digest Display impl
We don't need to use BASE64 here on our own, B3Digest has a Display
impl.

This will also make sure the `b3:` digest is present in field values.

Change-Id: I0ce6ee0f7e7e99fb9b16872953a1b742e99be291
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11192
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-03-18 16:10:05 +00:00
Florian Klink
8d177a18a5 feat(tvix/boot): make VM cmdline and asserted VM output configurable
This can be used in the future to boot proper NixOS VMs, which want a
init= in their cmdline.

Change-Id: Iad4a25d5081f3a6af1c7f62f15853f3afaae4a0c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11190
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-03-18 16:10:05 +00:00
Florian Klink
898830b332 fix(tvix/boot/tests): disable otlp
We don't have a OTLP collector inside the Nix build, this removes some
noise from the log about not being able to reach it.

Change-Id: I058839c8f214821b536306c8c30e1a8e7192153b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11189
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-18 16:10:05 +00:00
Florian Klink
997b59e452 feat(tvix/boot): support seeding closures via nar-bridge
This updates the parameters mkBootTest can be called with. It now
accepts a `path`, and then either importPathName needs to be set, or
isClosure needs to be set to true.

The former activates the existing functionality, tvix-store import is
used to import contents  as a NAR-addressed store path.
The latter uploads the path as a closure (so including its references,
and keeping the store paths intact) to tvix-store.
We use nar-bridge, and the HTTP interface it provides to do this.
As `nix copy` can't be used inside a Nix build, we use
`pkgs.mkBinaryCache` to come up with the .narinfo and .nar files that
would be in a binary cache, and then use a bit of GNU Parallel and bash
to upload store paths ourselves.

Change-Id: Icfa5c0af0c22ab5418686947aa2c060f5987b873
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11188
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-03-18 16:10:05 +00:00
Florian Klink
8fb1d0ad4b docs(tvix/boot): document we could use a smaller kernel here.
We currently use a pretty big kernel to boot the tvix VMs, with a lot
of drivers.

It can probably be slimmed down significantly, at least on the hardware
driver and filesystem front.

Change-Id: I4c7f73fb2dafbf2bcdec8057d2b14a9b0e9b3275
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11187
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-03-18 16:10:05 +00:00
Florian Klink
fca3926a62 feat(tvix/boot/tests): make urls and paths configurable
Fork off a long-living tvix-store daemon in the background, and have
both the import, as well as the virtiofs invocation connect to that one,
rather than use the underlying storage directly.

This also opens up the potential to have other things seed the store
(like nar-bridge).

Change-Id: Ie13aedaf7ab31e5ebe78a7be42f144af3920fa9c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11186
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-18 16:10:05 +00:00
Florian Klink
05bdb68523 feat(tvix/blobservice/object_store) more logging
Have derive_{blob,chunk}_path emit trace-level events for both the
values they're called with, as well as the return value.

With RUST_LOG in place, it doesn't get lost in other unrelated noise.

Change-Id: Id2451e3657324eff482841eb26a22d19e22bde30
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11136
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-03-18 16:10:05 +00:00
Florian Klink
c0e16059e6 feat(tvix/store): support RUST_LOG env var
This allows selectively increasing the log level for only parts of the
stack.

For example, the following RUST_LOG env var enables "tracing" level
logging for `tvix_store` and `tvix_castore`, while keeping it at "info"
for the rest of the stack:

export RUST_LOG='info,tvix_store=trace,tvix_castore=trace'

It only affects logs, not traces (if enabled).

Change-Id: Ib936bd132a405f216e75c843db83fbd71d20a18a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11182
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-18 16:10:05 +00:00
Florian Klink
82f8ce8b7d feat(tvix/castore/blobsvc/grpc): read data in chunks
Whenever this encounters an open_read(), it'll first check for more
granular chunking. If there's more granular chunking data available, a
ChunkedReader is constructed (which supports seeking backwards).

This currently is still a bit stupid, and doesn't compose, as
`ChunkedReader` uses `self` as the `BlobService` to ask for the
individual chunks.

In store composition future, we might want to compose this differently,
essentially constructing `ChunkedReader` with another `BlobService`
representing the entire hierarchy, so there's a chance to locally cache
things, and do less requests.

Change-Id: I22e0df4d6245f666d083b4f0b7114d3ac41d1dce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11185
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-18 16:10:05 +00:00
Florian Klink
50c81d7838 feat(tvix/nar-bridge): support listening on unix sockets
This simply checks for the address to contain slashes, and if so, opens
a unix socket, rather than a tcp one. We'll use this in //tvix/boot
tests to simplify waiting for nar-bridge to be up.

Change-Id: I7184f548d57142b1c5f698a1f0c30343489373a5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11184
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
2024-03-18 14:57:30 +00:00
Florian Klink
70bbf23767 refactor(tvix/castore/src/blobservice): remove useless else case
Change-Id: I09000371a1d8ff212ab46050d1a480509c6ffe70
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11183
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-03-18 14:25:37 +00:00
Florian Klink
3b1c9172f6 feat(tvix/castore): impl Debug for B3Digest
Use the same format as Display, b3: followed by the base64
representation. This makes the debug implementation of everything
containing a b3 digest much nicer to read.

Change-Id: I3ca3154d0b6fb07781c8f9c83ece3ff1a6957902
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11181
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-03-17 22:18:20 +00:00
Vincent Ambo
fa8e706b9b fix(3p/overlays): upgrade tpm2-pkcs11, but add unmerged patch
Instead of pinning to an old version, move forward but with a fix for
the critical bug that's been preventing me from upgrading.

The project seems to be unmaintained upstream, but I took the fix from
the open pull requests.

Change-Id: I85c8f780b1e363bac4060dd89b1930a6e59ce2a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11145
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
2024-03-17 11:30:00 +00:00
sterni
e220d80727 chore(3p/sources): Bump channels & overlays
- agenix has not been updated (https://github.com/ryantm/agenix/pull/241).

- //tvix: regenerate protobuf files

- //tvix:clippy: work around https://github.com/rust-lang/rust-clippy/issues/12281
  which exclusively causes false positives in our code at the moment.

Change-Id: I38d2f4c0e6d1abc92be360b06f58e6d40e7732a3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11127
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <tazjin@tvl.su>
2024-03-17 10:56:07 +00:00
Profpatsch
32c5674962 feat(users/Profpatsch/whatcd-resolver): render anon json+ld objs
Some json+ld objects cannot be expanded any further, they are missing
the `@id` tag. Now we also render them as objects.

Change-Id: I1c8f26f3c34e69420c349e66a3ce5a36dc55a1ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11173
Reviewed-by: Profpatsch <mail@profpatsch.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2024-03-17 01:36:56 +00:00
Profpatsch
3b9fb1aa60 feat(users/Profpatsch/whatcd-resolver): add autorefresh
Adds a little polling mechanism that compares against an ID that is
generated anew every time the server is restarted.

Works well together with shortcuttable.

Change-Id: Icc6745b599e43881c14349794feaf5794cfe6777
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11172
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2024-03-17 01:36:56 +00:00
Profpatsch
efa5fe1239 feat(users/Profpatsch/whatcd-resolver): add services & readme
Gives a story for running both jaeger and the reverse proxy, and the
program proper.

`postgres` is still run from the client.

Adds a little readme.

Change-Id: I11185028541c0214e152fd76c9c72b7e2eb11122
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11171
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
2024-03-17 01:21:14 +00:00
Profpatsch
8ca42dc0cd feat(users/Profpatsch): add nix-run, nix-run-bin, nix-eval
Change-Id: I76f6fa4866c5792c4b9babe2c3a16401162ad833
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11170
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2024-03-17 01:21:14 +00:00
Profpatsch
11a2098e0b feat(users/Profpatsch/my-prelude): update libraries
The latest and greatest!

Change-Id: I34c0e9f41b3b3cc727d9ea89c7ce6a43271b3170
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11169
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2024-03-16 22:36:55 +00:00
Profpatsch
8335076173 feat(users/Profpatsch): add shortcuttable
Initial mockup for a tool which restarts its subprocess on SIGHUP, and
stops everything on SIGTERM and SIGINT.

Change-Id: Ie5260d73c2663d1821eb6623e2bc61d16f6c92cd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11168
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2024-03-16 22:36:55 +00:00
Profpatsch
16461c4100 feat(users/Profpatsch/execline): add setsid
Small wrapper around `setsid(2)` that does not exist in the
execline-style tooling.

Is probably gonna fail iff our process is a session leader already,
but I haven’t tried that out yet.

Change-Id: I1a820f1c6d65ddc29c30995bfd56a760a9d6b341
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11167
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
2024-03-16 22:36:55 +00:00
Profpatsch
981c7fef0e feat(users/Profpatsch/whatcd-resolver): log json+ld fetching
This traces the target of a json+ld fetch.

We also simplify the telemetry stuff by using a pseudo-class
`MonadOtel` everywhere.

I wonder if we can get rid of passing the span to `assertM`, because
it’s kind of an antipattern to be honest.

Change-Id: I1448d643c909a29684fa1ae54037177ba2c20639
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11166
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
2024-03-16 22:36:55 +00:00
Profpatsch
0b06dda9a6 refactor(users/Profpatsch/whatcd-resolver): move AppT & Html out
These functions are just general setup and html helpers, the main file
is getting a bit long otherwise.

Change-Id: I194e9f7f4caa4ce204d510c885dcf5af63d0e76e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11165
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2024-03-16 22:36:55 +00:00
Profpatsch
803d726ed5 chore(users/Profpatsch): remove more really dumb hlints
Change-Id: Id7ca7e5716aba619bfe178c5c0831f2aa16c0257
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11163
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2024-03-16 22:36:55 +00:00
Profpatsch
7e63addad3 chore(third_party/haskell): pa-label: 0.1.0.1 -> 0.1.1.0
chore(users/Profpatsch/whatcd-resolver): use more from Label

New label version contains these helpers.

Change-Id: I3ef2ae9b92f2c798842ae001588b2c6d9f3ab971
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11162
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
2024-03-16 22:36:55 +00:00
Florian Klink
dbf87f3057 chore(tvix): bump tonic to 0.11.0
This bumps tonic and surrounding crates to 0.11.x.

We added support for tonic 0.11.x into tokio-listener
(https://github.com/vi/tokio-listener/pull/4), so that's bumped as well.

Change-Id: Icfade5894403228299836fefb21b2f9ae59dbebb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11156
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-03-16 17:04:12 +00:00
sterni
212f5afdcf fix(tvix/eval): builtins.unsafeGetAttrPos returns path as a string
Change-Id: Ia1058a5ea687f413fbb62365da199032bdbd3fd2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11155
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
2024-03-16 12:52:30 +00:00
Profpatsch
0e0f69881c chore(users/Profpatsch): fix cabal.project & hie.yaml
Change-Id: I39bebe6ce2bde85d26a68b017c0e6d872e9b9705
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11161
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
2024-03-16 12:42:21 +00:00
Profpatsch
28a83478bc fix(users/Profpatsch/httzip): fix PATH
Change-Id: I6f27a947fe609f7d6ce004ce5aaae76b32dbe4fc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11160
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
2024-03-16 12:42:16 +00:00
Florian Klink
fdf9657654 fix(tvix): don't emit rerun-if-changed
`build.rs` emits rerun-if-changed statements for all proto files, as
well as all include paths we pass it.

Unfortunately, due to protobufs include path rules, we need to specify
the path to the depot root itself as an include path, at least when
building impurely with `cargo`. This causes cargo to essentially always
rebuild, as it also puts its own temporary files in there.

Unfortunately, tonic-build does not chase down to individual .proto
files that are included.

Disable emitting these `rerun-if-changed` statements for now.

This could cause cargo to not rebuild protos every time, causing stale
data until the next local `cargo clean`, but considering the protos
change not that frequently, and it'll immediately surface if trying to
build via Nix (either locally or in CI), it's a good-enough compromise.

Change-Id: Ifd279a2216222ef3fc0e70c5a2fe6f87997f562e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11157
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
2024-03-16 09:34:10 +00:00
Florian Klink
56dc4f735b docs(tvix): fix some docstrings
Change-Id: Ife599387d0472cd746b992bd6755a2fb6a0e0dc4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11158
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-03-15 23:16:01 +00:00
Vincent Ambo
d3534ee051 fix(depotfmt): only exclude *imported* tvix tests
When we added the Nix language test suite in cl/6126, we excluded the
whole tvix tests folder from Nix formatting. This is unintentional, as
we probably want *our* tests to be formatted correctly.

Change-Id: I2b66d79e30fae17e75d5a1f8c44e279886091c5f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11154
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2024-03-15 14:22:49 +00:00
sterni
d5b6704d3d chore: move protoCheck into extraStep of //nix/bufCheck
Resolves b/385. I have a feeling bufCheck should be moved to
//tools as well.

Change-Id: I2a2b63d135a2f8bcc982aa1138ff3213c6012f20
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11152
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2024-03-15 11:55:54 +00:00
Florian Klink
5fccbe5939 feat(nix-compat/wire): add read_bytes[_unchecked]
This introduces a version reading sized byte packets. Both read_bytes,
accepting a range of allowed sizes, as well as read_bytes_unchecked,
which doesn't care, are added, including tests.

Co-Authored-By: picnoir <picnoir@alternativebit.fr>
Change-Id: I9fc1c61eb561105e649eecca832af28badfdaaa8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11150
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
Tested-by: BuildkiteCI
2024-03-15 10:35:24 +00:00
Florian Klink
c364c0b4de docs(nix-compat/wire): update docstrings
These are not streams, but AsyncRead and AsyncWrite.

Change-Id: I7d988fa0490800b72862f4f0fcac3dceac70ec26
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11149
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-03-15 10:23:12 +00:00
Florian Klink
907ecff999 feat(nix-compat/wire): add low-level wire format primitives code
This brings some initial Nix wire format parsing code, used in the nix
daemon protocol, remote store/builder protocol, as well as the NAR
format itself (note we already have more specialized code for the last
one).

Thanks to embr, this code already exists, in
https://codeberg.org/gorgon/gorgon/src/branch/main/nix-daemon/src/wire.rs,
and we can vendor it into here, as EUPL is compatible with GPL (in that
direction).

The code uses the tokio::io Reader and Writer traits, not the ones from
the `futures` crate, as they provide some more convenient `read_u64_le`
functions.

More application-specific parsing code, as well as code to read strings,
or bytes are left out for now, as we want to be be more restrictive
w.r.t allowed max sizes, and need to parse bytes, not strings.

The code slightly diverges, as we have clippy looped into CI.
`Ok(…?)` can be turned into just the inner expression, and
some .and_then can be expressed in a simpler fashion.

Change-Id: Ie3adcb485e9d66786673b1962a08d4e5df3781d9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11148
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-03-15 10:23:12 +00:00
Florian Klink
905a79308e refactor(nix-compat/derivation): emphasize aterm_bytes
derivation_or_fod_hash constructs ATerm bytes and feeds them to sha256.

input_derivations being slightly modified is an implementation detail,
so move the BTreeMap construction inline, and have aterm_bytes in a
let binding (and feed it to the hash function directly while
constructing it).

This makes it a bit more understandable what's going on.

Change-Id: I2f5cfbd1c964fd39ac731ca39e76cfc168f4c7d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11147
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: John Ericson <git@johnericson.me>
2024-03-14 23:10:54 +00:00
Florian Klink
142c72e070 refactor(nix-compat/store_path): add from_name_and_digest_fixed
Allow constructing a StorePath with a fixed-size digest.

Change-Id: Id7d0b0152f6c55660a8973a02c84afa9188ce3ba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11144
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: John Ericson <git@johnericson.me>
Tested-by: BuildkiteCI
2024-03-14 23:10:54 +00:00
Florian Klink
98e6936301 refactor(nix-compat/store_path/utils): move helper function in
This is only used inside this function, in 2 of the match cases.

Change-Id: Ib361f5ee0e3b203802f7d05b9a7f332d14bbcf80
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11143
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: John Ericson <git@johnericson.me>
Tested-by: BuildkiteCI
2024-03-14 17:38:47 +00:00
Florian Klink
43c851bc84 refactor(nix-compat/store_path): take [u8;32] for outer fingerprint
The outer fingerprint used for store path calculation is always a sha256
digest. This includes both input and output-addressed store paths.

We used a NixHash here, which can also represent other hash types, and
that had a bunch of annoyances:

 - Whenever we had the bytes, we had to wrap them in a NixHash::Sha256().
 - Things like AtermWriteable had to be implemented on NixHash,
   even though we then had an assertion it was only called in the
   NixHash::Sha256 case.

Change-Id: Ic895503d9b071800d2e52ae057666f44bd0ab9d6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11142
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: John Ericson <git@johnericson.me>
Reviewed-by: picnoir picnoir <picnoir@alternativebit.fr>
2024-03-14 16:52:42 +00:00
Florian Klink
35f636b684 feat(ops/terraform/deploy-nixos): support argstr map
This allows setting argstr to a map of keys and values.
We use jq to construct "--argstr k v" arguments, which are passed to
nix-instantiate.

Change-Id: I720a597ca2276364bc7005c156064d938f143041
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11141
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
2024-03-14 07:14:55 +00:00
Florian Klink
a9db1455f5 refactor(ops/terraform/deploy-nixos): downcase bash variables
These are not environment variables, so let's shout less.

Change-Id: I3132844937ee78b7230a46afc0240e0225a99f3e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11140
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2024-03-14 06:03:48 +00:00
Florian Klink
8f19ffc4fd chore(ops/modules/open_eid): use nativeMessagingHosts.packages
trace: warning: The `programs.firefox.nativeMessagingHosts.euwebid`
option is deprecated, please add `web-eid-app` to
`programs.firefox.nativeMessagingHosts.packages` instead.

Change-Id: Ic2518957eb8e9151f2fdb13a5f25dfbc12c577a2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11138
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-03-13 18:06:19 +00:00
sterni
dc9c2c9b0c docs(tvix/eval): C++ Nix now has deduplicated inherit (from)
Change-Id: Icf74c699130c2ab774973a7e98bc44ffba8c5ec2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11139
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: flokli <flokli@flokli.de>
2024-03-12 21:23:32 +00:00
Florian Klink
514edc2ea5 feat(tvix/boot/tests): add simple objectstore test
This makes BLOB_SERVICE_ADDR configurable, and creates a flavor setting
it to objectstore+file://$PWD/blobs.

Change-Id: I68c21367f83f68b4dee701c5678f438c7d8fbe43
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11137
Autosubmit: flokli <flokli@flokli.de>
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
2024-03-11 22:42:26 +00:00
Florian Klink
f22d5b3d11 feat(tvix/castore/blobsvc/from_addr): support object_store
The object_store crate supports a ton of different stores, with different schemes.

For now, use a objectstore+ scheme prefix to enable these.

Change-Id: I946f76e32a0fb0867ef59060217894cda5b959b9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11080
Tested-by: BuildkiteCI
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Autosubmit: flokli <flokli@flokli.de>
2024-03-11 22:42:01 +00:00
Florian Klink
1c2db676a0 feat(tvix/castore/blobsvc): add object storage implementation
This uses the `object_store` crate to expose a tvix-castore BlobService
backed by object storage.

It's using FastCDC to chunk blobs into smaller chunks when writing to
it.

These are exposed at the .chunks() method.

Change-Id: I2858c403d4d6490cdca73ebef03c26290b2b3c8e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11076
Reviewed-by: Connor Brewster <cbrewster@hey.com>
Tested-by: BuildkiteCI
Reviewed-by: Brian Olsen <me@griff.name>
2024-03-11 22:42:01 +00:00