docs(tvix/docs/TODO): expand Store Composition config section

Write down some of the thoughts after brainstorming with yuka.

Change-Id: I01c94474dc643b8c4993db80e50d3ec65f5c17f4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11749
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: yuka <yuka@yuka.dev>
This commit is contained in:
Florian Klink 2024-06-05 10:15:04 +02:00 committed by clbot
parent 9371f97449
commit c3572048d5

View file

@ -107,9 +107,33 @@ logs etc, but this is something requiring a lot of designing.
### Store composition ### Store composition
- Combinators: list-by-priority, first-come-first-serve, cache - Combinators: list-by-priority, first-come-first-serve, cache
- How do describe hierarchies. URL format too one-dimensional, but we might get - Store composition hierarchies (@yuka).
quite far with a similar "substituters" concept that Nix uses, to construct - URL format too one-dimensional.
the composed stores. - We want to have nice and simple user-facing substituter config, including
sensible default wrappers for caching, retries, fallbacks, as well as
granular control for power-users.
- Current design idea:
- Have a concept similar to rclone config (map with store aliases as
keys, allowing to refer to stores by their alias from other parts of
the config).
It allows both referring to by name, as well as ad-hoc definition:
https://rclone.org/docs/#syntax-of-remote-paths
- Each store needs to be aware of its "instance name", so it can be
included in logs, metrics, …
- Have a "instantiation function" traversing such a config data structure,
creating store instances and plugging them together, ultimately returning
a dyn …Service interface.
- No reconfiguration/reconcilation for now
- Making URLs the primary data format would get ugly quite easily (hello
multiple layers of escaping!), so best to convert the existing URL
syntax to our new config format on the fly and then use one codepath
to instantiate/assemble. Similarly, something like the "user-facing
substituter config" mentioned above could aalso be converted to such a
config format under the hood.
- Maybe add a ?cache=$other_url parameter support to the URL syntax, to
easily wrap a store with a caching frontend, using $other_url as the
"near" store URL.
### Store Config ### Store Config
There's already serde for some store options (bigtable uses `serde_qs`). There's already serde for some store options (bigtable uses `serde_qs`).
We might also have common options global over all backends, like chunking We might also have common options global over all backends, like chunking