8b09ae54b1
This adds the materials for the lightning talk held at All Systems Go Conference 2023, Berlin. Talk lives at https://media.ccc.de/v/all-systems-go-2023-245-tvix-store Change-Id: I114b1aec9f1953c148dd29ca88888c16b9fc655d Reviewed-on: https://cl.tvl.fyi/c/depot/+/9751 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI
138 lines
3.5 KiB
Markdown
138 lines
3.5 KiB
Markdown
---
|
|
author:
|
|
- Florian Klink
|
|
date: 2023-09-09
|
|
title: "tvix-store: A content-addressed file system and sync protocol"
|
|
theme: moon
|
|
revealOptions:
|
|
transition: 'fade'
|
|
---
|
|
|
|
## tvix-store
|
|
### A content-addressed file system and sync protocol
|
|
|
|
2023-09-13
|
|
|
|
Florian Klink / flokli
|
|
|
|
---
|
|
|
|
## Whoami
|
|
|
|
- <!-- .element: class="fragment" -->
|
|
flokli
|
|
- <!-- .element: class="fragment" -->
|
|
Nix/NixOS contributor
|
|
- maintain systemd, nss and more low-level stuff there
|
|
- <!-- .element: class="fragment" -->
|
|
Freelance Nix/DevOps consultant
|
|
|
|
Note: more Kubernetes/DevOps exposure with work
|
|
|
|
---
|
|
|
|
## What is tvix-store?
|
|
- <!-- .element: class="fragment" -->
|
|
A new implementation of a content-addressed "storage system"
|
|
- <!-- .element: class="fragment" -->
|
|
part of the Tvix Project, a (WIP) reimplementation of Nix and auxillary components in Rust
|
|
- <!-- .element: class="fragment" -->
|
|
Storage model: think about git trees and its Merkle DAG…
|
|
- <!-- .element: class="fragment" -->
|
|
… but with nicer wire format (`.proto`) and hash function (blake3)
|
|
|
|
---
|
|
|
|
## Storage model
|
|
- <!-- .element: class="fragment" -->
|
|
Once you know the root: everything else is content-addressed
|
|
- <!-- .element: class="fragment" -->
|
|
No timestamps, no uid/gid, no xattrs, only one way to represent the same tree
|
|
- <!-- .element: class="fragment" -->
|
|
Automatic dedup of identical subtrees in different file system trees
|
|
- <!-- .element: class="fragment" -->
|
|
Automatic dedup of identical blobs (and you can do more chunking underneath too)
|
|
|
|
---
|
|
|
|
## Storage model (cont.)
|
|
- <!-- .element: class="fragment" -->
|
|
Granular seekable access into blobs
|
|
- <!-- .element: class="fragment" -->
|
|
verified streaming thanks to BLAKE3 and Bao, faulty data is detected early on
|
|
- <!-- .element: class="fragment" -->
|
|
Everything below can be retrieved from anyone without having to trust (P2P substitution, CDNs, …)
|
|
|
|
---
|
|
|
|
## Usecases
|
|
- <!-- .element: class="fragment" -->
|
|
File system tree delivery
|
|
- <!-- .element: class="fragment" -->
|
|
Container image delivery
|
|
- <!-- .element: class="fragment" -->
|
|
Backing store for VCS
|
|
- <!-- .element: class="fragment" -->
|
|
Granular access into large datasets
|
|
|
|
---
|
|
|
|
## Status
|
|
- <!-- .element: class="fragment" -->
|
|
In-memory backend, a local K/V backend (Sled)
|
|
- <!-- .element: class="fragment" -->
|
|
FUSE filesystem
|
|
- <!-- .element: class="fragment" -->
|
|
A gRPC API to transfer things, bindings for golang and rust
|
|
- <!-- .element: class="fragment" -->
|
|
some object storage backends in development (GCS, NATS)
|
|
- <!-- .element: class="fragment" -->
|
|
FUTUREWORK: more storage backends / store composition / in-kernel module?
|
|
|
|
Notes: of course you can use your own network protocol too, like HTTP CAS or iroh....plug different stores together to represent caches, blobfs
|
|
|
|
---
|
|
|
|
## Contributing
|
|
|
|
- <!-- .element: class="fragment" -->
|
|
Join the IRC channel (`#tvl` on `hackint`), bridged to Matrix and XMPP
|
|
- <!-- .element: class="fragment" -->
|
|
Check our issue tracker
|
|
- <!-- .element: class="fragment" -->
|
|
Try to use it and tell us how you broke it!
|
|
|
|
Note: if this sounds useful to you, reach out!
|
|
|
|
---
|
|
|
|
# Thanks!
|
|
|
|
<style>
|
|
.container{
|
|
display: flex;
|
|
}
|
|
.col{
|
|
flex: 1;
|
|
}
|
|
</style>
|
|
|
|
<div class="container">
|
|
|
|
<div class="col">
|
|
Florian Klink / <a href="https://flokli.de">flokli.de</a><br />
|
|
<img src="qrcode-flokli.svg" />
|
|
</div>
|
|
|
|
<div class="col">
|
|
Tvix / <a href="https://tvix.dev">tvix.dev</a><br />
|
|
<img src="qrcode-tvix.svg" />
|
|
</div>
|
|
|
|
</div>
|
|
|
|
---
|
|
|
|
## Structure
|
|
|
|
[tvix-store graph](tvix-store-graph-blob-directory.svg)
|