tvl-depot/users/flokli/presentations/2023-09-13-asg-tvix-store/presentation.md
Florian Klink 8b09ae54b1 feat(users/flokli): add ASG Lightning talk presentation
This adds the materials for the lightning talk held at All Systems Go
Conference 2023, Berlin.

Talk lives at https://media.ccc.de/v/all-systems-go-2023-245-tvix-store

Change-Id: I114b1aec9f1953c148dd29ca88888c16b9fc655d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9751
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-10-16 11:49:06 +00:00

138 lines
3.5 KiB
Markdown

---
author:
- Florian Klink
date: 2023-09-09
title: "tvix-store: A content-addressed file system and sync protocol"
theme: moon
revealOptions:
transition: 'fade'
---
## tvix-store
### A content-addressed file system and sync protocol
2023-09-13
Florian Klink / flokli
---
## Whoami
- <!-- .element: class="fragment" -->
flokli
- <!-- .element: class="fragment" -->
Nix/NixOS contributor
- maintain systemd, nss and more low-level stuff there
- <!-- .element: class="fragment" -->
Freelance Nix/DevOps consultant
Note: more Kubernetes/DevOps exposure with work
---
## What is tvix-store?
- <!-- .element: class="fragment" -->
A new implementation of a content-addressed "storage system"
- <!-- .element: class="fragment" -->
part of the Tvix Project, a (WIP) reimplementation of Nix and auxillary components in Rust
- <!-- .element: class="fragment" -->
Storage model: think about git trees and its Merkle DAG…
- <!-- .element: class="fragment" -->
… but with nicer wire format (`.proto`) and hash function (blake3)
---
## Storage model
- <!-- .element: class="fragment" -->
Once you know the root: everything else is content-addressed
- <!-- .element: class="fragment" -->
No timestamps, no uid/gid, no xattrs, only one way to represent the same tree
- <!-- .element: class="fragment" -->
Automatic dedup of identical subtrees in different file system trees
- <!-- .element: class="fragment" -->
Automatic dedup of identical blobs (and you can do more chunking underneath too)
---
## Storage model (cont.)
- <!-- .element: class="fragment" -->
Granular seekable access into blobs
- <!-- .element: class="fragment" -->
verified streaming thanks to BLAKE3 and Bao, faulty data is detected early on
- <!-- .element: class="fragment" -->
Everything below can be retrieved from anyone without having to trust (P2P substitution, CDNs, …)
---
## Usecases
- <!-- .element: class="fragment" -->
File system tree delivery
- <!-- .element: class="fragment" -->
Container image delivery
- <!-- .element: class="fragment" -->
Backing store for VCS
- <!-- .element: class="fragment" -->
Granular access into large datasets
---
## Status
- <!-- .element: class="fragment" -->
In-memory backend, a local K/V backend (Sled)
- <!-- .element: class="fragment" -->
FUSE filesystem
- <!-- .element: class="fragment" -->
A gRPC API to transfer things, bindings for golang and rust
- <!-- .element: class="fragment" -->
some object storage backends in development (GCS, NATS)
- <!-- .element: class="fragment" -->
FUTUREWORK: more storage backends / store composition / in-kernel module?
Notes: of course you can use your own network protocol too, like HTTP CAS or iroh....plug different stores together to represent caches, blobfs
---
## Contributing
- <!-- .element: class="fragment" -->
Join the IRC channel (`#tvl` on `hackint`), bridged to Matrix and XMPP
- <!-- .element: class="fragment" -->
Check our issue tracker
- <!-- .element: class="fragment" -->
Try to use it and tell us how you broke it!
Note: if this sounds useful to you, reach out!
---
# Thanks!
<style>
.container{
display: flex;
}
.col{
flex: 1;
}
</style>
<div class="container">
<div class="col">
Florian Klink / <a href="https://flokli.de">flokli.de</a><br />
<img src="qrcode-flokli.svg" />
</div>
<div class="col">
Tvix / <a href="https://tvix.dev">tvix.dev</a><br />
<img src="qrcode-tvix.svg" />
</div>
</div>
---
## Structure
[tvix-store graph](tvix-store-graph-blob-directory.svg)