Florian Klink 8b09ae54b1 feat(users/flokli): add ASG Lightning talk presentation

This adds the materials for the lightning talk held at All Systems Go
Conference 2023, Berlin.

Talk lives at https://media.ccc.de/v/all-systems-go-2023-245-tvix-store

Change-Id: I114b1aec9f1953c148dd29ca88888c16b9fc655d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9751
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI

2023-10-16 11:49:06 +00:00

3.5 KiB

Raw Blame History

author

date

title

theme

revealOptions

Florian Klink

2023-09-09

tvix-store: A content-addressed file system and sync protocol

moon

transition
fade

tvix-store

A content-addressed file system and sync protocol

2023-09-13

Florian Klink / flokli

Whoami

flokli
Nix/NixOS contributor
- maintain systemd, nss and more low-level stuff there
Freelance Nix/DevOps consultant

Note: more Kubernetes/DevOps exposure with work

What is tvix-store?

A new implementation of a content-addressed "storage system"
- part of the Tvix Project, a (WIP) reimplementation of Nix and auxillary components in Rust
- Storage model: think about git trees and its Merkle DAG…
- … but with nicer wire format (.proto) and hash function (blake3)

Storage model

Once you know the root: everything else is content-addressed
- No timestamps, no uid/gid, no xattrs, only one way to represent the same tree
Automatic dedup of identical subtrees in different file system trees
Automatic dedup of identical blobs (and you can do more chunking underneath too)

Storage model (cont.)

Granular seekable access into blobs
verified streaming thanks to BLAKE3 and Bao, faulty data is detected early on
Everything below can be retrieved from anyone without having to trust (P2P substitution, CDNs, …)

Usecases

File system tree delivery
Container image delivery
Backing store for VCS
Granular access into large datasets

Status

In-memory backend, a local K/V backend (Sled)
FUSE filesystem
A gRPC API to transfer things, bindings for golang and rust
some object storage backends in development (GCS, NATS)
FUTUREWORK: more storage backends / store composition / in-kernel module?

Notes: of course you can use your own network protocol too, like HTTP CAS or iroh....plug different stores together to represent caches, blobfs

Contributing

Join the IRC channel (#tvl on hackint), bridged to Matrix and XMPP
Check our issue tracker
Try to use it and tell us how you broke it!

Note: if this sounds useful to you, reach out!

Thanks!

Florian Klink / flokli.de

Tvix / tvix.dev

Structure

tvix-store graph

3.5 KiB Raw Blame History