docs: More documentation of nix daemon protocol

Uses `Set` to mark that the given list is actually a set.
The format on stream is no different from `List` but is nice to
know.

Best effort document string formats.

Documents default values for fields not used in old protocol
versions.

Document handshake, AddMultipleToStore format, Export format and
Import paths format.

Change-Id: I6cbe4c7de3770cc5caafc26229f99a07353ee3e2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11984
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: Brian Olsen <me@griff.name>

tvix/docs/src/SUMMARY.md

@@ -39,7 +39,8 @@
 - [Nix language version history](./lang-version.md)
 - [Value Pointer Equality](./value-pointer-equality.md)
 - [Daemon Protocol](./nix-daemon/index.md)
-  - [Changelog](./nix-daemon/changelog.md)
+  - [Handshake](./nix-daemon/handshake.md)
   - [Logging](./nix-daemon/logging.md)
   - [Operations](./nix-daemon/operations.md)
   - [Serialization](./nix-daemon/serialization.md)
+  - [Changelog](./nix-daemon/changelog.md)

tvix/docs/src/nix-daemon/changelog.md

@@ -34,7 +34,7 @@ So this means that any version of Nix 2.x can't talk to Nix 0.x.
 | --------------- | -- | -------------- | -------- | ----------- | ----- |
 | *Quit           | 0  | [a711689368][a711689368] || 0.11 | Became dead code in [7951c3c54][7951c3c54] (Nix 0.11) and removed in [d3c61d83b][d3c61d83b] (Nix 1.8) |
 | IsValidPath     | 1  | [a711689368][a711689368] || 0.11 ||
-| HasSubstitutes  | 3  | [0565b5f2b3][0565b5f2b3] || 0.11 ||
+| HasSubstitutes  | 3  | [0565b5f2b3][0565b5f2b3] || 0.11 | Obsolete [09a6321aeb][09a6321aeb]<br>Nix 1.2 Protocol 1.12 |
 | QueryPathHash   | 4  | [0565b5f2b3][0565b5f2b3] || 0.11 | Obsolete [e0204f8d46][e0204f8d46]<br>Nix 2.0 Protocol 1.16 |
 | QueryReferences | 5  | [0565b5f2b3][0565b5f2b3] || 0.11 | Obsolete [e0204f8d46][e0204f8d46]<br>Nix 2.0 Protocol 1.16 |
 | QueryReferrers  | 6  | [0565b5f2b3][0565b5f2b3] || 0.11 ||

tvix/docs/src/nix-daemon/handshake.md

@@ -0,0 +1,32 @@
+
+
+## client -> server
+- 0x6e697863 :: [Int](#int) (hardcoded, 'nixc' in ASCII)
+
+## server -> client
+- 0x6478696f :: [Int](#int) (hardcoded, 'dxio' in ASCII)
+- protocolVersion :: [Int](#int)
+
+## client -> server
+- clientVersion :: [Int](#int)
+
+### If clientVersion is 1.14 or later
+- sendCpu :: [Bool](#bool) (hardcoded to false in client)
+#### If sendCpu is true
+- cpuAffinity :: [Int](#int) (obsolete and ignored)
+
+### If clientVersion is 1.11 or later
+- reserveSpace :: [Bool](#bool) (obsolete, ignored and set to false)
+
+
+## server -> client
+
+### If clientVersion is 1.33 or later
+- nixVersion :: String
+
+### If clientVersion is 1.35 or later
+- trusted :: OptTrusted
+
+## server -> client
+- send logs
+- [operation](./operations.md) :: Int

tvix/docs/src/nix-daemon/logging.md

@@ -13,20 +13,21 @@ While not in this state between request reading and response sending all
 messages and activities are buffered until next time the logger can send data.
 
 The logging messages supported are:
-- `STDERR_LAST`
+- [`STDERR_LAST`](#stderr_last)
-- `STDERR_ERROR`
+- [`STDERR_ERROR`](#stderr_error)
-- `STDERR_NEXT`
+- [`STDERR_NEXT`](#stderr_next)
-- `STDERR_READ`
+- [`STDERR_READ`](#stderr_read)
-- `STDERR_WRITE`
+- [`STDERR_WRITE`](#stderr_write)
-- `STDERR_START_ACTIVITY`
+- [`STDERR_START_ACTIVITY`](#stderr_start_activity)
-- `STDERR_STOP_ACTIVITY`
+- [`STDERR_STOP_ACTIVITY`](#stderr_stop_activity)
-- `STDERR_RESULT`
+- [`STDERR_RESULT`](#stderr_result)
 
 
 ### `STDERR_LAST`
 Marks the end of the logs, normal processing can resume.
 
-- 0x616c7473 :: [UInt64][se-UInt64] (hardcoded)
+- 0x616c7473 :: [UInt64][se-UInt64] (hardcoded, 'alts' in ASCII)
+
 
 ### `STDERR_ERROR`
 This also marks the end of this log "session" and so it
@@ -34,20 +35,20 @@ has the same effect as `STDERR_LAST`.
 On the client the error is thrown as an exception and no response is read.
 
 #### If protocol version is 1.26 or newer
-- 0x63787470 :: [UInt64][se-UInt64] (hardcoded)
+- 0x63787470 :: [UInt64][se-UInt64] (hardcoded, 'cxtp' in ASCII)
 - error :: [Error][se-Error]
 
 #### If protocol version is older than 1.26
-- 0x63787470 :: [UInt64][se-UInt64] (hardcoded)
+- 0x63787470 :: [UInt64][se-UInt64] (hardcoded, 'cxtp' in ASCII)
-- msg :: [String][se-String]
+- msg :: [String][se-String] (If logger is JSON, invalid UTF-8 is replaced with U+FFFD)
 - exitStatus :: [Int][se-Int]
 
 
 ### `STDERR_NEXT`
 Normal string log message.
 
-- 0x6f6c6d67 :: [UInt64][se-UInt64] (hardcoded)
+- 0x6f6c6d67 :: [UInt64][se-UInt64] (hardcoded, 'olmg' in ASCII)
-- msg :: [String][se-String]
+- msg :: [String][se-String] (If logger is JSON, invalid UTF-8 is replaced with U+FFFD)
 
 
 ### `STDERR_READ`
@@ -56,15 +57,17 @@ It works by sending a desired buffer length and then on the receiver stream it
 reads bytes buffer of that length. If it receives 0 bytes it sees this as an
 unexpected EOF.
 
-- 0x64617461 :: [UInt64][se-UInt64] (hardcoded)
+- 0x64617461 :: [UInt64][se-UInt64] (hardcoded, 'data' in ASCII)
 - desiredLen :: [Size][se-Size]
 
+
 ### `STDERR_WRITE`
 Writer interface used by ExportPath. Simply writes a buffer.
 
 - 0x64617416 :: [UInt64][se-UInt64] (hardcoded)
 - buffer :: [Bytes][se-Bytes]
 
+
 ### `STDERR_START_ACTIVITY`
 Begins an activity. In other tracing frameworks this would be called a span.
 
@@ -73,11 +76,11 @@ versions of the protocol instead of sending an `STDERR_START_ACTIVITY`
 the level is checked against enabled logging level and the text field is
 sent as a simple log message with `STDERR_NEXT`.
 
-- 0x53545254 :: [UInt64][se-UInt64] (hardcoded)
+- 0x53545254 :: [UInt64][se-UInt64] (hardcoded, 'STRT' in ASCII)
 - act :: [UInt64][se-UInt64]
 - level :: [Verbosity][se-Verbosity]
 - type :: [ActivityType][se-ActivityType]
-- text :: [String][se-String]
+- text :: [String][se-String] (If logger is JSON, invalid UTF-8 is replaced with U+FFFD)
 - fields :: [List][se-List] of [Field][se-Field]
 - parent :: [UInt64][se-UInt64]
 
@@ -92,7 +95,7 @@ Just sends `ActivityId`.
 Implemented in protocol 1.20. When backwards compatible with older versions of
 the protocol and this message would have been sent it is instead ignored.
 
-- 0x53544f50 :: [UInt64][se-UInt64] (hardcoded)
+- 0x53544f50 :: [UInt64][se-UInt64] (hardcoded, 'STOP' in ASCII)
 
 
 ### `STDERR_RESULT`
@@ -101,14 +104,13 @@ Sends results for a given activity.
 Implemented in protocol 1.20. When backwards compatible with older versions of
 the protocol and this message would have been sent it is instead ignored.
 
-- 0x52534c54 :: [UInt64][se-UInt64] (hardcoded)
+- 0x52534c54 :: [UInt64][se-UInt64] (hardcoded, 'RSLT' in ASCII)
 - act :: [UInt64][se-UInt64]
 - type :: [ResultType][se-ResultType]
 - fields :: [List][se-List] of [Field][se-Field]
 
 
 
-
 [se-UInt64]: ./serialization.md#uint64
 [se-Int]: ./serialization.md#int
 [se-Size]: ./serialization.md#size

tvix/docs/src/nix-daemon/operations.md

@@ -14,11 +14,9 @@
 | [FindRoots](#findroots)                                     | 14 |
 | [SetOptions](#setoptions)                                   | 19 |
 | [CollectGarbage](#collectgarbage)                           | 20 |
-| [QuerySubstitutablePathInfo](#querysubstitutablepathinfo)   | 21 |
 | [QueryAllValidPaths](#queryallvalidpaths)                   | 23 |
 | [QueryPathInfo](#querypathinfo)                             | 26 |
 | [QueryPathFromHashPart](#querypathfromhashpart)             | 29 |
-| [QuerySubstitutablePathInfos](#querysubstitutablepathinfos) | 30 |
 | [QueryValidPaths](#queryvalidpaths)                         | 31 |
 | [QuerySubstitutablePaths](#querysubstitutablepaths)         | 32 |
 | [QueryValidDerivers](#queryvalidderivers)                   | 33 |
@@ -41,16 +39,18 @@
 ## Obsolete operations
 
 | Operation                                                   | Id |
-| --------------------------------------------------------- | -- |
+| ----------------------------------------------------------- | -- |
 | [QueryPathHash](#querypathhash)                             | 4  |
 | [QueryReferences](#queryreferences)                         | 5  |
 | [AddTextToStore](#addtexttostore)                           | 8  |
 | [SyncWithGC](#syncwithgc)                                   | 13 |
 | [ExportPath](#exportpath)                                   | 16 |
 | [QueryDeriver](#queryderiver)                               | 18 |
+| [QuerySubstitutablePathInfo](#querysubstitutablepathinfo)   | 21 |
 | [QueryDerivationOutputs](#queryderivationoutputs)           | 22 |
 | [ImportPaths](#importpaths)                                 | 27 |
 | [QueryDerivationOutputNames](#queryderivationoutputnames)   | 28 |
+| [QuerySubstitutablePathInfos](#querysubstitutablepathinfos) | 30 |
 
 
 ## Removed operations
@@ -81,6 +81,7 @@ As the name says checks that a store path is valid i.e. in the store.
 
 This is a pretty core operation used everywhere.
 
+
 ### Inputs
 path :: [StorePath][se-StorePath]
 
@@ -92,6 +93,9 @@ isValid :: [Bool][se-Bool]
 
 **Id:** 3<br>
 **Introduced:** Nix 0.11<br>
+**Obsolete** Protocol 1.12, Nix 1.2<br>
+
+Replaced by QuerySubstitutablePaths.
 
 Checks if we can substitute the input path from a substituter. Uses
 QuerySubstitutablePaths under the hood :/
@@ -115,7 +119,7 @@ Retrieves the base16 NAR hash of a given store path.
 path :: [StorePath][se-StorePath]
 
 ### Outputs
-hash :: [String][se-String] (base16-encoded NAR hash without algorithm prefix)
+hash :: [NARHash][se-NARHash]
 
 
 ## QueryReferences
@@ -130,7 +134,7 @@ Retrieves the references of a given path
 path :: [StorePath][se-StorePath]
 
 ### Outputs
-references :: [List][se-List] of [StorePath][se-StorePath]
+references :: [Set][se-Set] of [StorePath][se-StorePath]
 
 
 ## QueryReferrers
@@ -144,7 +148,7 @@ Retrieves the referrers of a given path.
 path :: [StorePath][se-StorePath]
 
 ### Outputs
-referrers :: [List][se-List] of [StorePath][se-StorePath]
+referrers :: [Set][se-Set] of [StorePath][se-StorePath]
 
 
 ## AddToStore
@@ -156,16 +160,16 @@ Add a new path to the store.
 
 ### Before protocol version 1.25
 #### Inputs
-- baseName :: [String][se-String]
+- baseName :: [StorePathName][se-StorePathName]
 - fixed :: [Bool64][se-Bool64]
 - recursive :: [FileIngestionMethod][se-FileIngestionMethod]
-- hashAlgo :: [String][se-String]
+- hashAlgo :: [HashAlgorithm][se-HashAlgorithm]
 - NAR dump
 
 If fixed is `true`, hashAlgo is forced to `sha256` and recursive is forced to
-`Recursive`.
+`NixArchive`.
 
-Only `Flat` and `Recursive` values are supported for the recursive input
+Only `Flat` and `NixArchive` values are supported for the recursive input
 parameter.
 
 #### Outputs
@@ -173,9 +177,9 @@ path :: [StorePath][se-StorePath]
 
 ### Protocol version 1.25 or newer
 #### Inputs
-- name :: [String][se-String]
+- name :: [StorePathName][se-StorePathName]
 - camStr :: [ContentAddressMethodWithAlgo][se-ContentAddressMethodWithAlgo]
-- refs :: [List][se-List] of [StorePath][se-StorePath]
+- refs :: [Set][se-Set] of [StorePath][se-StorePath]
 - repairBool :: [Bool64][se-Bool64]
 - [Framed][se-Framed] NAR dump
 
@@ -197,9 +201,9 @@ to [AddToStore](#addtostore). And so this corresponds to calling
 wrapped as a NAR.
 
 ### Inputs
-- suffix :: [String][se-String]
+- suffix :: [StorePathName][se-StorePathName]
 - text :: [Bytes][se-Bytes]
-- refs :: [List][se-List] of [StorePath][se-StorePath]
+- refs :: [Set][se-Set] of [StorePath][se-StorePath]
 
 ### Outpus
 path :: [StorePath][se-StorePath]
@@ -213,15 +217,15 @@ path :: [StorePath][se-StorePath]
 Build (or substitute) a list of derivations.
 
 ### Inputs
-paths :: [List][se-List] of [DerivedPath][se-DerivedPath]
+paths :: [Set][se-Set] of [DerivedPath][se-DerivedPath]
 
 #### Protocol 1.15 or newer
-mode :: [BuildMode][se-BuildMode]
+mode :: [BuildMode][se-BuildMode] (defaults to Normal)
 
 Check that connection is trusted before allowing Repair mode.
 
 ### Outputs
-1 :: [Int][se-Int] (hardcoded)
+1 :: [Int][se-Int] (hardcoded and ignored by client)
 
 
 ## EnsurePath
@@ -235,7 +239,7 @@ Checks if a path is valid. Note: it may be made valid by running a substitute.
 path :: [StorePath][se-StorePath]
 
 ### Outputs
-1 :: [Int][se-Int] (hardcoded)
+1 :: [Int][se-Int] (hardcoded and ignored by client)
 
 
 ## AddTempRoot
@@ -253,7 +257,7 @@ deleting store paths that the client is actively doing something with.
 path :: [StorePath][se-StorePath]
 
 ### Outputs
-1 :: [Int][se-Int] (hardcoded)
+1 :: [Int][se-Int] (hardcoded and ignored by client)
 
 
 ## AddIndirectRoot
@@ -267,10 +271,10 @@ created by [AddPermRoot](#addpermroot).
 Only ever sent on the local unix socket nix daemon.
 
 ### Inputs
-path :: [String][se-String]
+path :: [Path][se-Path]
 
 ### Outputs
-1 :: [Int][se-Int] (hardcoded)
+1 :: [Int][se-Int] (hardcoded and ignored by client)
 
 
 ## SyncWithGC
@@ -308,7 +312,7 @@ thing.
 Find the GC roots.
 
 ### Outputs
-roots :: [Map][se-Map] of [String][se-String] to [StorePath][se-StorePath]
+roots :: [Map][se-Map] of [Path][se-Path] to [StorePath][se-StorePath]
 
 The key is the link pointing to the given store path.
 
@@ -333,7 +337,8 @@ Export a store path in the binary format nix-store --import expects. See impleme
 - sign :: [Int][se-Int] (ignored and hardcoded to 0 in client)
 
 ### Outputs
-Uses `STDERR_WRITE` to send dump in export format
+Uses [`STDERR_WRITE`](./logging.md#stderr_write) to send dump in
+[export format][se-ExportFormat]
 
 After dump it outputs.
 
@@ -373,18 +378,18 @@ Only ever used right after the handshake.
 
 ### Inputs
 
-- keepFailed :: [Int][se-Int]
+- keepFailed :: [Bool][se-Bool]
-- keepGoing :: [Int][se-Int]
+- keepGoing :: [Bool][se-Bool]
-- tryFallback :: [Int][se-Int]
+- tryFallback :: [Bool][se-Bool]
 - verbosity :: [Verbosity][se-Verbosity]
-- maxbuildJobs :: [Int][se-Int]
+- maxBuildJobs :: [Int][se-Int]
-- maxSilentTime :: [Int][se-Int]
+- maxSilentTime :: [Time][se-Time]
 - useBuildHook :: [Bool][se-Bool] (ignored and hardcoded to true in client)
 - verboseBuild :: [Verbosity][se-Verbosity]
 - logType :: [Int][se-Int] (ignored and hardcoded to 0 in client)
 - printBuildTrace :: [Int][se-Int] (ignored and hardcoded to 0 in client)
 - buildCores :: [Int][se-Int]
-- useSubstitutes :: [Int][se-Int]
+- useSubstitutes :: [Bool][se-Bool]
 
 ### Protocol 1.12 or newer
 otherSettings :: [Map][se-Map] of [String][se-String] to [String][se-String]
@@ -399,7 +404,7 @@ Find the GC roots.
 
 ### Inputs
 - action :: [GCAction][se-GCAction]
-- pathsToDelete :: [List][se-List] of [StorePath][se-StorePath]
+- pathsToDelete :: [Set][se-Set] of [StorePath][se-StorePath]
 - ignoreLiveness :: [Bool64][se-Bool64]
 - maxFreed :: [UInt64][se-UInt64]
 - removed :: [Int][se-Int] (ignored and hardcoded to 0 in client)
@@ -407,18 +412,19 @@ Find the GC roots.
 - removed :: [Int][se-Int] (ignored and hardcoded to 0 in client)
 
 ### Outputs
-- pathsDeleted :: [List][se-List] of [String][se-String]
+- pathsDeleted :: [Set][se-Set] of [Path][se-Path]
 - bytesFreed :: [UInt64][se-UInt64]
-- 0 :: [UInt64][se-UInt64] (hardcoded)
+- 0 :: [UInt64][se-UInt64] (hardcoded, obsolete and ignored by client)
 
-Depending on the action pathsDeleted is, the GC roots, or the paths that would
+Depending on the value of the action input the value of output pathsDeleted
-be or have been deleted.
+is either, the GC roots, or the paths that would be or have been deleted.
 
 
 ## QuerySubstitutablePathInfo
 
 **Id:** 21<br>
 **Introduced:** Protocol 1.02, Nix 0.12<br>
+**Obsolete:** Protocol 1.12, Nix 1.2<br>
 
 Retrieves the various substitutable paths infos for a given path.
 
@@ -429,10 +435,7 @@ path :: [StorePath][se-StorePath]
 found :: [Bool][se-Bool]
 
 #### If found is true
-- deriver :: [OptStorePath][se-OptStorePath]
+- info :: [SubstitutablePathInfo][se-SubstitutablePathInfo]
-- references :: [List][se-List] of [StorePath][se-StorePath]
-- downloadSize :: [UInt64][se-UInt64]
-- narSize :: [UInt64][se-UInt64]
 
 
 ## QueryDerivationOutputs
@@ -447,7 +450,7 @@ Retrieves all the outputs paths of a given derivation.
 path :: [StorePath][se-StorePath] (must point to a derivation)
 
 ### Outputs
-derivationOutputs :: [List][se-List] of [StorePath][se-StorePath]
+derivationOutputs :: [Set][se-Set] of [StorePath][se-StorePath]
 
 
 ## QueryAllValidPaths
@@ -458,7 +461,7 @@ derivationOutputs :: [List][se-List] of [StorePath][se-StorePath]
 Retrieves all the valid paths contained in the store.
 
 ### Outputs
-paths :: [List][se-List] of [StorePath][se-StorePath]
+paths :: [Set][se-Set] of [StorePath][se-StorePath]
 
 
 ## QueryFailedPaths (removed)
@@ -498,7 +501,7 @@ success :: [Bool64][se-Bool64]
 pathInfo :: [UnkeyedValidPathInfo][se-UnkeyedValidPathInfo]
 
 #### If protocol version is older than 1.17
-If info not found return error with `STDERR_ERROR`
+If info not found return error with [`STDERR_ERROR`](./logging.md#stderr_error)
 
 pathInfo :: [UnkeyedValidPathInfo][se-UnkeyedValidPathInfo]
 
@@ -516,7 +519,7 @@ before the metadata about the store path and so you would typically have
 to store the NAR in memory or temporarily on disk before processing it.
 
 ### Inputs
-List of NAR dumps coming from the ExportPaths operations.
+[List of NAR dumps][se-ImportPaths] coming from one or more ExportPath operations.
 
 ### Outputs
 importedPaths :: [List][se-List] of [StorePath][se-StorePath]
@@ -534,7 +537,7 @@ Retrieves the name of the outputs of a given derivation. EG. out, dev, etc.
 path :: [StorePath][se-StorePath] (must be a derivation path)
 
 ### Outputs
-names :: [List][se-List] of [String][se-String]
+names :: [Set][se-Set] of [OutputName][se-OutputName]
 
 
 ## QueryPathFromHashPart
@@ -542,11 +545,10 @@ names :: [List][se-List] of [String][se-String]
 **Id:** 29<br>
 **Introduced:** Protocol 1.11, Nix 1.1<br>
 
-Retrieves a store path from a base16 (input) hash. Returns "" if no path was
+Retrieves a store path from a nixbase32 (input) hash.
-found.
 
 ### Inputs
-hashPart :: [String][se-String]  (must be a base-16 hash)
+hashPart :: [StorePathHash][se-StorePathHash]
 
 ### Outputs
 path :: [OptStorePath][se-OptStorePath]
@@ -556,18 +558,22 @@ path :: [OptStorePath][se-OptStorePath]
 
 **Id:** 30<br>
 **Introduced:** Protocol 1.12*, Nix 1.2<br>
+**Obsolete:** Protocol 1.19*, Nix 2.0<br>
 
 Retrieves the various substitutable paths infos for set of store paths.
 
+Only ever used in the fallback for QueryMissing which means that if protocol is 1.19 or later
+it is never sent and is therefore obsolete after that.
+
 ### Inputs
 #### If protocol version is 1.22 or newer
 paths :: [Map][se-Map] of [StorePath][se-StorePath] to [OptContentAddress][se-OptContentAddress] 
 
 #### If protocol version older than 1.22
-paths :: [List][se-List] of [StorePath][se-StorePath]
+paths :: [Set][se-Set] of [StorePath][se-StorePath]
 
 ### Outputs
-infos :: [List][se-List] of [SubstitutablePathInfo][se-SubstitutablePathInfo]
+infos :: [Map][se-Map] of [StorePath][se-StorePath] to [SubstitutablePathInfo][se-SubstitutablePathInfo]
 
 
 ## QueryValidPaths
@@ -578,13 +584,13 @@ infos :: [List][se-List] of [SubstitutablePathInfo][se-SubstitutablePathInfo]
 Takes a list of store paths and returns a new list only containing the valid store paths
 
 ## Inputs
-paths :: [List][se-List] of [StorePath][se-StorePath]
+paths :: [Set][se-Set] of [StorePath][se-StorePath]
 
 ### If protocol version is 1.27 or newer
 substitute :: [Bool][se-Bool] (defaults to false if not sent)
 
 ## Outputs
-paths :: [List][se-List] of [StorePath][se-StorePath]
+paths :: [Set][se-Set] of [StorePath][se-StorePath]
 
 
 ## QuerySubstitutablePaths
@@ -599,10 +605,10 @@ In versions of the protocol prior to 1.12 [HasSubstitutes](#hassubstitutes)
 is used to implement the functionality that this operation provides.
 
 ### Inputs
-paths :: [List][se-List] of [StorePath][se-StorePath]
+paths :: [Set][se-Set] of [StorePath][se-StorePath]
 
 ### Outputs
-paths :: [List][se-List] of [StorePath][se-StorePath]
+paths :: [Set][se-Set] of [StorePath][se-StorePath]
 
 
 ## QueryValidDerivers
@@ -616,7 +622,7 @@ Retrieves the derivers of a given path.
 path :: [StorePath][se-StorePath]
 
 ### Outputs
-derivers :: [List][se-List] of [StorePath][se-StorePath]
+derivers :: [Set][se-Set] of [StorePath][se-StorePath]
 
 
 ## OptimiseStore
@@ -627,7 +633,7 @@ derivers :: [List][se-List] of [StorePath][se-StorePath]
 Optimise store by hardlinking files with the same content.
 
 ### Outputs
-1 :: [Int][se-Int] (hardcoded)
+1 :: [Int][se-Int] (hardcoded and ignored by client)
 
 
 ## VerifyStore
@@ -677,14 +683,14 @@ buildResult :: [BuildResult][se-BuildResult]
 **Id:** 37<br>
 **Introduced:** Protocol 1.16, Nix 2.0<br>
 
-Add the signatures associated to a given path.
+Add the signatures associated to a given path. Used by `nix store copy-sigs` and `nix store sign`.
 
 ### Inputs
 - path :: [StorePath][se-StorePath]
-- signatures :: [List][se-List] of [String][se-String]
+- signatures :: [Set][se-Set] of [Signature][se-Signature]
 
 ### Outputs
-1 :: [Int][se-Int] (hardcoded)
+1 :: [Int][se-Int] (hardcoded and ignored by client)
 
 
 ## NarFromPath
@@ -717,12 +723,12 @@ Dumps a path as a NAR
 ### Inputs
 - path :: [StorePath][se-StorePath]
 - deriver :: [OptStorePath][se-OptStorePath]
-- narHash :: [String][se-String] SHA256 NAR hash base 16
+- narHash :: [NARHash][se-NARHash]
-- references :: [List][se-List] of [StorePath][se-StorePath]
+- references :: [Set][se-Set] of [StorePath][se-StorePath]
 - registrationTime :: [Time][se-Time]
 - narSize :: [UInt64][se-UInt64]
 - ultimate :: [Bool64][se-Bool64]
-- signatures :: [List][se-List] of [String][se-String]
+- signatures :: [Set][se-Set] of [Signature][se-Signature]
 - ca :: [OptContentAddress][se-OptContentAddress]
 - repair :: [Bool64][se-Bool64]
 - dontCheckSigs :: [Bool64][se-Bool64]
@@ -731,7 +737,7 @@ Dumps a path as a NAR
 [Framed][se-Framed] NAR dump
 
 #### If protocol version is between 1.21 and 1.23
-NAR dump sent using `STDERR_READ`
+NAR dump sent using [`STDERR_READ`](./logging.md#stderr_read)
 
 #### If protocol version is older than 1.21
 NAR dump sent raw on stream
@@ -746,9 +752,9 @@ NAR dump sent raw on stream
 targets :: [List][se-List] of [DerivedPath][se-DerivedPath]
 
 ### Outputs
-- willBuild :: [List][se-List] of [StorePath][se-StorePath]
+- willBuild :: [Set][se-Set] of [StorePath][se-StorePath]
-- willSubstitute :: [List][se-List] of [StorePath][se-StorePath]
+- willSubstitute :: [Set][se-Set] of [StorePath][se-StorePath]
-- unknown :: [List][se-List] of [StorePath][se-StorePath]
+- unknown :: [Set][se-Set] of [StorePath][se-StorePath]
 - downloadSize :: [UInt64][se-UInt64]
 - narSize :: [UInt64][se-UInt64]
 
@@ -764,7 +770,7 @@ Retrieves an associative map outputName -> storePath for a given derivation.
 path :: [StorePath][se-StorePath]  (must be a derivation path)
 
 ### Outputs
-outputs :: [Map][se-Map] of [String][se-String] to [OptStorePath][se-OptStorePath]
+outputs :: [Map][se-Map] of [OutputName][se-OutputName] to [OptStorePath][se-OptStorePath]
 
 
 ## RegisterDrvOutput
@@ -795,10 +801,10 @@ outputId :: [DrvOutput][se-DrvOutput]
 
 ### Outputs
 #### If protocol is 1.31 or newer
-realisations :: [List][se-List] of [Realisation][se-Realisation]
+realisations :: [Set][se-Set] of [Realisation][se-Realisation]
 
 #### If protocol is older than 1.31
-outPaths :: [List][se-List] of [BaseStorePath][se-BaseStorePath]
+outPaths :: [Set][se-Set] of [StorePath][se-StorePath]
 
 
 ## AddMultipleToStore
@@ -816,7 +822,7 @@ for each small NAR was costly.
 ### Inputs
 - repair :: [Bool64][se-Bool64]
 - dontCheckSigs :: [Bool64][se-Bool64]
-- [Framed][se-Framed] stream of add multiple NAR dump
+- [Framed][se-Framed] stream of [add multiple NAR dump][se-AddMultipleToStore]
 
 
 ## AddBuildLog
@@ -827,11 +833,11 @@ for each small NAR was costly.
 Attach some build logs to a given build.
 
 ### Inputs
-- path :: [String][se-String] (might be [BaseStorePath][se-BaseStorePath])
+- path :: [BaseStorePath][se-BaseStorePath]
 - [Framed][se-Framed] stream of log lines
 
 ### Outputs
-1 :: [Int][se-Int] (hardcoded)
+1 :: [Int][se-Int] (hardcoded and ignored by client)
 
 
 ## BuildPathsWithResults
@@ -856,10 +862,10 @@ results :: [List][se-List] of [KeyedBuildResult][se-KeyedBuildResult]
 
 ### Inputs
 - storePath :: [StorePath][se-StorePath]
-- gcRoot :: [String][se-String]
+- gcRoot :: [Path][se-Path]
 
 ### Outputs
-gcRoot :: [String][se-String]
+gcRoot :: [Path][se-Path]
 
 
 
@@ -884,6 +890,7 @@ gcRoot :: [String][se-String]
 [se-DrvOutput]: ./serialization.md#drvoutput
 [se-Realisation]: ./serialization.md#realisation
 [se-List]: ./serialization.md#list-of-x
+[se-Set]: ./serialization.md#set-of-x
 [se-Map]: ./serialization.md#map-of-x-to-y
 [se-SubstitutablePathInfo]: ./serialization.md#substitutablepathinfo
 [se-ValidPathInfo]: ./serialization.md#validpathinfo
@@ -892,3 +899,6 @@ gcRoot :: [String][se-String]
 [se-KeyedBuildResult]: ./serialization.md#keyedbuildresult
 [se-BasicDerivation]: ./serialization.md#basicderivation
 [se-Framed]: ./serialization.md#framed
+[se-AddMultipleToStore]: ./serialization.md#addmultipletostore-format
+[se-ExportFormat]: ./serialization.md#export-path-format
+[se-ImportPaths]: ./serialization.md#import-paths-format

tvix/docs/src/nix-daemon/serialization.md

@@ -6,7 +6,7 @@ Little endian byte order
 
 - len :: [UInt64](#uint64)
 - len bytes of content
-- padding with zeros to ensure 64 bit alignment of content with padding
+- padding with zeros to ensure 64 bit alignment of content + padding
 
 
 ## Int serializers
@@ -16,6 +16,9 @@ Little endian byte order
 
 ### Int64
 [UInt64](#uint64) cast to C `int64_t` with upper bounds checking.
+This means that negative numbers can be written but not read.
+Since this is only used for cpuSystem and cpuUser it is fine that
+negative numbers aren't supported.
 
 ### UInt8
 [UInt64](#uint64) cast to C `uint8_t` with upper bounds checking.
@@ -25,7 +28,8 @@ Little endian byte order
 
 ### Time
 [UInt64](#uint64) cast to C `time_t` with upper bounds checking.
-s
+This means that negative numbers can be written but not read.
+
 ### Bool
 Sent as an [Int](#int) where 0 is false and everything else is true.
 
@@ -36,9 +40,9 @@ Sent as an [UInt64](#uint64) where 0 is false and everything else is true.
 An [UInt8](#uint8) enum with the following possible values:
 
 | Name       | Int |
-| --------- | --- |
+| ---------- | --- |
 | Flat       |  0  |
-| Recursive |  1  |
+| NixArchive |  1  |
 
 ### BuildMode
 An [Int](#int) enum with the following possible values:
@@ -137,6 +141,15 @@ An [Int](#int) enum with the following possible values:
 | Int    |  0  |
 | String |  1  |
 
+### OptTrusted
+An [UInt8](#uint8) optional enum with the following possible values:
+
+| Name             | Int |
+| ---------------- | --- |
+| None             |  0  |
+| Some(Trusted)    |  1  |
+| Some(NotTrusted) |  2  |
+
 
 ## Bytes serializers
 
@@ -144,51 +157,107 @@ An [Int](#int) enum with the following possible values:
 Simply a [Bytes](#bytes) that has some UTF-8 string like semantics sometimes.
 
 ### StorePath
-String representation of a full store path.
+[String](#string) representation of a full store path including the store directory.
 
 ### BaseStorePath
-String representation of the basename of a store path. That is the store path
+[String](#string) representation of the basename of a store path. That is the store path
 without the /nix/store prefix.
 
+### StorePathName
+[String](#string) representation of the name part of a base store path. This is the part
+of the store path after the nixbase32 hash and '-'
+
+It must have the following format:
+- Deny ".", "..", or those strings followed by '-'
+- Otherwise check that each character is 0-9, a-z, A-Z or one of +-._?=
+
+### StorePathHash
+[String](#string) representation of the hash part of a base store path. This is the part
+of the store path at the beginning and before the '-' and is in nixbase32 format.
+
+
+### OutputName
+[String](#string) representation of the name of a derivation output.
+This is usually combined with the name in the derivation to form the store path name for the
+store path with this output.
+
+Since output name is usually combined to form a store path name its format must follow the
+same rules as [StorePathName](#storepathname):
+- Deny ".", "..", or those strings followed by '-'
+- Otherwise check that each character is 0-9, a-z, A-Z or one of +-._?=
+
+
 ### OptStorePath
 Optional store path.
 
 If no store path this is serialized as the empty string otherwise it is the same as
 [StorePath](#storepath).
 
-### ContentAddressMethodWithAlgo
+### Path
-One of the following strings:
+[String](#string) representation of an absolute path.
-- text:`hash algorithm`
-- fixed:r:`hash algorithm`
-- fixed:`hash algorithm`
 
-### DerivedPath
+### NARHash
-#### If protocol is 1.30 or newer
+[String](#string) base16-encoded NAR SHA256 hash without algorithm prefix.
-        return DerivedPath::parseLegacy(store, s);
+
-#### If protocol is older than 1.30
+### Signature
-        return parsePathWithOutputs(store, s).toDerivedPath();
+[String](#string) with a signature for the given store path or realisation. This should be
+in the format `name`:`base 64 encoded signature` but this is not enforced in the protocol.
+
+### HashAlgorithm
+[String](#string) with one of the following values:
+- md5
+- sha1
+- sha256
+- sha512
+
+### HashDigest
+[String](#string) with a hash digest in any encoding
+
+### OptHashDigest
+Optional version of [HashDigest](#hashdigest) where empty string means
+no value.
+
+
+### ContentAddressMethodWithAlgo
+[String](#string) with one of the following formats:
+- text:[HashAlgorithm](#hashalgorithm)
+- fixed:r:[HashAlgorithm](#hashalgorithm)
+- fixed:[HashAlgorithm](#hashalgorithm)
+
+### OptContentAddressMethodWithAlgo
+Optional version of [ContentAddressMethodWithAlgo](#contentaddressmethodwithalgo)
+where empty string means no value.
 
 ### ContentAddress
-String with the format:
+[String](#string) with the format:
-- [ContentAddressMethodWithAlgo](#contentaddressmethodwithalgo):`hash`
+- [ContentAddressMethodWithAlgo](#contentaddressmethodwithalgo):[HashDigest](#hashdigest)
 
 ### OptContentAddress
 Optional version of [ContentAddress](#contentaddress) where empty string means
 no content address.
 
+### DerivedPath
+#### If protocol is 1.30 or newer
+output-names = [OutputName](#outputname), { "," [OutputName](#outputname) }<br>
+output-spec = "*" | output-names<br>
+derived-path = [StorePath](#storepath), [ "!", output-spec ]<br>
+
+#### If protocol is older than 1.30
+[StorePath](#storepath), [ "!", [OutputName](#outputname), { "," [OutputName](#outputname) } ]
+
 ### DrvOutput
-String with format:
+[String](#string) with format:
-- `hash with any prefix`!`output name`
+- `hash with any prefix` "!" [OutputName](#outputname)
 
 ### Realisation
-A JSON object sent as a string.
+A JSON object sent as a [String](#string).
 
 The JSON object has the following keys:
 | Key                   | Value                            |
-| --------------------- | ----------------------- |
+| --------------------- | -------------------------------- |
 | id                    | [DrvOutput](#drvoutput)          |
 | outPath               | [StorePath](#storepath)          |
-| signatures            | Array of String         |
+| signatures            | Array of [Signature](#signature) |
 | dependentRealisations | Object where key is [DrvOutput](#drvoutput) and value is [StorePath](#storepath) |
 
 
@@ -200,20 +269,22 @@ A list is encoded as a [Size](#size) length n followed by n encodings of x
 ### Map of x to y
 A map is encoded as a [Size](#size) length n followed by n encodings of pairs of x and y
 
+### Set of x
+A set is encoded as a [Size](#size) length n followed by n encodings of x
 
 ### BuildResult
 - status :: [BuildStatus](#buildstatus)
 - errorMsg :: [String](#string)
 
 #### Protocol 1.29 or newer
-- timesBuilt :: [Int](#int)
+- timesBuilt :: [Int](#int) (defaults to 0)
-- isNonDeterministic :: [Bool64](#bool64)
+- isNonDeterministic :: [Bool64](#bool64) (defaults to false)
-- startTime :: [Time](#time)
+- startTime :: [Time](#time) (defaults to 0)
-- stopTime :: [Time](#time)
+- stopTime :: [Time](#time) (defaults to 0)
 
 #### Protocol 1.37 or newer
-- cpuUser :: [OptMicroseconds](#optmicroseconds)
+- cpuUser :: [OptMicroseconds](#optmicroseconds) (defaults to none)
-- cpuSystem :: [OptMicroseconds](#optmicroseconds)
+- cpuSystem :: [OptMicroseconds](#optmicroseconds) (defaults to none)
 
 #### Protocol 1.28 or newer
 builtOutputs ::  [Map](#map-of-x-to-y) of [DrvOutput](#drvoutput) to [Realisation](#realisations)
@@ -232,23 +303,22 @@ Optional microseconds.
 
 
 ### SubstitutablePathInfo
-- storePath :: [StorePath](#storepath)
 - deriver :: [OptStorePath](#optstorepath)
-- references :: [List](#list-of-x) of [StorePath](#storepath)
+- references :: [Set][#set-of-x] of [StorePath](#storepath)
 - downloadSize :: [UInt64](#uint64)
 - narSize :: [UInt64](#uint64)
 
 
 ### UnkeyedValidPathInfo
 - deriver :: [OptStorePath](#optstorepath)
-- narHash :: [String](#string) SHA256 NAR hash base16 encoded
+- narHash :: [NARHash](#narhash)
-- references :: [List](#list-of-x) of [StorePath](#storepath)
+- references :: [Set](#set-of-x) of [StorePath](#storepath)
 - registrationTime :: [Time](#time)
 - narSize :: [UInt64](#uint64)
 
 #### If protocol version is 1.16 or above
-- ultimate :: [Bool64](#bool64)
+- ultimate :: [Bool64](#bool64) (defaults to false)
-- signatures :: [List](#list-of-x) of [String](#string)
+- signatures :: [Set](#set-of-x) of [Signature](#signature)
 - ca :: [OptContentAddress](#optcontentaddress)
 
 
@@ -257,13 +327,13 @@ Optional microseconds.
 - info :: [UnkeyedValidPathInfo](#unkeyedvalidpathinfo)
 
 ### DerivationOutput
-- path :: [String](#string)
+- path :: [OptStorePath](#optstorepath)
-- hashAlgo :: [String](#string)
+- hashAlgo :: [OptContentAddressMethodWithAlgo](#optcontentaddressmethodwithalgo)
-- hash :: [String](#string)
+- hash :: [OptHashDigest](#opthashdigest)
 
 ### BasicDerivation
-- outputs :: [Map](#map-of-x-to-y) of [String](#string) to [DerivationOutput](#derivationoutput)
+- outputs :: [Map](#map-of-x-to-y) of [OutputName](#outputname) to [DerivationOutput](#derivationoutput)
-- inputSrcs :: [List](#list-of-x) of [StorePath](#storepath)
+- inputSrcs :: [Set](#set-of-x) of [StorePath](#storepath)
 - platform :: [String](#string)
 - builder :: [String](#string)
 - args :: [List](#list-of-x) of [String](#string)
@@ -271,13 +341,13 @@ Optional microseconds.
 
 ### TraceLine
 - havePos :: [Size](#size) (hardcoded to 0)
-- hint :: [String](#string)
+- hint :: [String](#string) (If logger is JSON, invalid UTF-8 is replaced with U+FFFD)
 
 ### Error
 - type :: [String](#string) (hardcoded to `Error`)
 - level :: [Verbosity](#verbosity)
 - name :: [String](#string) (removed and hardcoded to `Error`)
-- msg :: [String](#string)
+- msg :: [String](#string) (If logger is JSON, invalid UTF-8 is replaced with U+FFFD)
 - havePos :: [Size](#size) (hardcoded to 0)
 - traces :: [List](#list-of-x) of [TraceLine](#traceline)
 
@@ -297,9 +367,43 @@ At protocol 1.23 [AddToStoreNar](./operations.md#addtostorenar) introduced a
 framed streaming for sending the NAR dump and later versions of the protocol
 also used this framing for other operations.
 
-At its core the framed streaming is just a series of [Bytes](#bytes) of
+At its core the framed streaming is just a series of [UInt64](#uint64) `size`
-varying length and terminated by an empty [Bytes](#bytes).
+followed by `size` bytes. The stream is terminated when `size` is zero.
+
+Unlike [Bytes](#bytes), frames are *NOT* padded.
 
 This method of sending data has the advantage of not having to parse the data
 to find where it ends. Older versions of the protocol would potentially parse
 the NAR twice.
+
+
+## AddMultipleToStore format
+
+Paths must be topologically sorted.
+
+- expected :: [UInt64](#uint64)
+
+### Repeated expected times
+- info :: [ValidPathInfo](#validpathinfo)
+- NAR dump
+
+
+## Export path format
+- NAR dump
+- 0x4558494es :: [Int](#int) (hardcoded, 'EXIN' in ASCII)
+- path :: [StorePath](#storepath)
+- references :: [Set](#set-of-x) of [StorePath](#storepath)
+- deriver :: [OptStorePath](#optstorepath)
+- hasSignature :: [Int](#int) (hardcoded to 0 in newer versions)
+
+#### If hasSignature is 1
+- signature :: [String](#string) (ignored)
+
+
+## Import paths format
+
+- hasNext :: [UInt64](#uint64)
+
+### While hasNext is 1
+- [Export path format](#export-path-format)
+- hasNext :: [UInt64](#uint64)