It turns out that the netencode spec requiring to ignore *later*
entries meant that every parser has to do an extra check for each
element, instead of just overriding the key in the hash map.
This leads to a situation where the simple implementation is the wrong
one, which would lead to very subtle problems in parsers (see also the
infamous “json duplicate record entry” problem which has been used for
various exploits in the past).
To be fair, exploits are still possible, but at least a `Map.fromList`
will be the right implementation (provided it folds from the left) now
instead of the wrong one.
Examples of the trivial implementation being now right:
Python:
> dict([("foo", 1), ("foo", 2)])
{'foo': 2}
Rust:
> println!("{:?}", HashMap::from([
("foo", 1),
("foo", 2)
]));
{"foo": 2}
Haskell:
> Data.Map.fromList [ ("foo", 1), ("foo", 2) ]
fromList [("foo",2)]
Change-Id: Ife9593956f4718e5e720f4f348c227e4f3a71e2d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5108
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: Profpatsch <mail@profpatsch.de>
`write` returns the written usize; now I wonder why rustc didn’t at
least produce a warning because the result was unused. Do we need to
add any flags to `rustSimple`?
Change-Id: If8d51d95c993dec6c92e46dbc82cd8cdd398f441
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3056
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
It’s the inverse of record-splice-env! It sucks up the environment and
prints it as a netencode dict! Only the utf-8 clean parts at least.
Change-Id: I96c19fc5ea3a67a23e238f15f4d0fa783081859c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2527
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
In order to arbitrarily split netencode over multiple reads, we need
to make the parser completely streaming, so that it recognizes all
cases where it needs more input.
Luckily, this is fairly trivial, after working around a bunch of
overeager parsing.
The tricky part was the giant `alt`, where inner parsers would start
consuming input and thus become incomplete when they fail afterwards.
Sinc the format *always* starts the different types with one
discriminator char, we can use that to instantly return the parser and
try the next one instead.
The other tricky part was that lists and records would parse all inner
elements and then choke on the empty string after the last element,
because the inner parser would consume at least the descriminator, and
an empty string is always `Incomplete`. We wrap these into a small
combinator which plays nice with `many0` in that regard.
Change-Id: Ib8d15d9a7cab19d432c6b24a35fcad6a5a72b246
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2704
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tries to decode the inner type, turning it into an Option.
Change-Id: I29d1286fe873c28d7c4a4b71f220acaf2d23f8e1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2522
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
`Text` and `Binary` should be self-explaining, they just match on the
primitive and throw an error otherwise.
OneOf is cool, because it allows the user to match on the
result type of decoding `inner`, and give a list of values that should
be allowed as the result type (the associated type `A` in the
`Decoder` trait).
Change-Id: Ia252e25194610555c17c37640a96953142f0a165
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2498
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
This fell out of us moving the `U::List` to a `Vec`.
I noticed that now we have deep recursion for `U`s, which originally
wasn’t intended; reverting to contain `&[u8]` might be a good
experiment, as long as the lists stay a `Vec<&'a [u8]`, which was the
thing preventing us from parsing lists without allocating memory.
Change-Id: I4900c5dea460fa69a78ce0dbed5708495af5d2e1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2495
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
`dec::RecordDot` accesses a specific field of a netencode record.
In order to implement this, either we’d have to introduce a type-level
string, but in all honesty this kind of typelevel circlejerking never
leads anywhere, so let’s change the trait to use `&self` after all.
Usage is pretty much the same, except actually more like you’d expect.
Change-Id: I5a7f1a3f587256c50df1b65c2969e5a7194bba70
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2494
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Since we don’t necessarily need to decode deeply, we can make the
decoders take a `U` instead of a `T`.
Change-Id: I9704a21edb3922d58411e6807d027d684b18d390
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2492
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Also change the toplevel `encode()` to take a `&U` instead of an owned
`U`.
Change-Id: I8e51540cc531e70ae1c94e3676f4dd88da7a924d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2491
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
`U::Record` is required to be a hash map (later keys should be
ignored), so why not do the hash map immediately.
This surfaced a problem with read-http, because duplicate headers in
http are possible, but before they’d be silently ignored.
Now we merge them into a `U::List` in case, to be handled by
consumers of read-http.
Change-Id: Ifd594916f76e5acf9d08e705e0dec2c10a0081c9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2490
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Earlier we left the next level of values unencoded, since lists are
just concatenated netencode values. But I noticed that you can’t write
e.g. a `t_to_u` function, because only in the case of lists you need
to allocate memory.
Turns out that if we read the next level of values, everything is
handled the same as in `Record` and things suddenly start working.
We can also throw away some of the strange and ad-hoc parser helpers
we needed before, `skip` and `list_take`, since now those are just
normal `Vec::iter().skip()` and take.
Change-Id: Ibc476e028102944a65c2b64621047086cfc09aa5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2488
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Since `Text` is a scalar, it doesn’t make sense to delay the utf-8
verification to the consumer.
Change-Id: I36e4d228fbf35374d7c1addb4b24828cf6e927e5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2478
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
a044a87084 removed boxes in T::List, but
the tests were not adjusted accordingly.
Seems like netencode fell victim to CI not recursing into attrsets not
generated by readTree in pipeline generation.
Change-Id: I65d58a82881059983f7d6bc7a32263c6671ccbba
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2486
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
There is this semantic exit code schema championed by execline and
skaware tooling, and we refined and documented it a bit in lorri
d1d673d420/src/ops/mod.rs (L24-L35)
in the past.
This just transcribes the error messages into simple helper functions.
Applies the functions to the places where we would panic or die
`sys::exit()` instead.
Change-Id: I15ca05cd6f99a25a3378518be94110eab416354e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2475
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Decoders are implemented not directly on output types, but on trivial
proxy types, so that we can easily combine those into a decoder, and
then the associated type is the actual return value of the decoder.
Change-Id: Ibce98fa09fc944e02ab327112ec7ffbc09815830
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2455
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Apparently HashMap and Vec already do internal boxing, so the extra
indirection in the value isn’t needed.
Then, in order to make things uniform, move the boxing of `Sum` into
the `Tag` value. No extra boxing in the recursion! \o/
Change-Id: Ic21d2e3e6ac0c6e1f045bf2c9d3e9c5af446fcff
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2443
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
A little executable, combining the netencode and mustache libraries to
make easy templating from the command line possible.
Combined with the nix netencode generators, it’s now trivial to
populate a mustache template with (nearly) arbitrary data.
Yay.
Change-Id: I5b892c38fbc33dd826a26174dd9567f0b72e6322
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2320
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
The netencode standard, a no-nonsense extension of netstrings for
structured data.
Includes a nix generator module and a rust parsing library.
Imported from
e409df3861/pkgs/profpatsch/netencode
Original license GPLv3, but I’m the sole author, so I transfer it to
whatever license depot uses.
Change-Id: I4f6fa97120a0fd861eeef35085a3dd642ab7c407
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2319
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
