Commit graph

537 commits

Author SHA1 Message Date
zseri
536f7c134a feat(depot-scanner): forward non-processed stderr output
previously, depot-scanner swallowed/ignored all non-processed
stderr output of nix-instantiate, which makes diagnosing
failures of nix-instantiate (e.g. failed with exit status 1)
difficult. This commit fixes that by always forwarding
the remaining stderr messages.

Example previous error message:

  panic: nix-instantiate failed: exit status 1

  goroutine 1 [running]:
  main.main()
	/nix/store/8vb2j13bd7j5ipl7dhsnwvgr7nrrsqsi-main.go:160 +0xeb4

Example new error message:

  nix-inst> error: unrecognised flag '--trace-file-access'
  nix-inst> Try '/run/current-system/sw/bin/nix-instantiate --help' for more information.
  panic: nix-instantiate failed: exit status 1

  goroutine 1 [running]:
  main.main()
	/nix/store/qy7v79a3harddirzmc0432vbzqhyf91i-main.go:165 +0xeb4

Change-Id: I666f3490fc648f77a5384b95edd74f6115f7920d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4553
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-12-25 05:19:14 +00:00
Vincent Ambo
4c23a9c1d4 feat(depotfmt): run formatting checks in CI
Right now this only includes Go, but more is to come.

Change-Id: Idd8fc27c0eb25e82688ef8337ba20810d834f4b6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4504
Reviewed-by: zseri <zseri.devel@ytrizja.de>
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
Autosubmit: tazjin <mail@tazj.in>
2021-12-24 19:26:00 +00:00
Vincent Ambo
9c61d64a11 feat(depotfmt): add terraform formatting to depotfmt
Change-Id: I235e7fcbd49f11e71127ce31da9c11da4d220ff6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4584
Autosubmit: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2021-12-24 19:25:48 +00:00
Jérôme Petazzoni
7433d620bb feat: add /tmp
Examples of programs that fail when /tmp doesn't exist:
- terraform
- anything using mktemp and similar helpers
2021-12-24 22:19:55 +03:00
Ethan Davidson
15f79e1364 docs: mention arm64 metapackage 2021-12-24 21:42:02 +03:00
Jérôme Petazzoni
aaf5370344 chore: fix env var name in error message
The error message shows the wrong variable name, which might
be confusing for new users.
2021-12-24 20:06:50 +03:00
Jérôme Petazzoni
1dd3421615 docs: update installation instructions
These instructions were not up-to-date (they didn't mention
the different storage backends, and some variables were
tagged as optional while they were mandatory). With this
update, they should (hopefully) be more accurate! :)

I also added instructions if someone wants to run Nixery
outside of the container image (I found it convenient when
working on Nixery's code).
2021-12-24 18:49:35 +03:00
Vincent Ambo
29db849849 fix(depot-scanner): Reenable CI runs
(zseri:) It seems like relativePath should be replaced with relativeFlag,
as no variable with the former name exists, a boolean is as far as I can
tell expected, and a boolean with a similar name exists. Lets give it a
try in the CI.

Change-Id: I0e7e522a41a517a38222dcda3b66731344613c1e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3581
Reviewed-by: tazjin <mail@tazj.in>
Autosubmit: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2021-12-24 10:18:40 +00:00
Griffin Smith
d4ce1607c2 fix(tvl.el): Change prompt for different commands
Prompting with "Checkout CL" for the gerrit cherry-pick command doesn't
make any sense.

Change-Id: I51495e7975202146fae2da0807e525596f2d490e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4516
Autosubmit: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-12-21 10:28:22 +00:00
Vincent Ambo
d018484376 feat(tools/depotfmt): add a treefmt wrapper with depot configuration
Right now the only included formatter is gofmt, but we will extend
this over time.

The version of treefmt is bumped to 0.3.0 (which supports custom
config files) until this lands in nixpkgs.

Change-Id: I1e1aafd05ec7427c616f90c90490c528ecb2615c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4399
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
2021-12-18 20:26:26 +00:00
sterni
3090f61713 fix(tools/magrathea): prevent trailing slash for // in path
All targets would have no trailing slash, so it was at least ugly that
// had one as the only legal target.

Change-Id: I1b60850ac86d8c550f262841694fb00c518413b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4404
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-12-18 20:23:35 +00:00
sterni
4c07a97af3 feat(tools/cheddar): shortlink depot revisions
r/3000 will now be shortlinked to show the commit in cgit. Going via
atward probably doesn't make sense at this point, since the depot refs
are not available in sourcegraph at all (for reasons I can't
repeat). Switching to atward might be interesting when/if we introduce
support for shortlinking // paths.

Fixes: b/163
Change-Id: I57c1a7d02d881e4f8b3ee1f71755dd7930925dc4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4402
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: tazjin <mail@tazj.in>
2021-12-18 19:16:47 +00:00
Vincent Ambo
8fb90b9e5e feat(tools/magrathea): let users override the repository root
for global installations of magrathea, setting MG_ROOT can be a way to
switch quickly between different repositories (esp. in combination
with `cd (mg path)`).

Change-Id: I4627fe78b7cc112b75ab57e7806ffd85c6d38aee
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4396
Tested-by: BuildkiteCI
Reviewed-by: wpcarro <wpcarro@gmail.com>
2021-12-18 15:03:02 +00:00
Vincent Ambo
46a4e25550 feat(tools/magrathea): add a path command
this command prints the absolute directory for a given target. it can
be combined with shell aliases to add quick navigation commands.

unfortunately due to the nature of computers implementing something
like `mg cd` directly is not possible.

Change-Id: Icc88eb97384812c620c49fe2de8fa331f4d7153b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4395
Tested-by: BuildkiteCI
Reviewed-by: wpcarro <wpcarro@gmail.com>
2021-12-18 15:03:02 +00:00
Vincent Ambo
589480a925 feat(tools/magrathea): wire up parsed targets with commands
magrathea now does what it says on the tin - build and shell commands
can be used with the targets specified on the command line.

implementation notes:

* string representation of target has been changed to look like the
  target spec format, this is now used in user-facing messages

* errors returned by the target parser make the program exit with
  status 1

* normalisation could be done better (for example, maybe it makes
  sense to always do it) but it's good enough for now

Change-Id: Ib85f389a5cec92b3c2f3b9c0b40764435bbcc68b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4394
Tested-by: BuildkiteCI
Reviewed-by: wpcarro <wpcarro@gmail.com>
2021-12-18 15:03:02 +00:00
Vincent Ambo
cecc249c96 feat(tools/magrathea): add weirdly overengineered target spec parser
i'm not sure what happened here, but it works (yes, the fancy target
printing is completely unnecessary, but oh well):

    #;152> (parse-target "foo")
    #target(foo)

    #;153> (parse-target "//foo")
    #target(//foo)

    #;154> (parse-target "//foo/bar")
    #target(//foo/bar)

    #;155> (parse-target "//foo/bar/")
    #target(//foo/bar)

    #;156> (parse-target "//foo/bar:baz")
    #target(//foo/bar:baz)

    #;157> (parse-target "//foo/bar/:baz")
    #target(//foo/bar:baz)

    #;158> (parse-target "//foo/bar:")
    (error . "unexpected end of input while parsing virtual target")

    #;159> (parse-target "//foo//")
    (error . "unexpected root-anchor while parsing normal target")

the most notable thing is that trailing slashes are allowed in the
physical targets, since people may be autocompleting these on the
shell from folder names.

Change-Id: I32975ad77fe2a327130dc9574011fe92cce49f84
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4393
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: wpcarro <wpcarro@gmail.com>
2021-12-17 20:48:35 +00:00
Vincent Ambo
564ad52b30 feat(tools/magrathea): bootstrap a tool for working with depot
this is going to be a serious version of //tools/depot-build.

right now it doesn't support parsing any target specs yet, so only
shells and builds for the physical project of the current folder work.

Change-Id: I4308e29da940571622ff9e539fbb8ededd27aca7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4335
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: grfn <grfn@gws.fyi>
2021-12-17 20:48:35 +00:00
Griffin Smith
47f1f3be5f fix(tvl.el): Replace fifth with nth 4
as before, fifth doesn't exist on all emacsen, but nth definitely does

Change-Id: Ic0e4e3790402d960d1546d37187758a4d9ca33c2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4346
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-12-15 23:30:59 +00:00
sterni
6b285f008d chore(3p): bump NixOS channels to 2021-12-15
* Switch from (unused) NixOS 21.05 to newly released NixOS 21.11
  channel.

* grfn/machines/yeren: Linux 5.14.x -> 5.15.x

* 3p/gerrit, 3p/gerrit_plugin/oauth: update dependency bundle output
  hashes to what whitby comes up with now.

Change-Id: Id3bfe3d07b9223f77ec628954783a6b0a48f4086
Reviewed-on: https://cl.tvl.fyi/c/depot/+/4310
Tested-by: BuildkiteCI
Autosubmit: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: tazjin <mail@tazj.in>
2021-12-15 17:43:52 +00:00
Vincent Ambo
db117176f2 fix(passively): Load known terms on startup
Change-Id: Ia32f0a1c158255a098e7d4017bb585b5c4e3c657
2021-12-13 16:36:59 +03:00
Vincent Ambo
40888c9630 fix(tvl.el): Fix use of label command in refs
The l= is part of the command, not of the shape of commands, and the
previous command concatenation logic was wrong because of that.

Fix is done in the most obvious way: Make the l= part of the command.

Change-Id: Ia3c08c3da60fe5fc38f29a2d94adcd123e4f3052
2021-12-10 21:08:48 +00:00
Vincent Ambo
1b94d2c0ba feat(tvl.el): Add autosubmit feature to magit-gerrit-rubberstamp
This makes this function a true rubberstamp again, leading to
rubberstamped CLs automatically being merged after CI passes.

This is similar to the initial functionality we had last year, where
this directly submitted changes, but with the addition of the CI
checks.

Change-Id: I946b074b968eb18a64c4edb0043f7a4af28759b4
2021-12-10 22:00:39 +03:00
Vincent Ambo
7cb9b204e9 feat(tvl.el): Add shortcut for push with Autosubmit+1
This almost makes for a sort of fire&forget button, except we don't
have a way to automatically pick reviewers yet :)

Change-Id: I6f446270f8aaf0409ccb6321bdbb5c349079cd19
2021-12-10 18:54:32 +00:00
Griffin Smith
d4c765743e feat(tvl.el): Add magit-gerrit-cherry-pick
Bound to `A g`, this behaves similarly to `magit-gerrit-checkout` - it
prompts for a CL number, then cherry-picks the latest patchset of that
CL number

Change-Id: Ieef970b99d96170e8c960cc7687ead9022948f8b
2021-12-09 02:08:42 +00:00
Griffin Smith
9246425407 fix(tvl.el): s/fourth/cadddr
Apparently some emacsen don't have functions like `fourth` etc.

Change-Id: I3d8b698685ce3b1757b427b32d8e27938cc26661
2021-12-07 19:19:18 +00:00
Vincent Ambo
1b41e34e79 feat(tools/passively): Implement automatic passive learning in Emacs
Adds all the functionality described in the README in cl/4066.

This code is very closely related to //users/tazjin/russian/russian.el

Change-Id: I14f1052cebfbe4886e75e8efc730eacbf8773f29
2021-12-06 11:50:55 +00:00
Vincent Ambo
4adab5f7b9 feat(tools/passively): Bootstrap new Emacs package 'passively'
Passively is a tool to help people learn information via Emacs,
designed for language learning.

As of this CL, the actual implementation still lives in
//users/tazjin/russian/russian.el but I am generalising it here.

Change-Id: Iac5a8cfc78415496637a7ba5ddc4c2a1aa6bee26
2021-12-03 11:26:12 +00:00
Vincent Ambo
4f1249e46f refactor(readTree): Move 'drvTargets' into readTree
This function is also generally useful for readTree consumers that
have the concept of subtargets.

Change-Id: Ic7fc03380dec6953fb288763a28e50ab3624d233
2021-11-23 14:42:08 +00:00
Griffin Smith
023d6d6a0d feat(tvl.el): Add magit-gerrit-checkout command
Add a new magit-gerrit-checkout command, which prompts for a CL number
then fetches and checks out the latest patchset of that CL with a
detached HEAD.

Change-Id: I88b8209d40017479d97ed40ecbd5fd1ccd7cf650
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3880
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2021-11-20 21:41:00 +00:00
Vincent Ambo
fc62b90514 chore: Bump all Go dependencies
Result of 'go get -u && go mod tidy'
2021-10-29 19:18:27 +02:00
Vincent Ambo
485b8aa929 chore: Bump nixpkgs pin to nixos-unstable 2021-10-29 2021-10-29 18:24:57 +02:00
Vincent Ambo
f4daffbb50 chore(docs): Bump included nix-1p version
... basically never updated this, oops.
2021-10-29 17:41:01 +02:00
Vincent Ambo
9929891f56 docs: Remove note about unsupported Google projects
I no longer work at Google and the repo has moved, so this is no
longer relevant.
2021-10-29 17:32:40 +02:00
sterni
a3f8d2b84b refactor(tools/rust-crates-advisory): move advisory-db to 3p
Change-Id: Iaaed35de078292c0c99a7c83de9ca5fdf27b8135
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3711
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-10-12 13:35:14 +00:00
sterni
bb0df3fb93 chore(tools/cheddar): update dependencies
Change-Id: I64c305c60d2187633cb84e8642e4639f1402325c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3725
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-10-11 14:28:43 +00:00
Jérôme Petazzoni
dd778e7766 revert: "feat(storage): Add generic support for content-types"
This reverts commit 7db252f36a68d875429a25e06d88fbfc804d84fd.

Superseded by the implementation in #127.
2021-10-08 13:44:02 +03:00
sterni
d904724adf refactor(rust-crates-advisory): check type instead of blacklisting
`our-crates` can just check if the attributes in question are
derivation (i. e. have an `outPath`) instead of blacklisting the
`__readTree` attribute specifically.

Change-Id: I472692e89c0e9eff551372c72a73ab765b0b6599
2021-09-15 22:35:23 +00:00
Vincent Ambo
5e08f9b6c6 refactor(cheddar): Support multiple shortlinks & custom link sets
This makes it possible for users of cheddar as a library to supply
their own shortlinks. In practice it is unlikely anyone will do this,
but the change also allows us to (relatively) easily add additional
shortlinks to the set used by TVL.

Note that Cheddar is primarily intended for use by TVL and the default
rendering function interfaces have not changed, and will default to
using TVL shortlinks.

A new public function `format_markdown_with_shortlinks` has been added
with which users can use an alternative set of shortlinks. This
function should not be used in TVL depot code.

Change-Id: I4ddab28cbcf45d07c51323b7b730b96e62922816
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3083
Tested-by: BuildkiteCI
Reviewed-by: lukegb <lukegb@tvl.fyi>
2021-09-10 11:32:33 +00:00
Vincent Ambo
af337010e9 feat(prepare-image): Ensure /usr/bin/env is always present
This is required by common patterns in shell scripts.

There are some caveats around this. Adding logic to filter whether
coreutils is included in an image would slow down the Nix evaluation,
so the link is currently created even in cases where it doesn't point
to anything.

Fixes #109
2021-08-25 16:49:05 +03:00
Vincent Ambo
0ed3c48884 fix(tvl.el): Fix compilation warnings
This fixes all compilation warnings except the one about 'tvl lacking
a parent group, which we can look into later (it doesn't matter that much).

Change-Id: Iaff5e7f5f251f0670afb0a47031ccf197de69818
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3408
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-08-24 17:22:47 +00:00
Griffin Smith
db3fdd5ee8 feat(tvl.el): Make target branch configurable
I'm using gerrit at work now, and would like to use tvl.el to interact
with it via Emacs, but we use a different default branch than "canon".
This makes it configurable, and also marks it as safe so I can configure
it in .dir-locals.el

Change-Id: I66d4c7ce94351f2df863ec49dbc3e1d1d6d1547a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3369
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-08-17 19:07:17 +00:00
Vincent Ambo
02455bd0fd chore(build): Allow passing in a specific commit hash when building
Required for builds where the full repository isn't available (e.g.
from a tarball).
2021-08-06 14:27:21 +03:00
sterni
5a3a495956 feat(tools/depot-nixpkgs-update): tool to update our pin
Very simplistic tool to aid with updating our nixpkgs channel pins:
This tool prints the correct two `*Hashes` sets to stdout, so you can
easily delete the appropriate lines in `third_party/nixpkgs/default.nix`,
run

    ./bin/depot-nixpkgs-update | wl-copy

and paste it into your editor.

Doing this fully automatically would be possible, but would either
a) require changing `default.nix`, so it is regex-able more easily,
b) touching something like rnix-parser which I have no motivation to
at the moment or
c) searching for the old hashes and replacing them with the new
ones. This may be a simple and worthwhile improvement in the future.

Change-Id: I4df44e3827ce9ff6e4fe2d336c08016d799e21a7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3252
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2021-07-28 12:14:52 +00:00
Vincent Ambo
84fb380f57 docs: Update build badge in README
Moves the build badge to point at Github Actions, instead of the old (failing) Travis build
2021-07-15 16:23:42 +02:00
Jérôme Petazzoni
94e04a76b6 feat(storage): Store blob content-type in extended attributes
After the discussion in #116, this stores the blob content types
in extended attributes when using the filesystem backend.

If the underlying filesystem doesn't support extended attributes,
storing blobs won't work; also, if extended attributes get removed,
blobs won't be served anymore. We can relax this behavior if
needed (i.e. log errors but still accept to store or serve blobs).
However, since the Docker Engine (and possibly other container
engines) won't accept to pull images from a registry that doesn't
use correct content types for manifest files, it could be argued
that it's better to give a hard fail. (Otherwise, the container
engine gives cryptic error messages like "missing signature key".)

I can change that behavior (and log errors but still store/serve
blobs to the filesystem) if you think it's better.
2021-06-26 01:27:43 +02:00
Florian Klink
3efbbfcd4e feat(ci): don't mount /var/cache/nixery from tmpfs into docker container
With https://github.com/google/nixery/pull/127, nixery will use extended
attributes to store metadata (when using local storage).

Right now, our integration test mounts a tmpfs to /var/cache/nixery.
However, *user* xattrs aren't supported with tmpfs [1], so setting
xattrs would fail.

To workaround this, use a folder in the current working directory and
hope it's backed by something supporting user xattrs (which is the case
for GitHub Actions).

[1]: https://man7.org/linux/man-pages/man5/tmpfs.5.html#NOTES
2021-06-20 18:33:53 +02:00
Vincent Ambo
65be8f20e0 chore(nixpkgs): Bump channels to 2021-05-25
* users/grfn/system/home/yeren: remove obsolete awscli2 overrides

* ops: make new isSystemUser || isNormalUser assertion happy

* users/grfn/system/system/mugwump: make buildkite agents system users

* users/tazjin/nixos/camden: set isSystemUser = true for git

* users/tazjin/emacs: Remove missing & broken packages

* third_party/openldap: remove, as the argon2 module is now enabled upstream

* third_party/gerrit_plugins: Pinned new unstable hashes

* third_party/nix, third_party/grpc: Disabled CI as these are broken

* third_party/overlays/emacs: Bumped version to stay in sync with channel

* third_party/buzz: Update LIBCLANG_PATH to reference libclang.lib,
  since libclang's default output no longer contains libclang.so

* users/grfn/system/home: Install julia-stable instead of julia (which
  aliases to julia-lts), as the latter depends on an insecure version of
  libgit

Change-Id: Iff33b0ecb0ef07a82d1de35e23c40d2f4bf0f8ed
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3001
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
2021-05-25 17:09:28 +00:00
Vincent Ambo
3c7d4c74be chore(3p/overlays): Remove LLVM11 bump from overlay
* This was mostly for //third_party/nix and its dependencies which now
  have been set to use llvmPackages_11 manually.

* For //users/grfn/achilles we also manually select the newer LLVM version.

* //tools/cheddar doesn't seem to need llvm anymore.

* //third_party/buzz also compiles with clang 7.1.0

* replace clang-tools everywhere with new attribute clang-tools_11

For the future we may want to have something similar again, but it may
not be necessary to invest too much time into it: nixpkgs is set to
upgrade their default llvmPackages to LLVM 11 as well at some point in
the near future.

Co-Authored-By: sterni <sternenseemann@systemli.org>
Change-Id: Id83868dbc476a6c776b59518b856c933f30ea79d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3135
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: sterni <sternenseemann@systemli.org>
Reviewed-by: grfn <grfn@gws.fyi>
2021-05-24 18:15:05 +00:00
Vincent Ambo
797212077d fix(hash-password): Ensure OpenLDAP with correct module is used
This is fixed in upstream nixpkgs, but we're not yet at a commit where
it's used, so it's important to use the OpenLDAP from //third_party

Change-Id: I7c033cd23f45a95c4a4af864ffe561c496833a0d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3143
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-05-23 13:58:10 +00:00
Profpatsch
952afb7da9 feat(tools): add rust-crates-advisory
We have a bunch of crates in `third_party/rust-crates`; it would be
great if we could check them for existing CVEs.

This tool does that, it takes the rust security advisory database,
parses the applicable CVEs, and cross-checks them against the actual
crate versions we list in our package database.

The dumb parser we wrote is tested against all entries in the
database, so we will notice when upstream breaks their shit.
Checking the semver stuff is easy enough with the semver crate.

If an advisory matches, it prints the whole thing and fails the build.

Change-Id: I9e912c43d37a685d9d7a4424defc467a171ea3c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2818
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-05-17 23:00:57 +00:00
sterni
c8ecea1032 docs(cheddar): fix typo in build.rs comment
Change-Id: Id323b4a187b3837d60ac6b8917f3e8571954b8e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3119
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2021-05-12 15:38:57 +00:00
Vincent Ambo
a9902dadcf feat(cheddar): Implement highlighting of CL and bug shortlinks
Implements highlighting of shortlinks like cl/123, or b/123.

Highlighting works by replacing the input Markdown using a simple
regular expression replacement.

We also considered parsing and replacing these links in the Markdown
AST, but it would have been significantly more complex (due to arena
allocation) for little upside and no obvious performance benefit.

Change-Id: I53f03fb17491046d89d0b7f605929571c11ee9a8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3082
Tested-by: BuildkiteCI
Reviewed-by: eta <eta@theta.eu.org>
2021-05-04 15:50:26 +00:00
Vincent Ambo
5e6685c689 feat(cheddar): Add a build script that injects BAT_SYNTAXES
The BAT_SYNTAXES environment variable is injected during Nix builds
and points to the build result of //third_party/bat_syntaxes, which is
included in the cheddar binary.

During development this is a bit annoying: Opening a Nix shell will
set the right variable, but it will also mess with the Cargo
environment in ways that make it annoying to use (*and* require a
pre-build of all dependencies as required by naersk).

For an easier development flow, this adds a build script that
automatically runs Nix, fetches the path to the syntax set and sets
the environment variable. It is a no-op if the variable is already
set.

This makes for a smoother development experience when working on
cheddar in a depot checkout.

Change-Id: Ib5435518897e3443247edd9f362821c0b9ff756b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3081
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2021-05-03 20:18:33 +00:00
Vincent Ambo
d311af9bc0 refactor(cheddar): Split out a library with rendering logic
Splits `main.rs` into `lib.rs` and `bin/cheddar.rs`, which enables
reuse of cheddar's rendering logic in other Rust applications.

Change-Id: Ifd1a44a8d1620c595550a0a497a25b0563e917ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3060
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2021-05-03 13:57:43 +00:00
Vincent Ambo
768f3986a9 feat(build): Run go vet as a step in the GitHub Actions workflow 2021-04-30 13:27:59 +02:00
Vincent Ambo
13d97c9e51 refactor(build): Pin dependencies using Go modules
Drops the go2nix configuration in favour of pkgs.buildGoModule.

Note that the go.sum file is bloated by issues with cyclic
dependencies in some Google projects, but this large number of
dependencies is not actually built.
2021-04-30 13:27:59 +02:00
Vincent Ambo
5c2db7b8ce chore(build): Use current git commit hash as build version 2021-04-30 13:27:59 +02:00
Vincent Ambo
7520f2cb96 chore: Update default NixOS channel to nixos-20.09 2021-04-30 12:28:17 +02:00
Vincent Ambo
8a1add9ef8 chore(ci): Remove unnecessary commands from new CI setup
* remove a step that was not supposed to be committed ("Do we have
  Docker?")
* remove setup of old temporary storage directory (now done in
  integration script test instead)
* skip creation of out-link for initial Nixery build (to avoid
  cache-busting on the second build)
2021-04-30 11:02:38 +02:00
Florian Klink
7e8295189b docs: document unset GOOGLE_APPLICATION_CREDENTIALS
In case the `GOOGLE_APPLICATION_CREDENTIALS` environment variable is not
set, a redirect to storage.googleapis.com is issued, which means the
underlying bucket objects need to be publicly accessible.

This wasn't really obvious until now, so further clarify it.
2021-04-29 23:55:34 +02:00
Florian Klink
970f492235 feat(ci): add integration tests to GitHub Actions, remove .travis.yaml
This copies the integration tests from `.travis.yaml` into a script,
documents the assumptions it makes, and wires it into GitHub Actions.

Contrary to the travis version, we don't use Nixery's GCS backend, as
handing out access to the bucket used, especially for PRs, needs to be
done carefully.

Adding back GCS to the integration test can be done at a later point,
either by using a mock server, or by only exposing the credentials for
master builds (and have the test script decide on whether
GOOGLE_APPLICATION_CREDENTIALS is set or not).

The previous travis version had some complicated post-mortem log
gathering - instead of doing this, we can just `docker run` nixery, but
fork it into the background with the shell - causing it to still be able
to log its output as it's running.

An additional `--rm` is appended, so the container gets cleaned up on
termination - this allows subsequent runs on non-CI infrastructure (like
developer laptops), without having to manually clean up containers.

Fixes #119.
2021-04-29 23:44:42 +02:00
Florian Klink
ee48bd891c feat(ci): remove unneeded permissions: read-all
We don't intend to label, authenticate or whatever with the
GITHUB_TOKEN, so there's not really a reason to give any broader
permissions than the defaults.
2021-04-29 20:23:15 +02:00
Vincent Ambo
d2767bbe8a feat(ci): Configure initial GitHub Actions setup
Travis is being deprecated, and this might be the best option for now.
2021-04-28 22:50:58 +02:00
Jerome Petazzoni
f172107ef1 feat(storage): Add generic support for content-types
When serving a manifest, it is important to set the content-type
correctly (otherwise pulling an image is likely to give a cryptic
error message, "Error response from daemon: missing signature key").

This makes sure that we set the content-type properly for both
manifests and layers.
2021-04-27 15:39:58 +02:00
Jerome Petazzoni
954953d8ba chore(nix): update channel URL
It looks like NixPkgs channels have moved. Fixing this URL allows
using nixos-20.09, for instance.
2021-04-14 14:10:53 +02:00
Vincent Ambo
9d95751302 feat(tools/emacs-pkgs): Add new builder for internal packages
Adds a new internal builder that makes it possible to override the
`emacsPackages` passed to our Emacs packages, which in turn makes it
possible to inject them into the emacsPackages fixpoint and use them
with features like Emacs native compilation.

Change-Id: I80dad57115c83cf5693ae6ba4e4cf3105d103d5e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/3003
Tested-by: BuildkiteCI
Reviewed-by: adisbladis <adisbladis@gmail.com>
Reviewed-by: grfn <grfn@gws.fyi>
2021-04-13 22:56:38 +00:00
Griffin Smith
6266c5d32f refactor(users/glittershark): Rename to grfn
Rename my //users directory and all places that refer to glittershark to
grfn, including nix references and documentation.

This may require some extra attention inside of gerrit's database after
it lands to allow me to actually push things.

Change-Id: I4728b7ec2c60024392c1c1fa6e0d4a59b3e266fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2933
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: glittershark <grfn@gws.fyi>
2021-04-12 14:45:51 +00:00
Vincent Ambo
d62fab4168 refactor(cheddar): Consistent use of depot.third_party vs. pkgs
In preparation for the solution of b/108, we need to consistently use
`depot.third_party` for packages that are only packed in the TVL depot
and `pkgs` for things that come from nixpkgs.

Change-Id: I49d82726b2f3bd7d4923effdd9a7e3f67ddc0659
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2916
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-10 12:06:40 +00:00
Vincent Ambo
8361b82d0a refactor(third_party): Consistent use of depot.third_party vs. pkgs
In preparation for the solution of b/108, we need to consistently use
`depot.third_party` for packages that are only packed in the TVL depot
and `pkgs` for things that come from nixpkgs.

This commit cleans up a huge chunk of these uses in //third_party

Change-Id: Ic382c0cdea7330a84d5f0b7d109c824ddceb94e7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2912
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-10 11:48:55 +00:00
Vincent Ambo
057331aa64 feat(cheddar): Set front_matter_delimiter = '---' extension
This chops off "front matter", which we have - for example - in Tvix
documentation that is rendered through Pandoc.

Change-Id: I65c7a4ad8171423a50aaf47c046dab8c774346d8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2717
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2021-03-31 22:20:49 +00:00
Vincent Ambo
ca578c04ec chore(cheddar): Bump crate dependencies
This required some minor Comrak-related refactoring.

Change-Id: I5c5898eb895bd5d8743949458ee9406087fcff22
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2690
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
Reviewed-by: lukegb <lukegb@tvl.fyi>
Reviewed-by: flokli <flokli@flokli.de>
2021-03-31 22:06:50 +00:00
Vincent Ambo
c710509078 test(cheddar): Add simple Markdown rendering tests
Covers some of the odd things we do, specifically callouts and code
rendering.

Change-Id: Ib8542373b434b53d277b0d8c9ddb78ac7c5176a5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2689
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2021-03-31 22:06:50 +00:00
Vincent Ambo
3e1d63ccb3 docs: Update README with a link to the NixCon talk 2020-12-05 14:34:09 +01:00
Dave Nicponski
8ad5c55ad2 docs(config): Fix comment typo 2020-12-04 14:17:21 +01:00
Luke Granger-Brown
1409b9c37b feat(gerrit-update): Add helper script for updating Gerrit schema
I've been running a script similar to this after doing Gerrit version
bumps to make sure the schema is up to date, but in the spirit of making
sure someone other that myself can do this task I'm formalising it into
the depot, where I should've put it in the first place.

Change-Id: I50a198e798e2ff26989b01e4bdd0571d85ab62aa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/2203
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-11-29 11:50:58 +00:00
Vincent Ambo
cc35bf0fc3 feat(storage): Add support for content-types (GCS only)
Extends storage.Persist to accept a Content-Type argument, which in
the GCS backend is persisted with the object to ensure that the object
is served back with this content-type.

This is not yet implemented for the filesystem backend, where the
parameter is simply ignored.

This should help in the case of clients which expect the returned
objects to have content-types set when, for example, fetching layers
by digest.
2020-10-29 17:07:52 +01:00
Vincent Ambo
8a5c446bab docs: Add a note about a Nix-native builder to the roadmap
... if I don't mention this somewhere I'll probably never do it!
2020-10-27 23:07:31 +01:00
Vincent Ambo
9e5ebb2f4f feat(main): Implement caching of manifests in CAS
To ensure that registry clients which attempt to pull manifests by
their content hash can interact with Nixery, this change implements
persisting image manifests in the CAS in the same way as image layers.

In combination with the previous refactorings this means that Nixery's
serving flow is now compatible with containerd.

I have verified this locally, but CI currently only runs against
Docker and not containerd, which is something I plan to address in a
subsequent PR.

This fixes #102
2020-10-27 15:29:11 +01:00
Vincent Ambo
94570aa83f feat(main): Implement serving of manifests by digest
Modifies the layer serving endpoint to be a generic blob-serving
endpoint that can handle both manifest and layer object "types".

Note that this commit does not yet populate the CAS with any
manifests.
2020-10-27 15:29:11 +01:00
Vincent Ambo
cbbf45b5cb refactor(storage): Rename ServeLayer -> Serve
This is going to be used for general content-addressed objects, and is
not layer specific anymore.
2020-10-27 15:29:11 +01:00
Vincent Ambo
5ce745d104 refactor(main): Split HTTP handlers into separate functions
There is a new handler coming up to fix #102 and I want to avoid
falling into the classic Go trap of creating thousand-line functions.
2020-10-27 14:03:14 +01:00
Vincent Ambo
4ce32adfe8 fix(build): Work around arbitrary new maxLayers restriction 2020-10-27 13:16:38 +01:00
Vincent Ambo
ad0541940f fix(build): Completely remove Cachix from build setup
Installing Cachix started failing on ARM64.
2020-10-27 13:16:38 +01:00
Vincent Ambo
5e22faec85 fix(notable): Truncate first lines if they don't fit the window
Change-Id: I00746aa285d7854b8e35b6d00aa7c608ccf8ce77
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1988
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-09-18 23:32:15 +00:00
Vincent Ambo
30066b4ba0 refactor(notable): Remove dependency on dash
It was only used for #'-each, for which the built-in dolist is sufficient.

Change-Id: I326d8b733b36b2edfebd4e9384e70b8a570fee0f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1987
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-09-18 23:32:15 +00:00
Vincent Ambo
426a31b7f2 feat(notable): Add note archival function
Archiving notes is done by just changing the filename to an `archive-`
instead of `note-` prefix.

Unarchiving is not yet implemented and should be done by moving the
note to a *new note ID*.

Archiving is bound to 'a' in the note list.

Change-Id: I8c225a25bdac5147a26030f47f24edee497f69df
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1986
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-09-18 23:32:15 +00:00
Vincent Ambo
f94dd5e932 feat(notable): Add a mode for displaying single notes
This mode displays the note's timestamp in its header line, and the
content in its body.

Some rudimentary key bindings are set up in the list view to view
notes in full (RET), and in both modes to quit the current buffer (q).

Change-Id: Icb35baec220a9efcfa2b86617b16178669df996f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1985
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-09-18 23:32:15 +00:00
Vincent Ambo
b1a84f0019 feat(notable): Add note listing functionality
Adds a very simple initial display of all notes in chronologically
descending order.

Currently this is a read-only view with no available actions.

Change-Id: I6d47363909f1a9ab8d1d18a1c44966883aa758ad
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1983
Reviewed-by: tazjin <mail@tazj.in>
Tested-by: BuildkiteCI
2020-09-17 10:51:05 +00:00
Vincent Ambo
40aeba6281 feat(notable): Add a function for listing existing notes
Change-Id: I23697b4798ee4d4e94d3f7c1a4e4e9abf5115345
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1982
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-09-17 10:51:05 +00:00
Vincent Ambo
6b16e5c1ef feat(emacs-pkgs/notable): Begin implementation by adding note-taking
I'm writing a note taking system because I don't like any of the
existing ones.

This adds a note-taking function which prompts the user for note
content and persists it in a user-configurable directory.

Next steps are roughly:

* listing notes
* editing notes
* archiving notes

At which point I would consider this functional. The idea is to keep
adding incremental improvements that fit my workflows after that, and
if it ends up being useful for anyone else - great.

Change-Id: I7ae0eb4015511ebdbaeff92d8d32502726eb57c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1981
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-09-17 10:51:05 +00:00
Vincent Ambo
11430f4a4b fix(term-switcher.el): Explicitly fail if buffers are missing
Since upgrading to Emacs 27 I have observed a strange behaviour where
this terminal switcher sometimes fails to select a valid buffer, in
which case it falls through to the case that just opens a new buffer
instead.

This is kind of annoying and to aid in debugging this change makes the
creation of new buffers explicit and fails if no matching buffer is
found.

Note that this is likely not a fix for the issue itself, but it will
help debug what is going on.

Change-Id: I906869aba7d25156aaf92c090b169ce02785b85e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1930
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-09-05 11:45:33 +00:00
Vincent Ambo
7e6b3e1b42 chore: Add meta.ci=false to various broken projects
These projects, which are not currently included in CI runs, don't
build at the moment.

Upcoming logic changes would mean that we would start including them
in CI, which is undesirable until they're fixed - but I'm not going to
be doing that now.

Change-Id: I7c337e098be8bff00db6d99fc7236a695f5a85f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1850
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-26 23:49:32 +00:00
Vincent Ambo
468ffcd4b6 fix(tvlc): Do not call mkDerivation in default.nix
This should actually just be an attribute set.

Change-Id: Idea1a9f7cfbb2eecd7e6342c6b5aeb66d3f3441a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1845
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-26 23:49:32 +00:00
Kane York
a9650041bb feat(tools/tvlc): pass list of derivations to depot-scanner
You can now provide a list of Nix derivations to tvlc to get a git worktree + sparse-checkout containing only the paths needed to build the specified derivations.

Known bugs: even though //third_party is only passed to readdir(), git doesn't know this and includes all of //third_party/*.

Change-Id: I9dccebd3fbff4bb04ebd568175cf0a7e37d71ab3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1826
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-25 22:34:02 +00:00
Kane York
dfc351b463 feat(tools/depot-scanner): init project
depot-scanner is a tool that runs Nix and parses the --trace-file-access output to deduce what files are necessary to evaluate a derivation.

Take DEPOT_ROOT from the environment. If depotRoot doesn't exist, print an error early.

Fix the build of the protobuf library. Switch to the GRPC build rule, as a service is in this proto file.

Create the PathType enum and parse it from cmdline flags.

Change-Id: I537b5c6bceecf76ca510f7ac04ab9dad7785feb1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1769
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
Reviewed-by: lukegb <lukegb@tvl.fyi>
2020-08-20 03:05:40 +00:00
Kane York
92d4554b62 feat(tools/tvlc): init project
tvlc is a tool for managing sparse git worktrees of the TVL depot.
It is still in development; near-term tasks include a setup script,
Nix dependency resolution, worktree removal, and the dispatch
script.

See cs.tvl.fyi/depot/docs/designs/SPARSE_CHECKOUTS.md for more info.

Change-Id: Iad96656f0206178980fe7dcadd3dffe70d690f8f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1760
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-19 01:23:54 +00:00
Vincent Ambo
c9d5262aa7 fix(nsfv-setup): Resample pipeline at 48kHz with pulseaudio
This gives better overall results and may also prevent drift, which
was an issue I've had previously.

See also this upstream commit for NSFV:
34003bd9ab

Change-Id: I45534cb60130755949cdbfbc39aed2478a249c72
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1717
Tested-by: BuildkiteCI
Reviewed-by: glittershark <grfn@gws.fyi>
2020-08-12 22:15:53 +00:00
edef
e6ccbc9f35 fix(tools/depot-build): handle multiple levels of subdirectories
Change-Id: I99511c17d635b0222e1c933432007e6f8ea8cf85
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1655
Tested-by: BuildkiteCI
Reviewed-by: tazjin <mail@tazj.in>
2020-08-05 22:31:47 +00:00
Vincent Ambo
87d70d2a80 style(cheddar): Apply rustfmt
Change-Id: I378b5ae618b01244edc090ad1495fbd56857d7c0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/1562
Tested-by: BuildkiteCI
Reviewed-by: kanepyork <rikingcoding@gmail.com>
2020-08-02 21:57:35 +00:00
Vincent Ambo
c194c5662b fix(build): Don't use Cachix as the binary cache during builds
Permission changes in the Travis CI Nix builders have caused this to
start failing, as the build user now has insufficient permissions to
use caches.

There may be a way to change the permissions instead, but in the
meantime we will just cause things to rebuild.
2020-07-25 14:43:21 +01:00