mdebray/tvl-depot
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
tvl-depot/tools/rust-crates-advisory/default.nix

201 lines
4.7 KiB
Nix
Raw Normal View History

feat(tools): add rust-crates-advisory We have a bunch of crates in `third_party/rust-crates`; it would be great if we could check them for existing CVEs. This tool does that, it takes the rust security advisory database, parses the applicable CVEs, and cross-checks them against the actual crate versions we list in our package database. The dumb parser we wrote is tested against all entries in the database, so we will notice when upstream breaks their shit. Checking the semver stuff is easy enough with the semver crate. If an advisory matches, it prints the whole thing and fails the build. Change-Id: I9e912c43d37a685d9d7a4424defc467a171ea3c4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2818 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-04 04:04:20 +02:00
{ depot, pkgs, lib, ... }:
let
bins =
feat(tools/rust-crates-advisory): also check all our Cargo.locks check-all-our-lock-files works very similarly to //users/sterni/nixpkgs-crate-holes, even reusing some parts of it, but is much simpler since we don't need to extract the lock files — they are already in tree. It is implemented as a very simple script which just traverses the subtree of the current directory, collecting all warnings. When executing this script in buildkite via extraSteps, it never fails, instead annotating the pipeline run with a warning. Change-Id: I0a0bc26deffe7b20b99f5aa7238fb3c3bb9deb92 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3721 Reviewed-by: sterni <sternenseemann@systemli.org> Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2021-10-11 14:24:02 +02:00
depot.nix.getBins pkgs.s6-portable-utils [ "s6-ln" "s6-cat" "s6-echo" "s6-mkdir" "s6-test" "s6-touch" "s6-dirname" ]
refactor(rust-crates-advisory): split code for buildkite and report Rename check-all-our-lock-files to tree-lock-file-report and pull out all the buildkite-specific code which makes the code less awkward. check-all-our-lock-files is then only executed in extraSteps and runs tree-lock-file-report on depot, adding it as a warning to the pipeline if it is non-empty. Change-Id: If6bd236d90cc680cba0ed4e988f2f28ddb8012d6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/5229 Tested-by: BuildkiteCI Reviewed-by: Profpatsch <mail@profpatsch.de>
2022-02-04 19:46:48 +01:00
// depot.nix.getBins pkgs.coreutils [ "printf" ]
feat(tools): add rust-crates-advisory We have a bunch of crates in `third_party/rust-crates`; it would be great if we could check them for existing CVEs. This tool does that, it takes the rust security advisory database, parses the applicable CVEs, and cross-checks them against the actual crate versions we list in our package database. The dumb parser we wrote is tested against all entries in the database, so we will notice when upstream breaks their shit. Checking the semver stuff is easy enough with the semver crate. If an advisory matches, it prints the whole thing and fails the build. Change-Id: I9e912c43d37a685d9d7a4424defc467a171ea3c4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2818 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-04 04:04:20 +02:00
// depot.nix.getBins pkgs.lr [ "lr" ]
feat(tools/rust-crates-advisory): also check all our Cargo.locks check-all-our-lock-files works very similarly to //users/sterni/nixpkgs-crate-holes, even reusing some parts of it, but is much simpler since we don't need to extract the lock files — they are already in tree. It is implemented as a very simple script which just traverses the subtree of the current directory, collecting all warnings. When executing this script in buildkite via extraSteps, it never fails, instead annotating the pipeline run with a warning. Change-Id: I0a0bc26deffe7b20b99f5aa7238fb3c3bb9deb92 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3721 Reviewed-by: sterni <sternenseemann@systemli.org> Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2021-10-11 14:24:02 +02:00
// depot.nix.getBins pkgs.cargo-audit [ "cargo-audit" ]
// depot.nix.getBins pkgs.jq [ "jq" ]
// depot.nix.getBins pkgs.findutils [ "find" ]
// depot.nix.getBins pkgs.gnused [ "sed" ]
style: format entire depot with nixpkgs-fmt This CL can be used to compare the style of nixpkgs-fmt against other formatters (nixpkgs, alejandra). Change-Id: I87c6abff6bcb546b02ead15ad0405f81e01b6d9e Reviewed-on: https://cl.tvl.fyi/c/depot/+/4397 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: lukegb <lukegb@tvl.fyi> Reviewed-by: wpcarro <wpcarro@gmail.com> Reviewed-by: Profpatsch <mail@profpatsch.de> Reviewed-by: kanepyork <rikingcoding@gmail.com> Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: cynthia <cynthia@tvl.fyi> Reviewed-by: edef <edef@edef.eu> Reviewed-by: eta <tvl@eta.st> Reviewed-by: grfn <grfn@gws.fyi>
2022-01-30 17:06:58 +01:00
;
feat(tools): add rust-crates-advisory We have a bunch of crates in `third_party/rust-crates`; it would be great if we could check them for existing CVEs. This tool does that, it takes the rust security advisory database, parses the applicable CVEs, and cross-checks them against the actual crate versions we list in our package database. The dumb parser we wrote is tested against all entries in the database, so we will notice when upstream breaks their shit. Checking the semver stuff is easy enough with the semver crate. If an advisory matches, it prints the whole thing and fails the build. Change-Id: I9e912c43d37a685d9d7a4424defc467a171ea3c4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2818 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-04 04:04:20 +02:00
refactor(tools/rust-crates-advisory): move advisory-db to 3p Change-Id: Iaaed35de078292c0c99a7c83de9ca5fdf27b8135 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3711 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in>
2021-10-09 17:22:44 +02:00
crate-advisories = "${depot.third_party.rustsec-advisory-db}/crates";
feat(tools): add rust-crates-advisory We have a bunch of crates in `third_party/rust-crates`; it would be great if we could check them for existing CVEs. This tool does that, it takes the rust security advisory database, parses the applicable CVEs, and cross-checks them against the actual crate versions we list in our package database. The dumb parser we wrote is tested against all entries in the database, so we will notice when upstream breaks their shit. Checking the semver stuff is easy enough with the semver crate. If an advisory matches, it prints the whole thing and fails the build. Change-Id: I9e912c43d37a685d9d7a4424defc467a171ea3c4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2818 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-04 04:04:20 +02:00
refactor(rust-crates-advisory): check type instead of blacklisting `our-crates` can just check if the attributes in question are derivation (i. e. have an `outPath`) instead of blacklisting the `__readTree` attribute specifically. Change-Id: I472692e89c0e9eff551372c72a73ab765b0b6599
2021-09-15 13:25:08 +02:00
our-crates = lib.filter (v: v ? outPath)
(builtins.attrValues depot.third_party.rust-crates);
feat(tools): add rust-crates-advisory We have a bunch of crates in `third_party/rust-crates`; it would be great if we could check them for existing CVEs. This tool does that, it takes the rust security advisory database, parses the applicable CVEs, and cross-checks them against the actual crate versions we list in our package database. The dumb parser we wrote is tested against all entries in the database, so we will notice when upstream breaks their shit. Checking the semver stuff is easy enough with the semver crate. If an advisory matches, it prints the whole thing and fails the build. Change-Id: I9e912c43d37a685d9d7a4424defc467a171ea3c4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2818 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-04 04:04:20 +02:00
feat(rust-crates-advisories): check 3p crates together w/ lock files Instead of the strict check-all-our-crates, generate a fake Cargo.lock and add it to the report generated by check-all-our-lock-files. check-all-our-crates was a reimplementation of cargo-audit anyways and prevented us from updating the advisory db due to its strict model (failing on any advisory). Change-Id: I264a7f1a5058a527cbc46d26225352ecd437a22b Reviewed-on: https://cl.tvl.fyi/c/depot/+/5230 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-02-04 19:54:53 +01:00
our-crates-lock-file = pkgs.writeText "our-crates-Cargo.lock"
(lib.concatMapStrings
(crate: ''
[[package]]
name = "${crate.crateName}"
version = "${crate.version}"
source = "registry+https://github.com/rust-lang/crates.io-index"
'')
our-crates);
style: format entire depot with nixpkgs-fmt This CL can be used to compare the style of nixpkgs-fmt against other formatters (nixpkgs, alejandra). Change-Id: I87c6abff6bcb546b02ead15ad0405f81e01b6d9e Reviewed-on: https://cl.tvl.fyi/c/depot/+/4397 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: lukegb <lukegb@tvl.fyi> Reviewed-by: wpcarro <wpcarro@gmail.com> Reviewed-by: Profpatsch <mail@profpatsch.de> Reviewed-by: kanepyork <rikingcoding@gmail.com> Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: cynthia <cynthia@tvl.fyi> Reviewed-by: edef <edef@edef.eu> Reviewed-by: eta <tvl@eta.st> Reviewed-by: grfn <grfn@gws.fyi>
2022-01-30 17:06:58 +01:00
check-security-advisory = depot.nix.writers.rustSimple
{
name = "parse-security-advisory";
dependencies = [
depot.third_party.rust-crates.toml
depot.third_party.rust-crates.semver
];
}
(builtins.readFile ./check-security-advisory.rs);
feat(tools): add rust-crates-advisory We have a bunch of crates in `third_party/rust-crates`; it would be great if we could check them for existing CVEs. This tool does that, it takes the rust security advisory database, parses the applicable CVEs, and cross-checks them against the actual crate versions we list in our package database. The dumb parser we wrote is tested against all entries in the database, so we will notice when upstream breaks their shit. Checking the semver stuff is easy enough with the semver crate. If an advisory matches, it prints the whole thing and fails the build. Change-Id: I9e912c43d37a685d9d7a4424defc467a171ea3c4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2818 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-04 04:04:20 +02:00
# $1 is the directory with advisories for crate $2 with version $3
check-crate-advisory = depot.nix.writeExecline "check-crate-advisory" { readNArgs = 3; } [
style: format entire depot with nixpkgs-fmt This CL can be used to compare the style of nixpkgs-fmt against other formatters (nixpkgs, alejandra). Change-Id: I87c6abff6bcb546b02ead15ad0405f81e01b6d9e Reviewed-on: https://cl.tvl.fyi/c/depot/+/4397 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: lukegb <lukegb@tvl.fyi> Reviewed-by: wpcarro <wpcarro@gmail.com> Reviewed-by: Profpatsch <mail@profpatsch.de> Reviewed-by: kanepyork <rikingcoding@gmail.com> Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: cynthia <cynthia@tvl.fyi> Reviewed-by: edef <edef@edef.eu> Reviewed-by: eta <tvl@eta.st> Reviewed-by: grfn <grfn@gws.fyi>
2022-01-30 17:06:58 +01:00
"pipeline"
[ bins.lr "-0" "-t" "depth == 1" "$1" ]
"forstdin"
"-0"
"-Eo"
"0"
"advisory"
"if"
[ depot.tools.eprintf "advisory %s\n" "$advisory" ]
check-security-advisory
"$advisory"
"$3"
feat(tools): add rust-crates-advisory We have a bunch of crates in `third_party/rust-crates`; it would be great if we could check them for existing CVEs. This tool does that, it takes the rust security advisory database, parses the applicable CVEs, and cross-checks them against the actual crate versions we list in our package database. The dumb parser we wrote is tested against all entries in the database, so we will notice when upstream breaks their shit. Checking the semver stuff is easy enough with the semver crate. If an advisory matches, it prints the whole thing and fails the build. Change-Id: I9e912c43d37a685d9d7a4424defc467a171ea3c4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2818 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-04 04:04:20 +02:00
];
# Run through everything in the `crate-advisories` repository
# and check whether we can parse all the advisories without crashing.
style: format entire depot with nixpkgs-fmt This CL can be used to compare the style of nixpkgs-fmt against other formatters (nixpkgs, alejandra). Change-Id: I87c6abff6bcb546b02ead15ad0405f81e01b6d9e Reviewed-on: https://cl.tvl.fyi/c/depot/+/4397 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: lukegb <lukegb@tvl.fyi> Reviewed-by: wpcarro <wpcarro@gmail.com> Reviewed-by: Profpatsch <mail@profpatsch.de> Reviewed-by: kanepyork <rikingcoding@gmail.com> Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: cynthia <cynthia@tvl.fyi> Reviewed-by: edef <edef@edef.eu> Reviewed-by: eta <tvl@eta.st> Reviewed-by: grfn <grfn@gws.fyi>
2022-01-30 17:06:58 +01:00
test-parsing-all-security-advisories = depot.nix.runExecline "check-all-our-crates" { } [
"pipeline"
[ bins.lr "-0" "-t" "depth == 1" crate-advisories ]
"if"
[
feat(tools): add rust-crates-advisory We have a bunch of crates in `third_party/rust-crates`; it would be great if we could check them for existing CVEs. This tool does that, it takes the rust security advisory database, parses the applicable CVEs, and cross-checks them against the actual crate versions we list in our package database. The dumb parser we wrote is tested against all entries in the database, so we will notice when upstream breaks their shit. Checking the semver stuff is easy enough with the semver crate. If an advisory matches, it prints the whole thing and fails the build. Change-Id: I9e912c43d37a685d9d7a4424defc467a171ea3c4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2818 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-04 04:04:20 +02:00
# this will succeed as long as check-crate-advisory doesn’t `panic!()` (status 101)
style: format entire depot with nixpkgs-fmt This CL can be used to compare the style of nixpkgs-fmt against other formatters (nixpkgs, alejandra). Change-Id: I87c6abff6bcb546b02ead15ad0405f81e01b6d9e Reviewed-on: https://cl.tvl.fyi/c/depot/+/4397 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: lukegb <lukegb@tvl.fyi> Reviewed-by: wpcarro <wpcarro@gmail.com> Reviewed-by: Profpatsch <mail@profpatsch.de> Reviewed-by: kanepyork <rikingcoding@gmail.com> Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: cynthia <cynthia@tvl.fyi> Reviewed-by: edef <edef@edef.eu> Reviewed-by: eta <tvl@eta.st> Reviewed-by: grfn <grfn@gws.fyi>
2022-01-30 17:06:58 +01:00
"forstdin"
"-0"
"-E"
"-x"
"101"
"crate_advisories"
check-crate-advisory
"$crate_advisories"
"foo"
"0.0.0"
feat(tools): add rust-crates-advisory We have a bunch of crates in `third_party/rust-crates`; it would be great if we could check them for existing CVEs. This tool does that, it takes the rust security advisory database, parses the applicable CVEs, and cross-checks them against the actual crate versions we list in our package database. The dumb parser we wrote is tested against all entries in the database, so we will notice when upstream breaks their shit. Checking the semver stuff is easy enough with the semver crate. If an advisory matches, it prints the whole thing and fails the build. Change-Id: I9e912c43d37a685d9d7a4424defc467a171ea3c4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2818 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-04 04:04:20 +02:00
]
style: format entire depot with nixpkgs-fmt This CL can be used to compare the style of nixpkgs-fmt against other formatters (nixpkgs, alejandra). Change-Id: I87c6abff6bcb546b02ead15ad0405f81e01b6d9e Reviewed-on: https://cl.tvl.fyi/c/depot/+/4397 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: lukegb <lukegb@tvl.fyi> Reviewed-by: wpcarro <wpcarro@gmail.com> Reviewed-by: Profpatsch <mail@profpatsch.de> Reviewed-by: kanepyork <rikingcoding@gmail.com> Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: cynthia <cynthia@tvl.fyi> Reviewed-by: edef <edef@edef.eu> Reviewed-by: eta <tvl@eta.st> Reviewed-by: grfn <grfn@gws.fyi>
2022-01-30 17:06:58 +01:00
"importas"
"out"
"out"
bins.s6-touch
"$out"
feat(tools): add rust-crates-advisory We have a bunch of crates in `third_party/rust-crates`; it would be great if we could check them for existing CVEs. This tool does that, it takes the rust security advisory database, parses the applicable CVEs, and cross-checks them against the actual crate versions we list in our package database. The dumb parser we wrote is tested against all entries in the database, so we will notice when upstream breaks their shit. Checking the semver stuff is easy enough with the semver crate. If an advisory matches, it prints the whole thing and fails the build. Change-Id: I9e912c43d37a685d9d7a4424defc467a171ea3c4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2818 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-04 04:04:20 +02:00
];
refactor(rust-crates-advisory): move report generation into script This script is somewhat usable by humans (it even has a help screen!) and can be reused in //users/sterni/nixpkgs-crate-holes. We are using bash since that allows us to exit with the actual exit code of cargo-audit - something that's not possible in execline. Change-Id: I3331ae8222a20e23b8e30dc920ab48af78f0247c Reviewed-on: https://cl.tvl.fyi/c/depot/+/5228 Tested-by: BuildkiteCI Reviewed-by: Profpatsch <mail@profpatsch.de>
2022-02-04 17:28:09 +01:00
lock-file-report = pkgs.writers.writeBash "lock-file-report" ''
set -u
if test "$#" -lt 2; then
echo "Usage: $0 IDENTIFIER LOCKFILE [CHECKLIST [MAINTAINERS]]" >&2
echo 2>&1
echo " IDENTIFIER Unique string describing the lock file" >&2
echo " LOCKFILE Path to Cargo.lock file" >&2
echo " CHECKLIST Whether to use GHFM checklists in the output (true or false)" >&2
echo " MAINTAINERS List of @names to cc in case of advisories" >&2
exit 100
fi
"${bins.cargo-audit}" audit --json --no-fetch \
--db "${depot.third_party.rustsec-advisory-db}" \
--file "$2" \
| "${bins.jq}" --raw-output --join-output \
--from-file "${./format-audit-result.jq}" \
--arg maintainers "''${4:-}" \
--argjson checklist "''${3:-false}" \
--arg attr "$1"
exit "''${PIPESTATUS[0]}" # inherit exit code from cargo-audit
'';
refactor(rust-crates-advisory): split code for buildkite and report Rename check-all-our-lock-files to tree-lock-file-report and pull out all the buildkite-specific code which makes the code less awkward. check-all-our-lock-files is then only executed in extraSteps and runs tree-lock-file-report on depot, adding it as a warning to the pipeline if it is non-empty. Change-Id: If6bd236d90cc680cba0ed4e988f2f28ddb8012d6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/5229 Tested-by: BuildkiteCI Reviewed-by: Profpatsch <mail@profpatsch.de>
2022-02-04 19:46:48 +01:00
tree-lock-file-report = depot.nix.writeExecline "tree-lock-file-report"
{
readNArgs = 1;
} [
feat(tools/rust-crates-advisory): also check all our Cargo.locks check-all-our-lock-files works very similarly to //users/sterni/nixpkgs-crate-holes, even reusing some parts of it, but is much simpler since we don't need to extract the lock files — they are already in tree. It is implemented as a very simple script which just traverses the subtree of the current directory, collecting all warnings. When executing this script in buildkite via extraSteps, it never fails, instead annotating the pipeline run with a warning. Change-Id: I0a0bc26deffe7b20b99f5aa7238fb3c3bb9deb92 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3721 Reviewed-by: sterni <sternenseemann@systemli.org> Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2021-10-11 14:24:02 +02:00
"backtick"
"-E"
"report"
[
"pipeline"
refactor(rust-crates-advisory): split code for buildkite and report Rename check-all-our-lock-files to tree-lock-file-report and pull out all the buildkite-specific code which makes the code less awkward. check-all-our-lock-files is then only executed in extraSteps and runs tree-lock-file-report on depot, adding it as a warning to the pipeline if it is non-empty. Change-Id: If6bd236d90cc680cba0ed4e988f2f28ddb8012d6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/5229 Tested-by: BuildkiteCI Reviewed-by: Profpatsch <mail@profpatsch.de>
2022-02-04 19:46:48 +01:00
[ bins.find "$1" "-name" "Cargo.lock" "-and" "-type" "f" "-print0" ]
feat(tools/rust-crates-advisory): also check all our Cargo.locks check-all-our-lock-files works very similarly to //users/sterni/nixpkgs-crate-holes, even reusing some parts of it, but is much simpler since we don't need to extract the lock files — they are already in tree. It is implemented as a very simple script which just traverses the subtree of the current directory, collecting all warnings. When executing this script in buildkite via extraSteps, it never fails, instead annotating the pipeline run with a warning. Change-Id: I0a0bc26deffe7b20b99f5aa7238fb3c3bb9deb92 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3721 Reviewed-by: sterni <sternenseemann@systemli.org> Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2021-10-11 14:24:02 +02:00
"forstdin"
"-E"
"-0"
"lockFile"
"backtick"
"-E"
"depotPath"
[
"pipeline"
[ bins.s6-dirname "$lockFile" ]
bins.sed
"s|^\\.|/|"
]
refactor(rust-crates-advisory): move report generation into script This script is somewhat usable by humans (it even has a help screen!) and can be reused in //users/sterni/nixpkgs-crate-holes. We are using bash since that allows us to exit with the actual exit code of cargo-audit - something that's not possible in execline. Change-Id: I3331ae8222a20e23b8e30dc920ab48af78f0247c Reviewed-on: https://cl.tvl.fyi/c/depot/+/5228 Tested-by: BuildkiteCI Reviewed-by: Profpatsch <mail@profpatsch.de>
2022-02-04 17:28:09 +01:00
lock-file-report
feat(tools/rust-crates-advisory): also check all our Cargo.locks check-all-our-lock-files works very similarly to //users/sterni/nixpkgs-crate-holes, even reusing some parts of it, but is much simpler since we don't need to extract the lock files — they are already in tree. It is implemented as a very simple script which just traverses the subtree of the current directory, collecting all warnings. When executing this script in buildkite via extraSteps, it never fails, instead annotating the pipeline run with a warning. Change-Id: I0a0bc26deffe7b20b99f5aa7238fb3c3bb9deb92 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3721 Reviewed-by: sterni <sternenseemann@systemli.org> Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2021-10-11 14:24:02 +02:00
"$depotPath"
refactor(rust-crates-advisory): move report generation into script This script is somewhat usable by humans (it even has a help screen!) and can be reused in //users/sterni/nixpkgs-crate-holes. We are using bash since that allows us to exit with the actual exit code of cargo-audit - something that's not possible in execline. Change-Id: I3331ae8222a20e23b8e30dc920ab48af78f0247c Reviewed-on: https://cl.tvl.fyi/c/depot/+/5228 Tested-by: BuildkiteCI Reviewed-by: Profpatsch <mail@profpatsch.de>
2022-02-04 17:28:09 +01:00
"$lockFile"
feat(tools/rust-crates-advisory): omit GHFM checklist in buildkite Buildkite doesn't understand GitHub Flavored Markdown and having a read only checklist in there is probably not much use. Change-Id: I41538487087e8c817b1a5e653f077bb0fbe6eb47 Reviewed-on: https://cl.tvl.fyi/c/depot/+/5201 Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: tazjin <tazjin@tvl.su> Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2022-02-03 14:50:13 +01:00
"false"
feat(tools/rust-crates-advisory): also check all our Cargo.locks check-all-our-lock-files works very similarly to //users/sterni/nixpkgs-crate-holes, even reusing some parts of it, but is much simpler since we don't need to extract the lock files — they are already in tree. It is implemented as a very simple script which just traverses the subtree of the current directory, collecting all warnings. When executing this script in buildkite via extraSteps, it never fails, instead annotating the pipeline run with a warning. Change-Id: I0a0bc26deffe7b20b99f5aa7238fb3c3bb9deb92 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3721 Reviewed-by: sterni <sternenseemann@systemli.org> Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2021-10-11 14:24:02 +02:00
]
"if"
refactor(rust-crates-advisory): split code for buildkite and report Rename check-all-our-lock-files to tree-lock-file-report and pull out all the buildkite-specific code which makes the code less awkward. check-all-our-lock-files is then only executed in extraSteps and runs tree-lock-file-report on depot, adding it as a warning to the pipeline if it is non-empty. Change-Id: If6bd236d90cc680cba0ed4e988f2f28ddb8012d6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/5229 Tested-by: BuildkiteCI Reviewed-by: Profpatsch <mail@profpatsch.de>
2022-02-04 19:46:48 +01:00
[ bins.printf "%s\n" "$report" ]
feat(tools/rust-crates-advisory): also check all our Cargo.locks check-all-our-lock-files works very similarly to //users/sterni/nixpkgs-crate-holes, even reusing some parts of it, but is much simpler since we don't need to extract the lock files — they are already in tree. It is implemented as a very simple script which just traverses the subtree of the current directory, collecting all warnings. When executing this script in buildkite via extraSteps, it never fails, instead annotating the pipeline run with a warning. Change-Id: I0a0bc26deffe7b20b99f5aa7238fb3c3bb9deb92 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3721 Reviewed-by: sterni <sternenseemann@systemli.org> Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2021-10-11 14:24:02 +02:00
# empty report implies success (no advisories)
refactor(rust-crates-advisory): split code for buildkite and report Rename check-all-our-lock-files to tree-lock-file-report and pull out all the buildkite-specific code which makes the code less awkward. check-all-our-lock-files is then only executed in extraSteps and runs tree-lock-file-report on depot, adding it as a warning to the pipeline if it is non-empty. Change-Id: If6bd236d90cc680cba0ed4e988f2f28ddb8012d6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/5229 Tested-by: BuildkiteCI Reviewed-by: Profpatsch <mail@profpatsch.de>
2022-02-04 19:46:48 +01:00
bins.s6-test
"-z"
"$report"
];
check-all-our-lock-files = depot.nix.writeExecline "check-all-our-lock-files" { } [
"backtick"
"-EI"
"report"
[
feat(rust-crates-advisories): check 3p crates together w/ lock files Instead of the strict check-all-our-crates, generate a fake Cargo.lock and add it to the report generated by check-all-our-lock-files. check-all-our-crates was a reimplementation of cargo-audit anyways and prevented us from updating the advisory db due to its strict model (failing on any advisory). Change-Id: I264a7f1a5058a527cbc46d26225352ecd437a22b Reviewed-on: https://cl.tvl.fyi/c/depot/+/5230 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-02-04 19:54:53 +01:00
"foreground"
[
lock-file-report
"//third_party/rust-crates"
our-crates-lock-file
"false"
]
refactor(rust-crates-advisory): split code for buildkite and report Rename check-all-our-lock-files to tree-lock-file-report and pull out all the buildkite-specific code which makes the code less awkward. check-all-our-lock-files is then only executed in extraSteps and runs tree-lock-file-report on depot, adding it as a warning to the pipeline if it is non-empty. Change-Id: If6bd236d90cc680cba0ed4e988f2f28ddb8012d6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/5229 Tested-by: BuildkiteCI Reviewed-by: Profpatsch <mail@profpatsch.de>
2022-02-04 19:46:48 +01:00
tree-lock-file-report
"."
]
"ifelse"
feat(tools/rust-crates-advisory): also check all our Cargo.locks check-all-our-lock-files works very similarly to //users/sterni/nixpkgs-crate-holes, even reusing some parts of it, but is much simpler since we don't need to extract the lock files — they are already in tree. It is implemented as a very simple script which just traverses the subtree of the current directory, collecting all warnings. When executing this script in buildkite via extraSteps, it never fails, instead annotating the pipeline run with a warning. Change-Id: I0a0bc26deffe7b20b99f5aa7238fb3c3bb9deb92 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3721 Reviewed-by: sterni <sternenseemann@systemli.org> Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2021-10-11 14:24:02 +02:00
[
bins.s6-test
refactor(rust-crates-advisory): split code for buildkite and report Rename check-all-our-lock-files to tree-lock-file-report and pull out all the buildkite-specific code which makes the code less awkward. check-all-our-lock-files is then only executed in extraSteps and runs tree-lock-file-report on depot, adding it as a warning to the pipeline if it is non-empty. Change-Id: If6bd236d90cc680cba0ed4e988f2f28ddb8012d6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/5229 Tested-by: BuildkiteCI Reviewed-by: Profpatsch <mail@profpatsch.de>
2022-02-04 19:46:48 +01:00
"-z"
"$report"
]
[
"exit"
"0"
feat(tools/rust-crates-advisory): also check all our Cargo.locks check-all-our-lock-files works very similarly to //users/sterni/nixpkgs-crate-holes, even reusing some parts of it, but is much simpler since we don't need to extract the lock files — they are already in tree. It is implemented as a very simple script which just traverses the subtree of the current directory, collecting all warnings. When executing this script in buildkite via extraSteps, it never fails, instead annotating the pipeline run with a warning. Change-Id: I0a0bc26deffe7b20b99f5aa7238fb3c3bb9deb92 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3721 Reviewed-by: sterni <sternenseemann@systemli.org> Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2021-10-11 14:24:02 +02:00
]
"pipeline"
[
"printf"
"%s"
"$report"
]
"buildkite-agent"
"annotate"
"--style"
"warning"
"--context"
"check-all-our-lock-files"
];
style: format entire depot with nixpkgs-fmt This CL can be used to compare the style of nixpkgs-fmt against other formatters (nixpkgs, alejandra). Change-Id: I87c6abff6bcb546b02ead15ad0405f81e01b6d9e Reviewed-on: https://cl.tvl.fyi/c/depot/+/4397 Tested-by: BuildkiteCI Reviewed-by: sterni <sternenseemann@systemli.org> Reviewed-by: lukegb <lukegb@tvl.fyi> Reviewed-by: wpcarro <wpcarro@gmail.com> Reviewed-by: Profpatsch <mail@profpatsch.de> Reviewed-by: kanepyork <rikingcoding@gmail.com> Reviewed-by: tazjin <tazjin@tvl.su> Reviewed-by: cynthia <cynthia@tvl.fyi> Reviewed-by: edef <edef@edef.eu> Reviewed-by: eta <tvl@eta.st> Reviewed-by: grfn <grfn@gws.fyi>
2022-01-30 17:06:58 +01:00
in
depot.nix.readTree.drvTargets {
feat(tools): add rust-crates-advisory We have a bunch of crates in `third_party/rust-crates`; it would be great if we could check them for existing CVEs. This tool does that, it takes the rust security advisory database, parses the applicable CVEs, and cross-checks them against the actual crate versions we list in our package database. The dumb parser we wrote is tested against all entries in the database, so we will notice when upstream breaks their shit. Checking the semver stuff is easy enough with the semver crate. If an advisory matches, it prints the whole thing and fails the build. Change-Id: I9e912c43d37a685d9d7a4424defc467a171ea3c4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2818 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-04 04:04:20 +02:00
inherit
feat(rust-crates-advisories): check 3p crates together w/ lock files Instead of the strict check-all-our-crates, generate a fake Cargo.lock and add it to the report generated by check-all-our-lock-files. check-all-our-crates was a reimplementation of cargo-audit anyways and prevented us from updating the advisory db due to its strict model (failing on any advisory). Change-Id: I264a7f1a5058a527cbc46d26225352ecd437a22b Reviewed-on: https://cl.tvl.fyi/c/depot/+/5230 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-02-04 19:54:53 +01:00
test-parsing-all-security-advisories
feat(tools): add rust-crates-advisory We have a bunch of crates in `third_party/rust-crates`; it would be great if we could check them for existing CVEs. This tool does that, it takes the rust security advisory database, parses the applicable CVEs, and cross-checks them against the actual crate versions we list in our package database. The dumb parser we wrote is tested against all entries in the database, so we will notice when upstream breaks their shit. Checking the semver stuff is easy enough with the semver crate. If an advisory matches, it prints the whole thing and fails the build. Change-Id: I9e912c43d37a685d9d7a4424defc467a171ea3c4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2818 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-04 04:04:20 +02:00
check-crate-advisory
refactor(rust-crates-advisory): move report generation into script This script is somewhat usable by humans (it even has a help screen!) and can be reused in //users/sterni/nixpkgs-crate-holes. We are using bash since that allows us to exit with the actual exit code of cargo-audit - something that's not possible in execline. Change-Id: I3331ae8222a20e23b8e30dc920ab48af78f0247c Reviewed-on: https://cl.tvl.fyi/c/depot/+/5228 Tested-by: BuildkiteCI Reviewed-by: Profpatsch <mail@profpatsch.de>
2022-02-04 17:28:09 +01:00
lock-file-report
feat(tools): add rust-crates-advisory We have a bunch of crates in `third_party/rust-crates`; it would be great if we could check them for existing CVEs. This tool does that, it takes the rust security advisory database, parses the applicable CVEs, and cross-checks them against the actual crate versions we list in our package database. The dumb parser we wrote is tested against all entries in the database, so we will notice when upstream breaks their shit. Checking the semver stuff is easy enough with the semver crate. If an advisory matches, it prints the whole thing and fails the build. Change-Id: I9e912c43d37a685d9d7a4424defc467a171ea3c4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2818 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-04 04:04:20 +02:00
;
feat(tools/rust-crates-advisory): also check all our Cargo.locks check-all-our-lock-files works very similarly to //users/sterni/nixpkgs-crate-holes, even reusing some parts of it, but is much simpler since we don't need to extract the lock files — they are already in tree. It is implemented as a very simple script which just traverses the subtree of the current directory, collecting all warnings. When executing this script in buildkite via extraSteps, it never fails, instead annotating the pipeline run with a warning. Change-Id: I0a0bc26deffe7b20b99f5aa7238fb3c3bb9deb92 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3721 Reviewed-by: sterni <sternenseemann@systemli.org> Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2021-10-11 14:24:02 +02:00
refactor(rust-crates-advisory): split code for buildkite and report Rename check-all-our-lock-files to tree-lock-file-report and pull out all the buildkite-specific code which makes the code less awkward. check-all-our-lock-files is then only executed in extraSteps and runs tree-lock-file-report on depot, adding it as a warning to the pipeline if it is non-empty. Change-Id: If6bd236d90cc680cba0ed4e988f2f28ddb8012d6 Reviewed-on: https://cl.tvl.fyi/c/depot/+/5229 Tested-by: BuildkiteCI Reviewed-by: Profpatsch <mail@profpatsch.de>
2022-02-04 19:46:48 +01:00
tree-lock-file-report = tree-lock-file-report // {
feat(tools/rust-crates-advisory): also check all our Cargo.locks check-all-our-lock-files works very similarly to //users/sterni/nixpkgs-crate-holes, even reusing some parts of it, but is much simpler since we don't need to extract the lock files — they are already in tree. It is implemented as a very simple script which just traverses the subtree of the current directory, collecting all warnings. When executing this script in buildkite via extraSteps, it never fails, instead annotating the pipeline run with a warning. Change-Id: I0a0bc26deffe7b20b99f5aa7238fb3c3bb9deb92 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3721 Reviewed-by: sterni <sternenseemann@systemli.org> Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2021-10-11 14:24:02 +02:00
meta.ci.extraSteps.run = {
feat(rust-crates-advisories): check 3p crates together w/ lock files Instead of the strict check-all-our-crates, generate a fake Cargo.lock and add it to the report generated by check-all-our-lock-files. check-all-our-crates was a reimplementation of cargo-audit anyways and prevented us from updating the advisory db due to its strict model (failing on any advisory). Change-Id: I264a7f1a5058a527cbc46d26225352ecd437a22b Reviewed-on: https://cl.tvl.fyi/c/depot/+/5230 Tested-by: BuildkiteCI Reviewed-by: tazjin <tazjin@tvl.su>
2022-02-04 19:54:53 +01:00
label = "Check all crates used in depot for advisories";
feat(tools/rust-crates-advisory): also check all our Cargo.locks check-all-our-lock-files works very similarly to //users/sterni/nixpkgs-crate-holes, even reusing some parts of it, but is much simpler since we don't need to extract the lock files — they are already in tree. It is implemented as a very simple script which just traverses the subtree of the current directory, collecting all warnings. When executing this script in buildkite via extraSteps, it never fails, instead annotating the pipeline run with a warning. Change-Id: I0a0bc26deffe7b20b99f5aa7238fb3c3bb9deb92 Reviewed-on: https://cl.tvl.fyi/c/depot/+/3721 Reviewed-by: sterni <sternenseemann@systemli.org> Autosubmit: sterni <sternenseemann@systemli.org> Tested-by: BuildkiteCI
2021-10-11 14:24:02 +02:00
alwaysRun = true;
command = check-all-our-lock-files;
};
};
feat(tools): add rust-crates-advisory We have a bunch of crates in `third_party/rust-crates`; it would be great if we could check them for existing CVEs. This tool does that, it takes the rust security advisory database, parses the applicable CVEs, and cross-checks them against the actual crate versions we list in our package database. The dumb parser we wrote is tested against all entries in the database, so we will notice when upstream breaks their shit. Checking the semver stuff is easy enough with the semver crate. If an advisory matches, it prints the whole thing and fails the build. Change-Id: I9e912c43d37a685d9d7a4424defc467a171ea3c4 Reviewed-on: https://cl.tvl.fyi/c/depot/+/2818 Tested-by: BuildkiteCI Reviewed-by: tazjin <mail@tazj.in> Reviewed-by: sterni <sternenseemann@systemli.org>
2021-04-04 04:04:20 +02:00
}
Reference in a new issue Copy permalink