tvl-depot/tvix/scripts/bench.sh

#!/usr/bin/env nix-shell
#!nix-shell -i bash ../.. -A tvix.shell

# Benchmark script that runs inside the Windtunnel build agent

set -euo pipefail

echo "Running benchmarks for tvix/eval..."
cd "$(dirname "$(dirname "$0")")/eval"
cargo bench
windtunnel-cli report -f criterion-rust .

echo "Running tvix macrobenchmarks..."
cargo build --release --bin tvix
hyperfine --export-json ./results.json \
    -n 'eval-nixpkgs-hello' "target/release/tvix -E '(import ../../nixpkgs {}).hello.outPath'" \
    -n 'eval-nixpkgs-cross-hello' "target/release/tvix -E '(import ../../nixpkgs {}).pkgsCross.aarch64-multiplatform.hello.outPath'"
windtunnel-cli report -f hyperfine-json ./results.json
feat(tvix): Benchmark nixpkgs eval with hyperfine Add some hyperfine benchmarks to Tvix's windtunnel benchmark script for evaluating the outPath of hello and a cross-compiled hello. Change-Id: I9d76e5ce0a3fd7d9c125c36c5fced675b660a8a8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10248 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI Autosubmit: grfn <grfn@gws.fyi> 2023-12-11 04:51:43 +01:00			`#!/usr/bin/env nix-shell`
			`#!nix-shell -i bash ../.. -A tvix.shell`
feat(tvix): Add script for running benchmarks in Windtunnel Currently this just uses a Docker container, which is gross but works fine for now since we don't have the ability to build benchmarks in nix as of cl/7538 Change-Id: I48e317f44bc2c73533d7663403786a3a37c7952f Reviewed-on: https://cl.tvl.fyi/c/depot/+/10189 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi> Autosubmit: grfn <grfn@gws.fyi> 2023-12-05 17:58:20 +01:00
			`# Benchmark script that runs inside the Windtunnel build agent`

			`set -euo pipefail`

			`echo "Running benchmarks for tvix/eval..."`
refactor(tvix): Make benchmark script directory-agnostic Allow running the benchmark script from any directory - primarily so Windtunnel can point to the /tvix josh workspace rather than the depot overall Change-Id: Ie5fc3ef995bf8114277298ae5c5010e6a0bf13ac Reviewed-on: https://cl.tvl.fyi/c/depot/+/10205 Reviewed-by: grfn <grfn@gws.fyi> Autosubmit: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI 2023-12-06 17:14:16 +01:00			`cd "$(dirname "$(dirname "$0")")/eval"`
feat(tvix): Benchmark nixpkgs eval with hyperfine Add some hyperfine benchmarks to Tvix's windtunnel benchmark script for evaluating the outPath of hello and a cross-compiled hello. Change-Id: I9d76e5ce0a3fd7d9c125c36c5fced675b660a8a8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10248 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI Autosubmit: grfn <grfn@gws.fyi> 2023-12-11 04:51:43 +01:00			`cargo bench`
feat(tvix): Add script for running benchmarks in Windtunnel Currently this just uses a Docker container, which is gross but works fine for now since we don't have the ability to build benchmarks in nix as of cl/7538 Change-Id: I48e317f44bc2c73533d7663403786a3a37c7952f Reviewed-on: https://cl.tvl.fyi/c/depot/+/10189 Reviewed-by: flokli <flokli@flokli.de> Tested-by: BuildkiteCI Reviewed-by: grfn <grfn@gws.fyi> Autosubmit: grfn <grfn@gws.fyi> 2023-12-05 17:58:20 +01:00			`windtunnel-cli report -f criterion-rust .`
feat(tvix): Benchmark nixpkgs eval with hyperfine Add some hyperfine benchmarks to Tvix's windtunnel benchmark script for evaluating the outPath of hello and a cross-compiled hello. Change-Id: I9d76e5ce0a3fd7d9c125c36c5fced675b660a8a8 Reviewed-on: https://cl.tvl.fyi/c/depot/+/10248 Reviewed-by: grfn <grfn@gws.fyi> Tested-by: BuildkiteCI Autosubmit: grfn <grfn@gws.fyi> 2023-12-11 04:51:43 +01:00
			`echo "Running tvix macrobenchmarks..."`
			`cargo build --release --bin tvix`
			`hyperfine --export-json ./results.json \`
			`-n 'eval-nixpkgs-hello' "target/release/tvix -E '(import ../../nixpkgs {}).hello.outPath'" \`
			`-n 'eval-nixpkgs-cross-hello' "target/release/tvix -E '(import ../../nixpkgs {}).pkgsCross.aarch64-multiplatform.hello.outPath'"`
			`windtunnel-cli report -f hyperfine-json ./results.json`