38ec27e834
The number of jobs in the depot pipeline is reaching the limits of the Buildkite backend's ability for a single pipeline upload. Based on a conversation with their support my understanding is that this has to do with internal locking mechanisms at Buildkite. To work around this, we can instead chunk the pipeline into several smaller chunks that are uploaded serially. This commit introduces logic to chunk the pipeline accordingly. The chunk size chosen is 256 for now (a multiple of our number of agents, which is useful if we can get builds from the first chunk to start before the next ones are uploaded). Note that this chunk size is significantly below even the current number of targets (~460 as of this commit), but choosing a lower chunk size might alleviate problems we've been seeing with timeouts during pipeline uploads. Change-Id: I77030aaf8b874c330218b78c77d15216e13b9af7 Reviewed-on: https://cl.tvl.fyi/c/depot/+/4332 Tested-by: BuildkiteCI Reviewed-by: wpcarro <wpcarro@gmail.com> Autosubmit: tazjin <mail@tazj.in> |
||
|---|---|---|
| .. | ||
| besadii | ||
| deploy-whitby | ||
| dns | ||
| gerrit-tvl | ||
| journaldriver | ||
| kontemplate | ||
| machines | ||
| modules | ||
| mq_cli | ||
| pipelines | ||
| posix_mq.rs | ||
| secrets | ||
| users | ||
| nixos.nix | ||