Devconf.cz 2023

Our team is responsible for detecting performance regressions in QEMU. Each build takes about 8-12 hours to complete, so we cannot afford checking every little change. Instead, we rely on weekly checks. When a performance regression is found we have to analyze what caused it. But within a week's time, lots of things can change besides the code under test; for example, multiple packages in a distro might be updated (multiple times). Identifying the change that caused the regression requires maintaining a detailed checklist and is both time-consuming and error-prone.

This talk describes how we inspired by "git bisect" to build a simple multi-axis arbitrary bisection tool https://github.com/ldoktor/bisecter and how to use it in (not only) Jenkins pipelines, which has greatly relieved the developer burden and has increased the speed and reliability of identifying the event responsible for the regression.