GitGoodBench: A Novel Benchmark for Evaluating Agentic Performance on Git

Published in REALM Workshop at ACL 2025 (Spotlight), 2025

GitGoodBench evaluates how well code agents perform version control tasks. It covers three core Git scenarios sourced from real open-source repositories, with datasets of 900, 120, and 17,469 samples respectively. In the paper’s evaluation, even frontier models reached only a 21% overall solve rate, showing how much room remained for improvement on practical Git workflows.

Recommended citation: Lindenbauer, T., Bogomolov, E., & Zharov, Y. (2025). "GitGoodBench: A Novel Benchmark for Evaluating Agentic Performance on Git." Proceedings of the 1st Workshop for Research on Agent Language Models (REALM) at ACL 2025.
Download Paper