Google expands Android Bench with new evaluation framework and eight additional AI models

Google has updated Android Bench, its benchmark for evaluating large language models (LLMs) on Android development tasks, introducing a new evaluation methodology, broader model coverage, and additional transparency features as AI-assisted software development continues to evolve. The changes are part of the benchmark’s July release and are intended to provide a more accurate assessment of how AI models perform on real-world Android coding tasks.

The most significant update is the adoption of the Harbor framework, replacing the benchmark’s previous evaluation approach based on the mini-swe-agent v1. Harbor provides a standardized benchmarking framework that enables developers, researchers, and model providers to reproduce evaluations, compare different configurations, and share results more consistently. According to Google, the transition improves the rigor and transparency of Android Bench while aligning it with newer industry standards for AI evaluation.

Because of the methodological change, Google re-ran every model currently included in Android Bench to establish a new baseline. While this results in modest score changes across the leaderboard, historical benchmark results remain available through an archive, allowing developers to compare performance over time.

The latest release also expands Android Bench’s leaderboard with eight additional large language models, broadening the range of proprietary and open-weight models available for comparison. Google said the expansion reflects growing demand from developers seeking to evaluate different AI coding assistants for Android-specific workflows rather than relying solely on general-purpose programming benchmarks.

Android Bench was introduced earlier this year to measure how effectively AI models solve practical Android development challenges, including navigating large codebases, understanding project dependencies, fixing bugs, and implementing platform-specific features. Unlike general coding benchmarks, the evaluation focuses exclusively on Android software engineering tasks using real-world scenarios derived from Android projects.

In recent months, Google has also expanded the benchmark beyond raw capability scores by introducing cost and efficiency metrics, allowing developers to compare not only model quality but also the resources required to achieve those results. Open-weight models have likewise been added to the leaderboard, giving developers greater visibility into locally deployable alternatives alongside commercial AI services.

Google said Android Bench will continue evolving as AI models and evaluation techniques advance. Future updates are expected to further refine testing methodologies and expand model coverage, with the goal of providing Android developers and AI providers with a transparent benchmark for measuring real-world coding performance across the Android ecosystem.