How much more productive are developers who rely on AI code tools? Recently, there has been a lot of speculation that AI makes developers 2, 3, or even 5 times more productive. A report predicts multiply developer productivity tenfold by 2030.
The irony, however, is that the engineering community, for the most part, has not been able to agree on a universal way to measure productivity of engineering. Some have even rejected the idea outright, arguing that most metrics are flawed or imperfect. Most of the claims about AI productivity improvement today are qualitative, based on surveys and anecdotes, rather than quantitative data.
How can we make judgments about AI without first agreeing how to measure productivity? If we learned anything from the remote work experiment, it's that we stagger without data to inform our decisions, shifting between office, remote, and hybrid strategies based on dogmas and ideologies rather than data and measurement.
We are on our way to do it again with AI. To move forward, we must first understand and quantify its impact.
The risk of being left behind
The current hype around AI may give some of us a reason to pause, due to the unknown impact on quality, potential risk of plagiarism, and other factors. The more cautious companies have entered a holding pattern, waiting to see how this new scenario plays out.
However, for technology-enabled companies, the risk of being left behind is existential. AI is a double accelerator, affecting both the what and how companies build. Companies investing in AI today have the potential to double down on investment by bringing not only new AI-powered products to market, but products to market faster and cheaper.
Most companies have focused on the what but AI could be the engine of how, creating the engineering team with 10x or even 100x production. Companies that figure out how to quickly cross the chasm, by optimizing AI tools in the most efficient and impactful way, and reach the productivity plateau faster will gain a head start for years to come. The risk of doing nothing is too high.
Understand the tradeoffs
To someone with a hammer, everything looks like a nail. The same goes for AI.
According to a recent github report, the main benefit of the AI coding tools cited by the developers was to improve their coding language skills. Another key benefit is the automation of tasks, such as writing boilerplate code. A recent experiment by the company CodeCov showed that ChatGPT works well when writing simple tests for trivial functions and relatively simple code.
But like any technology, AI has its pros and cons. For example, generative AI and LLMs are not capable of highly complex or creative tasks, such as migrating a codebase from one architecture to another or incorporating business logic into a new feature. They could even open unforeseen security holes or legal questions if developers accept AI-generated suggestions without plagiarism checks, license restrictions, or AI-generated bypasses (so-called hallucinations).
But AI doesn't need to be a panacea to change the way software is written and put into production. Even with these limitations, when used in just a few precise ways to help with the right tasks, it can open new doors that make those tasks 10 times easier, faster, or cheaper.
Quantifying the impact
Most developers are already using AI-powered tools. According to GitHub, 92% of users in a survey have used AI coding tools at work or in their spare time. That makes it critical for companies to establish baselines as soon as possible to help compare productivity before and after implementing new AI tools.
Even simple proxy metrics can provide quantitative information about the impact of new tools. For example in an investigation in a community From over 400.000 developers, it appears that those using GitHub Copilot are inserting 1,3 times as many characters and 1,22 times as many lines of code in the same amount of time as developers not using the AI coding assistant. . While changing more lines of code doesn't necessarily equate to improved productivity, the increasing rate at which developers using GitHub Copilot can write code, whether it's unit tests, functions, or other boilerplate code, provides evidence that they're saving time and effort on repetitive tasks.
Similarly, measuring how many features a team can deliver before and after investing in AI tools can help quantify the impact. Improvements in the number of features delivered per engineer (along with a lower cost per feature delivered) make a stronger business case for continuing to invest in AI tools.
Positive returns on AI investments: Investing in AI tools can unlock the development of new features. Image Credits: Software.com
Comparing delivered features to remove or refactor can also help you understand the impact AI has on quality. Companies that experience quality issues as a result of implementing new AI coding tools will spend more time on code rotation by removing inefficient code and refactoring and less time developing new features.
While a certain amount of updating and refactoring is necessary, an increasing amount may indicate that the ROI of those tools is less compelling and significantly impacts quality.
AI impact on quality: Increased turnover and refactoring is a sign of AI quality issues. Image Credits: Software.com
Making the right investments
In order to answer whether AI improves developer productivity, one must first gain visibility into its impact: where, when, and how in the development process AI is likely to have the greatest return on investment. With so much hype and excitement surrounding generative AI, it's more important than ever to uncover the right data to get to the truth.
As companies invest in AI development tools, they can empower their teams by applying insight to software development. Observability not only helps you learn faster about the limits and opportunities with AI, but also opens the door to conversations about other factors that affect productivity, such as remote work, DevOps, and internal development platforms.
There can be little doubt that AI has a role in improving productivity, but it requires examining the facts to understand how, and then making informed decisions.
