Spoiler alert: according to new research from Microsoft, the answer to this article’s title appears to be yes.

Generative AI (Gen AI) tools like GitHub Copilot continue to gain momentum, and engineering leaders should understand how these tools impact developers’ beliefs and work habits. Microsoft partnered with the Institute of Work Life to study GitHub Copilot adoption at a large enterprise at a large enterprise They used a series of surveys and three weeks of daily dev diaries to explore how exposure to Gen AI tools influenced developers' perceptions, productivity, and attitudes toward their work. 

The findings indicate a significant improvement in developer perception over the three weeks and offer valuable insights to illustrate why developers find value in Gen AI tools.

Research Context

The research was conducted at a large, multinational software company that maintains one of the world’s largest codebases. It has more than 1 billion lines of code and spans decades, with some parts as old as 35 years. It supports a wide range of platforms and services and requires a high engineering proficiency to manage its complexity. In the summer of 2022, the company introduced Copilot, though it initially experienced very low adoption rates. This low awareness created an ideal setting for studying perception changes toward Copilot, as the organization was primed for adoption but had limited familiarity with the tool.

The researchers designed the experiment to explore four questions:

  • What were developers’ initial perceptions of Gen AI tools?
  • How did sustained use of Gen AI tools influence their beliefs?
  • What specific use cases and challenges emerged?
  • How did Gen AI affect developers' daily work and sense of purpose?

The researchers conducted pre- and post-trial surveys and a three-week period in which developers created daily diaries. They also measured a handful of coding activity metrics. They sought to capture the quantitative and qualitative impact of Copilot usage on developers’ day-to-day work and self-perception.

Initial Developer Sentiment Towards Gen AI

Initially, the study found that most developers were aware of Copilot but were skeptical about its effectiveness. 45% of the respondents had heard of Copilot but never used it, while 16% were consistent users. Developers most commonly cited a lack of time as the primary reason they hadn’t adopted Copilot (17%). They also mentioned skepticism about how well it works (12%), technical issues using it (10%), and a general lack of necessity to use it (10%).

However, the survey participants with prior Gen AI experience were significantly more likely to believe the tools were helpful in their job. 86% of respondents with prior experience said Gen AI tools are helpful versus only 44% without Gen AI experience. A similar trend appeared when developers were asked if they liked Gen AI tools.

Additionally, most developers—regardless of experience—were cautious about trusting AI-generated code, with only about 20% expressing confidence in its accuracy. This finding underscores the prevalent hesitation to depend on AI for critical tasks without human oversight, an attitude that persisted throughout the trial.

Gen AI Usage Results in Developer Perception Improvements

The study's high-level summary is that developers like Gen AI more after using it and are curious to find new ways to leverage it, but there is still a significant amount of distrust in the output of Gen AI tools.

Increased utility and enjoyment: Developers' ratings for statements like "I like using AI coding tools" rose significantly. Many participants found that these tools accelerated routine coding tasks and made their workdays more enjoyable. Specifically, developers appreciated the tool’s support for boilerplate code, documentation, and even code comments, with some expressing surprise at how accurately it could generate structured documentation.

No significant change in trust: There was a slight improvement in the perception that Gen AI tools generate correct code, and developers grew to appreciate Gen AI’s utility during the trial, even when it was wrong. However, their trust in the accuracy of AI-generated code remained unchanged. This underscores a vital insight for engineering leaders: while AI can save time on repetitive tasks, critical review, and validation are essential to avoid potential errors or security vulnerabilities.

Greater openness to future use: Some developers recognized AI’s role in shaping the future of coding, with comments highlighting an increased curiosity about how these tools might continue evolving. This positive outlook is particularly relevant as engineering leaders consider long-term AI adoption strategies.

Real-World Use Cases

Two major use cases emerged from the daily diaries during the trial period. The first is that Gen AI quickly replaces web searches. Developers increasingly used Copilot instead of search engines for coding queries, especially for learning new languages, resolving debugging issues, or identifying specific code patterns. This unexpected shift indicates a broader role for Gen AI in knowledge retrieval and learning within the IDE, bypassing traditional, fragmented web searches.

The second is that there was a major perceived productivity boost while working on repetitive tasks. Gen AI proved most useful for tasks like code documentation, test creation, and even code comments, where developers reported significant time savings. This aligns with the notion that AI can handle routine tasks, freeing developers to focus on higher-value work.

Gen AI Adoption Challenges

The study also shed light on some challenges developers face when first adopting Gen AI tools. First, developers expressed many concerns about code accuracy. A common challenge was code that looked correct but contained subtle errors. This issue made some developers question the productivity gains of AI-generated code, as they needed to spend time validating results. For leaders, this highlights the need for robust testing frameworks and critical thinking when incorporating AI tools.

Gen AI also had varied success when being applied across languages and contexts. While Copilot worked well for widely used languages like Python and JavaScript, it struggled with less common languages and configuration files. This limitation is important for leaders to consider when planning deployment and training, especially in diverse technical environments.

Lack of Quantitative Impact and Suggestions for Future Studies

Despite significant qualitative shifts in developers' perceptions, the study did not find measurable improvements in quantitative metrics, such as lines of code written, pull requests submitted, or coding hours. This lack of quantitative impact suggests two things.

The first is that the study duration might need to be increased to measure quantitative impact. Three weeks may not provide enough time for developers to acclimate to AI tools and realize potential productivity gains. Research on similar AI tools suggests that 11 weeks may be the tipping point for noticeable improvements, so extending the trial period might offer a more accurate reflection of the long-term effects of Gen AI on productivity.

The second, perhaps more critical, improvement is that there is room to incorporate additional metrics that holistically measure developer productivity. The study focused on metrics like lines of code change, total PRs, cycle time, and build time, which aren’t typically enough to quantify developer productivity. Additional metrics like planning accuracy, investment profile, and change failure rate would provide a more detailed analysis of Gen AI's impact on productivity.

Balancing AI’s Benefits and Challenges

The study highlights the potential of Gen AI tools to transform coding workflows by enhancing productivity and making work more engaging. However, while developers quickly recognized the utility of tools like Copilot, trust in AI-generated code remains limited, emphasizing the need for vigilance and continuous evaluation.

For engineering leaders, this research provides actionable insights into the role of AI in the developer ecosystem. To maximize the benefits of AI tools, companies should:

  1. Foster awareness and training: Address skepticism through training programs highlighting AI’s benefits and limitations, helping developers understand when and how to rely on AI.
  2. Emphasize critical evaluation: Encourage developers to scrutinize AI-generated code rigorously to prevent subtle errors, particularly in complex projects.
  3. Consider the long-term impact on developer roles: As AI becomes an integral part of development, leaders should facilitate discussions around career development and skills that will complement AI capabilities.

By approaching AI integration thoughtfully, companies can empower developers to use these tools effectively, creating a productive, engaged, and future-ready workforce.