There has been relatively extensive research about the factors that make developers more productive. Specifically, research indicates high satisfaction positively correlates with higher perceived productivity, and happy developers are better problem solvers with improved analytical abilities. Additionally, positive emotions enhance developers' flow state and ability to do good work, and satisfaction is significantly influenced by developers' ability to control their workday and minimize unplanned disruptions.
Despite much research on measuring developer productivity and Developer Experience (DevEx), little time has been spent on understanding the holistic factors contributing to a "bad day" for developers. A recent mixed-methods study conducted by Microsoft, Purdue University, and the University of Victoria delved into the factors contributing to "bad days" for developers. This study provides valuable insights for engineering leaders seeking to support their teams effectively. Additionally, the study addresses a gap in tying developer feedback with quantitative metrics, aiming to build a more comprehensive understanding of what contributes to bad days for developers.
How Microsoft Analyzed Bad Days
Microsoft’s study focused on identifying factors contributing to bad days for developers, employing interviews, surveys, daily diaries, and telemetry analysis. The researchers aimed to understand the developers' perception of bad days rather than providing a rigid definition. They also wanted to triangulate qualitative experiences with quantitative metrics to build a more comprehensive understanding of DevEx.
- Interviews: Twenty-two developers representing a range of seniority were interviewed. Developers shared their experiences of what constitutes a bad day, how it impacts their work, and potential coping strategies.
- Survey: 214 developers participated in a survey informed by the interview results, focusing on identifying common factors that lead to bad days and their frequency.
- Daily Diaries: 79 developers logged their experiences over 30 days, noting factors contributing to bad days in real-time.
- Telemetry Data: Data from 131 developers were analyzed to validate the factors causing bad days. The study compared PR and build metrics between those who reported bad days and those who did not.
The Major Contributing Factors to Bad Days
DevEx is complex, and technical infrastructure isn’t the only thing contributing to bad days. Three key themes emerged as the primary contributors to bad days for Microsoft developers:
- Tooling and Infrastructure: Unreliable tools were the most consistent cause of bad days. Developers cited issues with flaky tests, slow builds, outdated user interfaces, and broken tools that hindered their workflow.
- Process Inefficiencies: Challenges such as unclear project ownership, lack of documentation, and rapidly changing priorities contributed significantly to bad days, particularly for senior developers.
- Team Dynamics and Communication: Collaboration difficulties, communication breakdowns, and interpersonal conflicts were frequently reported, with junior developers significantly impacted by the dynamics of the teams they worked closely with.
The survey revealed that pull request (PR) delays outside of the team’s control were the top factor contributing to bad days, with other subjective factors like team dynamics and lack of support also playing significant roles. The daily diaries echoed these findings, with tooling and infrastructure issues being the most common cause of bad days.
How Seniority Impacts A Bad Day Experience
The study found notable differences between how senior and junior developers described the impact of bad days. Bad days caused senior developers to experience frustration, annoyance, and anger that would turn into disillusionment and cynicism if they experienced frequent bad days. Some senior developers indicate they often start looking for a new job as a direct result of this frustration. The study suggests more opportunities for co-creation activities aligned with improving organizational processes as a solution to reduce bad days for senior developers.
On the other hand, junior developers frequently internalized the challenges, attributing bad days to their incompetence. This led to self-doubt and guilt and sometimes contributed to imposter syndrome. The study suggested mentoring programs as an effective way to increase confidence, knowledge, and productivity.
Developers Accurately Perceive (Some) PR and Build Metrics
The study combined the qualitative data from the survey with quantitative telemetry metrics to analyze the connection between developer perception and standardized efficiency metrics. The researchers tracked Pull Request (PR) and build time metrics because they are easy to gather systems data that can be compared across individuals.
Developers who reported PRs as a cause of bad days experienced a 23.8% longer total PR cycle time and a 48.8% increase in pickup time compared to those who did not report PRs as a bad day factor. PR review time was not correlated with bad days, which is a bit surprising given that we know that pickup and review times are consistently the biggest blockers in the typical software delivery lifecycle. They also failed to find a correlation between the number of reviewers and bad days.
Note: LinearB uses terminology that's slightly different from the paper's authors.
Developers who identified build times as a contributor to bad days had a 26.3% longer total build time and a 28% longer core build time. However, the research did not see a significant impact from non-core build times or the build success rate.
This data indicates to some extent that developers can sense when code reviews or build pipelines are significant sources of frustration, but it would be nice to see more details at the team level, mainly related to PR review time. Considering that their junior developers cited issues that varied from team to team, it would be interesting to see if a more granular analysis of PR metrics would lead to further discoveries.
Quantifying the Impact of DevEx on Productivity
This research paper successfully used qualitative insights to identify the factors contributing to developers having bad days. More importantly, it ties those factors to quantifiable metrics that track productivity. The correlation between qualitative insights and build and PR metrics is a great starting point, but it would be interesting to dive deeper into the quantitative side of this research. Specifically, tracking long-term investments into DevEx would be interesting to see if these investments reduce the rate or the severity of bad day reports.
It would also be interesting to analyze the impact of planning and capacity accuracy on the frequency of bad days. Specifically, how much does unplanned work, work-in-progress overload, and burnout risk result in developers experiencing bad days? There are many ways to quantify various aspects of a developer’s day, and many of them could be augmented by direct qualitative feedback.
