Measuring and improving productivity is a constant challenge. New research sheds light on Meta’s approach to quantifying developer efficiency through what they internally call "Diff Authoring Time," but what is more commonly known as coding time: the time developers spend creating code changes.
What is Coding Time?
Coding time is a conceptually simple but powerful metric measuring how long engineers develop changes. Meta captures a developer's active time on a particular code change using telemetry systems integrated with version control, IDEs, and their developers’ operating systems.
Unlike traditional productivity metrics such as lines of code or change count, coding time focuses on the effort in authoring code rather than just the output. This metric provides an accurate representation of development complexity and efficiency.
How Meta Measures Coding Time
Meta's implementation of coding time tracking is remarkably sophisticated:
- Precise Matching: The core algorithm maps IDE activity to specific commits in version control, attributing time spent in coding tools to the appropriate diff.
- Anchor Sessions: Additional heuristics capture related activities leading to a commit, such as terminal work that precedes coding sessions.
- Thorough Validation: Meta validated their coding time metrics through observational studies, large-scale surveys, visualizations, and descriptive statistics, achieving over 90% accuracy compared to ground truth measurements.
Three Real-World Productivity Improvements
Meta has used coding time measurements in more than 20 projects, with three notable case studies showing significant productivity enhancements:
1. Typed Mocking in Hack (14% Improvement)
Meta conducted a controlled experiment testing the impact of typed mocking in Hack (one of their server-side languages). By introducing static type checking for mock objects, they reduced coding time by 14%. This improvement stemmed from catching type errors earlier in the development process, eliminating the delay of discovering issues only after code had been committed to CI.
This experiment represents one of the first large-scale, real-world experiments demonstrating the productivity benefits of static typing in industry.
2. Auto-Memoization in React (33% Improvement)
Meta's React team developed a compiler feature for automatic memoization of components, eliminating the need for developers to implement memoization manually, a time-consuming and error-prone process.
Through a mixed-effect regression model that accounted for variables like lines of code, developer tenure, organization, and time, they measured a remarkable 33% reduction in coding time for diffs using auto-memoization compared to manual approaches.
3. Cross-Platform Code Sharing (Over 50% Improvement)
Perhaps most impressive was Meta's analysis of code-sharing frameworks, which allow developers to write code once and deploy it across multiple platforms and applications. By comparing actual coding time against an estimated counterfactual (what it would have taken without code sharing), they found:
- Over 50% improvement in coding time
- Thousands of hours saved annually
- Precise quantification of previously anecdotal efficiency claims
Why Coding Time Matters
The adoption of coding time as a metric has enabled Meta to:
- Measure Previously Unmeasurable Improvements: Quantify the impact of language features, tooling improvements, and framework efficiencies that were previously only supported by anecdotal evidence.
- Build a Data-Driven Engineering Culture: Move internal development toward an experiment-driven approach similar to what's used in consumer-facing product development.
- Make Better Investment Decisions: Balance the initial development cost of new features against the long-term productivity savings they provide.
Looking Forward
Meta continues to evolve their coding time measurement approach, expanding coverage to include more tools and IDE environments. As AI advances in software development, coding time remains a relevant metric that can help optimize new AI-assisted development workflows.
By focusing on accurate, objective measurement of development time, Meta has created a robust framework for making data-driven decisions about engineering productivity investments. This approach has proven remarkably effective at quantifying the value of previously difficult-to-measure improvements.
