Companies Without Productivity Measurement Tools Rely on Their CTO's Experience and Intuition

As engineering teams grow, relying on intuition and experience to assess productivity becomes unsustainable. Traditional metrics like story points or commit counts are flawed and easily manipulated. Our research at Stanford suggests a more effective approach: analyzing code downstream to uncover team dynamics and bottlenecks. While manual code reviews can yield accurate insights, they’re slow and costly. That’s why we developed a groundbreaking tool that evaluates code with precision, speed, and cost-efficiency—providing data-driven support to enhance developer productivity at scale.

Yegor Denisov-Blanch

Content

3 mins

Companies without developer productivity tools rely heavily on their tech leaderships’ experience and intuition. This may work for small teams, but falls apart as teams grow. Without data-driven productivity measures, evaluations become inconsistent, and biases can creep in.

Experienced tech leaders rely on check-ins and discussions to assess productivity, identifying bottlenecks or issues as they emerge. However, staying 'close to the work' becomes increasingly hard as teams grow.

Metrics like story points or counting commits are a popular, but have significant problems that make them ineffective. A major problem is that they can be easily gamed--and our research shows they often are.

Measuring productivity in software engineering is hard. Our research suggests that aspects reflecting the broader software team’s performance ultimately show up in the code. For instance, unclear requirements from a business analyst slows the team’s progress. This is visible in the code.

Rather than measuring numerous metrics upstream (which have confounding variables and are an implementation nightmare), a more pragmatic approach is to measure downstream by analyzing the code being shipped.

This can be done through manual code reviews. Our research shows that backward-estimating coding and implementation time is highly accurate. However, it's slow (5-15 minutes per commit), not scalable (requires a panel of independent reviewers to avoid bias), and costly—each review could run $100+ per commit.

Imagine a tool that could evaluate code as accurately as a panel of independent reviewers, 1,000 times faster, and at a fraction of the cost. After over two years of research at Stanford, we built exactly that.

Productivity is about supporting developers, not just measuring them. As teams scale, data-backed tools paired with seasoned judgment create the best foundation for sustainable, productive environments.

If your company is grappling with how to balance intuition and metrics in assessing developer productivity, we’re exploring just that at Stanford!