How to Measure Quality

In order to measure quality, first we must determine why we want to measure quality. What are we getting out of it? What decisions are we making as a result?

Some typical decisions measures of quality help answer:

  • Do we need to spend time addressing tech debt?
  • Are there quality layers or tests we are missing?
  • How confident are we really?

Product quality is made of three components:

  • Code Quality, which involves things such as:
    • Maintainability
    • Extensibility
  • Functionality Quality, which answers the question of if the code actually does what it is required to do (beautifully written code can still not perform the job it needs to)
  • Fit for Use Quality, which answers if the customers use it as expected as even if code is written perfectly and performs entirely as expected, it still might not be as the user needs

The support bucket from the Swiss Cheese Model catches a lot which should go into quality metrics, since we define an overflowing bucket as an indicator of when to revisit the strategy. The bucket itself typically refers to:

  • amount of unplanned work compared to planned work (some is expected, but is it taking too much time?)
  • amount of bugs
  • how long it takes to fix bugs
  • customer complaints
  • changes in time it takes to develop (eg, is development time slowing down due to dealing with low quality code)

So with these definitions and expectations, let’s consider each component in quality and ways we can measure.

Code Quality

This part of quality is all about the code. Martin Fowler has a really good post about code quality and why it is useful. In summary, code quality has an upfront investment and results in quicker delivery within just a few weeks.

Thankfully these days there are several tools already created and ready for use for several aspects. Some of the more common ones follow in the metrics section.

Metrics around code quality

Static Analyzers (eg, Sonar Cloud)

  • Easy to implement
  • Can give many false positives
  • Useful to enforce agreed upon standards
  • Useful for catching some security concerns
  • Useful when working with juniors or others unfamiliar with good conventions and practices within a particular language/framework (especially eg, Sonar Cloud with its detailed explanations)

Development Time

  • If development time is increasing for all changes, regardless of complexity, then this might be an indication of lack of quality.
  • When you can confirm the time increases are due to things like code churn, needing to rewrite large portions of code to enable adding the new functionality, etc, then this points to a code quality problem which should be addressed

DORA Metrics

  • While I’ve called out development time on its own, the DORA metrics in general are also a good measure about quality. They focus on a team’s ability to deliver and code quality is directly correlated to this.

Functionality Quality

This category focuses on if the project behaves as expected, even under unexpected conditions. This is the area traditional QA focused on and is also what we write tests around. That said, measuring the tests does not actually measure functional quality.

Metrics for Functionality Quality

Code coverage

  • Easy to implement
  • Not actually a measure of quality, but can point to areas which need more work test-wise to protect the functionality
  • Most useful in legacy contexts where testing was minimal to help guide when writing additional tests is needed before refactoring or touching a new area

Number and severity of defects

  • If a lot of defects are cropping up, sign the strategy needs to be re-addressed

Fit for Use Quality

This section is very product focused, and my general recommendation is to work with product to identify what metrics are most useful and make them requirements within the project.

The questions we seek to answer here revolve around if the project is useful to the customers (in at least the ways we expect) and if the customers are actually using it. We might discover that even if the code executes exactly as expected, the solution just does not work in a way the users are willing to replace their existing processes for.

Metrics for Functionality Quality

Usage statistics

  • Trying to answer if customers are using it the way we expect them to
  • If there’s a separate product, can work with them to ensure they have this information

Conclusion

Identify the questions you want to answer and from there identify useful metrics. Even if the first metrics are not perfect, getting in the habit of gathering measurements and figuring out what is and is not useful will be valuable in the long run.

Several of these are metrics many teams already gather for other reasons, but only the defect number is frequently used in a quality context. This post shows how a variety of metrics can also be used to think about and guide quality decisions.