Product Risk Analysis (PRA): how to decide what you should test?

What is a Product Risk Analysis?

A Product Risk Analysis, usually shortened to PRA, helps you determine where a failure could have the greatest consequences. Those areas receive more attention. Where the risk is lower, a lighter test approach may be enough.

In the current TMAP terminology, this is called a Quality Risk Analysis. This term emphasises that you are looking at quality risks that could prevent a system from delivering the expected value.

Product Risk Analysis is the older term, but it is still widely used. Whether you come across PRA or QRA, the basic question is the same:

Where could things go seriously wrong, and how much confidence do we need?

Why can’t you test everything?

Because you do not have unlimited time, people, or budget. And because not every defect is equally serious.

A typo on an information page is annoying. A customer being charged twice is in a different league. Yet without a risk analysis, both may simply appear as two items on the same test list.

That is exactly where things can go wrong. Teams spend a great deal of time on areas that are easy to test, while difficult and business-critical parts receive less attention as the deadline approaches.

A PRA forces you to choose. Not based on gut feeling, but based on risk.

How do you determine Product Risk?

The simple version is:

Product risk = likelihood of failure × impact

The likelihood of failure tells you how likely it is that a part of the system will not work properly in practice. The impact describes what happens if it does fail.

Take an online shop. A poorly aligned product image deserves attention, but it is unlikely to bring the entire shop to a halt. A problem in the payment process could lead to lost revenue, duplicate charges, and a service desk having a very unpleasant Monday.

The payment process therefore receives a higher risk level. Not because it is guaranteed to fail, but because you would rather not take your chances with it.

What increases the likelihood of failure?

New technology, complicated logic, and numerous dependencies all make defects more likely. The same applies to parts of the system that have been changed repeatedly in a short period.

But technology is not the whole story. Frequency of use matters too.

A defect in an administration screen that is opened once a year is less likely to surface than a defect in a button used by thousands of people every day. The defect may exist in both cases. The likelihood of users actually encountering it is very different.

Consider factors such as:

• the complexity of the component;
• new or unfamiliar technology;
• recent changes;
• integrations with other systems;
• the team’s experience;
• how often the feature is used.

There is no need to turn this into advanced mathematics. The aim is to have an honest conversation about where things are most likely to go wrong.

What does “impact” mean in a Risk Analysis?

Impact is not only about money.

A failure can also cause reputational damage, privacy breaches, disruption to a business process, or customers losing trust. Sometimes the impact means that colleagues spend several days repairing data manually. That is not exactly free either.

What counts as serious differs between organisations. A reporting tool being unavailable for an hour may be inconvenient. A medical system being unavailable for an hour is a very different matter.

That is why a standard checklist will only take you so far. You need to understand how the system is used in the real world and who will be affected if something goes wrong.

You do not perform a Product Risk Analysis alone

You could ask the tester to determine all the risks. That happens quite often. But the tester does not automatically know what an hour of downtime costs, which customer processes are sacred, or which technical integration is being held together with sticky tape. 

You need several perspectives.

The product owner understands the business objectives. The developer sees the technical complexity. The tester considers defect likelihood and test coverage. Operations knows what happens in production. Users and stakeholders can explain what a failure means in practice.

Bring that knowledge together, and you get a far more realistic picture than when everyone creates a separate risk list.

And yes, this may lead to some discussion. That is not a problem. The discussion is often exactly why you perform the analysis.

Priority is not the same as risk

A high-priority user story does not automatically have a high product risk. Priority usually says something about business value or planning: when do we want this? Risk asks a different question: what happens if it does not work properly?

A feature can be important to the business while being technically straightforward. On the other hand, a rarely used administration feature could cause serious damage if someone accidentally deletes all customer data.

In short: Priority determines when you build something. Risk determines how much confidence you need.

The two may point in the same direction, but they do not have to.

How do you perform a Product Risk Analysis?

There is no mandatory PRA ceremony involving a sacred spreadsheet. You can organise a workshop, conduct interviews, or use Risk Poker. Choose an approach that gets people talking to each other.

A practical approach consists of five steps.

1. Decide what you are assessing

Choose specific business processes, user stories, system components, or quality characteristics. “We are going to analyse the entire platform” sounds ambitious, but usually produces an afternoon full of general statements.

2. Involve the right people

Make sure both business and IT are represented. You need people who can estimate the impact and people who understand the technical likelihood of failure.

3. Describe what could go wrong

“The payment process is risky” is too vague. “The customer pays, but the order is not registered” is useful. Now everyone knows which problem is being discussed.

4. Assess likelihood and impact

Use numerical scores or simple labels such as low, medium, and high. The exact scale matters less than everyone having roughly the same understanding of what the scores mean.

The result does not need to be accurate to three decimal places. A risk score is not a law of physics. The conversation behind it is more important.

5. Choose suitable quality measures

Decide what is needed to reduce the risk or make it visible. This could be a review, a proof of concept, an automated test, a specific test design technique, or additional monitoring.

This is the point where the risk analysis becomes genuinely useful.

From product risk to test strategy

A high risk does not automatically mean writing more test cases.

Sometimes the real problem is an unclear requirement. In that case, it is better to sit down together first than to test twenty variations afterwards. An unfamiliar technical solution may call for a proof of concept. A critical process may require a combination of test automation, exploratory testing, and monitoring.

The risk analysis therefore guides your entire quality approach. What will you test? How thoroughly will you test it? And which measures could prevent defects earlier?

Also record which measure addresses which risk. Otherwise, you may end up with a beautifully coloured risk matrix that no one knows what to do with two sprints later.

When should you update the risk analysis?

A PRA is not a document you create once and then file away with great ceremony.

New integrations, amended legislation, and unexpected user behaviour can all change the risk profile. The same applies to test results and production incidents.

Revisit the risks when the scope, technology, or usage situation changes. This does not require another half-day workshop. Sometimes ten minutes during refinement is enough.

Risk-Based Testing with TMAP

A Product Risk Analysis helps you spend limited test time deliberately. Not everything receives the same level of attention, because not every failure has the same consequences.

In the TMAP: Quality for Cross-Functional Teams e-learning course, you will learn how to identify quality risks together and translate them into an appropriate test strategy. You will discover how to include quality from the start, rather than simply hoping for the best just before release.

You cannot test everything. But you can make sure you test the things that matter most.