Efficiency testing with TMAP: how efficiently does software use resources?

What is efficiency in TMAP?

Efficiency describes the relationship between a system’s performance and the resources required to achieve it. Performance can be expressed in terms of speed and the number of transactions processed. Resources include CPU capacity, memory, database I/O, storage, and network traffic.

Efficiency does not mean that software should use as few resources as possible. An application that uses very little capacity but keeps users waiting is not efficient. It is about finding the right balance between speed, transaction volume, and resource utilization.

The central question is therefore:

How many resources does the system need to achieve the agreed level of performance?

What is the difference between performance and efficiency?

Performance and efficiency are easily confused, but they reveal different aspects of software quality.

A performance test might check whether an application responds within two seconds and can process one thousand transactions per minute. When testing efficiency, you also measure how much CPU, memory, database capacity, and network traffic are needed to achieve that result.

Imagine two systems that can both process one thousand transactions per minute. The first system uses 80 percent of the available CPU capacity, while the second uses 40 percent. Both systems deliver the same performance from the user’s perspective, but the second uses the available capacity more efficiently.

A performance test is therefore not complete if you only look at response times and transaction volumes. To assess efficiency, you must relate those results to the measured resource utilization.

Examples of inefficient resource tilization

Inefficient software is not always immediately noticeable. The functionality works, and response times remain within the agreed limits. The problem often becomes visible only when the number of users increases or cloud costs start to rise.

Memory usage continues to increase

An application does not fully release temporary memory after use. You may not notice this during a short test. During a soak test, however, you can see memory usage gradually increasing. Eventually, the application needs to be restarted or requires additional capacity.

A search request processes too much data

A user searches for a single customer, but the application reads a large part of the database. The correct result appears on time, so the functional test passes. Behind the scenes, however, the system is doing far more work than necessary.

Cloud resources do not scale down

During a peak, the platform automatically starts additional containers. Once demand drops, those containers remain active. The application stays fast and stable, but the organization continues to pay for unused capacity.

A release uses more resources

After a change, response times remain the same, but the application uses 25 percent more memory. Without comparing the results with the previous release, this deterioration can easily go unnoticed.

How can you make efficiency measurable?

A requirement such as “the application must use resources efficiently” is too general to test. A useful standard connects resource utilization to a clearly defined amount of work.

Possible metrics include:

• CPU usage per thousand transactions;
• memory usage per concurrent user;
• database operations per order;
• network traffic per API request;
• the number of transactions one container can process;
• resource recovery after a load peak.

The right metric depends on the system. Battery and data usage may be important for a mobile application. In a cloud environment, you are more likely to monitor memory usage, server capacity, and autoscaling. For embedded software, CPU capacity and available memory may be critical.

Do not try to measure everything. Focus on the resources whose excessive use poses a genuine risk to the product.

How do you test efficiency with TMAP?

Within TMAP: Quality for Cross-Functional Teams, quality is a shared responsibility. Efficiency is therefore not only the responsibility of a performance tester or operations engineer. Testers, developers, system administrators, operations engineers, and the product owner determine together which performance levels are required and what level of resource utilization is acceptable.

Tip: include efficiency in the Product Risk Analysis

First, determine whether inefficient resource utilization is a relevant product risk. This may be the case when a system has large numbers of users, processes high transaction volumes, runs on limited hardware, or operates in a cloud environment where additional capacity directly increases costs.

If the risk is significant, efficiency should have an explicit place in the test approach. This prevents the team from considering resource utilization only after the software has gone live.

Define measurable acceptance criteria

Combine a performance target and a resource limit in a single criterion. For example:

With five hundred concurrent users, 95 percent of transactions are completed within two seconds, while average memory usage remains below the agreed limit.

This prevents the system from meeting a performance target simply by using increasing amounts of capacity.

Measure resources during existing tests

Efficiency does not always require a separate test. You can monitor CPU, memory, I/O, and network utilization during functional tests, load tests, and soak tests.

Soak tests are particularly useful. Problems such as memory leaks and connections that remain open may only become visible after the application has been running for an extended period.

Use a baseline for each release

Record the resource utilization of a stable release and use it as a baseline for new versions. This shows whether the same functionality requires more capacity after a change.

A baseline is also useful when there is no fixed maximum value. A sudden increase compared with the previous release provides a clear reason for further investigation.

Check resource recovery after a load peak

Do not only measure what happens during the highest load. Check whether memory is released, connections are closed, and additional cloud capacity is scaled down afterwards.

A system that scales up successfully but does not recover afterwards continues to use resources unnecessarily.

Efficiency, cloud costs, and sustainability

Efficient resource utilization leaves more capacity available for growth and unexpected peaks. It can also prevent the organization from paying unnecessarily for additional servers, storage, or cloud capacity.

Efficiency is also related to sustainability. Software that requires less computing power and infrastructure to perform the same amount of work will generally use less energy. This gives the quality attribute a technical, financial, and environmental impact.

The testing principle remains the same: measure which resources are required to deliver the desired level of performance.

Learn how to test efficiency with Testlearning

In the TMAP: Quality for Cross-Functional Teams e-learning course, you will learn how to make quality attributes such as efficiency measurable and discuss them early in the development process. You will learn how to identify relevant product risks together with the entire team and translate them into an appropriate test approach.

This does not mean collecting every available metric. It means measuring what matters for your system.