In the field of Software Testing, Testers mainly concentrate on Black Box and White Box Testing. Under the Black Box testing, again there are different types of testing. The major types of testing are Functionality testing and Non-functional testing. Performance testing and types of performance testing comes under Non-functional testing.
What is Performance Testing?
In software, performance testing (also called Perf Testing) determines or validates the speed, scalability, or stability characteristics of the system or application under test. Performance is concerned with achieving response times, throughput, and resource-utilization levels that meet the performance objectives for the project or product.
Web application performance testing is conducted to mitigate the risk of availability, reliability, scalability, responsiveness, stability etc. of a system. Performance testing include a number of different types of testing like load testing, volume testing, stress testing, capacity testing, soak/endurance testing and spike testing each of which is designed to solve performance problems in a system.
Why Performance Testing?
In the current market performance and responsiveness applications play an important role in the success of a business. We conduct performance testing to address the bottlenecks of the system and to fine-tune the system by finding the root cause of performance issues. Performance testing answers to the questions like how many users the system could handle, how well the system could recover when the no. of users crossed the maximum users, what is the response time of the system under normal and peak loads.
We use performance testing tools to measure the performance of a system or application under test (AUT) and help in releasing high-quality software but it is not done to find defects in an application.
Load and performance testing will determine whether an application meets Speed, Scalability, and Stability requirements under expected workloads.
Speed: It determines whether an application responds quickly
Scalability: It determines maximum load an application can handle
Stability: It determines whether an application is stable under varying loads
Poorly performed applications gain a bad reputation and fail to meet the expected goals. So performance testing of an application is very important.
Types of Performance Testing
Capacity Testing:
Capacity Testing is to determine how many users a system/application can handle successfully before the performance goals become unacceptable. This allows us to avoid the potential problems in the future such as increased user base or increased volume of data. It helps users to identify a scaling strategy in order to determine whether a system should scale up or scale out. It is done majorly for eCommerce and Banking sites. are some examples. This testing is sometimes called Scalability testing.
Load Testing:
Load Testing is to verify that a system/application can handle the expected number of transactions and to verify the system/application behavior under both normal and peak load conditions (no. of users).
Volume Testing:
Volume Testing is to verify whether a system/application can handle a large amount of data. This testing focuses on Data Base. Performance tester who does volume testing has to populate a huge volume of data in a database and monitors the behavior of a system.
Stress Testing:
Stress Testing is to verify the behavior of the system once the load increases more than the system’s design expectations. This testing addresses which components fail first when we stress the system by applying the load beyond the design expectations. So that we can design a more robust system.
Soak/Endurance Testing:
Soak Testing is aka Endurance Testing. Running a system at high load for a prolonged period of time to identify the performance problems is called Soak Testing. It is to make sure the software can handle the expected load over a long period of time.
Spike Testing:
Spike Testing is to determine the behavior of the system under a sudden increase of load (a large number of users) on the system.
Objectives of performance Testing
The key objectives of performance testing are as follows:
- To gain the confidence that the changes to the existing chat application are in line with the defined Service Level Agreement.
- To ensure that the application performance, service availability, and the stability of the application are not impacted as a result of the new enhancements.
- Transaction Response Times remain within the acceptable tolerance over the increasing Load profile.
- JVMs show stable memory usage over the increasing load profiles.
Difference between Functional Testing and Non-functional Testing?
| FUNCTIONAL TESTING | NON-FUNCTIONAL TESTING |
| What the system actually does is functional testing | How well the system performs is non-functionality testing |
| To ensure that your product meets customer and business requirements and doesn’t have any major bugs | To ensure that the product stands up to customer expectations |
| To verify the accuracy of the software against expected output | To verify the behavior of the software at various load conditions |
| It is performed before non-functional testing | It is performed after functional testing |
| Example of functional test case is to verify the login functionality | Example of non-functional test case is to check whether the homepage is loading in less than 2 seconds |
| Testing types are • Unit testing • Smoke testing • User Acceptance • Integration Testing • Regression testing • Localization • Globalization • Interoperability | Testing types are • Performance Testing • Volume Testing • Scalability • Usability Testing • Load Testing • Stress Testing • Compliance Testing • Portability Testing • Disaster Recover Testing |
| It can be performed either manual or automated way | It can be performed efficiently if automated |
Difference between Performance Testing, Load Testing & Stress Testing
Performance Testing:In software, performance testing (also called Perf Testing) determines or validates the speed, scalability, and/or stability characteristics of the system or application under test. Performance is concerned with achieving response times, throughput, and resource-utilization levels that meet the performance objectives for the project or product. Performance testing is conducted to mitigate the risk of availability, reliability, scalability, responsiveness, stability etc. of a system.
Load Testing: Load Testing is to verify that a system/application can handle the expected number of transactions and to verify the system/application behavior under both normal and peak load conditions (no. of users).
Stress Testing: Stress Testing is to verify the behavior of the system once the load increases more than the system’s design expectations. This testing addresses which components fail first when we stress the system by applying the load beyond the design expectations. So that we can design a more robust system.
| PERFORMANCE TESTING | LOAD TESTING | STRESS TESTING |
| It is a superset of load and stress testing | It is a subset of performance testing | It is a subset of performance testing |
| Goal of performance testing is to set the benchmark and standards for the application | Goal of load testing is to identify the upper limit of the system, set SLA of the app and check how the system handles heavy load | Goal of stress testing is to find how the system behaves under extreme loads and how it recovers from failure |
| Load limit is both below and above the threshold of a break | Load limit is a threshold of a break | Load limit is above the threshold of a break |
| The attributes which are checked in performance testing are speed, response time, resource usage, stability, reliability and throughput | The attributes which are checked in a load testing are peak performance, server throughput, response time under various load levels, load balancing requirements etc. | The attributes which are checked in a stress testing are stability response time, bandwidth capacity etc., |
Difference between Performance Engineering & Performance Testing?
Performance engineering is a discipline that includes best practices and activities during every phase of the software development life cycle (SDLC) in order to test and tune the application with the intent of realizing the required performance.
Performance testing simulates the realistic end-user load to determine the speed, responsiveness, and stability of the system. It concerned with testing and reporting the current performance of an application under various parameters such as response time, concurrent user load, server throughput etc.
Performance Testing Process
Identify the test environment: Identify the physical test environment, production environment and know what testing tools are available. Before beginning the testing process, understand details of the hardware, software and network configurations. This process must be revisited periodically throughout the projects life cycle.
Identify performance acceptance criteria: This includes goals and constraints for response time, throughput and resource utilization. Response time is a user concern, throughput is a business concern, and resource utilization is a system concern. It is also necessary to identify project success criteria that may not be captured by those goals and constraints.
Plan & Design performance tests: Identify key scenarios to test for all possible use cases. Determine how to simulate that variability, define test data, and establish metrics to be gathered.
Configure the Test Environment: Prepare the test environment, arrange tools and other resources before execution
Implement the Test Design: Develop the performance tests in accordance with the test design
Execute the Test: Execute and monitor the tests
Analyze Results, Report, and Retest: Consolidate, analyze and share test results. Fine tune and retest to see if there is an improvement in performance. When all of the metric values are within acceptable limits then you have finished testing that particular scenario on that particular configuration.
Example of Performance Test Cases
Writing test cases for performance testing requires a different mindset compared to writing functional test cases.
- To verify whether an application is capable of handling a certain number of simultaneous users
- To verify whether the response time of an application under load is within an acceptable range when the network connectivity is slow
- To verify the response time of an application under low, normal, moderate and heavy load conditions
- To check whether the server remain functional without any crash under high load
- To verify whether an application reverts to normal behavior after a peak load
- To verify database server and CPU and memory usage of the application under peak load
How to choose the right Performance Testing Tool
There are many tools in the market to do performance testing. It is impossible to mention the best performance testing tool out of all the tools available. It is because every company has its own needs. What’s perfect for one company may not be suitable for another company. We have to do some analysis before choosing the right tool. Here are some factors we have to consider when choosing the best performance testing tool.
Some factors considered for choosing the best performance testing tool for Performance Testing.
- Budget (License cost)
- Types of license
- Protocol support
- Customer preference of load testing tool
- The cost involved in training employees on the selected tool
- Hardware/Software requirements of a loading tool
- Tool Vendor support and update policy
What are some popular Performance Testing Tools to do Performance Testing?
There are a lot of performance testing tools in the market. There are free website load testing tools, paid tools and freemium tools. Almost all the commercial performance testing tools have a free trial. You can get a chance to work hands-on before deciding which is the best tool for your needs.
Some of the popular performance testing tools are LoadRunner, Apache JMeter, NeoLoad, StresStimulus, LoadUI Pro, WebLOAD, Rational Performance Tester, AppLoader, SmartMeter.io, Silk Performer, StormRunner Load, LoadView.
Summary : In this chapter , we have covered most of the information required to understand Performance testing.