Everything You Wanted To Know About Regression Testing (But Were Afraid To Ask)

One of the most important types of application testing is performing so-called regression testing (regression tests). Often this group of tests is carried out not in full or not at all. The purpose of this article is to give a brief description of regression tests and to highlight its key notions.

Introduction to regression testing

Before we start talking about the methodology of regression testing let’s define what the term “regression testing” means. First thing that you should remember is that the regression testing is the selective retesting of the system. It is kind of test aimed to test the changes made to the application or the environment (fixing the bug, the merging of code, migration to a different operating system, database, web server or application server), to confirm the fact that the pre-existing functionality works the same way it has been working before.

Join us on Facebook to stay updated on all news around mobile app development and testing!

Types of regression testing

There are two main types of regression testing:

• Final regression testing. We perform this type of testing in order to validate the build that hadn’t undergone changes for a long period of time.
• Regression testing. This type of testing is performed to check whether the build had not damaged the other part of the application due to recent changes in the code for enhancement or for fixing the bug.

The final regression testing is carried out on “unchanged build for unspecified period of time” or for a period of time that has been chosen as cook-time for the release of the application. This type of testing is more crucial than any other type of stage of testing since only final regression testing gives you an opportunity to be sure that the same build of application that had been tested will go directly to the customers.

When you perform an ordinary regression testing you can use the builds for time frame that is necessary for the test cases to be executed. However, we strongly advise you to use the build that has not undergone any changes for each cycle of the regression testing process.

Selecting test cases for the regression testing

As it turns out, the considerable amount of defects which were found out and reported by the customers are caused by the last minute bug fixes. It has some serious side effects that is why the choice of test case for the regression testing sometimes can be a challenging task.

The main requirements to the test cases for the regression testing are follows:

• Waste knowledge about bugs fixing and its influence on the system;
• The area of frequent bugs should be included;
• The area where the code has been recently changed should be included as well and the area that is visible to your customers;
• Both the most significant features of your application that represent obligatory requirements of your customers.

While selecting the test cases you should not chose only those ones, that unsuccessful or ineffective or those ones that have nothing to do with bug fixing. You should select positive test cases rather than negative ones for the final circle of the regression testing. It is also highly advised that the regular test cycles before the start of regression testing process should have a proper combination of positive and negative test cases. In this context negative test cases can be considered those ones that had been implemented recently with the aim of breaking the system.

The preliminary planning and acting (in other words planning and acting at the very beginning of your project even before the test cycles) will do you a lot of good. You can also classify the test case into various Priorities in accordance to their role and customer usage.

We suggest that all your test case can be classified into three main categories:

1. Priority-0 – Rational test cases that check main functionality and are run for pre-system acceptance and when application undergoes major changes. These test cases have the very important meaning to both engineering department and customers.
2. Priority-1 – Uses the basic and normal setup and these test cases are crucial for both engineering and to customers.
3. Priority-2 – These test cases are less important. Executed parts of ST cycle are selected for regression testing on required basis.

Learn how you can manage test cases within Ubertesters platform.

There are few proper approaches to regression testing that are required to be decided on “case to case” basis:

• Case 1: If the criticality and influence of the bug fixes is not very high, then it is enough for a test engineer to select several test cases from TCDB and execute them. These test cases can have any priority (0, 1 or 2).
• Case 2: If bug fixes have medium criticality and impact, you would need to execute all Priority-0 and Priority-1 test cases. If bug fixes require additional test cases from Priority-2, then those test cases can also selected and used for regression testing. Selecting Priority-2 test cases in this context is desirable but not obligatory.
• Case 3: In the case where the seriousness and impact of the bug fixes is high, you would need to execute all Priority-0, Priority-1 and thoroughly chosen Priority-2 test cases.

Resetting the test cases for regression testing

In a large product release with the aid of a few rounds of testing, it is crucial to write down what test cases were executed with what build and related data. We can call these notes a test case result history.

It is not necessary to RESET the test case very often. Resetting of the test cases is required in following cases:

• a. When the product undergone significant changes;
• b. When the build procedure was changed and it has some impact on the product;
• c. In situation when you have a large release cycle in which some test cases were not executed for a long period of time;
• d. When you are having the final regression test cycle with a few selected test cases;
• e. In situation when the expected results of the test cases could differ greatly from previous cycles.

When the aforementioned recommendations are not implemented, you would better RERUN the test cases instead of resetting the results of the test cases. There are only few distinctions between RERUN and RESET states in test cases, either way the test cases are executed but in case of RESET one has to think “zero base” and expect different result than what was obtained in earlier cycles and therefore those test cases have their influence on the completion rate of testing. In case of RERUN you need not worry about completion rate since you can consider those test cases the complete except for a formality check and you are expected to get same results.

RESET is also selected according to the level of stability of the functionalities. If you are in Priority-1 and have reached a stage of convenient level on Priority-0 (for example more than 95% pass rate) then it is not necessary to RESET Priority-0 test cases unless there are major changes taking place. This is true with Priority-1 test cases when you are in Priority-2 test phase.

Concluding the results of a regression testing

In the process of regression testing it is strongly advised that you should use only one build for testing. It is expected that all 100% of those test cases pass using the same build. When the pass % is not 100, the tester can have a look at the previous results of the test case to conclude the expected result.

• a. If the outcome of a particular test case was PASSING when you were using the previous builds and FAIL in the current build, then regression failed. You would need to have a new build and start the testing from the very beginning after resetting the test cases.
• b. If the result of a definite test case was a FAIL using the previous builds and a PASS in the current build, then it is not difficult to suppose that the bug fixes worked.
• c. If the outcome of a definite test case was a FAIL while you were using the previous builds and a FAIL in the current build and if there are no bug fixes for this particular test case, it may denote that the outcome of this test case shouldn’t be considered for the pass %. This may also signify the fact that those test cases shouldn’t be chosen for regression.

Regression test guidelines for patch/upgrade releases

The regression recommendations can be used in both situations when:

1. You are preparing for a great release of a product, executed all system test cycles and now you are going to have a regression test cycle for fixing the bugs.
2. You are having a minor release of a product with only bug fixing, and you are thinking over the regression test cycles for dealing with those bug fixes.

There can be numerous cycles of regression testing that you can plan for each release. It happens because of the possibility of bugs fixes coming in phases or just because some bug fixes may not work as expected resulting in one more regression cycle.

Mapping Test cases for defect fixing

When assigning a “Fail” outcome to a test case, you should enter the defect number(s) along so that you will know what test cases to be executed when a bug fix arrives. Please pay attention that there can be numerous defects that can appear of a particular test case and a definite bug can have much serious impact than one test case.

Despite the fact that the usage of these mechanisms makes the mapping between test cases and defects much easier, the test cases that are to be executed for taking notice of side effects of bug fixes may remain in the function of a manual process.

Regression testing in an integral part of any testing process. Set up your own team of testers to make sure your next app release will work flawless!