A Problem from CTO
Our services are accessed
through a browser from a PC or Mac platform. It is also accessed through a
browser from tablets.
We will soon introduce an ios
app and later on Android.
I would like to be able to test
our applications (all have an English and French version) from a combination of
browsers and devices. At first count we have more than 60 combinations and
permutations.
Our manual regression test takes
about 2 days and our automated version take about 2.5 hours.
We run the automated regression
test with every deployment which happens most weekends
Please propose a strategy and
explain your choices.
Strategy Proposal.
In general quality (Q)
can be declared as Q = (R*T)/S where R - resources; T - time; S - scope.
In order to be able to finish
regression testing in time and not lose quality I have the following options:
- Test Automation
- reduce the scope;
- use available time better;
- get appropriate help with
testing within and/or outside scrum team(s).
Strategies:
0. Test Automation
Manual testing should be
minimized by extending:
a) Unit testing.
No code line of code should be
left without a unit test.
b) Functional testing.
c) Regression testing.
d) Performance and load testing.
The growing regression testing
suite helps increase stability of the application quality dramatically.
This is long term strategy that
will help to convert mini water-fall projects into right iterative development
process.
I. Reduce the scope.
Approach 1.1
Collect statistics about what
device/OS/browser combinations are used.
Select device/OS/browser
combinations that exceed a base line value, set by business for mandatory
testing. For example use for testing only device/OS/browser combinations that
exceed 5%.
Prioritize with business
stakeholders the selected device/OS/browser combinations with High, Medium, and
Low priority.
Approach 1.2
For the selected combinations
exclude an OS parameter. An operation system would not affect the behavior of a
web application radically, when on the same browser for the same device.
For example:
Combination1: PC/Windows
7/Chrome 30
Combination2: PC/Windows
XP/Chrome 30
Combination3: PC/Windows
8/Chrome 30
It can be reliable to do full
manual regression test only on one of those combinations.
Approach 1.3.
Use different device/OS/browser
combinations for full manual regression testing during different releases.
Use automated regression testing
for the rest of device/OS/browser combinations.
Approach 1.4
Limit browser support.
Majority of the browsers support
automatic updates regardless of the operation system. It is secure to support
only the latests versions of a browser.
Approach 1.5
Test different parts of the
application using different device/OS/browser combination.
The result of those approaches
can be summarized in a table. See Table 1
as an example.
If required a similar table can
be prepared for each release.
Table 1. NOTE: THIS TABLE IS
JUST AN ILLUSTRATION.
Device Type
|
Operation
System
|
Browser
|
Priority
|
Feature
|
Test Type
|
PC
|
Windows XP
|
IE8
|
High
|
Full
regression
|
Automated
|
PC
|
Windows XP
|
IE9
|
Low
|
N/A
|
|
PC
|
Windows 7
|
IE 8
|
Low
|
N/A
|
|
PC
|
Windows 7
|
IE 9
|
High
|
Sign in;
|
Manual
|
Mac
|
10.9:
"Mavericks"
|
Chrome
|
Low
|
Full
regression
|
Automated
|
Mac
|
Version 10.8:
"Mountain Lion"
|
Safari
|
High
|
Full
regression
|
Manual
|
Tablet
|
Windows 8
|
IE 10
|
High
|
Feature 1;
Feature 2; …., Feature N
|
Manual
|
Tablet
|
Android4:
Jelly Bean
|
Native
|
High
|
Full
regression
|
Automated
|
Tablet
|
iOS 6
|
Safari
|
High
|
Full
regression
|
Automated
|
II Use available time more
efficiently
Approach 2.1
Run automated regression tests
more often (for example: every night) against several device/OS/browser
combinations.
Create automated tests to cover
bugs. Include the tests into automated regression suite. Automated regression
test suite should be constantly growing.
Approach 2.2
Make sure an estimation of a
story, an improvement, a bug etc. includes testing related tasks.
For example, the complexity of a
story is estimated as 5 points. Does this estimation consider testing?
Better estimation will lead to
better prioritization.
III Get help within or outside a
scrum team.
Approach 3.1
Ask developers to help with
testing.
By the end of a sprint, some
developers may help with manual regression testing. It can help with the
overall quality as well. It also can help to reduce segregation between
Development and Testing in a scrum.
Approach 3.2
Ask business stakeholders to
help do regression testing.
REFERENCES
http://www.netmarketshare.com/
