MobIle TestIng is a Challenge, Meet It!
The greater the obstacle, the more glory in overcoming it.
Every problem has a gift for you in its hands.
— Richard Bach
Sorrow prepares you for joy.
— Mevlana Rumi
Just after getting used to the terms “mobility” and “mobile context,” we need to spend some time on the challenges. Yes, you have heard it correctly: as you can guess, there are a number of challenges ahead of you before you can be successful in mobile testing, and these challenges are really tough.
Mobile testers generally underestimate this fact. They need to know that without clearly defining the challenges, one cannot say that he or she is mastering mobile testing. Project management courses also underline this fact very clearly. Know your challenges to build up your strategy and a fitting approach. A trustful action plan is nothing but a good reflection/resolution of the challenges ahead.
Well then, let’s now try to list the challenges of mobile testing. In order to be more clear here, I will divide these challenges into two areas. One will be the “generic challenges,” which include management, strategy, and human resources (HR) issues. The other area encompasses “test-specific challenges,” which are more related to testing itself.
Number of platforms — When we observe the timeline of the most recent mobile operating system (OS) releases, we see that iOS, Android, Windows, and BlackBerry dominate the market. They have the most recent versions running in numerous devices with frequent OS updates and device-specific features. As these common platforms are widely used, they should be tested accordingly. I advise every tester to regularly check the “Application Developers Alliance” website and stay informed about the recent releases, news, and statistics of mobile platforms.
Number of devices and displays — According to a research done by Application Developers Alliance last year, there were about fourteen thousand distinct Android devices in the market! Too many, isn’t it? When we look at it from a tester’s perspective, this can be an unreachable and unrealistic target. Since we, the testers, have limited time and budget, we need to be selective in our tests. In the following blog posts, I will propose some methods to cope with device selection and coverage.
Surely, distinct devices also mean different displays. Mobile testers should also take this into account. Different display size means different testing and different priorities. So I suggest you regularly search for “list of displays by pixel density” on Wikipedia so that you can keep yourself up-to-date regarding market devices and their display specifications.
Mobile hardware is complex — Believe it or not, mobile devices are complex, especially the hardware. Within a very small space, there are numerous microelectronic devices, and all should be interoperable. There is a processor, a solid-state disk, backlights, loudspeakers and earpiece, a Bluetooth module, USB peripherals, a battery, a GPS antenna, a touchscreen, and several other sensors and microchips.
When you add these up, you have a complex formula. As a mobile tester, you need to understand the hardware complexity while you are testing any mobile device. In order to better understand the hardware side, you may find that regularly checking ifixit.com is useful in understanding the physical parts and their importance in testing.
Mobile software is complex — Not only is the hardware complex, but mobile software often possesses detailed features and architecture—especially if we are talking about server-client applications. Many distinct technologies should be taken into consideration. How should server and client talk to each other? How should the database be integrated and accessed? How are the web services oriented. How many screens are there? Can the application work offline? Does it have authentication/authorization? The list can be endless.
If you are testing a mobile application, you at least need a basic understanding of the technologies that have been used in developing such an application, the architecture that the app relies on, and the feature set that the application brings. Without this information, testing cannot be done effectively, especially the lower level tests such as unit and integration testing.
Mobile security is critical — Security has always been a headache for software producers. When we are talking about mobile applications, it is one of the biggest challenges. Every year, statistics indicate a common fact: mobile operating systems are being targeted by malware, and this equates to an increased security risk. When we consider the fact that each smartphone or tablet uses about one hundred times more data than a standard feature phone, this equation becomes even more critical.
The risks associated with any software are a main driver of software testing; the mobile testers need to focus on this high-risk subject, that is, security testing.
Carriers/Operators are important — Due to lack of resources, know-how, and funding, mobile testing is usually done from noncarrier networks, for example, WAN, LAN, or Wi-Fi. This is a great problem since there are many network/carrier-related factors influencing mobile apps while they are running. Without moving some part of your testing into the wild by using real carrier networks, you cannot be 100 percent sure about your test coverage.
We all know that mobile users are on the go and not always connected to Wi-Fi. Therefore, we need to include the carrier network dynamics in our testing activities. We need to test while we are using 4G, 3G or 2G, we need to test while we are roaming, we need to test while we are tethering to the Internet, we need to test while we are having limited or poor access, and we need to test the interruptions caused by incoming calls, SMSs, MMSs, e-mails, and so on. All these carrier issues are important and are indispensable for mobile testing activities.
Mobile devices have limitations — Some people think that a mobile device is a portable/miniature computer. I agree with this perception but only to a certain extent. We need to know that these devices have some significant limitations when compared to PCs. They have smaller screens and reduced bandwidth, they have no mouse or standard-size keyboards, their processors and memories are less capable, their power management is different, they have smaller disk spaces, they mostly have touchscreens, they generally run a single application from a single screen, they have shorter sessions, and their users are moving.
While you are testing a mobile device, you need to remember these limitations. Do not forget to define your limitations and differences before starting your tests. In this way, you can be more confident about your testing and its outcomes.
Native vs. mobile web — One common discussion in mobile application development is the following: “Should we have native apps, or shall we use mobile web?” The answer is not so easy. A political answer like “it depends” fits here.
Both ways bring advantages and disadvantages. For example, a native app may have more advanced UI features, and it has more access to device-specific features (e.g., vibration, localization, backlights, camera, sensors, etc.). On the other hand, mobile web has upgrade flexibility and code portability. As testers, we need to understand the needs of our users/organizations, and we need to test the mobile apps accordingly, especially by taking into consideration the related technologies and their differences.
We have just defined our framework, so now let’s move on to the test-specific challenges.
Need for shorter development life cycles — Since mobility translates into very demanding and ever-changing user behavior, software producers should adapt themselves to this context. In order to fulfill customers’ needs and penetrate the market quicker, mobile application development should be done with shorter life cycles with fewer requirements and features. Having fewer requirements per cycle significantly reduces the risk of having unhappy users. In order to achieve this, companies try to adopt agile development and be more responsive. They do frequent releases with narrow scopes instead of a wide-ranging one. Mobile testing should be adapted to this very demanding and agile context, and testers should be more responsive and creative. As we always complain about the lack of remaining time for testing, we need to be careful here. We need to deploy a risk-based testing approach and try to focus our testing efforts on the most commonly used features and business rules.
Need for regression testing — After we talk about shorter life cycles, we need to talk about regression testing. As we will have more frequent deliveries, we should do more regression testing. With each and every change in the code, we need to be sure about the status of the legacy (predecessor) functions and/or features. These changes can be new features and/or defect fixes, and both should be very carefully tested.
In the following blog posts, I will further emphasize the importance of mobile test automation and will show how it fits with regression testing by referring to several case studies.
Need for back-end testing — Unfortunately, mobile testers focus a lot more on the UI (front end) rather than APIs or any back-end layers. When I quantify this figure through several case studies, I observe a ratio of around 90 percent. Generally, mobile applications are parts of bigger systems. For example, consider a mobile banking application: the back-end code is likely to have dependencies, coupling, and shared code with the core banking system.
If we test a mobile application with a similar architecture, we need to test the back end as well as the front end. Especially, application programming interfaces (APIs) should be checked by test cases, and in this way, business and database layers should also be checked.
Need for performance testing — One other widely neglected test activity is performance testing. As nonfunctional requirements are rarely collected, developed, and traced in software projects, nonfunctional testing is unfortunately positioned as a low-priority issue. If you ask mobile software producers, none will agree with this, but you cannot hide the truth. Almost all the performance-related bugs are being resolved in production environments! In other words, companies are not readily investing in performance testing unless they experience a hurtful performance failure in their live systems.
Mobile testers should be aware of this fact, and they need to promote performance testing. Lots of research is being done in this area, and the results show that mobile users are impatient. You cannot tell them to wait for ten seconds for your mobile app to launch and that afterward they will ascend to heaven. Industry benchmarks show that mobile users want apps to launch in two seconds! Yes, you are reading it correctly: just two seconds. Simply put, mobile users want us to do performance testing; otherwise they will do it for us and then just delete the apps we have developed.
Need for mobile testing tools — This is a generic point in any software testing problem and tool selection. When we shift our focus to mobile testing, it just grows bigger. In order to reduce the risk of mobile application failures and to improve traceability, coverage, reporting, and execution frequency, you need to select the right mobile testing tools at the right time.
Recently, we have observed significant improvements in this area. Mobile testing tools are on the rise, and every year new players are being introduced to the market. It seems that tool scarcity, which was the biggest challenge two to three years ago, is now out of the picture. However, this does not mean that tooling is no longer a challenge. We still know that organizations are suffering when choosing the right tools. They have more shelfware (not used/utilized software) than usable testing software. Since lots of solutions exist, there is no one right option. And mobile testers should own the responsibility to solve this equation and find the test tool they will use.
Need more time to test — Another big challenge in any tester’s life is the shortage of time. For several years, I have been delivering training on software testing and mobile testing and telling people to test early and often—but still, there’s not much improvement! Especially when I speak to upper managers and C-levels, they tell me that they are impressed and promise to give more time to testing, but it never happens. The latest life-cycle activity seems to be software testing, and if you are dealing with mobile applications, this behavior can make you suffer more.
A responsible tester should estimate rational efforts and promote quality rather than time-to-market. So never hesitate to ask for more testing time if you don’t feel comfortable. OK, you need to think twice if you are talking to your CIO, but you should know that thinking twice doesn’t mean never saying a word!
Need more devices and environment — Whether you do in-house testing or outsource it, it is always useful to have an in-house test environment (a lab or a simple test place) and to have some real devices in place. Surely you can use cloud solutions and emulators instead of real devices, but you need to know that nothing compares to a real mobile device. If you are doing system or user-acceptance levels of testing and the devices are owned by you, testing becomes more and more real, and you capture more real bugs.
Need for skilled mobile testers — Mobile testing is a very big and a promising market, but companies still suffer from a lack of talented people. If you want to be famous, you can try your luck and go for a mobile tester position! Since traditional approaches and regular ways of testing will probably not make anyone happy, creative, talented, and educated testers are needed.
Need to be on the move — When people invite me to their mobile testing environments, I observe lots of tables full of mobile devices, and testers are running their tests from their chairs and marking their results on the test management tool. Most of the time they use a stationary testing approach with no movement at all! The sad truth is that they are not aware of what they are missing. Mobile applications are being used by people who are on the move. I don’t see any people around me who stop walking and sit on a chair to watch a video, tweet, or send an e-mail. All these things are done on the go.
Since we mobile testers are defenders of mobility, we need to devote some time to doing at least some of our testing on the move. If a testing activity does not reflect, exercise, or contain any real user behavior, then we can not conclude that it is 100 percent real testing.
After reading this blog post, if you feel that being a mobile tester will be a real pain for you, you can try your chances at several other jobs. As a famous saying goes, “If you are afraid of failure, you don’t deserve to be successful.” Alternatively, I can be more constructive and end this post with words from a famous U2 song, “Miracle Drug”:
Of science and the human heart, there is no limit
There is no failure here, sweetheart, just when you quit…
IT professional with 15+ years of experience as IT Consultant, Software Engineer, Software Developer and Software Tester for many different organizations. Published a number of papers and books within the Software Engineering profession and contribute to the field by regularly attending conferences as a public speaker.
- Think Wider, Test Better!
- Convincing Your Manager To Fund Testing
- An Overview of Test Design Techniques
- User Acceptance Testing Has Many Meanings
- Testers Never Assume; Testers Ask Questions
- Performance Testing Tools Need Calibration!
- Testing in a Cross-Cultural Team
- Be “A Little” More Flexible in Hiring Junior Testers
- Buy Tools if You Really Need Them
- Mobile Testing & The Importance of Test Metrics
- Measure Your Testing
- Improve Your Test Processes by Improving Your Test Data
- Negative Testing Can Be Really Positive
- 20 Practical Advice To Help You Better Test Your Mobile Apps
- Testers Can (!) Stop Bad Software
- Mobile Usability Testing is Not Rocket Science!