The post-PC era is already over. Mobile usage will surpass the PCs in lightning fast mode. By default, all business organizations started developing mobile apps for their customers. Below is the key statistics about mobile usage from Business Insider and Nielson 2012.
- In 2011, there were 835 million smartphone users, 5.6 billion feature phone users – Business Insider
- Global internet usage will more than double by 2015 – Business Insider
- 64% of mobile phone time is spent on apps – Nielson 2012
- 13.4% texting, 11.1% browser, 5.5% social networking apps, 5.4% dialer, 5.3% email/IM, 2.3% music/video, 1.1% camera, 55.8% other apps – Nielson 2012
From the above statistics, it is crystal clear that users spending more than 50% of their time on apps. Popular stores like Apple Apps Store, Google Play store has more than 650,000 and 600,000 apps respectively. Mobile app revenue will be $46 billion by 2016. Mobile Apps is an emerging digital industry today.
It is very much critical to validate the functionality and the performance of the apps before go-to-market. Just pick few of the low rated apps in the Apple Apps Store or in the Google Play Store or in any other store, the key reason of low ratings is because of lack in non-functional aspects of the apps.
Validating the non-functional aspects plays a vital role in Quality Assurance. This whitepaper focuses on mobile apps performance testing and its approach. It stresses more on the native apps aspects.
What are mobile apps (applications)?
Mobile app is software which accomplish specific task. E.g. Calendar, Music, Videos etc. It saves time and effort of the users by performing simple touch, pinch and zoom, tap, swipe, slide etc.
Mobile Load Testing
Mobile Load Testing helps us to study the behavior of the apps under various loads. This study reveals how the application handles concurrent users effectively. It is not possible to test the performance using real human users. By simulating virtual users (VUsers) using performance testing tool, it is simple to reproduce the huge number of concurrent users.
Difference between mobile testing and mobile apps performance testing
Before start mobile load testing, it is important to design the strategy. There are difference between mobile testing and mobile apps performance testing.
Mobile testing is a process of validating the functional aspects of the mobile device. Mobile apps performance testing is a process of validating the non-functional aspects of the apps installed in the mobile device.
Types of apps
There are three main types of apps in the mobile device. It is challenging to identify the type of performance testing is required and the type of mobile application under test. Below are the types of apps for the mobile device.
Apps which are developed by mobile OS, e.g. apps developed in Objective-C for iPhone, apps developed in Java for Android mobiles etc. Native apps came first; famous examples of native apps are Address Book, Calendar, Music, Videos, and Games etc.
Hybrid apps designed to access web applications which are embedded in the mobile. E.g. Twitter app, Apple’s Apps Store, LinkedIn app etc.
Challenges in Mobile Load Testing
There are few challenges in mobile load testing when compared to traditional load testing. Mobile devices access internet via various mobile technologies. Following are the key differences:
Users access internet using various mobile bandwidths and network. In US, people use 3G, 4G LTE technologies, in India people prefers 2G network, because 3G and 4G are emerging networks. Following are the few points play a critical role:
- Simulating bandwidth and speed
- Slower quality connection has high response time which extends the user sessions, and also utilizes more battery power.
- 2G connection will be in Kbps where as 3G and 4G in Mbps
- More number of devices
- New mobile devices hitting the market every day. iOS, Android, Windows etc. It is important to validate all the aspects of all the devices.
- Users Growth
- Exponential growth in mobile users, hence it is important to predict the business growth by forecasting in advance.
Addressing mobile apps load testing
Using traditional tools like HP LoadRunner, IBM Rational Performance Tester, Apache JMeter, it is possible to simulate the mobile web applications by modifying the user agent string in the request headers.
Below is the snapshot of LinkedIn.com in IE 8 (default user agent string).
It is because of the modified user agent string in the request headers. Based on the user agent string, server will redirect to the appropriate URLs.
Mobile website applications can be easily load tested by modifying the user agents. But the important drawback here is we cannot create realistic real time load. Because users have different devices, different carriers, upload/download speed, different signal strength and different technologies to access internet via wireless protocol. Following are the important point needs to be considered while designed the workload model:
- Signal Strength [Poor, Average, Good, Very Good]
- Mobile technology [EDGE, 2G, 3G, 4G LTE, WIFI]
- Bandwidth [Low, Medium, High, Very High]
- Download Speed
- Upload Speed
- Battery Power
- Geographical Location
Earlier native apps performs purely client side activity i.e. stand alone apps, but modern apps interacts with the server. E.g. Big Fish Casino game app in Apple store, when you launch this app, it connects with the game server to update your profile, score, chips and when you play with others, it sends/receives requests/responses.
To validate the client side parameter such as CPU usage, memory utilization, 2D, 3D graphics rendering etc. there are tools available in the market such as Quadrant, Smart bench, AnTuTu System Benchmark, CF-Bench, Linpack, GL Benchmark etc.,
Using above tools, validation of client side parameters can be validated and analyzed. As I mentioned, modern native apps will make connections with the server to accomplish the tasks.
Strategy to conduct performance tests on the native apps:
Following strategy helps us to conduct a performance tests on the native apps.
- Requirements Elicitation and Understanding
- Effective Workload Model
- Creating and Conducting realistic load tests
- Result Analysis and Benchmarking
Crystal clear requirements elicitation and thorough understanding helps testers to validate and verify the functional and non-functional aspects of the apps.
Defining the objective guides testers to create effective realistic workload model for performance testing. Testers need to identify the different set of configurations of hardware/software, platforms, and the test objective.
Workload should be based on real-time scenario, not all users will have 4G LTE connection, hence the workload model should consists of combination of bandwidth settings, carriers, signal strength, geographical location etc.
Following are the unrealistic workload model, which is also testers should keep in mind:
Why above scenarios are unrealistic?
4G is not yet deployed completely in India. Only cities like Bangalore and Kolkata has 4G, other cities yet to deploy. While creating workload, it is important to know about the geographical locations.
Also, mobile web applications don’t need any upgrades or maintenance. Because it use browser to access the contents, whereas native apps needs to be upgraded whenever the newer version is available. Scenarios should cover upgrades, installation and uninstallation also.
Modern tools have the capability of all customizing the bandwidth profile, signal strength, upload speed, download speed etc. See below snapshot of Neotys’s NeoLoad Bandwidth profiles picker.
Image Courtesy: NeoLoad
Effective creation and conducting load tests will reveal behavior of the mobile apps. Also, if the concurrent users are heavy, the probability of mobile apps failures will be more.
Following metrics should be captured and compared with the benchmarks:
- Response time
- Network Utilization
- Packets/Bytes/Hits sent and received
Constructive analysis of above metrics will helps to fix the issues identified in the mobile apps.
Mobile market is bright in future, also mobile usage is exponentially increasing among internet users, it is important to design and deploy mobile apps in the market. Also it is very challenging to address various components of mobile apps such as functionality, usability, performance issues from native apps perspective. However, by defining and designing effective strategy and workload, it will be stress-free task to validate and verify the apps.
Testing should start along with the development life cycle, to save the costs. By following above said approach, it will transform the challenges and produce commendable result.
- Unboxing HPE LoadRunner 12.55 - August 20, 2017
- Measure Client-side Performance using Lighthouse - August 10, 2017
- Perf Calculator – free iOS app for performance testers - July 16, 2017
- How to performance test AngularJS or ReactJS applications? - June 25, 2017
- How to run Apache JMeter tests with Visual Studio Team Services? - June 6, 2017