Close

December 25, 2012

Mobile Apps Performance Testing – Whitepaper

Mobile apps performance testing - LinkedIn in IE

The post-PC era is already over. Mobile usage will surpass the PCs in lightning fast mode. By default, all business organizations started developing mobile apps for their customers. Below is the key statistics about mobile usage from Business Insider and Nielson 2012.

  • In 2011, there were 835 million smartphone users, 5.6 billion feature phone users – Business Insider
  • Global internet usage will more than double by 2015 – Business Insider
  • 64% of mobile phone time is spent on apps – Nielson 2012
  • 13.4% texting, 11.1% browser, 5.5% social networking apps, 5.4% dialer, 5.3% email/IM, 2.3% music/video, 1.1% camera, 55.8% other apps – Nielson 2012

From the above statistics, it is crystal clear that users spending more than 50% of their time on apps. Popular stores like Apple Apps Store, Google Play store has more than 650,000 and 600,000 apps respectively. Mobile app revenue will be $46 billion by 2016. Mobile Apps is an emerging digital industry today.

It is very much critical to validate the functionality and the performance of the apps before go-to-market. Just pick few of the low rated apps in the Apple Apps Store or in the Google Play Store or in any other store, the key reason of low ratings is because of lack in non-functional aspects of the apps.

Validating the non-functional aspects plays a vital role in Quality Assurance. This whitepaper focuses on mobile apps performance testing and its approach. It stresses more on the native apps aspects.

What are mobile apps (applications)?

Mobile app is software which accomplish specific task. E.g. Calendar, Music, Videos etc. It saves time and effort of the users by performing simple touch, pinch and zoom, tap, swipe, slide etc.

Mobile Load Testing

Mobile Load Testing helps us to study the behavior of the apps under various loads. This study reveals how the application handles concurrent users effectively. It is not possible to test the performance using real human users. By simulating virtual users (VUsers) using performance testing tool, it is simple to reproduce the huge number of concurrent users.

Difference between mobile testing and mobile apps performance testing

Before start mobile load testing, it is important to design the strategy. There are difference between mobile testing and mobile apps performance testing.

Mobile testing is a process of validating the functional aspects of the mobile device. Mobile apps performance testing is a process of validating the non-functional aspects of the apps installed in the mobile device.

Types of apps

There are three main types of apps in the mobile device. It is challenging to identify the type of performance testing is required and the type of mobile application under test. Below are the types of apps for the mobile device.

Native Apps:

Apps which are developed by mobile OS, e.g. apps developed in Objective-C for iPhone, apps developed in Java for Android mobiles etc. Native apps came first; famous examples of native apps are Address Book, Calendar, Music, Videos, and Games etc.

Web Apps:

Web apps designed in languages like HTML, JavaScript, and PHP etc. in a way that can be accessed via internet connection. E.g. Facebook can be accessed in mobile using http://m.facebook.com and Gmail can be accessed in http://m.gmail.com

Hybrid Apps:

Hybrid apps designed to access web applications which are embedded in the mobile. E.g. Twitter app, Apple’s Apps Store, LinkedIn app etc.

Challenges in Mobile Load Testing

There are few challenges in mobile load testing when compared to traditional load testing. Mobile devices access internet via various mobile technologies. Following are the key differences:

Users access internet using various mobile bandwidths and network. In US, people use 3G, 4G LTE technologies, in India people prefers 2G network, because 3G and 4G are emerging networks. Following are the few points play a critical role:

  • Simulating bandwidth and speed
    • Slower quality connection has high response time which extends the user sessions, and also utilizes more battery power.
    • 2G connection will be in Kbps where as 3G and 4G in Mbps
  • More number of devices
    • New mobile devices hitting the market every day. iOS, Android, Windows etc. It is important to validate all the aspects of all the devices.
  • Users Growth
    • Exponential growth in mobile users, hence it is important to predict the business growth by forecasting in advance.

Addressing mobile apps load testing

Using traditional tools like HP LoadRunner, IBM Rational Performance Tester, Apache JMeter, it is possible to simulate the mobile web applications by modifying the user agent string in the request headers.

Below is the snapshot of LinkedIn.com in IE 8 (default user agent string).

Mobile apps performance testing - LinkedIn in IE

Below is the snapshot of LinkedIn.com in iPhone 4S. LoadRunner is automatically redirecting from http://www.linkedin.com to https://touch.www.linkedin.com

Mobile apps performance testing - LinkedIn in iPhone 4s

It is because of the modified user agent string in the request headers. Based on the user agent string, server will redirect to the appropriate URLs.

Mobile website applications can be easily load tested by modifying the user agents. But the important drawback here is we cannot create realistic real time load. Because users have different devices, different carriers, upload/download speed, different signal strength and different technologies to access internet via wireless protocol. Following are the important point needs to be considered while designed the workload model:

  1. Signal Strength [Poor, Average, Good, Very Good]
  2. Mobile technology [EDGE, 2G, 3G, 4G LTE, WIFI]
  3. Bandwidth [Low, Medium, High, Very High]
  4. Download Speed
  5. Upload Speed
  6. Battery Power
  7. Geographical Location

Earlier native apps performs purely client side activity i.e. stand alone apps, but modern apps interacts with the server. E.g. Big Fish Casino game app in Apple store, when you launch this app, it connects with the game server to update your profile, score, chips and when you play with others, it sends/receives requests/responses.

To validate the client side parameter such as CPU usage, memory utilization, 2D, 3D graphics rendering etc. there are tools available in the market such as Quadrant, Smart bench, AnTuTu System Benchmark, CF-Bench, Linpack, GL Benchmark etc.,

Using above tools, validation of client side parameters can be validated and analyzed. As I mentioned, modern native apps will make connections with the server to accomplish the tasks.

Strategy to conduct performance tests on the native apps:

Following strategy helps us to conduct a performance tests on the native apps.

  1. Requirements Elicitation and Understanding
  2. Objective
  3. Effective Workload Model
  4. Creating and Conducting realistic load tests
  5. Result Analysis and Benchmarking

Crystal clear requirements elicitation and thorough understanding helps testers to validate and verify the functional and non-functional aspects of the apps.

Defining the objective guides testers to create effective realistic workload model for performance testing. Testers need to identify the different set of configurations of hardware/software, platforms, and the test objective.

Workload should be based on real-time scenario, not all users will have 4G LTE connection, hence the workload model should consists of combination of bandwidth settings, carriers, signal strength, geographical location etc.

Mobile apps performance testing - Realistic Workload

Following are the unrealistic workload model, which is also testers should keep in mind:

Mobile apps performance testing - Table Two - Unrealistic WorkLoad

Why above scenarios are unrealistic?

4G is not yet deployed completely in India. Only cities like Bangalore and Kolkata has 4G, other cities yet to deploy. While creating workload, it is important to know about the geographical locations.

Also, mobile web applications don’t need any upgrades or maintenance. Because it use browser to access the contents, whereas native apps needs to be upgraded whenever the newer version is available. Scenarios should cover upgrades, installation and uninstallation also.

Modern tools have the capability of all customizing the bandwidth profile, signal strength, upload speed, download speed etc. See below snapshot of Neotys’s NeoLoad Bandwidth profiles picker.

Mobile apps performance testing - NeoLoad

Image Courtesy: NeoLoad

Effective creation and conducting load tests will reveal behavior of the mobile apps. Also, if the concurrent users are heavy, the probability of mobile apps failures will be more.

Following metrics should be captured and compared with the benchmarks:

  • Response time
  • Throughput
  • Network Utilization
  • CPU
  • Memory
  • Packets/Bytes/Hits sent and received

Constructive analysis of above metrics will helps to fix the issues identified in the mobile apps.

Conclusion:

Mobile market is bright in future, also mobile usage is exponentially increasing among internet users, it is important to design and deploy mobile apps in the market. Also it is very challenging to address various components of mobile apps such as functionality, usability, performance issues from native apps perspective. However, by defining and designing effective strategy and workload, it will be stress-free task to validate and verify the apps.

Testing should start along with the development life cycle, to save the costs. By following above said approach, it will transform the challenges and produce commendable result.

About NaveenKumar Namachivayam

Hi there! is a passionate and experienced Performance Test Analyst based in Chicago, USA with skill sets in Manual Testing, Test Automation, Performance Testing, and Test Estimation. He loves to learn and experiment new trends and models in Software Testing. Please stop by at his personal blogs: Excel Blog, DealsBrook, NaveenKumarN.in, and Affiliate Insights blog and Software Testing Memes.

2 Comments on “Mobile Apps Performance Testing – Whitepaper

web load testing tools free
February 14, 2013 at 10:50 pm

Please tell me that youre heading to keep this up! Its so great and so important. I cant wait to read a lot more from you. I just feel like you know so substantially and know how to make people listen to what you might have to say. This blog is just too cool to become missed. Terrific stuff, genuinely. Please, PLEASE keep it up!

Reply
Appvance Inc.
April 4, 2013 at 3:39 am

Do you know if the TCP strategy itself, that is number of HTTP requests per persistent connection differs between Mobile and Web? I know the content will be different, as you describe, when the HTTP headers are manipulated but I wondered if any other behavior changes on either the HTTP Client or Server. I am thinking connection strategy might change for Mobile.

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *