The issue of acceptable application functioning is becoming more and more essential within the mobile testing industry. Various platforms, operating systems, screen sizes and mobile devices complicate the process of app adaptation and cause new issues. As a result, mobile cloud services are becoming a more popular option for mobile device testing. However, there are several things to keep in mind when choosing a cloud service for this function; what are the advantages and disadvantages, and is the ROI worth it?
Cloud services can provide great packages of functions that simplify and optimize mobile app testing. For example:
• Remote use of mobile devices available for testing
• A plethora of device options with different screen sizes, and functioning on different platforms, operation systems, etc.
• One-click function imitation (incoming calls, incoming messages, connection loss, etc.)
• Establishment of mobile testing environment: logging, screenshots and one-click video record
The above functions offer considerable advantages for mobile testing. To be specific, you can:
• Test devices that haven`t come out to market yet (for example, devices made only for the U.S.)
• Work with the most popular devices, which may be experiencing launch delays in some regions
• Work with outdated or expensive devices; although, this does not make much sense if the company has only a limited number of tests to be done
• Work with a real connection to foreign telecom operators (AT&T, T-Mobile, Verizon, Vodafone, etc.)
• Work with various types of Internet connections (Wi-Fi, 3G, LTE, Edge, GPRS)
Nevertheless, there are also some disadvantages to keep in mind when operating with cloud services:
• Low connection speed
• Weak device response
• Constant server and device glitches
• Expensive subscription for paid-for service
Today, the most widely used services are Perfecto Mobile’s MobileCloud project and Keynote Systems’ DeviceAnywhere project.
MobileCloud by Perfecto Mobile
Perfecto Mobile is one of the biggest players in the cloud services industry. The company`s primary offering is a SaaS MobileCloud platform, which includes MobileCloud-Interactive, MobileCloud-Automation and MobileCloud-Monitoring services. This tool package is a universal system for application testing and monitoring.
When using the MobileCloud platform, you can perform real-time testing of smartphones, phones and tablets. The company proposes more than 500 devices, operating on Android, Blackberry, iOS, Symbian and WinPhone platforms for testing. Almost all devices have a real connection to major U.S. mobile operators (AT&T, T-Mobile and Verizon), and UK operators (O2, Orange and Vodafone). Devices are also connected to Indian, Canadian, Israeli and other mobile companies.
In my research, I have found advantages of the MobileCloud platform to include:
• Full control over the device (real and virtual keyboard, touch and slide functions, accelerometer, turn on/off functions)
• Possibility to make calls, send text messages, access to the Internet (devices use tariff plans (service plans) of mobile operators)
• Automated procedures of app installation, incoming calls, incoming text messages, file upload, information input from OS clipboard and laptop keyboard
• Recommendations about device utilization in different countries
• Simultaneous automation testing on several devices
• Device sharing
Nevertheless, real-time operation with MobileCloud services is far from being ideal. First of all, you have a time limit of 60 minutes to use the free trial version. Other challenges include:
• Long response time
• Indistinct (often unreadable) image
• Constant device interaction failures (cannot process data/touch function challenges)
• Hot functions do not operate on most of the devices (incoming call and text messages simulation)
• Constant device malfunctioning (screen turns off, restarts, loses data, reports about mistakes)
Working with the trial session, you also have only six devices at your disposal: two iOS (iPhone 5S and iPad Air), four Android devices (HTC One Max, Nexus 5, Samsung Galaxy Note 3 and Samsung Galaxy Nexus 4.2.2). As a whole, even though the MobileCloud platform offers multiple tools and possibilities for mobile testing, the number of malfunctions I experienced made the process of testing quite challenging. The MobileCloud platform is also expensive when compared with other services.
DeviceAnywhere by Keynote Systems
Keynote Systems is a developer of cloud monitoring services. Its DeviceAnywhere platform includes complex solutions for mobile app testing and the automation of testing processes, along with development, analysis and certification processes.
In late 2012, DeviceAnywhere served more than 2,000 devices including smartphones and tablets, having a real-time connection to major mobile operators in the U.S., Canada, UK, France, Germany and other countries. The company cooperates with Dell, Salesforce, Google and Microsoft.
Among the services that DeviceAnywhere offers are:
• Manual application and website testing run on the company`s devices
• Website testing on multiple devices with URL input
• Application testing supported with outgoing and incoming text messages
• Calls and messages exchange among several virtual devices
• Full control over the device operation process (physical and virtual keypad, touch and slide functions, g-sensor, device restart, battery disconnection)
• Operation of an unlimited number of devices simultaneously (with an hourly fee)
• Fast screenshot export
• PC keyboard text input
• Scalable picture from device to PC
• Control over audio/video quality (important for slow Internet connection)
• Test case manager, business paper organizer
• Automation script creation and processing upon several devices (enterprise package)
The trial service version offers an unlimited free connection to a limited number of devices (10-15), and each session lasts for 10 minutes. In the trial version, you may work on iOS (iPhone 5S, iPhone 5), WinPhone (Nokia Lumia 925) and Android (smartphones by Motorola, Samsung and LG).
The picture is quite distinct – imaging and colors are clear, and text is readable. When watching HD video, you may experience some hovering. In my experience, cloud service DeviceAnywhere proved itself to be a near perfect free tool for testing websites imaging and running smoke tests for small ready apps. To hold more profound and serious testing, you would need to purchase a paid account, which would provide more devices, and sessions unlimited in length and quantity.
DeviceAnywhere shortcomings include a slower response that devices are demonstrating to user actions, as well as rather slow operation speed of the interface as a whole. In addition, compared with Perfecto Mobile, the web interface workspace is much less convenient to use. On the screenshot below, you can see that even at FULLHD resolution monitor does not fit all the details on the screen.
Mobile Cloud Services: Is it worth the ROI?
We’ve discussed the pros and cons. Now, I want to offer some issues to think over before buying a cloud platform account. First, which is truly better: a cloud service or a real device? From a tester`s point of view a REAL device is always better. But how does it look from the financial side? Let`s compare the prices depending upon the project category, workload, and terms:
Option 1: 10 hours per month
DeviceAnywhere offers 10 hours per month with one hour costing $18 + $100 registration fee. Perfecto Mobile is a bit cheaper – $17 per hour. As a result:
• If you need multiple devices, non-standard platforms, medium environment quality, get a package at $170 at Perfecto Mobile platform (it that is not enough, consider a package comprising 10 hours at $250 in price);
• If you need multiple devices, non-standard platforms, and you have free means for a good tool, then consider the package at $280 by DeviceAnywhere;
• If you have NO requirements to the number of devices, it’s better to get an inexpensive Android model and a Windows Phone, as they are always necessary.
Option 2: 50 hours per month
Need 50 hours per month for an efficient work flow? Then you should choose between a DeviceAnywhere account – $16 per hour + a fee; and a Perfecto Mobile account – $15 per hour. Let`s see what we get:
• A $900 account with DeviceAnywhere versus a $750 Perfecto Mobile account. If you have NO free means, then the Perfecto Mobile account is a good choice. If you HAVE means, than it`s obviously better to choose DeviceAnywhere account.
• However, the pricing for the accounts is equal to the price of a new iPhone. Therefore, it is worth buying a real device if the company doesn`t have it yet. Ultimately, the requirements define everything, and when a company needs a complex sample like “iPhone 5S + iPhone 5 + iPad Air + iPad 4”, cloud services is definitely a valid option.
Having analyzed the numbers and options, I would say that cloud services are better to use in the case where real devices are lacking. The prices for the accounts vary from $1,440 to $72,600 depending upon the hours, number of users and duration of use.
So, when is applying to cloud services REALLY worth doing?
1. You are engaged in a project that requires minimum time, e.g. a week or even one working day; and the tests are exercised upon a platform that your team is unfamiliar with.
OR, the test should be run on multiple device models, which are of little interest to your company. If there is no way out get a paid cloud service account, DeviceAnywhere showed the best effectiveness.
2. A project presupposes testing within some region or mobile operator (Verizon in the U.S., for example). If the client hasn`t found a vendor with real devices connected to the operator, then a paid cloud services account is the right choice.
3. In the event you need to buy a specific device in a specific region, first of all check the device availability. If the device truly is impossible to buy, and it`s unavailable in the support service, consider a paid account on one of the platforms. However, keep in mind that Perfecto Mobile and DeviceAnywhere are not museums of ancient devices, so no one can guarantee that it`s a solution to this specific problem.
4. If a client developed an app specifically for the newest device model, but it`s not on sale yet, and the deadline is growing closer. Then, first check the device availability within the cloud platform; if they have it, invest in an account.
With a few exceptions, it is usually better to widen the number of devices available to the company for testing. Mobile cloud services cannot compete with real devices – in price or testing environment quality.
However, you should not forget the free version of the service DeviceAnywhere. If 10-minute trial sessions are not a problem, then the DeviceAnywhere trial version can easily be justified when testing websites and small mobile applications.
In a nutshell, keep in mind that tests always show more realistic results on a REAL device. That is why, before choosing between real devices and cloud services you have to weigh all the pros and cons, and calculate expenditures, critical issues and lack of devices.
About the author:
Pavel Andreev is a quality assurance team lead with A1QA, an independent software testing company. His current responsibilities include analysis of technical and functional requirements, test documentation development, functional testing, GUI and usability testing, cross-platform and cross-browser compatibility testing, and localization testing. Pavel specializes in web-based project testing with complicated business logic and cloud services, and has experience testing mobile applications for iOS and Android, as well as standalone applications for Windows, Mac OS, and Linux.
